cotyledon-1.6.8/0000755000175000017500000000000013134343247014334 5ustar silehtsileht00000000000000cotyledon-1.6.8/test-requirements.txt0000644000175000017500000000075113037230124020567 0ustar silehtsileht00000000000000# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. hacking<0.11,>=0.10.0 coverage>=3.6 python-subunit>=0.0.18 oslotest>=1.10.0 # Apache-2.0 oslo.config>=3.14.0 # Apache-2.0 testrepository>=0.0.18 testscenarios>=0.4 testtools>=1.4.0 python-subunit>=0.0.18 os-testr # docs sphinx_rtd_theme sphinx oslosphinx>=2.5.0 # Apache-2.0 cotyledon-1.6.8/doc/0000755000175000017500000000000013134343247015101 5ustar silehtsileht00000000000000cotyledon-1.6.8/doc/source/0000755000175000017500000000000013134343247016401 5ustar silehtsileht00000000000000cotyledon-1.6.8/doc/source/examples.rst0000644000175000017500000000014712774716153020764 0ustar silehtsileht00000000000000======== Examples ======== .. literalinclude:: ../../cotyledon/tests/examples.py :language: python cotyledon-1.6.8/doc/source/non-posix-support.rst0000644000175000017500000000143713025030376022577 0ustar silehtsileht00000000000000============================ Note about non posix support ============================ On non-posix platform the lib have some limitation. When the master process receives a signal, the propagation to children processes is done manually on known pids instead of the process group. SIGHUP is of course not supported. Processes termination are not done gracefully. Even we use Popen.terminate(), children don't received SIGTERM/SIGBREAK as expected. The module multiprocessing doesn't allow to set CREATE_NEW_PROCESS_GROUP on new processes and catch SIGBREAK. Also signal handlers are only run every second instead of just after the signal reception because non-posix platform does not support signal.set_wakeup_fd correctly And to finish, the processes names are not set on non-posix platform. cotyledon-1.6.8/doc/source/oslo-service-migration.rst0000644000175000017500000001155413025030376023535 0ustar silehtsileht00000000000000=============================== Oslo.service migration examples =============================== This example shows the same application with oslo.service and cotyledon. It uses a wide range of API of oslo.service, but most applications don't really uses all of this. In most case cotyledon.ServiceManager don't need to inherited. It doesn't show how to replace the periodic task API, if you use it you should take a look to `futurist documentation`_ oslo.service typical application: .. code-block:: python import multiprocessing from oslo.service import service from oslo.config import cfg class MyService(service.Service): def __init__(self, conf): # called before os.fork() self.conf = conf self.master_pid = os.getpid() self.queue = multiprocessing.Queue() def start(self): # called when application start (parent process start) # and # called just after os.fork() if self.master_pid == os.getpid(): do_master_process_start() else: task = self.queue.get() do_child_process_start(task) def stop(self): # called when children process stop # and # called when application stop (parent process stop) if self.master_pid == os.getpid(): do_master_process_stop() else: do_child_process_stop() def restart(self): # called on SIGHUP if self.master_pid == os.getpid(): do_master_process_reload() else: # Can't be reach oslo.service currently prefers to # kill the child process for safety purpose do_child_process_reload() class MyOtherService(service.Service): pass class MyThirdService(service.Service): pass def main(): conf = cfg.ConfigOpts() service = MyService(conf) launcher = service.launch(conf, service, workers=2, restart_method='reload') launcher.launch_service(MyOtherService(), worker=conf.other_workers) # Obviously not recommanded, because two objects will handle the # lifetime of the masterp process but some application does this, so... launcher2 = service.launch(conf, MyThirdService(), workers=2, restart_method='restart') launcher.wait() launcher2.wait() # Here, we have no way to change the number of worker dynamically. Cotyledon version of the typical application: .. code-block:: python import cotyledon from cotyledon import oslo_config_glue class MyService(cotyledon.Service): name = "MyService fancy name that will showup in 'ps xaf'" # Everything in this object will be called after os.fork() def __init__(self, worker_id, conf, queue): self.conf = conf self.queue = queue def run(self): # Optional method to run the child mainloop or whatever task = self.queue.get() do_child_process_start(task) def terminate(self): do_child_process_stop() def reload(self): # Done on SIGHUP after the configuration file reloading do_child_reload() class MyOtherService(cotyledon.Service): name = "Second Service" class MyThirdService(cotyledon.Service): pass class MyServiceManager(cotyledon.ServiceManager): def __init__(self, conf) super(MetricdServiceManager, self).__init__() self.conf = conf oslo_config_glue.setup(self, self.conf, restart_method='reload') self.queue = multiprocessing.Queue() # the queue is explicitly passed to this child (it will live # on all of them due to the usage of os.fork() to create children) sm.add(MyService, workers=2, args=(self.conf, queue)) self.other_id = sm.add(MyOtherService, workers=conf.other_workers) sm.add(MyThirdService, workers=2) def run(self): do_master_process_start() super(MyServiceManager, self).run() do_master_process_stop() def reload(self): # The cotyledon ServiceManager have already reloaded the oslo.config files do_master_process_reload() # Allow to change the number of worker for MyOtherService self.reconfigure(self.other_id, workers=self.conf.other_workers) def main(): conf = cfg.ConfigOpts() MyServiceManager(conf).run() Other examples can be found here: * :doc:`examples` * https://github.com/openstack/gnocchi/blob/master/gnocchi/cli.py#L287 * https://github.com/openstack/ceilometer/blob/master/ceilometer/cmd/collector.py .. _futurist documentation: ` cotyledon-1.6.8/doc/source/contributing.rst0000644000175000017500000000011312706122176021635 0ustar silehtsileht00000000000000============ Contributing ============ .. include:: ../../CONTRIBUTING.rst cotyledon-1.6.8/doc/source/api.rst0000644000175000017500000000035413025030376017701 0ustar silehtsileht00000000000000======== API ======== .. autoclass:: cotyledon.Service :members: :special-members: __init__ .. autoclass:: cotyledon.ServiceManager :members: :special-members: __init__ .. autofunction:: cotyledon.oslo_config_glue.setup cotyledon-1.6.8/doc/source/index.rst0000644000175000017500000000111513025030376020233 0ustar silehtsileht00000000000000.. cotyledon documentation master file, created by sphinx-quickstart on Tue Jul 9 22:26:36 2013. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Welcome to cotyledon's documentation! ======================================================== Contents: ========= .. toctree:: :maxdepth: 2 installation api examples non-posix-support oslo-service-migration contributing .. include:: ../../README.rst Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` cotyledon-1.6.8/doc/source/installation.rst0000644000175000017500000000030412706122176021631 0ustar silehtsileht00000000000000============ Installation ============ At the command line:: $ pip install cotyledon Or, if you have virtualenvwrapper installed:: $ mkvirtualenv cotyledon $ pip install cotyledon cotyledon-1.6.8/doc/source/conf.py0000755000175000017500000000500713025030376017700 0ustar silehtsileht00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import sys sys.path.insert(0, os.path.abspath('../..')) # -- General configuration ---------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', ] # autodoc generation is a bit aggressive and a nuisance when doing heavy # text edit cycles. # execute "export SPHINX_DEBUG=1" in your terminal to disable # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = u'cotyledon' copyright = u'2016, Mehdi Abaakouk' # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). add_module_names = True # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # -- Options for HTML output -------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. # html_theme_path = ["."] # html_theme = '_theme' # html_static_path = ['static'] try: import sphinx_rtd_theme except ImportError: pass else: html_theme = 'sphinx_rtd_theme' html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] # Output file base name for HTML help builder. htmlhelp_basename = '%sdoc' % project # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass # [howto/manual]). latex_documents = [ ('index', '%s.tex' % project, u'%s Documentation' % project, u'Mehdi Abaakouk', 'manual'), ] # Example configuration for intersphinx: refer to the Python standard library. # intersphinx_mapping = {'http://docs.python.org/': None} cotyledon-1.6.8/README.rst0000644000175000017500000000522513122172631016021 0ustar silehtsileht00000000000000=============================== Cotyledon =============================== .. image:: https://img.shields.io/pypi/v/cotyledon.svg :target: https://pypi.python.org/pypi/cotyledon/ :alt: Latest Version .. image:: https://img.shields.io/pypi/dm/cotyledon.svg :target: https://pypi.python.org/pypi/cotyledon/ :alt: Downloads .. image:: https://travis-ci.org/sileht/cotyledon.png?branch=master :target: https://travis-ci.org/sileht/cotyledon .. image:: https://tea-ci.org/api/badges/sileht/cotyledon/status.svg :target: https://tea-ci.org/sileht/cotyledon Cotyledon provides a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more. * Free software: Apache license * Documentation: http://cotyledon.readthedocs.org/ * Source: https://github.com/sileht/cotyledon * Bugs: https://github.com/sileht/cotyledon/issues Why Cotyledon ------------- This library is mainly used in OpenStack Telemetry projects, in replacement of *oslo.service*. However, as *oslo.service* depends on *eventlet*, a different library was needed for project that do not need it. When an application do not monkeypatch the Python standard library anymore, greenlets do not in timely fashion. That made other libraries such as `Tooz `_ or `oslo.messaging `_ to fail with e.g. their heartbeat systems. Also, processes would not exist as expected due to greenpipes never being processed. *oslo.service* is actually written on top of eventlet to provide two main features: * periodic tasks * workers processes management The first feature was replaced by another library called `futurist `_ and the second feature is superseded by *Cotyledon*. Unlike *oslo.service*, **Cotyledon** have: * The same code path when workers=1 and workers>=2 * Reload API (on SIGHUP) hooks work in case of you don't want to restarting children * A separated API for children process termination and for master process termination * Seatbelt to ensure only one service workers manager run at a time. * Is signal concurrency safe. * Support non posix platform, because it's built on top of multiprocessing module instead of os.fork * Provide functional testing And doesn't: * facilitate the creation of wsgi application (sockets sharing between parent and children process). Because too many wsgi webserver already exists. *oslo.service* being impossible to fix and bringing an heavy dependency on eventlet, **Cotyledon** appeared. cotyledon-1.6.8/.coveragerc0000644000175000017500000000010612741741415016454 0ustar silehtsileht00000000000000[run] branch = True source = cotyledon [report] ignore_errors = True cotyledon-1.6.8/tox.ini0000644000175000017500000000131213025030376015637 0ustar silehtsileht00000000000000[tox] minversion = 2.0 skipsdist = True [testenv] usedevelop = True install_command = pip install -U {opts} {packages} setenv = VIRTUAL_ENV={envdir} deps = -r{toxinidir}/test-requirements.txt commands = {toxinidir}/tools/pretty_tox.sh {posargs} [testenv:pep8] deps = flake8 doc8 pygments commands = doc8 --ignore-path doc/source/rest.rst doc/source flake8 {posargs} [testenv:venv] commands = {posargs} [testenv:cover] commands = python setup.py test --coverage --testr-args='{posargs}' [testenv:docs] commands = python setup.py build_sphinx [testenv:debug] commands = oslo_debug_helper {posargs} [flake8] show-source = True builtins = _ exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build cotyledon-1.6.8/AUTHORS0000644000175000017500000000015113134343247015401 0ustar silehtsileht00000000000000Julien Danjou Mehdi ABAAKOUK Mehdi Abaakouk cotyledon-1.6.8/cotyledon.egg-info/0000755000175000017500000000000013134343247020026 5ustar silehtsileht00000000000000cotyledon-1.6.8/cotyledon.egg-info/not-zip-safe0000644000175000017500000000000113037226470022254 0ustar silehtsileht00000000000000 cotyledon-1.6.8/cotyledon.egg-info/entry_points.txt0000644000175000017500000000010513134343247023320 0ustar silehtsileht00000000000000[oslo.config.opts] cotyledon = cotyledon.oslo_config_glue:list_opts cotyledon-1.6.8/cotyledon.egg-info/top_level.txt0000644000175000017500000000001213134343247022551 0ustar silehtsileht00000000000000cotyledon cotyledon-1.6.8/cotyledon.egg-info/PKG-INFO0000644000175000017500000000774413134343247021137 0ustar silehtsileht00000000000000Metadata-Version: 1.1 Name: cotyledon Version: 1.6.8 Summary: Cotyledon provides a framework for defining long-running services. Home-page: https://github.com/sileht/cotyledon Author: Mehdi Abaakouk Author-email: sileht@sileht.net License: UNKNOWN Description: =============================== Cotyledon =============================== .. image:: https://img.shields.io/pypi/v/cotyledon.svg :target: https://pypi.python.org/pypi/cotyledon/ :alt: Latest Version .. image:: https://img.shields.io/pypi/dm/cotyledon.svg :target: https://pypi.python.org/pypi/cotyledon/ :alt: Downloads .. image:: https://travis-ci.org/sileht/cotyledon.png?branch=master :target: https://travis-ci.org/sileht/cotyledon .. image:: https://tea-ci.org/api/badges/sileht/cotyledon/status.svg :target: https://tea-ci.org/sileht/cotyledon Cotyledon provides a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more. * Free software: Apache license * Documentation: http://cotyledon.readthedocs.org/ * Source: https://github.com/sileht/cotyledon * Bugs: https://github.com/sileht/cotyledon/issues Why Cotyledon ------------- This library is mainly used in OpenStack Telemetry projects, in replacement of *oslo.service*. However, as *oslo.service* depends on *eventlet*, a different library was needed for project that do not need it. When an application do not monkeypatch the Python standard library anymore, greenlets do not in timely fashion. That made other libraries such as `Tooz `_ or `oslo.messaging `_ to fail with e.g. their heartbeat systems. Also, processes would not exist as expected due to greenpipes never being processed. *oslo.service* is actually written on top of eventlet to provide two main features: * periodic tasks * workers processes management The first feature was replaced by another library called `futurist `_ and the second feature is superseded by *Cotyledon*. Unlike *oslo.service*, **Cotyledon** have: * The same code path when workers=1 and workers>=2 * Reload API (on SIGHUP) hooks work in case of you don't want to restarting children * A separated API for children process termination and for master process termination * Seatbelt to ensure only one service workers manager run at a time. * Is signal concurrency safe. * Support non posix platform, because it's built on top of multiprocessing module instead of os.fork * Provide functional testing And doesn't: * facilitate the creation of wsgi application (sockets sharing between parent and children process). Because too many wsgi webserver already exists. *oslo.service* being impossible to fix and bringing an heavy dependency on eventlet, **Cotyledon** appeared. Platform: UNKNOWN Classifier: Intended Audience :: Information Technology Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 cotyledon-1.6.8/cotyledon.egg-info/dependency_links.txt0000644000175000017500000000000113134343247024074 0ustar silehtsileht00000000000000 cotyledon-1.6.8/cotyledon.egg-info/pbr.json0000644000175000017500000000005613134343247021505 0ustar silehtsileht00000000000000{"is_release": true, "git_version": "cfa0575"}cotyledon-1.6.8/cotyledon.egg-info/SOURCES.txt0000644000175000017500000000172213134343247021714 0ustar silehtsileht00000000000000.coveragerc .drone.yml .mailmap .testr.conf .travis.yml AUTHORS CONTRIBUTING.rst ChangeLog HACKING.rst LICENSE MANIFEST.in README.rst babel.cfg release.sh requirements.txt setup.cfg setup.py test-requirements.txt tox.ini cotyledon/__init__.py cotyledon/_service.py cotyledon/_service_manager.py cotyledon/_utils.py cotyledon/oslo_config_glue.py cotyledon.egg-info/PKG-INFO cotyledon.egg-info/SOURCES.txt cotyledon.egg-info/dependency_links.txt cotyledon.egg-info/entry_points.txt cotyledon.egg-info/not-zip-safe cotyledon.egg-info/pbr.json cotyledon.egg-info/requires.txt cotyledon.egg-info/top_level.txt cotyledon/tests/__init__.py cotyledon/tests/base.py cotyledon/tests/examples.py cotyledon/tests/test_functional.py cotyledon/tests/test_unit.py doc/source/api.rst doc/source/conf.py doc/source/contributing.rst doc/source/examples.rst doc/source/index.rst doc/source/installation.rst doc/source/non-posix-support.rst doc/source/oslo-service-migration.rst tools/pretty_tox.shcotyledon-1.6.8/cotyledon.egg-info/requires.txt0000644000175000017500000000006413134343247022426 0ustar silehtsileht00000000000000pbr>=1.6 [:(sys_platform != 'win32')] setproctitle cotyledon-1.6.8/CONTRIBUTING.rst0000644000175000017500000000024512744167024017001 0ustar silehtsileht00000000000000Bugs should be filed on Github: https://github.com/sileht/cotyledon/issues Contribution can be via Github pull requests: https://github.com/sileht/cotyledon/pulls cotyledon-1.6.8/LICENSE0000644000175000017500000002363712706002773015354 0ustar silehtsileht00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. cotyledon-1.6.8/MANIFEST.in0000644000175000017500000000013612706002773016072 0ustar silehtsileht00000000000000include AUTHORS include ChangeLog exclude .gitignore exclude .gitreview global-exclude *.pyc cotyledon-1.6.8/tools/0000755000175000017500000000000013134343247015474 5ustar silehtsileht00000000000000cotyledon-1.6.8/tools/pretty_tox.sh0000755000175000017500000000065213025030376020252 0ustar silehtsileht00000000000000#!/usr/bin/env bash set -o pipefail TESTRARGS=$1 # --until-failure is not compatible with --subunit see: # # https://bugs.launchpad.net/testrepository/+bug/1411804 # # this work around exists until that is addressed if [[ "$TESTARGS" =~ "until-failure" ]]; then python setup.py testr --slowest --testr-args="$TESTRARGS" else python setup.py testr --slowest --testr-args="--subunit $TESTRARGS" | subunit-trace -f fi cotyledon-1.6.8/HACKING.rst0000644000175000017500000000024112706122231016117 0ustar silehtsileht00000000000000cotyledon Style Commandments =============================================== Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/ cotyledon-1.6.8/PKG-INFO0000644000175000017500000000774413134343247015445 0ustar silehtsileht00000000000000Metadata-Version: 1.1 Name: cotyledon Version: 1.6.8 Summary: Cotyledon provides a framework for defining long-running services. Home-page: https://github.com/sileht/cotyledon Author: Mehdi Abaakouk Author-email: sileht@sileht.net License: UNKNOWN Description: =============================== Cotyledon =============================== .. image:: https://img.shields.io/pypi/v/cotyledon.svg :target: https://pypi.python.org/pypi/cotyledon/ :alt: Latest Version .. image:: https://img.shields.io/pypi/dm/cotyledon.svg :target: https://pypi.python.org/pypi/cotyledon/ :alt: Downloads .. image:: https://travis-ci.org/sileht/cotyledon.png?branch=master :target: https://travis-ci.org/sileht/cotyledon .. image:: https://tea-ci.org/api/badges/sileht/cotyledon/status.svg :target: https://tea-ci.org/sileht/cotyledon Cotyledon provides a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more. * Free software: Apache license * Documentation: http://cotyledon.readthedocs.org/ * Source: https://github.com/sileht/cotyledon * Bugs: https://github.com/sileht/cotyledon/issues Why Cotyledon ------------- This library is mainly used in OpenStack Telemetry projects, in replacement of *oslo.service*. However, as *oslo.service* depends on *eventlet*, a different library was needed for project that do not need it. When an application do not monkeypatch the Python standard library anymore, greenlets do not in timely fashion. That made other libraries such as `Tooz `_ or `oslo.messaging `_ to fail with e.g. their heartbeat systems. Also, processes would not exist as expected due to greenpipes never being processed. *oslo.service* is actually written on top of eventlet to provide two main features: * periodic tasks * workers processes management The first feature was replaced by another library called `futurist `_ and the second feature is superseded by *Cotyledon*. Unlike *oslo.service*, **Cotyledon** have: * The same code path when workers=1 and workers>=2 * Reload API (on SIGHUP) hooks work in case of you don't want to restarting children * A separated API for children process termination and for master process termination * Seatbelt to ensure only one service workers manager run at a time. * Is signal concurrency safe. * Support non posix platform, because it's built on top of multiprocessing module instead of os.fork * Provide functional testing And doesn't: * facilitate the creation of wsgi application (sockets sharing between parent and children process). Because too many wsgi webserver already exists. *oslo.service* being impossible to fix and bringing an heavy dependency on eventlet, **Cotyledon** appeared. Platform: UNKNOWN Classifier: Intended Audience :: Information Technology Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 cotyledon-1.6.8/.testr.conf0000644000175000017500000000047713037240077016431 0ustar silehtsileht00000000000000[DEFAULT] test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \ ${PYTHON:-python} -m subunit.run discover -t ./ . $LISTOPT $IDOPTION test_id_option=--load-list $IDFILE test_list_option=--list cotyledon-1.6.8/ChangeLog0000644000175000017500000001104213134343247016104 0ustar silehtsileht00000000000000CHANGES ======= 1.6.8 ----- * Fix terminate when workers fail to start * We don't care about child status when we exit * setup.cfg: Remove useless section * Fix a typo in service_manager docstring 1.6.7 ----- * Check user input * Increase 'fork too fast' sleep time 1.6.6 ----- * Fix sigchld for window 1.6.5 ----- * Use sigchld instead of polling children * Update sphinx 1.6.4 ----- * reduce cpu usage * Reword README 1.6.3 ----- * Fix forking too fast code * doc: Add note about futurist * Switch doc to rtd theme * doc enhancement * doc: hooks and window note 1.6.2 ----- * always run signal handling in main thread 1.6.1 ----- * window: Don't fail is the app use oslo.config * tests: fix py35 * tests: add a basic workflow for window * tests/doc/window: oslo.config limitation * tests: Don't use stdout * window: clarify SIGTERM/SIGALRM/SIGBREAK state * tests: window failfast * tests: Don't use signal 0 * window: Fix signals for children * tests: skip sighup tests * tests: fixing logging on window * fixing import order * Make option mutable * Add tea-ci.org supports * doc: Add some non-posix note * Close useless child pipe() * simplify the dead worker detection/restarting 1.6.0 ----- * doc: Add some non-posix note * add some py3 note * Bump the major version for the non-posix support * workaround killpg on non-posix platform * Simplify the running_services structure * Fix signal handling for windows * Make setproctitle optional for windows * Ignore setsid AttributeError on windows * Remove usage of os.waitpid * Use multiprocessing.Process instead of fork 1.5.0 ----- * Rework the oslo_config_glue * Implement sigalarm for servicemanager * Allow to register multiple hooks * Add on_new_worker hooks * Improve tests output and run doc8 * Tests: fix tests cleanup * Remove unused file * Improve docs * Split code base * Allow to register terminate/reload hooks * Share signals implementation * Allows to reconfigure a service * Revert "limit select.select() EINTR exception" * Don't wait for reload if terminate/reload already run * limit select.select() EINTR exception * Handle SIGALRM like other signals 1.4.6 ----- * set proc title for master process * doc: add some clarification 1.4.5 ----- * Rework signal handlers * Fixup code blocks 1.4.4 ----- * fix sporadic test failure * Set default python interpreter version to 3.5 * Move travis to py35 1.4.3 ----- * Empty the signal pipe 1.4.2 ----- * Uses signal.set_wakeup_fd() to ensure signal runs * Fix Graceful shutdown timeout with parent died 1.4.1 ----- * Rename oslo-config entry point 1.4.0 ----- * Add ServiceManager termination hook * Add new oslo.service/cotyledon diff 1.3.0 ----- * Fix oslo_config list_opts * Don't ship test binary * Add oslo config glue * Add graceful shutdown timeout * Ensure initialization is always called 1.2.8 ----- * Move release testing to py35 * fix time.sleep on darwin * Reduce sleep time while waiting for ever 1.2.7 ----- * Follow termination of our children only * Call atexit hook on parent only and with sys.exit * doc: Add examples * readme: typo * readme: typo * readme: Add more detail * Add Cotyledon story * Enhance description * Revert "Add py35 tests" 1.2.6 ----- * Remove useless classifier * Add py35 tests * Fix owner * Update README links 1.2.5 ----- * Only catch errno.ECHILD * switch unexpected pid from warn to error 1.2.4 ----- * release:Run py34 tests first * Allow non hashable args/kwargs * Run tests before release 1.2.3 ----- * re-re-handles signal during service init * Don't use oslosphinx 1.2.2 ----- * Wait all children, not only the first one * improve logging 1.2.1 ----- * pep8 fix * re-handles signal during service init 1.2.0 ----- * Don't call terminate twice on double child sigterm * handles signal during service init * docs: fix example 1.1.0 ----- * release: Allow to pass version * Allow to pass args and kwargs for services 1.0.3 ----- * Fix worker_id in tests 1.0.2 ----- * Start worker_id at 0 1.0.1 ----- * Add systemd notify support * Run atexit hooks * Some code deduplication * Remove hardcoded version 1.0.0 ----- * Fix release script * Fix tox and pep8 * Fix release script * Build universal wheel * Fix release script * create travis conf * Remove constraints stuffs * Set version * Better exception naming * remove useless code * Don't raise exception in default Service impl * doc: use pyclass in terminate() * doc: fix typo in reload() * doc: fix typo in terminate() * doc: fix missing backquots in ServiceManager * doc: fix type for worker_id parameter * More tests * Add some tests * Initial Ccotyledon Commit cotyledon-1.6.8/.drone.yml0000644000175000017500000000043113025030376016235 0ustar silehtsileht00000000000000build: image: teaci/msys32 shell: mingw32 pull: true commands: - pacman -S --needed --noconfirm --noprogressbar mingw-w64-i686-python2-pip - pip install -e . - pip install -r test-requirements.txt - python -m unittest discover -v -s cotyledon/tests -t . -f cotyledon-1.6.8/requirements.txt0000644000175000017500000000040213025030376017607 0ustar silehtsileht00000000000000# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. pbr>=1.6 setproctitle; sys_platform != 'win32' cotyledon-1.6.8/cotyledon/0000755000175000017500000000000013134343247016334 5ustar silehtsileht00000000000000cotyledon-1.6.8/cotyledon/__init__.py0000644000175000017500000000123513025030376020441 0ustar silehtsileht00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # Public API from cotyledon._service import Service # noqa from cotyledon._service_manager import ServiceManager # noqa cotyledon-1.6.8/cotyledon/_service_manager.py0000644000175000017500000003532613134342466022212 0ustar silehtsileht00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import collections import contextlib import logging import multiprocessing import os import signal import socket import sys import threading import time import uuid from cotyledon import _service from cotyledon import _utils LOG = logging.getLogger(__name__) class ServiceManager(_utils.SignalManager): """Manage lifetimes of services :py:class:`ServiceManager` acts as a master process that controls the lifetime of children processes and restart them if they die unexpectedly. It also propagate some signals (SIGTERM, SIGALRM, SIGINT and SIGHUP) to them. Each child process (:py:class:`ServiceWorker`) runs an instance of a :py:class:`Service`. An application must create only one :py:class:`ServiceManager` class and use :py:meth:`ServiceManager.run()` as main loop of the application. Usage:: class MyService(Service): def __init__(self, worker_id, myconf): super(MyService, self).__init__(worker_id) preparing_my_job(myconf) self.running = True def run(self): while self.running: do_my_job() def terminate(self): self.running = False gracefully_stop_my_jobs() def reload(self): restart_my_job() class MyManager(ServiceManager): def __init__(self): super(MyManager, self).__init__() self.register_hooks(on_reload=self.reload) conf = {'foobar': 2} self.service_id = self.add(MyService, 5, conf) def reload(self): self.reconfigure(self.service_id, 10) MyManager().run() This will create 5 children processes running the service MyService. """ _process_runner_already_created = False def __init__(self, wait_interval=0.01, graceful_shutdown_timeout=60): """Creates the ServiceManager object :param wait_interval: time between each new process spawn :type wait_interval: float """ if self._process_runner_already_created: raise RuntimeError("Only one instance of ServiceManager per " "application is allowed") ServiceManager._process_runner_already_created = True super(ServiceManager, self).__init__() # We use OrderedDict to start services in adding order self._services = collections.OrderedDict() self._running_services = collections.defaultdict(dict) self._forktimes = [] self._graceful_shutdown_timeout = graceful_shutdown_timeout self._wait_interval = wait_interval self._dead = threading.Event() # NOTE(sileht): Set it on startup, so first iteration # will spawn initial workers self._got_sig_chld = threading.Event() self._got_sig_chld.set() self._child_supervisor = None self._hooks = { 'terminate': [], 'reload': [], 'new_worker': [], } _utils.setproctitle("%s: master process [%s]" % (_utils.get_process_name(), " ".join(sys.argv))) # Try to create a session id if possible try: os.setsid() except (OSError, AttributeError): pass self._death_detection_pipe = multiprocessing.Pipe(duplex=False) signal.signal(signal.SIGINT, self._fast_exit) if os.name == 'posix': signal.signal(signal.SIGCHLD, self._signal_catcher) def register_hooks(self, on_terminate=None, on_reload=None, on_new_worker=None): """Register hook methods This can be callable multiple times to add more hooks, hooks are executed in added order. If a hook raised an exception, next hooks will be not executed. :param on_terminate: method called on SIGTERM :type on_terminate: callable() :param on_reload: method called on SIGHUP :type on_reload: callable() :param on_new_worker: method called in the child process when this one is ready :type on_new_worker: callable(service_id, worker_id, service_obj) If window support is planned, hooks callable must support to be pickle.pickle(). See CPython multiprocessing module documentation for more detail. """ if on_terminate is not None: _utils.check_callable(on_terminate, 'on_terminate') self._hooks['terminate'].append(on_terminate) if on_reload is not None: _utils.check_callable(on_reload, 'on_reload') self._hooks['reload'].append(on_reload) if on_new_worker is not None: _utils.check_callable(on_new_worker, 'on_new_worker') self._hooks['new_worker'].append(on_new_worker) def _run_hooks(self, name, *args, **kwargs): _utils.run_hooks(name, self._hooks[name], *args, **kwargs) def add(self, service, workers=1, args=None, kwargs=None): """Add a new service to the ServiceManager :param service: callable that return an instance of :py:class:`Service` :type service: callable :param workers: number of processes/workers for this service :type workers: int :param args: additional positional arguments for this service :type args: tuple :param kwargs: additional keywoard arguments for this service :type kwargs: dict :return: a service id :rtype: uuid.uuid4 """ _utils.check_callable(service, 'service') _utils.check_workers(workers, 1) service_id = uuid.uuid4() self._services[service_id] = _service.ServiceConfig( service_id, service, workers, args, kwargs) return service_id def reconfigure(self, service_id, workers): """Reconfigure a service registered in ServiceManager :param service_id: the service id :type service_id: uuid.uuid4 :param workers: number of processes/workers for this service :type workers: int :raises: ValueError """ try: sc = self._services[service_id] except KeyError: raise ValueError("%s service id doesn't exists" % service_id) else: _utils.check_workers(workers, minimum=(1 - sc.workers)) sc.workers = workers # Reset forktimes to respawn services quickly self._forktimes = [] def run(self): """Start and supervise services workers This method will start and supervise all children processes until the master process asked to shutdown by a SIGTERM. All spawned processes are part of the same unix process group. """ self._systemd_notify_once() self._child_supervisor = _utils.spawn(self._child_supervisor_thread) self._wait_forever() def _child_supervisor_thread(self): while not self._dead.is_set(): self._got_sig_chld.wait() self._got_sig_chld.clear() if self._dead.is_set(): return info = self._get_last_worker_died() while info is not None: service_id, worker_id = info self._start_worker(service_id, worker_id) info = self._get_last_worker_died() if self._dead.is_set(): return self._adjust_workers() def _on_signal_received(self, sig): if sig == _utils.SIGALRM: self._alarm() elif sig == signal.SIGTERM: self._shutdown() elif sig == _utils.SIGHUP: self._reload() elif sig == _utils.SIGCHLD: self._got_sig_chld.set() else: LOG.debug("unhandled signal %s" % sig) def _alarm(self): self._fast_exit(reason='Graceful shutdown timeout exceeded, ' 'instantaneous exiting of master process') def _reload(self): """reload all children posix only """ self._run_hooks('reload') # Reset forktimes to respawn services quickly self._forktimes = [] signal.signal(signal.SIGHUP, signal.SIG_IGN) os.killpg(0, signal.SIGHUP) signal.signal(signal.SIGHUP, self._signal_catcher) def _shutdown(self): LOG.info('Caught SIGTERM signal, graceful exiting of master process') signal.signal(signal.SIGTERM, signal.SIG_IGN) if self._graceful_shutdown_timeout > 0: if os.name == "posix": signal.alarm(self._graceful_shutdown_timeout) else: threading.Timer(self._graceful_shutdown_timeout, self._alarm).start() # NOTE(sileht): Stop the child supervisor self._dead.set() self._got_sig_chld.set() self._child_supervisor.join() self._run_hooks('terminate') LOG.debug("Killing services with signal SIGTERM") if os.name == 'posix': os.killpg(0, signal.SIGTERM) LOG.debug("Waiting services to terminate") for processes in self._running_services.values(): for process in processes: if os.name != "posix": # NOTE(sileht): we don't have killpg so we # kill all known processes instead # FIXME(sileht): We should use CTRL_BREAK_EVENT on windows # when CREATE_NEW_PROCESS_GROUP will be set on child # process process.terminate() process.join() LOG.debug("Shutdown finish") sys.exit(0) def _adjust_workers(self): for service_id, conf in self._services.items(): running_workers = len(self._running_services[service_id]) if running_workers < conf.workers: for worker_id in range(running_workers, conf.workers): self._start_worker(service_id, worker_id) elif running_workers > conf.workers: for worker_id in range(running_workers, conf.workers): self._stop_worker(service_id, worker_id) def _get_last_worker_died(self): """Return the last died worker information or None""" for service_id in self._running_services: # We copy the list to clean the orignal one processes = list(self._running_services[service_id].items()) for process, worker_id in processes: if not process.is_alive(): if process.exitcode < 0: sig = _utils.signal_to_name(process.exitcode) LOG.info('Child %(pid)d killed by signal %(sig)s', dict(pid=process.pid, sig=sig)) else: LOG.info('Child %(pid)d exited with status %(code)d', dict(pid=process.pid, code=process.exitcode)) del self._running_services[service_id][process] return service_id, worker_id def _fast_exit(self, signo=None, frame=None, reason='Caught SIGINT signal, instantaneous exiting'): if os.name == 'posix': signal.signal(signal.SIGINT, signal.SIG_IGN) signal.signal(signal.SIGALRM, signal.SIG_IGN) LOG.info(reason) os.killpg(0, signal.SIGINT) else: # NOTE(sileht): On windows killing the master process # with SIGINT kill automatically children LOG.info(reason) os._exit(1) def _slowdown_respawn_if_needed(self): # Limit ourselves to one process a second (over the period of # number of workers * 1 second). This will allow workers to # start up quickly but ensure we don't fork off children that # die instantly too quickly. expected_children = sum(s.workers for s in self._services.values()) if len(self._forktimes) > expected_children: if time.time() - self._forktimes[0] < expected_children: LOG.info('Forking too fast, sleeping') time.sleep(5) self._forktimes.pop(0) else: time.sleep(self._wait_interval) self._forktimes.append(time.time()) def _start_worker(self, service_id, worker_id): self._slowdown_respawn_if_needed() if os.name == "posix": fds = [self.signal_pipe_w, self.signal_pipe_r] else: fds = [] # Create and run a new service p = _utils.spawn_process( _service.ServiceWorker.create_and_wait, self._services[service_id], service_id, worker_id, self._death_detection_pipe, self._hooks['new_worker'], self._graceful_shutdown_timeout, fds_to_close=fds) self._running_services[service_id][p] = worker_id def _stop_worker(self, service_id, worker_id): for process, _id in self._running_services[service_id].items(): if _id == worker_id: # FIXME(sileht): We should use CTRL_BREAK_EVENT on windows # when CREATE_NEW_PROCESS_GROUP will be set on child process process.terminte() @staticmethod def _systemd_notify_once(): """Send notification once to Systemd that service is ready. Systemd sets NOTIFY_SOCKET environment variable with the name of the socket listening for notifications from services. This method removes the NOTIFY_SOCKET environment variable to ensure notification is sent only once. """ notify_socket = os.getenv('NOTIFY_SOCKET') if notify_socket: if notify_socket.startswith('@'): # abstract namespace socket notify_socket = '\0%s' % notify_socket[1:] sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) with contextlib.closing(sock): try: sock.connect(notify_socket) sock.sendall(b'READY=1') del os.environ['NOTIFY_SOCKET'] except EnvironmentError: LOG.debug("Systemd notification failed", exc_info=True) cotyledon-1.6.8/cotyledon/tests/0000755000175000017500000000000013134343247017476 5ustar silehtsileht00000000000000cotyledon-1.6.8/cotyledon/tests/base.py0000644000175000017500000000143212770420456020765 0ustar silehtsileht00000000000000# -*- coding: utf-8 -*- # Copyright 2010-2011 OpenStack Foundation # Copyright (c) 2013 Hewlett-Packard Development Company, L.P. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslotest import base class TestCase(base.BaseTestCase): """Test case base class for all unit tests.""" cotyledon-1.6.8/cotyledon/tests/test_unit.py0000644000175000017500000000474613042135103022065 0ustar silehtsileht00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import mock import cotyledon from cotyledon.tests import base class FakeService(cotyledon.Service): pass class SomeTest(base.TestCase): def setUp(self): super(SomeTest, self).setUp() cotyledon.ServiceManager._process_runner_already_created = False def test_forking_slowdown(self): sm = cotyledon.ServiceManager() sm.add(FakeService, workers=3) with mock.patch('time.sleep') as sleep: sm._slowdown_respawn_if_needed() sm._slowdown_respawn_if_needed() sm._slowdown_respawn_if_needed() # We simulatge 3 more spawn sm._slowdown_respawn_if_needed() sm._slowdown_respawn_if_needed() sm._slowdown_respawn_if_needed() self.assertEqual(6, len(sleep.mock_calls)) def test_invalid_service(self): sm = cotyledon.ServiceManager() self.assertRaisesMsg( ValueError, "'service' must be a callable", sm.add, u"foo") self.assertRaisesMsg( ValueError, "'workers' must be an int >= 1, not: None (NoneType)", sm.add, FakeService, workers=None) self.assertRaisesMsg( ValueError, "'workers' must be an int >= 1, not: -2 (int)", sm.add, FakeService, workers=-2) oid = sm.add(FakeService, workers=3) self.assertRaisesMsg( ValueError, "'workers' must be an int >= -2, not: -5 (int)", sm.reconfigure, oid, workers=-5) self.assertRaisesMsg( ValueError, "notexists service id doesn't exists", sm.reconfigure, "notexists", workers=-1) def assertRaisesMsg(self, exc, msg, func, *args, **kwargs): try: func(*args, **kwargs) except exc as e: self.assertEqual(msg, str(e)) else: self.assertFalse(True, "%r have not been raised" % exc) cotyledon-1.6.8/cotyledon/tests/test_functional.py0000644000175000017500000003650013134342466023257 0ustar silehtsileht00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import os import re import signal import socket import subprocess import threading import time import unittest from cotyledon import oslo_config_glue from cotyledon.tests import base if os.name == 'posix': def pid_exists(pid): """Check whether pid exists in the current process table.""" import errno if pid < 0: return False try: os.kill(pid, 0) except OSError as e: return e.errno == errno.EPERM else: return True else: def pid_exists(pid): import ctypes kernel32 = ctypes.windll.kernel32 SYNCHRONIZE = 0x100000 process = kernel32.OpenProcess(SYNCHRONIZE, 0, pid) if process != 0: kernel32.CloseHandle(process) return True else: return False class Base(base.TestCase): def setUp(self): super(Base, self).setUp() self.lines = [] self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.sock.bind(("127.0.0.1", 0)) self.t = threading.Thread(target=self.readlog) self.t.daemon = True self.t.start() examplepy = os.path.join(os.path.dirname(__file__), "examples.py") if os.name == 'posix': kwargs = { 'preexec_fn': os.setsid } else: kwargs = { 'creationflags': subprocess.CREATE_NEW_PROCESS_GROUP } self.subp = subprocess.Popen(['python', examplepy, self.name, str(self.sock.getsockname()[1])], **kwargs) def readlog(self): try: while True: data, addr = self.sock.recvfrom(65565) self.lines.append(data.strip()) except socket.error: pass def tearDown(self): if self.subp.poll() is None: self.subp.kill() super(Base, self).tearDown() def get_lines(self, number=None): if number is not None: while len(self.lines) < number: time.sleep(0.1) lines = self.lines[:number] del self.lines[:number] return lines else: self.subp.wait() # Wait children to terminate return self.lines @staticmethod def hide_pids(lines): return [re.sub(b"Child \d+", b"Child XXXX", re.sub(b" \[[^\]]*\]", b" [XXXX]", line)) for line in lines] @staticmethod def get_pid(line): try: return int(line.split()[-1][1:-1]) except Exception: raise Exception('Fail to find pid in %s' % line.split()) class TestCotyledon(Base): name = 'example_app' def assert_everything_has_started(self): lines = sorted(self.get_lines(7)) self.pid_heavy_1 = self.get_pid(lines[0]) self.pid_heavy_2 = self.get_pid(lines[1]) self.pid_light_1 = self.get_pid(lines[2]) lines = self.hide_pids(lines) self.assertEqual([ b'DEBUG:cotyledon._service:Run service heavy(0) [XXXX]', b'DEBUG:cotyledon._service:Run service heavy(1) [XXXX]', b'DEBUG:cotyledon._service:Run service light(0) [XXXX]', b'ERROR:cotyledon.tests.examples:heavy init', b'ERROR:cotyledon.tests.examples:heavy init', b'ERROR:cotyledon.tests.examples:heavy run', b'ERROR:cotyledon.tests.examples:heavy run' ], lines) self.assert_everything_is_alive() def assert_everything_is_alive(self): self.assertTrue(pid_exists(self.subp.pid)) self.assertTrue(pid_exists(self.pid_light_1)) self.assertTrue(pid_exists(self.pid_heavy_1)) self.assertTrue(pid_exists(self.pid_heavy_2)) def assert_everything_is_dead(self, status=0): self.assertEqual(status, self.subp.poll()) self.assertFalse(pid_exists(self.subp.pid)) self.assertFalse(pid_exists(self.pid_light_1)) self.assertFalse(pid_exists(self.pid_heavy_1)) self.assertFalse(pid_exists(self.pid_heavy_2)) @unittest.skipIf(os.name == 'posix', 'no window support') def test_workflow_window(self): # NOTE(sileht): The window workflow is a bit different because # SIGTERM doesn't really exists and processes are killed with SIGINT # FIXME(sileht): Implements SIGBREAK to have graceful exists self.assert_everything_has_started() # Ensure we restart with terminate method exit code os.kill(self.pid_heavy_1, signal.SIGTERM) lines = self.get_lines(4) lines = self.hide_pids(lines) self.assertEqual([ b'INFO:cotyledon._service_manager:Child XXXX exited ' b'with status 15', b'ERROR:cotyledon.tests.examples:heavy init', b'DEBUG:cotyledon._service:Run service heavy(0) [XXXX]', b'ERROR:cotyledon.tests.examples:heavy run', ], lines) # Kill master process os.kill(self.subp.pid, signal.SIGTERM) time.sleep(1) lines = self.get_lines() lines = sorted(self.hide_pids(lines)) self.assertEqual([ b'ERROR:cotyledon.tests.examples:heavy terminate', b'ERROR:cotyledon.tests.examples:heavy terminate', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service heavy(0) [XXXX]', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service heavy(1) [XXXX]', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service light(0) [XXXX]', b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, heavy(0) [XXXX] exiting', b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, heavy(1) [XXXX] exiting', b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, light(0) [XXXX] exiting', ], lines) self.assertEqual(15, self.subp.poll()) @unittest.skipIf(os.name != 'posix', 'no posix support') def test_workflow(self): self.assert_everything_has_started() # Ensure we just call reload method os.kill(self.pid_heavy_1, signal.SIGHUP) self.assertEqual([b"ERROR:cotyledon.tests.examples:heavy reload"], self.get_lines(1)) # Ensure we restart because reload method is missing os.kill(self.pid_light_1, signal.SIGHUP) lines = self.get_lines(3) self.pid_light_1 = self.get_pid(lines[-1]) lines = self.hide_pids(lines) self.assertEqual([ b'INFO:cotyledon._service:Caught SIGTERM signal, graceful ' b'exiting of service light(0) [XXXX]', b'INFO:cotyledon._service_manager:Child XXXX exited ' b'with status 0', b'DEBUG:cotyledon._service:Run service light(0) [XXXX]' ], lines) # Ensure we restart with terminate method exit code os.kill(self.pid_heavy_1, signal.SIGTERM) lines = self.get_lines(6) self.pid_heavy_1 = self.get_pid(lines[-2]) lines = self.hide_pids(lines) self.assertEqual([ b'INFO:cotyledon._service:Caught SIGTERM signal, graceful ' b'exiting of service heavy(0) [XXXX]', b'ERROR:cotyledon.tests.examples:heavy terminate', b'INFO:cotyledon._service_manager:Child XXXX exited ' b'with status 42', b'ERROR:cotyledon.tests.examples:heavy init', b'DEBUG:cotyledon._service:Run service heavy(0) [XXXX]', b'ERROR:cotyledon.tests.examples:heavy run', ], lines) # Ensure we restart when no terminate method os.kill(self.pid_light_1, signal.SIGTERM) lines = self.get_lines(3) self.pid_light_1 = self.get_pid(lines[-1]) lines = self.hide_pids(lines) self.assertEqual([ b'INFO:cotyledon._service:Caught SIGTERM signal, graceful ' b'exiting of service light(0) [XXXX]', b'INFO:cotyledon._service_manager:Child XXXX exited with status 0', b'DEBUG:cotyledon._service:Run service light(0) [XXXX]', ], lines) # Ensure everthing is still alive self.assert_everything_is_alive() # Kill master process os.kill(self.subp.pid, signal.SIGTERM) lines = self.get_lines() self.assertEqual(b'DEBUG:cotyledon._service_manager:Shutdown finish', lines[-1]) time.sleep(1) lines = sorted(self.hide_pids(lines)) self.assertEqual([ b'DEBUG:cotyledon._service_manager:Killing services with ' b'signal SIGTERM', b'DEBUG:cotyledon._service_manager:Shutdown finish', b'DEBUG:cotyledon._service_manager:Waiting services to terminate', b'ERROR:cotyledon.tests.examples:heavy terminate', b'ERROR:cotyledon.tests.examples:heavy terminate', b'ERROR:cotyledon.tests.examples:master terminate hook', b'ERROR:cotyledon.tests.examples:master terminate2 hook', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service heavy(0) [XXXX]', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service heavy(1) [XXXX]', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service light(0) [XXXX]', b'INFO:cotyledon._service_manager:Caught SIGTERM signal, ' b'graceful exiting of master process', ], lines) self.assert_everything_is_dead() @unittest.skipIf(os.name != 'posix', 'http://bugs.python.org/issue18040') def test_sigint(self): self.assert_everything_has_started() os.kill(self.subp.pid, signal.SIGINT) time.sleep(1) lines = sorted(self.get_lines()) lines = self.hide_pids(lines) self.assertEqual([ b'INFO:cotyledon._service_manager:Caught SIGINT signal, ' b'instantaneous exiting', ], lines) self.assert_everything_is_dead(1) @unittest.skipIf(os.name != 'posix', 'no posix support') def test_sighup(self): self.assert_everything_has_started() os.kill(self.subp.pid, signal.SIGHUP) time.sleep(0.5) lines = sorted(self.get_lines(6)) lines = self.hide_pids(lines) self.assertEqual([ b'DEBUG:cotyledon._service:Run service light(0) [XXXX]', b'ERROR:cotyledon.tests.examples:heavy reload', b'ERROR:cotyledon.tests.examples:heavy reload', b'ERROR:cotyledon.tests.examples:master reload hook', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service light(0) [XXXX]', b'INFO:cotyledon._service_manager:Child XXXX exited with status 0' ], lines) os.kill(self.subp.pid, signal.SIGINT) time.sleep(0.5) self.assert_everything_is_dead(1) @unittest.skipIf(os.name != 'posix', 'no posix support') def test_sigkill(self): self.assert_everything_has_started() self.subp.kill() time.sleep(1) lines = sorted(self.get_lines()) lines = self.hide_pids(lines) self.assertEqual([ b'ERROR:cotyledon.tests.examples:heavy terminate', b'ERROR:cotyledon.tests.examples:heavy terminate', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service heavy(0) [XXXX]', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service heavy(1) [XXXX]', b'INFO:cotyledon._service:Caught SIGTERM signal, ' b'graceful exiting of service light(0) [XXXX]', b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, heavy(0) [XXXX] exiting', b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, heavy(1) [XXXX] exiting', b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, light(0) [XXXX] exiting', ], lines) self.assert_everything_is_dead(-9) class TestBadlyCodedCotyledon(Base): name = "badly_coded_app" @unittest.skipIf(os.name != 'posix', 'no posix support') def test_badly_coded(self): time.sleep(0.5) self.subp.terminate() time.sleep(0.5) self.assertEqual(0, self.subp.poll(), self.get_lines()) self.assertFalse(pid_exists(self.subp.pid)) class TestBuggyCotyledon(Base): name = "buggy_app" @unittest.skipIf(os.name != 'posix', 'no posix support') def test_graceful_timeout_term(self): lines = self.get_lines(1) childpid = self.get_pid(lines[0]) self.subp.terminate() time.sleep(2) self.assertEqual(0, self.subp.poll()) self.assertFalse(pid_exists(self.subp.pid)) self.assertFalse(pid_exists(childpid)) lines = self.hide_pids(self.get_lines()) self.assertNotIn('ERROR:cotyledon.tests.examples:time.sleep done', lines) self.assertEqual([ b'INFO:cotyledon._service:Graceful shutdown timeout (1) exceeded, ' b'exiting buggy(0) [XXXX] now.', b'DEBUG:cotyledon._service_manager:Shutdown finish' ], lines[-2:]) @unittest.skipIf(os.name != 'posix', 'no posix support') def test_graceful_timeout_kill(self): lines = self.get_lines(1) childpid = self.get_pid(lines[0]) self.subp.kill() time.sleep(2) self.assertEqual(-9, self.subp.poll()) self.assertFalse(pid_exists(self.subp.pid)) self.assertFalse(pid_exists(childpid)) lines = self.hide_pids(self.get_lines()) self.assertNotIn('ERROR:cotyledon.tests.examples:time.sleep done', lines) self.assertEqual([ b'INFO:cotyledon._service:Parent process has died ' b'unexpectedly, buggy(0) [XXXX] exiting', b'INFO:cotyledon._service:Caught SIGTERM signal, graceful exiting ' b'of service buggy(0) [XXXX]', b'INFO:cotyledon._service:Graceful shutdown timeout (1) exceeded, ' b'exiting buggy(0) [XXXX] now.', ], lines[-3:]) class TestOsloCotyledon(Base): name = "oslo_app" def test_options(self): options = oslo_config_glue.list_opts() self.assertEqual(1, len(options)) self.assertEqual(None, options[0][0]) self.assertEqual(2, len(options[0][1])) lines = self.get_lines(1) self.assertIn( b'DEBUG:cotyledon.oslo_config_glue:Full set of CONF:', lines) self.subp.terminate() cotyledon-1.6.8/cotyledon/tests/examples.py0000644000175000017500000000646313134342466021701 0ustar silehtsileht00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import logging import os import signal import socket import sys import threading import time from oslo_config import cfg import cotyledon from cotyledon import _utils from cotyledon import oslo_config_glue if len(sys.argv) >= 3: s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(("127.0.0.1", int(sys.argv[2]))) if os.name == "posix": stream = os.fdopen(s.fileno(), 'w') else: stream = s.makefile() logging.basicConfig(level=logging.DEBUG, stream=stream) else: logging.basicConfig(level=logging.DEBUG) LOG = logging.getLogger("cotyledon.tests.examples") # We don't want functional tests to wait for this: cotyledon.ServiceManager._slowdown_respawn_if_needed = lambda *args: True class FullService(cotyledon.Service): name = "heavy" def __init__(self, worker_id): super(FullService, self).__init__(worker_id) self._shutdown = threading.Event() LOG.error("%s init" % self.name) def run(self): LOG.error("%s run" % self.name) self._shutdown.wait() def terminate(self): LOG.error("%s terminate" % self.name) self._shutdown.set() sys.exit(42) def reload(self): LOG.error("%s reload" % self.name) class LigthService(cotyledon.Service): name = "light" class BuggyService(cotyledon.Service): name = "buggy" graceful_shutdown_timeout = 1 def terminate(self): time.sleep(60) LOG.error("time.sleep done") class BadlyCodedService(cotyledon.Service): def run(self): raise Exception("so badly coded service") class OsloService(cotyledon.Service): name = "oslo" class WindowService(cotyledon.Service): name = "window" def on_terminate(): LOG.error("master terminate hook") def on_terminate2(): LOG.error("master terminate2 hook") def on_reload(): LOG.error("master reload hook") def example_app(): p = cotyledon.ServiceManager() p.add(FullService, 2) service_id = p.add(LigthService, 5) p.reconfigure(service_id, 1) p.register_hooks(on_terminate, on_reload) p.register_hooks(on_terminate2) p.run() def buggy_app(): p = cotyledon.ServiceManager() p.add(BuggyService) p.run() def oslo_app(): conf = cfg.ConfigOpts() conf([], project='openstack-app', validate_default_values=True, version="0.1") p = cotyledon.ServiceManager() oslo_config_glue.setup(p, conf) p.add(OsloService) p.run() def window_sanity_check(): p = cotyledon.ServiceManager() p.add(LigthService) t = _utils.spawn(p.run) time.sleep(10) os.kill(os.getpid(), signal.SIGTERM) t.join() def badly_coded_app(): p = cotyledon.ServiceManager() p.add(BadlyCodedService) p.run() if __name__ == '__main__': globals()[sys.argv[1]]() cotyledon-1.6.8/cotyledon/tests/__init__.py0000644000175000017500000000000012706122176021575 0ustar silehtsileht00000000000000cotyledon-1.6.8/cotyledon/_utils.py0000644000175000017500000001470213042134502020177 0ustar silehtsileht00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import collections import contextlib import errno import logging import multiprocessing import os import select import signal import sys import threading import time if os.name == 'posix': import fcntl LOG = logging.getLogger(__name__) _SIGNAL_TO_NAME = dict((getattr(signal, name), name) for name in dir(signal) if name.startswith("SIG") and name not in ('SIG_DFL', 'SIG_IGN')) def signal_to_name(sig): return _SIGNAL_TO_NAME.get(sig) def spawn(target, *args, **kwargs): t = threading.Thread(target=target, args=args, kwargs=kwargs) t.daemon = True t.start() return t def check_workers(workers, minimum): if not isinstance(workers, int) or workers < minimum: raise ValueError("'workers' must be an int >= %d, not: %s (%s)" % (minimum, workers, type(workers).__name__)) def check_callable(thing, name): if not hasattr(thing, "__call__"): raise ValueError("'%s' must be a callable" % name) def _bootstrap_process(target, *args, **kwargs): if "fds_to_close" in kwargs: for fd in kwargs["fds_to_close"]: os.close(fd) del kwargs["fds_to_close"] target(*args, **kwargs) def spawn_process(*args, **kwargs): p = multiprocessing.Process(target=_bootstrap_process, args=args, kwargs=kwargs) p.start() return p try: from setproctitle import setproctitle except ImportError: def setproctitle(*args, **kwargs): pass def get_process_name(): return os.path.basename(sys.argv[0]) def run_hooks(name, hooks, *args, **kwargs): try: for hook in hooks: hook(*args, **kwargs) except Exception: LOG.exception("Exception raised during %s hooks" % name) @contextlib.contextmanager def exit_on_exception(): try: yield except SystemExit as exc: os._exit(exc.code) except BaseException: LOG.exception('Unhandled exception') os._exit(2) if os.name == "posix": SIGALRM = signal.SIGALRM SIGHUP = signal.SIGHUP SIGCHLD = signal.SIGCHLD SIBREAK = None else: SIGALRM = SIGHUP = None SIGCHLD = "fake sigchld" SIGBREAK = signal.SIGBREAK class SignalManager(object): def __init__(self, master=False): # Setup signal fd, this allows signal to behave correctly if os.name == 'posix': self.signal_pipe_r, self.signal_pipe_w = os.pipe() self._set_nonblock(self.signal_pipe_r) self._set_nonblock(self.signal_pipe_w) signal.set_wakeup_fd(self.signal_pipe_w) self._signals_received = collections.deque() signal.signal(signal.SIGINT, signal.SIG_DFL) if os.name == 'posix': signal.signal(signal.SIGCHLD, signal.SIG_DFL) signal.signal(signal.SIGTERM, self._signal_catcher) signal.signal(signal.SIGALRM, self._signal_catcher) signal.signal(signal.SIGHUP, self._signal_catcher) else: # currently a noop on window... signal.signal(signal.SIGTERM, self._signal_catcher) # FIXME(sileht): should allow to catch signal CTRL_BREAK_EVENT, # but we to create the child process with CREATE_NEW_PROCESS_GROUP # to make this work, so current this is a noop for later fix signal.signal(signal.SIGBREAK, self._signal_catcher) @staticmethod def _set_nonblock(fd): flags = fcntl.fcntl(fd, fcntl.F_GETFL, 0) flags = flags | os.O_NONBLOCK fcntl.fcntl(fd, fcntl.F_SETFL, flags) def _signal_catcher(self, sig, frame): # NOTE(sileht): This is useful only for python < 3.5 # in python >= 3.5 we could read the signal number # from the wakeup_fd pipe if sig in (SIGALRM, signal.SIGTERM): self._signals_received.appendleft(sig) else: self._signals_received.append(sig) def _wait_forever(self): # Wait forever while True: # Check if signals have been received if os.name == "posix": self._empty_signal_pipe() self._run_signal_handlers() if os.name == "posix": # NOTE(sileht): we cannot use threading.Event().wait(), # threading.Thread().join(), or time.sleep() because signals # can be missed when received by non-main threads # (https://bugs.python.org/issue5315) # So we use select.select() alone, we will receive EINTR or # will read data from signal_r when signal is emitted and # cpython calls PyErr_CheckSignals() to run signals handlers # That looks perfect to ensure handlers are run and run in the # main thread try: select.select([self.signal_pipe_r], [], []) except select.error as e: if e.args[0] != errno.EINTR: raise else: # NOTE(sileht): here we do only best effort # and wake the loop periodically, set_wakeup_fd # doesn't work on non posix platform so # 1 seconds have been picked with the advice of a dice. time.sleep(1) # NOTE(sileht): We emulate SIGCHLD, _service_manager # will just check often for dead child self._signals_received.append(SIGCHLD) def _empty_signal_pipe(self): try: while os.read(self.signal_pipe_r, 4096) == 4096: pass except (IOError, OSError): pass def _run_signal_handlers(self): while True: try: sig = self._signals_received.popleft() except IndexError: return self._on_signal_received(sig) def _on_signal_received(self, sig): pass cotyledon-1.6.8/cotyledon/_service.py0000644000175000017500000001767613037231434020522 0ustar silehtsileht00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import logging import os import random import signal import sys import threading from cotyledon import _utils LOG = logging.getLogger(__name__) class Service(object): """Base class for a service This class will be executed in a new child process/worker :py:class:`ServiceWorker` of a :py:class:`ServiceManager`. It registers signals to manager the reloading and the ending of the process. Methods :py:meth:`run`, :py:meth:`terminate` and :py:meth:`reload` are optional. """ name = None """Service name used in the process title and the log messages in additionnal of the worker_id.""" graceful_shutdown_timeout = None """Timeout after which a gracefully shutdown service will exit. zero means endless wait. None means same as ServiceManager that launch the service""" def __init__(self, worker_id): """Create a new Service :param worker_id: the identifier of this service instance :type worker_id: int The identifier of the worker can be used for workload repartition because it's consistent and always the same. For example, if the number of workers for this service is 3, one will got 0, the second got 1 and the last got 2. if worker_id 1 died, the new spawned process will got 1 again. """ super(Service, self).__init__() self._initialize(worker_id) def _initialize(self, worker_id): if getattr(self, '_initialized', False): return self._initialized = True if self.name is None: self.name = self.__class__.__name__ self.worker_id = worker_id self.pid = os.getpid() self._signal_lock = threading.Lock() # Only used by oslo_config_glue for now, so we don't need # to have a list of hook self._on_reload_internal_hook = self._noop_hook def _noop_hook(self, service): pass def terminate(self): """Gracefully shutdown the service This method will be executed when the Service has to shutdown cleanly. If not implemented the process will just end with status 0. To customize the exit code, the :py:class:`SystemExit` exception can be used. Any exceptions raised by this method will be logged and the worker will exit with status 1. """ def reload(self): """Reloading of the service This method will be executed when the Service receives a SIGHUP. If not implemented the process will just end with status 0 and :py:class:`ServiceRunner` will start a new fresh process for this service with the same worker_id. Any exceptions raised by this method will be logged and the worker will exit with status 1. """ os.kill(os.getpid(), signal.SIGTERM) def run(self): """Method representing the service activity If not implemented the process will just wait to receive an ending signal. This method is ran into the thread and can block or return as needed Any exceptions raised by this method will be logged and the worker will exit with status 1. """ # Helper to run application methods in a safety way when signal are # received def _reload(self): with _utils.exit_on_exception(): if self._signal_lock.acquire(False): try: self._on_reload_internal_hook(self) self.reload() finally: self._signal_lock.release() def _terminate(self): with _utils.exit_on_exception(), self._signal_lock: self.terminate() sys.exit(0) def _run(self): with _utils.exit_on_exception(): self.run() class ServiceConfig(object): def __init__(self, service_id, service, workers, args, kwargs): self.service = service self.workers = workers self.args = args self.kwargs = kwargs self.service_id = service_id class ServiceWorker(_utils.SignalManager): """Service Worker Wrapper This represents the child process spawned by ServiceManager All methods implemented here, must run in the main threads """ @classmethod def create_and_wait(cls, *args, **kwargs): sw = cls(*args, **kwargs) sw.wait_forever() def __init__(self, config, service_id, worker_id, parent_pipe, started_hooks, graceful_shutdown_timeout): super(ServiceWorker, self).__init__() self._ready = threading.Event() _utils.spawn(self._watch_parent_process, parent_pipe) # Reseed random number generator random.seed() args = tuple() if config.args is None else config.args kwargs = dict() if config.kwargs is None else config.kwargs self.service = config.service(worker_id, *args, **kwargs) self.service._initialize(worker_id) if self.service.graceful_shutdown_timeout is None: self.service.graceful_shutdown_timeout = graceful_shutdown_timeout self.title = "%(name)s(%(worker_id)d) [%(pid)d]" % dict( name=self.service.name, worker_id=worker_id, pid=os.getpid()) # Set process title _utils.setproctitle( "%(pname)s: %(name)s worker(%(worker_id)d)" % dict( pname=_utils.get_process_name(), name=self.service.name, worker_id=worker_id)) # We are ready tell them self._ready.set() _utils.run_hooks('new_worker', started_hooks, service_id, worker_id, self.service) def _watch_parent_process(self, parent_pipe): # This will block until the write end is closed when the parent # dies unexpectedly parent_pipe[1].close() try: parent_pipe[0].recv() except EOFError: pass if self._ready.is_set(): LOG.info('Parent process has died unexpectedly, %s exiting' % self.title) if os.name == "posix": os.kill(os.getpid(), signal.SIGTERM) else: # Fallback to process signal later self._signals_received.appendleft(signal.SIGTERM) else: os._exit(0) def _alarm(self): LOG.info('Graceful shutdown timeout (%d) exceeded, ' 'exiting %s now.' % (self.service.graceful_shutdown_timeout, self.title)) os._exit(1) def _on_signal_received(self, sig): # Code below must not block to return to select.select() and catch # next signals if sig == _utils.SIGALRM: self._alarm() elif sig == signal.SIGTERM: LOG.info('Caught SIGTERM signal, ' 'graceful exiting of service %s' % self.title) if self.service.graceful_shutdown_timeout > 0: if os.name == "posix": signal.alarm(self.service.graceful_shutdown_timeout) else: threading.Timer(self.service.graceful_shutdown_timeout, self._alarm).start() _utils.spawn(self.service._terminate) elif sig == _utils.SIGHUP: _utils.spawn(self.service._reload) def wait_forever(self): LOG.debug("Run service %s" % self.title) _utils.spawn(self.service._run) super(ServiceWorker, self)._wait_forever() cotyledon-1.6.8/cotyledon/oslo_config_glue.py0000644000175000017500000000750413025030376022224 0ustar silehtsileht00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import copy import functools import logging import os from oslo_config import cfg LOG = logging.getLogger(__name__) service_opts = [ cfg.BoolOpt('log_options', default=True, mutable=True, help='Enables or disables logging values of all ' 'registered options when starting a service (at DEBUG ' 'level).'), cfg.IntOpt('graceful_shutdown_timeout', mutable=True, default=60, help='Specify a timeout after which a gracefully shutdown ' 'server will exit. Zero value means endless wait.'), ] def _load_service_manager_options(service_manager, conf): service_manager.graceful_shutdown_timeout = conf.graceful_shutdown_timeout if conf.log_options: LOG.debug('Full set of CONF:') conf.log_opt_values(LOG, logging.DEBUG) def _load_service_options(service, conf): service.graceful_shutdown_timeout = conf.graceful_shutdown_timeout if conf.log_options: LOG.debug('Full set of CONF:') conf.log_opt_values(LOG, logging.DEBUG) def _configfile_reload(conf, reload_method): if reload_method == 'reload': conf.reload_config_files() elif reload_method == 'mutate': conf.mutate_config_files() def _new_worker_hook(conf, reload_method, service_id, worker_id, service): def _service_reload(service): _configfile_reload(conf, reload_method) _load_service_options(service, conf) service._on_reload_internal_hook = _service_reload _load_service_options(service, conf) def setup(service_manager, conf, reload_method="reload"): """Load services configuration from oslo config object. It reads ServiceManager and Service configuration options from an oslo_config.ConfigOpts() object. Also It registers a ServiceManager hook to reload the configuration file on reload in the master process and in all children. And then when each child start or reload, the configuration options are logged if the oslo config option 'log_options' is True. On children, the configuration file is reloaded before the running the application reload method. Options currently supported on ServiceManager and Service: * graceful_shutdown_timeout :param service_manager: ServiceManager instance :type service_manager: cotyledon.ServiceManager :param conf: Oslo Config object :type conf: oslo_config.ConfigOpts() :param reload_method: reload or mutate the config files :type reload_method: str "reload/mutate" """ conf.register_opts(service_opts) # Set cotyledon options from oslo config options _load_service_manager_options(service_manager, conf) def _service_manager_reload(): _configfile_reload(conf, reload_method) _load_service_manager_options(service_manager, conf) if os.name != "posix": # NOTE(sileht): reloading can't be supported oslo.config is not pickle # But we don't care SIGHUP is not support on window return service_manager.register_hooks( on_new_worker=functools.partial( _new_worker_hook, conf, reload_method), on_reload=_service_manager_reload) def list_opts(): """Entry point for oslo-config-generator.""" return [(None, copy.deepcopy(service_opts))] cotyledon-1.6.8/.mailmap0000644000175000017500000000013112706002773015750 0ustar silehtsileht00000000000000# Format is: # # cotyledon-1.6.8/setup.py0000644000175000017500000000177712706002773016062 0ustar silehtsileht00000000000000# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT import setuptools # In python < 2.7.4, a lazy loading of package `pbr` will break # setuptools if some other modules registered functions in `atexit`. # solution from: http://bugs.python.org/issue15881#msg170215 try: import multiprocessing # noqa except ImportError: pass setuptools.setup( setup_requires=['pbr'], pbr=True) cotyledon-1.6.8/release.sh0000755000175000017500000000151413037233132016305 0ustar silehtsileht00000000000000#!/bin/bash set -e set -x version=$1 [ ! "$version"] && version=$(python setup.py --version | sed 's/\.dev.*//') status=$(git status -sz) [ -z "$status" ] || false git checkout master tox -epy35,py27,pep8 git push git tag -s $version -m "Release version ${version}" git checkout $version git clean -fd pbr_version=$(python setup.py --version) if [ "$version" != "$pbr_version" ]; then echo "something goes wrong pbr version is different from the provided one. ($pbr_version != $version)" exit 1 fi python setup.py sdist bdist_wheel set +x echo echo "release: Cotyledon ${version}" echo echo "SHA1sum: " sha1sum dist/* echo "MD5sum: " md5sum dist/* echo echo "uploading..." echo set -x read git push --tags twine upload -r pypi -s dist/cotyledon-${version}.tar.gz dist/cotyledon-${version}-py2.py3-none-any.whl git checkout master cotyledon-1.6.8/.travis.yml0000644000175000017500000000026413025030230016427 0ustar silehtsileht00000000000000sudo: false language: python python: 3.5 install: - pip install tox script: - tox env: - TOXENV=py27 - TOXENV=py35 - TOXENV=pep8 - TOXENV=docs cotyledon-1.6.8/setup.cfg0000644000175000017500000000167513134343247016166 0ustar silehtsileht00000000000000[metadata] name = cotyledon summary = Cotyledon provides a framework for defining long-running services. description-file = README.rst author = Mehdi Abaakouk author-email = sileht@sileht.net home-page = https://github.com/sileht/cotyledon classifier = Intended Audience :: Information Technology Intended Audience :: System Administrators License :: OSI Approved :: Apache Software License Operating System :: POSIX :: Linux Programming Language :: Python Programming Language :: Python :: 2 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.4 Programming Language :: Python :: 3.5 [files] packages = cotyledon [entry_points] oslo.config.opts = cotyledon = cotyledon.oslo_config_glue:list_opts [build_sphinx] source-dir = doc/source build-dir = doc/build all_files = 1 [upload_sphinx] upload-dir = doc/build/html [wheel] universal = 1 [egg_info] tag_build = tag_date = 0 cotyledon-1.6.8/babel.cfg0000644000175000017500000000002112706122330016043 0ustar silehtsileht00000000000000[python: **.py]