pax_global_header00006660000000000000000000000064132335733340014520gustar00rootroot0000000000000052 comment=b6b5688f31c2fc57a7e04a74cc524130b4096fe2 django-dbconn-retry-0.1.5/000077500000000000000000000000001323357333400153715ustar00rootroot00000000000000django-dbconn-retry-0.1.5/.gitignore000066400000000000000000000000731323357333400173610ustar00rootroot00000000000000.coverage *.iml *.pyc *.pyd build/ dist/ test/ *.egg-info/ django-dbconn-retry-0.1.5/.travis.yml000066400000000000000000000006511323357333400175040ustar00rootroot00000000000000language: python python: - "3.5" - "3.6" addons: postgresql: "9.6" services: - postgresql sudo: false install: - pip install tox before_script: - sudo -u postgres createuser -s -p 5432 dbconntest &>/dev/null - sudo -u postgres createdb -O dbconntest -p 5432 dbconntest script: - if [[ $TRAVIS_PYTHON_VERSION == 3* ]]; then tox; fi matrix: fast_finish: true notifications: email: false django-dbconn-retry-0.1.5/LICENSE000066400000000000000000000027321323357333400164020ustar00rootroot00000000000000Copyright (c) 2018, Jonas Maurus All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. django-dbconn-retry-0.1.5/MANIFEST.in000066400000000000000000000001251323357333400171250ustar00rootroot00000000000000recursive-include django_dbconn_retry/tests *.py recursive-include test_project *.py django-dbconn-retry-0.1.5/README.rst000066400000000000000000000117451323357333400170700ustar00rootroot00000000000000Django Database Connection Autoreconnect ======================================== .. image:: https://coveralls.io/repos/github/jdelic/django-dbconn-retry/badge.svg?branch=HEAD :target: https://coveralls.io/github/jdelic/django-dbconn-retry?branch=HEAD .. image:: https://travis-ci.org/jdelic/django-dbconn-retry.svg?branch=master :target: https://travis-ci.org/jdelic/django-dbconn-retry This library monkeypatches ``django.db.backends.base.BaseDatabaseWrapper`` so that when a database operation fails because the underlying TCP connection was already closed, it first tried to reconnect, instead of immediately raising an ``OperationException``. Why is this useful? ------------------- I use `HAProxy`_ as a load-balancer in front of my PostgreSQL databases all the time, sometimes in addition to ``pgbouncer``. Even though you can mostly prevent surprises by enabling TCP keep-alive packets through `tcpka`_, `clitcpka`_ and `srvtcpka`_, I still encounter situations where the underlying TCP connection has been closed through the load-balancer. Most often this results in .. code-block:: django.db.utils.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. This library patches Django such that it try to reconnect once before failing. Another application of this is when using `Hashicorp Vault`_, where credentials for a database connection can expire at any time and then need to be refreshed from Vault. How to install? --------------- Just pull the library in using ``pip install django-dbconn-retry``. Then add ``django_dbconn_retry`` to ``INSTALLED_APPS`` in your ``settings.py``. Signals ------- The library provides an interface for other code to plug into the process to, for example, allow `12factor-vault`_ to refresh the database credentials before the code tries to reestablish the database connection. These are implemented using `Django Signals`_. =========================== ================================================== Signal Description =========================== ================================================== ``pre_reconnect`` Installs a hook of the type ``Callable[[type, BaseDatabaseWrapper], None]`` that will be called before the library tries to reestablish a connection. 12factor-vault uses this to refresh the database credentials from Vault. ``post_reconnect`` Installs a hook of the type ``Callable[[type, BaseDatabaseWrapper], None]`` that will be called after the library tried to reestablish the connection. Success or failure has not been tested at this point. So the connection may be in any state. =========================== ================================================== License ======= Copyright (c) 2018, Jonas Maurus All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .. _12factor-vault: https://github.com/jdelic/12factor-vault/ .. _Django Signals: https://docs.djangoproject.com/en/dev/topics/signals/ .. _HAProxy: http://www.haproxy.org/ .. _tcpka: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#option%20tcpka .. _clitcpka: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-option%20clitcpka .. _srvtcpka: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#option%20srvtcpka .. _Hashicorp Vault: https://vaultproject.io/ django-dbconn-retry-0.1.5/django_dbconn_retry/000077500000000000000000000000001323357333400214035ustar00rootroot00000000000000django-dbconn-retry-0.1.5/django_dbconn_retry/__init__.py000066400000000000000000000065171323357333400235250ustar00rootroot00000000000000# -* encoding: utf-8 *- import logging from django.apps.config import AppConfig from django.db import utils as django_db_utils from django.db.backends.base import base as django_db_base from django.dispatch import Signal from typing import Union, Tuple, Callable, List # noqa. flake8 #247 _log = logging.getLogger(__name__) default_app_config = 'django_dbconn_retry.DjangoIntegration' pre_reconnect = Signal(providing_args=["dbwrapper"]) post_reconnect = Signal(providing_args=["dbwrapper"]) _operror_types = () # type: Union[Tuple[type], Tuple] _operror_types += (django_db_utils.OperationalError,) try: import psycopg2 except ImportError: pass else: _operror_types += (psycopg2.OperationalError,) try: import sqlite3 except ImportError: pass else: _operror_types += (sqlite3.OperationalError,) try: import MySQLdb except ImportError: pass else: _operror_types += (MySQLdb.OperationalError,) def monkeypatch_django() -> None: def ensure_connection_with_retries(self: django_db_base.BaseDatabaseWrapper) -> None: if self.connection is not None and hasattr(self.connection, 'closed') and self.connection.closed: _log.debug("failed connection detected") self.connection = None if self.connection is None and not hasattr(self, '_in_connecting'): with self.wrap_database_errors: try: self._in_connecting = True self.connect() except Exception as e: if isinstance(e, _operror_types): if hasattr(self, "_connection_retries") and self._connection_retries >= 1: _log.error("Reconnecting to the database didn't help %s", str(e)) del self._in_connecting post_reconnect.send(self.__class__, dbwrapper=self) raise else: _log.info("Database connection failed. Refreshing...") # mark the retry self._connection_retries = 1 # ensure that we retry the connection. Sometimes .closed isn't set correctly. self.connection = None del self._in_connecting # give libraries like 12factor-vault the chance to update the credentials pre_reconnect.send(self.__class__, dbwrapper=self) self.ensure_connection() post_reconnect.send(self.__class__, dbwrapper=self) else: _log.debug("Database connection failed, but not due to a known error for dbconn_retry %s", str(e)) del self._in_connecting raise else: # connection successful, reset the flag self._connection_retries = 0 del self._in_connecting _log.debug("django_dbconn_retry: monkeypatching BaseDatabaseWrapper") django_db_base.BaseDatabaseWrapper.ensure_connection = ensure_connection_with_retries class DjangoIntegration(AppConfig): name = "django_dbconn_retry" def ready(self) -> None: monkeypatch_django() django-dbconn-retry-0.1.5/django_dbconn_retry/tests/000077500000000000000000000000001323357333400225455ustar00rootroot00000000000000django-dbconn-retry-0.1.5/django_dbconn_retry/tests/__init__.py000066400000000000000000000066571323357333400246740ustar00rootroot00000000000000# -* encoding: utf-8 *- import sys import logging from unittest.mock import Mock from typing import Any import django_dbconn_retry as ddr from django.db.backends.base.base import BaseDatabaseWrapper from django.db import connection, OperationalError, transaction from django.test import TestCase logging.basicConfig(stream=sys.stderr) logging.getLogger("django_dbconn_retry").setLevel(logging.DEBUG) _log = logging.getLogger(__name__) class FullErrorTests(TestCase): """ This is SUPERHACKY. I couldn't find a better way to ensure that the database connections reliably fail. If I had been able to think of a better way, I'd have used it. """ def test_getting_root(self) -> None: self.client.get('/') def setUp(self) -> None: _log.debug("[FullErrorTests] patching for setup") self.s_connect = BaseDatabaseWrapper.connect BaseDatabaseWrapper.connect = Mock(side_effect=OperationalError('fail testing')) BaseDatabaseWrapper.connection = property(lambda x: None, lambda x, y: None) # type: ignore def tearDown(self) -> None: _log.debug("[FullErrorTests] restoring") BaseDatabaseWrapper.connect = self.s_connect del BaseDatabaseWrapper.connection def test_prehook(self) -> None: cb = Mock(name='pre_reconnect_hook') ddr.pre_reconnect.connect(cb) self.assertRaises(OperationalError, connection.ensure_connection) self.assertTrue(cb.called) del connection._connection_retries def test_posthook(self) -> None: cb = Mock(name='post_reconnect_hook') ddr.post_reconnect.connect(cb) self.assertRaises(OperationalError, connection.ensure_connection) self.assertTrue(cb.called) del connection._connection_retries def fix_connection(sender: type, *, dbwrapper: BaseDatabaseWrapper, **kwargs: Any) -> None: dbwrapper.connect = dbwrapper.s_connect class ReconnectTests(TestCase): @classmethod def tearDownClass(cls) -> None: return def test_ensure_closed(self) -> None: from django.db import connection connection.close() self.assertFalse(connection.is_usable()) # should be true after setUp def test_prehook(self) -> None: cb = Mock(name='pre_reconnect_hook') ddr.pre_reconnect.connect(fix_connection) ddr.pre_reconnect.connect(cb) from django.db import connection connection.close() connection.s_connect = connection.connect connection.connect = Mock(side_effect=OperationalError('reconnect testing')) connection.ensure_connection() ReconnectTests.cls_atomics['default'] = transaction.atomic(using='default') ReconnectTests.cls_atomics['default'].__enter__() self.assertTrue(cb.called) self.assertTrue(connection.is_usable()) def test_posthook(self) -> None: cb = Mock(name='post_reconnect_hook') ddr.pre_reconnect.connect(fix_connection) ddr.post_reconnect.connect(cb) from django.db import connection connection.close() connection.s_connect = connection.connect connection.connect = Mock(side_effect=OperationalError('reconnect testing')) connection.ensure_connection() ReconnectTests.cls_atomics['default'] = transaction.atomic(using='default') ReconnectTests.cls_atomics['default'].__enter__() self.assertTrue(cb.called) self.assertTrue(connection.is_usable()) django-dbconn-retry-0.1.5/requirements-test.txt000066400000000000000000000001311323357333400216250ustar00rootroot00000000000000coverage==4.4.1 coveralls==1.2.0 flake8==3.5.0 doc8==0.8.0 mypy==0.540 psycopg2==2.7.3.2 django-dbconn-retry-0.1.5/setup.py000066400000000000000000000021411323357333400171010ustar00rootroot00000000000000#!/usr/bin/env python # -* encoding: utf-8 *- import os from setuptools import setup HERE = os.path.dirname(__file__) try: long_description = open(os.path.join(HERE, 'README.rst')).read() except IOError: long_description = None setup( name="django-dbconn-retry", version="0.1.5", packages=[ 'django_dbconn_retry', 'django_dbconn_retry.tests', ], package_dir={ '': '.', }, classifiers=[ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "Intended Audience :: System Administrators", "Programming Language :: Python :: 3 :: Only", "License :: OSI Approved :: BSD License", "Operating System :: POSIX", ], url="https://github.com/jdelic/django-dbconn-retry/", author="Jonas Maurus (@jdelic)", author_email="jonas-dbconn-retry@gopythongo.com", maintainer="GoPythonGo.com", maintainer_email="info@gopythongo.com", description="Patch Django to retry a database connection first before failing.", long_description=long_description, install_requires=[ ], ) django-dbconn-retry-0.1.5/test_project/000077500000000000000000000000001323357333400200765ustar00rootroot00000000000000django-dbconn-retry-0.1.5/test_project/manage.py000066400000000000000000000014721323357333400217040ustar00rootroot00000000000000#!/usr/bin/env python import os import sys if __name__ == "__main__": os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_project.settings") try: from django.core.management import execute_from_command_line except ImportError: # The above import may fail for some other reason. Ensure that the # issue is really that Django is missing to avoid masking other # exceptions on Python 2. try: import django # noqa, flake8 except ImportError: raise ImportError( "Couldn't import Django. Are you sure it's installed and " "available on your PYTHONPATH environment variable? Did you " "forget to activate a virtual environment?" ) raise execute_from_command_line(sys.argv) django-dbconn-retry-0.1.5/test_project/test_project/000077500000000000000000000000001323357333400226035ustar00rootroot00000000000000django-dbconn-retry-0.1.5/test_project/test_project/__init__.py000066400000000000000000000000001323357333400247020ustar00rootroot00000000000000django-dbconn-retry-0.1.5/test_project/test_project/settings.py000066400000000000000000000076021323357333400250220ustar00rootroot00000000000000""" Django settings for test_project project. Generated by 'django-admin startproject' using Django 1.11.9. For more information on this file, see https://docs.djangoproject.com/en/1.11/topics/settings/ For the full list of settings and their values, see https://docs.djangoproject.com/en/1.11/ref/settings/ """ import os # Build paths inside the project like this: os.path.join(BASE_DIR, ...) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # Quick-start development settings - unsuitable for production # See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/ # SECURITY WARNING: keep the secret key used in production secret! SECRET_KEY = ')@(hymhz_m+mhgj$xnzmc=mx10we(-nhm*%jl(&bki7+ck7d_8' # SECURITY WARNING: don't run with debug turned on in production! DEBUG = True ALLOWED_HOSTS = [] # Application definition INSTALLED_APPS = [ 'django_dbconn_retry', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', ] MIDDLEWARE = [ 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', ] ROOT_URLCONF = 'test_project.urls' TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] WSGI_APPLICATION = 'test_project.wsgi.application' # Database # https://docs.djangoproject.com/en/1.11/ref/settings/#databases DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': 'dbconntest', 'USER': 'dbconntest', 'HOST': 'localhost', 'PASSWORD': 'dbconntest', } } # Password validation # https://docs.djangoproject.com/en/1.11/ref/settings/#auth-password-validators AUTH_PASSWORD_VALIDATORS = [ { 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', }, { 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', }, { 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', }, { 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', }, ] # Internationalization # https://docs.djangoproject.com/en/1.11/topics/i18n/ LANGUAGE_CODE = 'en-us' TIME_ZONE = 'UTC' USE_I18N = True USE_L10N = True USE_TZ = True # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/1.11/howto/static-files/ STATIC_URL = '/static/' LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { "simple": { "format": "%(asctime)s %(levelname)s %(message)s", }, }, "handlers": { "application_logs": { "level": "DEBUG", "formatter": "simple", "class": 'logging.StreamHandler', "stream": 'ext://sys.stderr', }, "server_logs": { "level": "INFO", "formatter": "simple", "class": 'logging.StreamHandler', "stream": 'ext://sys.stdout', }, }, "loggers": { "": { "handlers": ["application_logs"], "level": "DEBUG", "propagate": True, }, }, } django-dbconn-retry-0.1.5/test_project/test_project/urls.py000066400000000000000000000016141323357333400241440ustar00rootroot00000000000000"""test_project URL Configuration The `urlpatterns` list routes URLs to views. For more information please see: https://docs.djangoproject.com/en/1.11/topics/http/urls/ Examples: Function views 1. Add an import: from my_app import views 2. Add a URL to urlpatterns: url(r'^$', views.home, name='home') Class-based views 1. Add an import: from other_app.views import Home 2. Add a URL to urlpatterns: url(r'^$', Home.as_view(), name='home') Including another URLconf 1. Import the include() function: from django.conf.urls import url, include 2. Add a URL to urlpatterns: url(r'^blog/', include('blog.urls')) """ from django.conf.urls import url from django.http import HttpRequest, HttpResponse def testview(request: HttpRequest) -> HttpResponse: return HttpResponse("nothing here", content_type='text/plain', status=200) urlpatterns = [ url(r'^$', testview), ] django-dbconn-retry-0.1.5/test_project/test_project/wsgi.py000066400000000000000000000006221323357333400241260ustar00rootroot00000000000000""" WSGI config for test_project project. It exposes the WSGI callable as a module-level variable named ``application``. For more information on this file, see https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/ """ import os from django.core.wsgi import get_wsgi_application os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_project.settings") application = get_wsgi_application() django-dbconn-retry-0.1.5/tox.ini000066400000000000000000000012031323357333400167000ustar00rootroot00000000000000[tox] envlist = flake8, doc8, mypy, django{111,20}, [testenv] envdir = {toxinidir}/.toxenv usedevelop = true deps = -rrequirements-test.txt django111: Django>=1.11.0,<2.0 django20: Django>=2.0,<2.1 setenv = MYPYPATH=. passenv = COVERALLS_REPO_TOKEN commands = django{111,20}: coverage run --source=django_dbconn_retry test_project/manage.py test django{111,20}: coverage report -m django{111,20}: coveralls flake8: flake8 --max-line-length=120 django_dbconn_retry setup.py doc8: doc8 README.rst mypy: mypy --ignore-missing-imports --follow-imports=skip --disallow-untyped-calls --disallow-untyped-defs -p django_dbconn_retry