graypy-1.1.3/0000775000372000037200000000000013513205725013674 5ustar travistravis00000000000000graypy-1.1.3/README.rst0000664000372000037200000001302613513205276015366 0ustar travistravis00000000000000|PyPI_Status| |Build_Status| |Coverage_Status| Description =========== Python logging handlers that send messages in the Graylog Extended Log Format (GELF_). Installing ========== Using pip --------- Install the basic graypy python logging handlers .. code-block:: bash pip install graypy Install with requirements for ``GELFRabbitHandler`` .. code-block:: bash pip install graypy[amqp] Using easy_install ------------------ Install the basic graypy python logging handlers .. code-block:: bash easy_install graypy Install with requirements for ``GELFRabbitHandler`` .. code-block:: bash easy_install graypy[amqp] Usage ===== Messages are sent to Graylog2 using a custom handler for the builtin logging library in GELF format .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_logger.debug('Hello Graylog2.') Alternately, use ``GELFRabbitHandler`` to send messages to RabbitMQ and configure your Graylog2 server to consume messages via AMQP. This prevents log messages from being lost due to dropped UDP packets (``GELFUDPHandler`` sends messages to Graylog2 using UDP). You will need to configure RabbitMQ with a 'gelf_log' queue and bind it to the 'logging.gelf' exchange so messages are properly routed to a queue that can be consumed by Graylog2 (the queue and exchange names may be customized to your liking) .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFRabbitHandler('amqp://guest:guest@localhost/', exchange='logging.gelf') my_logger.addHandler(handler) my_logger.debug('Hello Graylog2.') Tracebacks are added as full messages .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) try: puff_the_magic_dragon() except NameError: my_logger.debug('No dragons here.', exc_info=1) For more detailed usage information please see the documentation provided within graypy's handler's docstrings. Using with Django ================= It's easy to integrate ``graypy`` with Django's logging settings. Just add a new handler in your ``settings.py``: .. code-block:: python LOGGING = { ... 'handlers': { 'graypy': { 'level': 'WARNING', 'class': 'graypy.GELFUDPHandler', 'host': 'localhost', 'port': 12201, }, }, 'loggers': { 'django.request': { 'handlers': ['graypy'], 'level': 'ERROR', 'propagate': True, }, }, } Custom fields ============= A number of custom fields are automatically added if available: * function * pid * process_name * thread_name You can disable these additional fields if you don't want them by adding an the ``debugging_fields=False`` to the handler: .. code-block:: python handler = graypy.GELFUDPHandler('localhost', 12201, debugging_fields=False) graypy also supports additional fields to be included in the messages sent to Graylog2. This can be done by using Python's LoggerAdapter_ and Filter_. In general, LoggerAdapter makes it easy to add static information to your log messages and Filters give you more flexibility, for example to add additional information based on the message that is being logged. Example using LoggerAdapter_ .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_adapter = logging.LoggerAdapter(logging.getLogger('test_logger'), {'username': 'John'}) my_adapter.debug('Hello Graylog2 from John.') Example using Filter_ .. code-block:: python import logging import graypy class UsernameFilter(logging.Filter): def __init__(self): # In an actual use case would dynamically get this # (e.g. from memcache) self.username = "John" def filter(self, record): record.username = self.username return True my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_logger.addFilter(UsernameFilter()) my_logger.debug('Hello Graylog2 from John.') Contributors: * Sever Banesiu * Daniel Miller * Tushar Makkar * Nathan Klapstein .. _GELF: http://docs.graylog.org/en/latest/pages/gelf.html .. _LoggerAdapter: http://docs.python.org/howto/logging-cookbook.html#using-loggeradapters-to-impart-contextual-information .. _Filter: http://docs.python.org/howto/logging-cookbook.html#using-filters-to-impart-contextual-information .. |Build_Status| image:: https://travis-ci.org/severb/graypy.svg?branch=master :target: https://travis-ci.org/severb/graypy .. |Coverage_Status| image:: https://codecov.io/gh/severb/graypy/branch/master/graph/badge.svg :target: https://codecov.io/gh/severb/graypy .. |PyPI_Status| image:: https://img.shields.io/pypi/v/graypy.svg :target: https://pypi.python.org/pypi/graypygraypy-1.1.3/setup.py0000775000372000037200000000550013513205276015412 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """setup.py for graypy""" import codecs import re import sys import os from setuptools import setup, find_packages from setuptools.command.test import test def find_version(*file_paths): with codecs.open(os.path.join(os.path.abspath(os.path.dirname(__file__)), *file_paths), 'r') as fp: version_file = fp.read() m = re.search(r"^__version__ = \((\d+), ?(\d+), ?(\d+)\)", version_file, re.M) if m: return "{}.{}.{}".format(*m.groups()) raise RuntimeError("Unable to find a valid version") VERSION = find_version("graypy", "__init__.py") class Pylint(test): def run_tests(self): from pylint.lint import Run Run(["graypy", "--persistent", "y", "--rcfile", ".pylintrc", "--output-format", "colorized"]) class PyTest(test): user_options = [('pytest-args=', 'a', "Arguments to pass to pytest")] def initialize_options(self): test.initialize_options(self) self.pytest_args = "-v --cov={}".format("graypy") def run_tests(self): import shlex # import here, cause outside the eggs aren't loaded import pytest errno = pytest.main(shlex.split(self.pytest_args)) sys.exit(errno) setup( name='graypy', version=VERSION, description="Python logging handler that sends messages in Graylog Extended Log Format (GLEF).", long_description=open('README.rst').read(), long_description_content_type="text/x-rst", keywords='logging gelf graylog2 graylog udp amqp', author='Sever Banesiu', author_email='banesiu.sever@gmail.com', url='https://github.com/severb/graypy', license='BSD License', packages=find_packages(), include_package_data=True, zip_safe=False, tests_require=[ "pytest>=2.8.7,<4.0.0", "pytest-cov<=2.6.0,<3.0.0", "pylint>=1.9.3,<2.0.0", "mock>=2.0.0,<3.0.0", "requests>=2.20.1,<3.0.0", "amqplib>=1.0.2,<2.0.0" ], extras_require={'amqp': ['amqplib==1.0.2']}, classifiers=[ 'License :: OSI Approved :: BSD License', 'Intended Audience :: Developers', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: System :: Logging', ], cmdclass={"test": PyTest, "lint": Pylint}, ) graypy-1.1.3/LICENSE0000664000372000037200000000274413513205276014711 0ustar travistravis00000000000000Copyright (c) 2011, Sever Băneşiu All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. graypy-1.1.3/setup.cfg0000664000372000037200000000010313513205725015507 0ustar travistravis00000000000000[bdist_wheel] universal = 1 [egg_info] tag_build = tag_date = 0 graypy-1.1.3/tests/0000775000372000037200000000000013513205725015036 5ustar travistravis00000000000000graypy-1.1.3/tests/helper.py0000664000372000037200000000372313513205276016675 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """helper functions for testing graypy These functions are used for both the integration and unit testing. """ import logging import pytest from graypy import GELFUDPHandler, GELFTCPHandler, GELFTLSHandler, \ GELFHTTPHandler TEST_CERT = "tests/config/localhost.cert.pem" KEY_PASS = "secret" TEST_TCP_PORT = 12201 TEST_UDP_PORT = 12202 TEST_HTTP_PORT = 12203 TEST_TLS_PORT = 12204 @pytest.fixture(params=[ GELFTCPHandler("127.0.0.1", TEST_TCP_PORT), GELFTCPHandler("127.0.0.1", TEST_TCP_PORT, extra_fields=True), GELFTCPHandler("127.0.0.1", TEST_TCP_PORT, extra_fields=True, debugging_fields=True), GELFTLSHandler("localhost", TEST_TLS_PORT), GELFTLSHandler("localhost", TEST_TLS_PORT, validate=True, ca_certs=TEST_CERT), GELFTLSHandler("127.0.0.1", TEST_TLS_PORT), GELFTLSHandler("127.0.0.1", TEST_TLS_PORT, validate=True, ca_certs=TEST_CERT), GELFHTTPHandler('127.0.0.1', TEST_HTTP_PORT), GELFHTTPHandler('127.0.0.1', TEST_HTTP_PORT, compress=False), GELFUDPHandler("127.0.0.1", TEST_UDP_PORT), GELFUDPHandler("127.0.0.1", TEST_UDP_PORT, compress=False), GELFUDPHandler("127.0.0.1", TEST_UDP_PORT, extra_fields=True), GELFUDPHandler("127.0.0.1", TEST_UDP_PORT, extra_fields=True, compress=False), GELFUDPHandler("127.0.0.1", TEST_UDP_PORT, extra_fields=True, debugging_fields=True), GELFUDPHandler("127.0.0.1", TEST_UDP_PORT, extra_fields=True, debugging_fields=True, compress=False), ]) def handler(request): return request.param @pytest.yield_fixture def logger(handler): logger_ = logging.getLogger("test_logger") logger_.addHandler(handler) yield logger_ logger_.removeHandler(handler) @pytest.yield_fixture def formatted_logger(handler): logger_ = logging.getLogger("formatted_test_logger") handler.setFormatter(logging.Formatter("%(levelname)s : %(message)s")) logger_.addHandler(handler) yield logger_ logger_.removeHandler(handler) graypy-1.1.3/tests/integration/0000775000372000037200000000000013513205725017361 5ustar travistravis00000000000000graypy-1.1.3/tests/integration/test_chunked_logging.py0000664000372000037200000000207113513205276024122 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests sending logs to a local Graylog instance that need to be chunked""" import logging import pytest from graypy.handler import SYSLOG_LEVELS, GELFUDPHandler from tests.helper import TEST_UDP_PORT from tests.integration import LOCAL_GRAYLOG_UP from tests.integration.helper import get_unique_message, get_graylog_response @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_chunked_logging(): """Test sending a common usage log that requires chunking to be fully sent""" logger = logging.getLogger("test_chunked_logger") handler = GELFUDPHandler("127.0.0.1", TEST_UDP_PORT, chunk_size=10) logger.addHandler(handler) message = get_unique_message() logger.error(message) graylog_response = get_graylog_response(message) assert message == graylog_response["message"] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert SYSLOG_LEVELS[logging.ERROR] == graylog_response["level"] graypy-1.1.3/tests/integration/helper.py0000664000372000037200000000252413513205276021216 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """helper functions for testing graypy with a local Graylog instance""" from time import sleep from uuid import uuid4 import requests def get_unique_message(): return str(uuid4()) DEFAULT_FIELDS = [ "message", "full_message", "source", "level", "func", "file", "line", "module", "logger_name", ] BASE_API_URL = 'http://127.0.0.1:9000/api/search/universal/relative?query=message:"{0}"&range=5&fields=' def get_graylog_response(message, fields=None): """Search for a given log message (with possible additional fields) within a local Graylog instance""" fields = fields if fields else [] api_resp = _get_api_response(message, fields) return _parse_api_response(api_resp) def _build_api_string(message, fields): return BASE_API_URL.format(message) + "%2C".join(set(DEFAULT_FIELDS + fields)) def _get_api_response(message, fields): sleep(2) url = _build_api_string(message, fields) api_response = requests.get( url, auth=("admin", "admin"), headers={"accept": "application/json"} ) return api_response def _parse_api_response(api_response): assert api_response.status_code == 200 print(api_response.json()) messages = api_response.json()["messages"] assert 1 == len(messages) return messages[0]["message"] graypy-1.1.3/tests/integration/test_debugging_fields.py0000664000372000037200000000346113513205276024260 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests validating the emitting of valid debugging fields for graypy loggers""" import pytest from tests.helper import logger, TEST_CERT, TEST_TCP_PORT, TEST_HTTP_PORT, \ TEST_TLS_PORT, TEST_UDP_PORT from tests.integration import LOCAL_GRAYLOG_UP from tests.integration.helper import get_graylog_response, get_unique_message from graypy import GELFUDPHandler, GELFTCPHandler, GELFTLSHandler, \ GELFHTTPHandler @pytest.fixture(params=[ GELFTCPHandler('127.0.0.1', TEST_TCP_PORT, debugging_fields=True), GELFUDPHandler('127.0.0.1', TEST_UDP_PORT, debugging_fields=True), GELFUDPHandler('127.0.0.1', TEST_UDP_PORT, compress=False, debugging_fields=True), GELFHTTPHandler('127.0.0.1', TEST_HTTP_PORT, debugging_fields=True), GELFHTTPHandler('127.0.0.1', TEST_HTTP_PORT, compress=False, debugging_fields=True), GELFTLSHandler('127.0.0.1', TEST_TLS_PORT, debugging_fields=True), GELFTLSHandler('127.0.0.1', TEST_TLS_PORT, debugging_fields=True, validate=True, ca_certs=TEST_CERT), ]) def handler(request): return request.param @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_debug_mode(logger): message = get_unique_message() logger.error(message) graylog_response = get_graylog_response(message, fields=["function", "pid", "thread_name"]) assert message == graylog_response['message'] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert graylog_response['file'].endswith("test_debugging_fields.py") assert 'test_debug_mode' == graylog_response['function'] assert 'line' in graylog_response assert "file" in graylog_response assert "pid" in graylog_response assert "thread_name" in graylog_response graypy-1.1.3/tests/integration/test_common_logging.py0000664000372000037200000000151713513205276023775 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests sending logs to a local Graylog instance""" import logging import pytest from graypy.handler import SYSLOG_LEVELS from tests.helper import handler, logger from tests.integration import LOCAL_GRAYLOG_UP from tests.integration.helper import get_unique_message, get_graylog_response @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_common_logging(logger): """Test sending a common usage log""" message = get_unique_message() logger.error(message) graylog_response = get_graylog_response(message) assert message == graylog_response["message"] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert SYSLOG_LEVELS[logging.ERROR] == graylog_response["level"] graypy-1.1.3/tests/integration/__init__.py0000664000372000037200000000073713513205276021502 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """integration pytests for :mod:`graypy` .. note:: These tests require an local instance of Graylog to send messages to. """ import requests def validate_local_graylog_up(): """Test to see if a localhost instance of Graylog is currently running""" try: requests.get("http://127.0.0.1:9000/api") return True except Exception: return False LOCAL_GRAYLOG_UP = validate_local_graylog_up() graypy-1.1.3/tests/integration/test_status_issue.py0000664000372000037200000000376313513205276023537 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests for addressing potential issues with adding an ``status`` extra field withing a given log and having the log failing to appear within graylog. Related issue: - Fails to log silently with specific extra field #85 URL: - https://github.com/severb/graypy/issues/85 """ import pytest from tests.helper import handler, logger from tests.integration import LOCAL_GRAYLOG_UP from tests.integration.helper import get_unique_message, get_graylog_response @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_non_status_field_log(logger): message = get_unique_message() logger.error(message, extra={"foo": "bar"}) graylog_response = get_graylog_response(message, fields=["foo"]) assert message == graylog_response["message"] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert "bar" == graylog_response["foo"] @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_status_field_issue(logger): message = get_unique_message() logger.error(message, extra={"status": "OK"}) graylog_response = get_graylog_response(message, fields=["status"]) assert message == graylog_response["message"] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert "OK" == graylog_response["status"] @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_status_field_issue_multi(logger): message = get_unique_message() logger.error(message, extra={"foo": "bar", "status": "OK"}) graylog_response = get_graylog_response(message, fields=["foo", "status"]) assert message == graylog_response["message"] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert "bar" == graylog_response["foo"] assert "OK" == graylog_response["status"] graypy-1.1.3/tests/integration/test_extra_fields.py0000664000372000037200000000415413513205276023450 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests for validating the addition of extra fields within GELF logs""" import logging import pytest from graypy import GELFTLSHandler, GELFTCPHandler, GELFUDPHandler, \ GELFHTTPHandler from tests.helper import TEST_CERT, TEST_TCP_PORT, TEST_HTTP_PORT, \ TEST_TLS_PORT, TEST_UDP_PORT from tests.integration import LOCAL_GRAYLOG_UP from tests.integration.helper import get_unique_message, get_graylog_response class DummyFilter(logging.Filter): def filter(self, record): record.ozzy = 'diary of a madman' record.van_halen = 1984 record.id = 42 return True @pytest.fixture(params=[ GELFTCPHandler('127.0.0.1', TEST_TCP_PORT, extra_fields=True), GELFUDPHandler('127.0.0.1', TEST_UDP_PORT, extra_fields=True), GELFUDPHandler('127.0.0.1', TEST_UDP_PORT, compress=False, extra_fields=True), GELFHTTPHandler('127.0.0.1', TEST_HTTP_PORT, extra_fields=True), GELFHTTPHandler('127.0.0.1', TEST_HTTP_PORT, compress=False, extra_fields=True), GELFTLSHandler('127.0.0.1', TEST_TLS_PORT, extra_fields=True), GELFTLSHandler('127.0.0.1', TEST_TLS_PORT, validate=True, ca_certs=TEST_CERT, extra_fields=True), ]) def handler(request): return request.param @pytest.yield_fixture def logger(handler): logger = logging.getLogger('test') dummy_filter = DummyFilter() logger.addFilter(dummy_filter) logger.addHandler(handler) yield logger logger.removeHandler(handler) logger.removeFilter(dummy_filter) @pytest.mark.skipif(not LOCAL_GRAYLOG_UP, reason="local Graylog instance not up") def test_dynamic_fields(logger): message = get_unique_message() logger.error(message) graylog_response = get_graylog_response(message, fields=['ozzy', 'van_halen']) assert message == graylog_response['message'] assert "long_message" not in graylog_response assert "timestamp" in graylog_response assert 'diary of a madman' == graylog_response['ozzy'] assert 1984 == graylog_response['van_halen'] assert 42 != graylog_response['_id'] assert 'id' not in graylog_response graypy-1.1.3/tests/__init__.py0000664000372000037200000000011713513205276017147 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests for :mod:`graypy`""" graypy-1.1.3/tests/unit/0000775000372000037200000000000013513205725016015 5ustar travistravis00000000000000graypy-1.1.3/tests/unit/helper.py0000664000372000037200000000056313513205276017653 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """helper functions for testing graypy with mocks of python logging and Graylog services""" import logging MOCK_LOG_RECORD_NAME = "MOCK_LOG_RECORD" MOCK_LOG_RECORD = logging.LogRecord( MOCK_LOG_RECORD_NAME, logging.INFO, pathname=None, lineno=None, msg="Log message", args=(), exc_info=None, ) graypy-1.1.3/tests/unit/test_ExcludeFilter.py0000664000372000037200000000214313513205276022166 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests for :class:`graypy.rabbitmq.ExcludeFilter`""" import pytest from graypy import ExcludeFilter from tests.unit.helper import MOCK_LOG_RECORD_NAME, MOCK_LOG_RECORD @pytest.mark.parametrize("name", [None, ""]) def test_invalid_name(name): """Test constructing:class:`graypy.rabbitmq.ExcludeFilter` with a invalid ``name`` argument""" with pytest.raises(ValueError): ExcludeFilter(name) @pytest.mark.parametrize("name", ["foobar", ".", " "]) def test_valid_name(name): """Test constructing :class:`graypy.rabbitmq.ExcludeFilter` with a valid ``name`` argument""" filter = ExcludeFilter(name) assert filter assert name == filter.name assert len(name) == filter.nlen def test_non_filtering_record(): filter = ExcludeFilter("NOT" + MOCK_LOG_RECORD_NAME) assert filter.filter(MOCK_LOG_RECORD) assert MOCK_LOG_RECORD.name != filter.name def test_filtering_record(): filter = ExcludeFilter(MOCK_LOG_RECORD_NAME) assert not filter.filter(MOCK_LOG_RECORD) assert MOCK_LOG_RECORD.name == filter.name graypy-1.1.3/tests/unit/__init__.py0000664000372000037200000000033313513205276020126 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """unit pytests for :mod:`graypy` .. note:: These tests mock sending to Graylog and thus, do not require a localhost instance of Graylog to successfully run. """ graypy-1.1.3/tests/unit/test_handler.py0000664000372000037200000002066613513205276021056 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests for the formatting and construction of Graylog GLEF logs by graypy .. note:: These tests mock sending to Graylog and do not require an active graylog instance to operate. """ import datetime import json import logging import socket import struct import sys import zlib import mock import pytest from graypy.handler import BaseGELFHandler, GELFHTTPHandler, GELFTLSHandler, \ ChunkedGELF from tests.helper import handler, logger, formatted_logger from tests.unit.helper import MOCK_LOG_RECORD, MOCK_LOG_RECORD_NAME UNICODE_REPLACEMENT = u"\ufffd" class TestClass(object): def __repr__(self): return "" @pytest.yield_fixture def mock_send(handler): try: with mock.patch.object(handler, "send") as mock_send: yield mock_send except Exception: with mock.patch.object(handler, "emit") as mock_send: yield mock_send def get_mock_send_arg(mock_send): assert mock_send.call_args_list != [] [[[arg], _]] = mock_send.call_args_list # TODO: this is inaccurate solution for mocking non-send handlers if isinstance(arg, logging.LogRecord): return json.loads(BaseGELFHandler(compress=False).makePickle(arg).decode("utf-8")) try: return json.loads(zlib.decompress(arg).decode("utf-8")) except zlib.error: # we have a uncompress message try: return json.loads(arg.decode("utf-8")) except Exception: # that is null terminated return json.loads(arg[:-1].decode("utf-8")) @pytest.mark.parametrize("message,expected", [ (u"\u20AC", u"\u20AC"), (u"\u20AC".encode("utf-8"), u"\u20AC"), (b"\xc3", UNICODE_REPLACEMENT), (["a", b"\xc3"], ["a", UNICODE_REPLACEMENT]), ]) def test_pack(message, expected): assert expected == json.loads(BaseGELFHandler._pack_gelf_dict(message).decode("utf-8")) def test_manual_exc_info_handler(logger, mock_send): """Check that a the ``full_message`` traceback info is passed when the ``exc_info=1`` flag is given within a log message""" try: raise SyntaxError("Syntax error") except SyntaxError: logger.error("Failed", exc_info=1) arg = get_mock_send_arg(mock_send) assert "Failed" == arg["short_message"] assert arg["full_message"].startswith("Traceback (most recent call last):") # GELFHTTPHandler mocking does not complete the stacktrace # thus a missing \n assert arg["full_message"].endswith("SyntaxError: Syntax error") or \ arg["full_message"].endswith("SyntaxError: Syntax error\n") def test_normal_exception_handler(logger, mock_send): try: raise SyntaxError("Syntax error") except SyntaxError: logger.exception("Failed") arg = get_mock_send_arg(mock_send) assert "Failed" == arg["short_message"] assert arg["full_message"].startswith("Traceback (most recent call last):") # GELFHTTPHandler mocking does not complete the stacktrace # thus a missing \n assert arg["full_message"].endswith("SyntaxError: Syntax error") or \ arg["full_message"].endswith("SyntaxError: Syntax error\n") def test_unicode(logger, mock_send): logger.error(u"Mensaje de registro espa\xf1ol") arg = get_mock_send_arg(mock_send) assert u"Mensaje de registro espa\xf1ol" == arg["short_message"] @pytest.mark.skipif(sys.version_info[0] >= 3, reason="python2 only") def test_broken_unicode_python2(logger, mock_send): # py3 record.getMessage() returns a binary string here # which is safely converted to unicode during the sanitization # process logger.error(b"Broken \xde log message") decoded = get_mock_send_arg(mock_send) assert u"Broken %s log message" % UNICODE_REPLACEMENT == decoded["short_message"] @pytest.mark.skipif(sys.version_info[0] < 3, reason="python3 only") def test_broken_unicode_python3(logger, mock_send): # py3 record.getMessage() returns somewhat broken "b"foo"" if the # message string is not a string, but a binary object: b"foo" logger.error(b"Broken \xde log message") decoded = get_mock_send_arg(mock_send) assert "b'Broken \\xde log message'" == decoded["short_message"] def test_extra_field(logger, mock_send): logger.error("Log message", extra={"foo": "bar"}) decoded = get_mock_send_arg(mock_send) assert "Log message" == decoded["short_message"] assert "bar" == decoded["_foo"] def test_list(logger, mock_send): logger.error("Log message", extra={"foo": ["bar", "baz"]}) decoded = get_mock_send_arg(mock_send) assert "Log message" == decoded["short_message"] assert ["bar", "baz"] == decoded["_foo"] def test_arbitrary_object(logger, mock_send): logger.error("Log message", extra={"foo": TestClass()}) decoded = get_mock_send_arg(mock_send) assert "Log message" == decoded["short_message"] assert "" == decoded["_foo"] def test_message_to_pickle_serializes_datetime_objects_instead_of_blindly_repring_them(logger, mock_send): timestamp = datetime.datetime(2001, 2, 3, 4, 5, 6, 7) logger.error("Log message", extra={"ts": timestamp}) decoded = get_mock_send_arg(mock_send) assert "datetime.datetime" not in decoded["_ts"] assert timestamp.isoformat() == decoded["_ts"] def test_status_field_issue(logger, mock_send): logger.error("Log message", extra={"status": "OK"}) decoded = get_mock_send_arg(mock_send) assert "Log message" == decoded["short_message"] assert "OK" == decoded["_status"] def test_add_level_name(): gelf_dict = dict() BaseGELFHandler._add_level_names(gelf_dict, MOCK_LOG_RECORD) assert "INFO" == gelf_dict["level_name"] def test_resolve_host(): """Test all posible resolutions of :meth:`BaseGELFHandler._resolve_host`""" assert socket.gethostname() == BaseGELFHandler._resolve_host(False, None) assert socket.getfqdn() == BaseGELFHandler._resolve_host(True, None) assert socket.getfqdn() == BaseGELFHandler._resolve_host(True, "localhost") assert "localhost" == BaseGELFHandler._resolve_host(False, "localhost") assert "" == BaseGELFHandler._resolve_host(False, "") def test_set_custom_facility(): gelf_dict = dict() facility = "test facility" BaseGELFHandler._set_custom_facility(gelf_dict, facility, MOCK_LOG_RECORD) assert MOCK_LOG_RECORD_NAME == gelf_dict["_logger"] assert "test facility" == gelf_dict["facility"] def test_formatted_logger(formatted_logger, mock_send): """Test the ability to set and modify the graypy handler's :class:`logging.Formatter` and have the resultant ``short_message`` be formatted by the set :class:`logging.Formatter`""" for handler in formatted_logger.handlers: if isinstance(handler, GELFHTTPHandler): pytest.skip("formatting not mocked for GELFHTTPHandler") formatted_logger.error("Log message") decoded = get_mock_send_arg(mock_send) assert "ERROR : Log message" == decoded["short_message"] def test_invalid_fqdn_localhost(): """Test constructing :class:`graypy.handler.BaseGELFHandler` with specifying conflicting arguments ``fqdn`` and ``localname``""" with pytest.raises(ValueError): BaseGELFHandler("127.0.0.1", 12202, fqdn=True, localname="localhost") def test_invalid_ca_certs(): """Test constructing :class:`graypy.handler.GELFTLSHandler` with incorrect arguments specifying server ca cert verification""" with pytest.raises(ValueError): GELFTLSHandler("127.0.0.1", validate=True) def test_invalid_client_certs(): """Test constructing :class:`graypy.handler.GELFTLSHandler` with incorrect arguments specifying client cert/key verification""" with pytest.raises(ValueError): # missing client cert GELFTLSHandler("127.0.0.1", keyfile="/dev/null") def test_glef_chunking(): """Testing the GELF chunking ability of :class:`graypy.handler.ChunkedGELF`""" message = b'12345' header = b'\x1e\x0f' chunks = list(ChunkedGELF(message, 2).__iter__()) expected = [ (struct.pack('b', 0), struct.pack('b', 3), b'12'), (struct.pack('b', 1), struct.pack('b', 3), b'34'), (struct.pack('b', 2), struct.pack('b', 3), b'5') ] assert len(chunks) == len(expected) for index, chunk in enumerate(chunks): expected_index, expected_chunks_count, expected_chunk = expected[index] assert header == chunk[:2] assert expected_index == chunk[10:11] assert expected_chunks_count == chunk[11:12] assert expected_chunk == chunk[12:] graypy-1.1.3/tests/unit/test_GELFRabbitHandler.py0000664000372000037200000000233313513205276022567 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """pytests for :class:`graypy.rabbitmq.GELFRabbitHandler`""" import json import pytest from graypy.rabbitmq import GELFRabbitHandler from graypy.handler import SYSLOG_LEVELS from tests.unit.helper import MOCK_LOG_RECORD def test_invalid_url(): """Test constructing :class:`graypy.rabbitmq.GELFRabbitHandler` with an invalid rabbitmq url""" with pytest.raises(ValueError): GELFRabbitHandler("BADURL") def test_valid_url(): """Test constructing :class:`graypy.rabbitmq.GELFRabbitHandler` with a valid rabbitmq url""" handler = GELFRabbitHandler("amqp://localhost") assert handler assert "amqp://localhost" == handler.url @pytest.mark.xfail(reason="rabbitmq service is not up") def test_socket_creation_failure(): """Test attempting to open a socket to a rabbitmq instance when no such service exists""" handler = GELFRabbitHandler("amqp://localhost") handler.makeSocket() def test_make_pickle(): handler = GELFRabbitHandler("amqp://localhost") pickle = json.loads(handler.makePickle(MOCK_LOG_RECORD)) assert "Log message" == pickle["short_message"] assert SYSLOG_LEVELS[MOCK_LOG_RECORD.levelno] == pickle["level"] graypy-1.1.3/graypy/0000775000372000037200000000000013513205725015207 5ustar travistravis00000000000000graypy-1.1.3/graypy/rabbitmq.py0000664000372000037200000001053513513205276017367 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """Logging Handler integrating RabbitMQ and Graylog Extended Log Format (GELF) handler""" import json from logging import Filter from logging.handlers import SocketHandler from amqplib import client_0_8 as amqp # pylint: disable=import-error from graypy.handler import BaseGELFHandler try: from urllib.parse import urlparse, unquote except ImportError: from urlparse import urlparse from urllib import unquote _ifnone = lambda v, x: x if v is None else v class GELFRabbitHandler(BaseGELFHandler, SocketHandler): """RabbitMQ / Graylog Extended Log Format handler .. note:: This handler ignores all messages logged by amqplib. """ def __init__(self, url, exchange='logging.gelf', exchange_type='fanout', virtual_host='/', routing_key='', **kwargs): """Initialize the GELFRabbitHandler :param url: RabbitMQ URL (ex: amqp://guest:guest@localhost:5672/) :type url: str :param exchange: RabbitMQ exchange. (default 'logging.gelf'). A queue binding must be defined on the server to prevent log messages from being dropped. :type exchange: str :param exchange_type: RabbitMQ exchange type (default 'fanout'). :type exchange_type: str :param virtual_host: :type virtual_host: str :param routing_key: :type routing_key: str """ self.url = url parsed = urlparse(url) if parsed.scheme != 'amqp': raise ValueError('invalid URL scheme (expected "amqp"): %s' % url) host = parsed.hostname or 'localhost' port = _ifnone(parsed.port, 5672) self.virtual_host = virtual_host if not unquote( parsed.path[1:]) else unquote(parsed.path[1:]) self.cn_args = { 'host': '%s:%s' % (host, port), 'userid': _ifnone(parsed.username, 'guest'), 'password': _ifnone(parsed.password, 'guest'), 'virtual_host': self.virtual_host, 'insist': False, } self.exchange = exchange self.exchange_type = exchange_type self.routing_key = routing_key BaseGELFHandler.__init__( self, **kwargs ) SocketHandler.__init__(self, host, port) self.addFilter(ExcludeFilter('amqplib')) def makeSocket(self, timeout=1): return RabbitSocket(self.cn_args, timeout, self.exchange, self.exchange_type, self.routing_key) def makePickle(self, record): message_dict = self._make_gelf_dict(record) return json.dumps(message_dict) class RabbitSocket(object): def __init__(self, cn_args, timeout, exchange, exchange_type, routing_key): self.cn_args = cn_args self.timeout = timeout self.exchange = exchange self.exchange_type = exchange_type self.routing_key = routing_key self.connection = amqp.Connection( connection_timeout=timeout, **self.cn_args) self.channel = self.connection.channel() self.channel.exchange_declare( exchange=self.exchange, type=self.exchange_type, durable=True, auto_delete=False, ) def sendall(self, data): msg = amqp.Message(data, delivery_mode=2) self.channel.basic_publish( msg, exchange=self.exchange, routing_key=self.routing_key ) def close(self): """Close the connection to the RabbitMQ socket""" try: self.connection.close() except Exception: pass class ExcludeFilter(Filter): """A subclass of :class:`logging.Filter` which should be instantiated with the name of the logger which, together with its children, will have its events excluded (filtered out)""" def __init__(self, name): """Initialize the ExcludeFilter :param name: Name to match for within a:class:`logging.LogRecord`'s ``name`` field for filtering. :type name: str """ if not name: raise ValueError('ExcludeFilter requires a non-empty name') Filter.__init__(self, name) def filter(self, record): return not (record.name.startswith(self.name) and ( len(record.name) == self.nlen or record.name[self.nlen] == ".")) graypy-1.1.3/graypy/__init__.py0000664000372000037200000000110513513205276017316 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """graypy Python logging handler that sends messages in Graylog Extended Log Format (GELF). Modules: + :mod:`.handler` - Logging Handlers that send messages in GELF + :mod:`.rabbitmq` - RabbitMQ and BaseGELFHandler Logging Handler composition """ from graypy.handler import GELFUDPHandler, GELFTCPHandler, GELFTLSHandler, \ GELFHTTPHandler, WAN_CHUNK, LAN_CHUNK try: from graypy.rabbitmq import GELFRabbitHandler, ExcludeFilter except ImportError: pass # amqplib is probably not installed __version__ = (1, 1, 3) graypy-1.1.3/graypy/handler.py0000664000372000037200000004557513513205276017217 0ustar travistravis00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """Logging Handlers that send messages in Graylog Extended Log Format (GELF)""" import abc import datetime import json import logging import math import random import socket import ssl import struct import sys import traceback import zlib from logging.handlers import DatagramHandler, SocketHandler WAN_CHUNK = 1420 LAN_CHUNK = 8154 if sys.version_info[0] == 3: # check if python3+ data, text = bytes, str else: data, text = str, unicode # pylint: disable=undefined-variable # fixes for using ABC if sys.version_info >= (3, 4): # check if python3.4+ ABC = abc.ABC else: ABC = abc.ABCMeta(str('ABC'), (), {}) try: import httplib except ImportError: import http.client as httplib SYSLOG_LEVELS = { logging.CRITICAL: 2, logging.ERROR: 3, logging.WARNING: 4, logging.INFO: 6, logging.DEBUG: 7, } class BaseGELFHandler(logging.Handler, ABC): """Abstract class noting the basic components of a GLEFHandler""" def __init__(self, chunk_size=WAN_CHUNK, debugging_fields=True, extra_fields=True, fqdn=False, localname=None, facility=None, level_names=False, compress=True): """Initialize the BaseGELFHandler. :param chunk_size: Message chunk size. Messages larger than this size will be sent to Graylog in multiple chunks. Defaults to ``WAN_CHUNK=1420``. :type chunk_size: int :param debugging_fields: If :obj:`True` add debug fields from the log record into the GELF logs to be sent to Graylog (:obj:`True` by default). :type debugging_fields: bool :param extra_fields: If :obj:`True` add extra fields from the log record into the GELF logs to be sent to Graylog (:obj:`True` by default). :type extra_fields: bool :param fqdn: If :obj:`True` use the fully qualified domain name of localhost to populate the ``host`` GELF field (:obj:`False` by default). :type fqdn: bool :param localname: If ``fqdn`` is :obj:`False` and ``localname`` is specified, used the specified hostname to populate the ``host`` GELF field. :type localname: str or None :param facility: If specified, replace the ``facility`` GELF field with the specified value. Additionally, the LogRecord.name will used populate the ``_logger`` GELF field. :type facility: str :param level_names: If :obj:`True` use string error level names instead of numerical values (:obj:`False` by default). :type level_names: bool :param compress: If :obj:`True` compress the GELF message before sending it to the server (:obj:`True` by default). :type compress: bool """ logging.Handler.__init__(self) self.debugging_fields = debugging_fields self.extra_fields = extra_fields self.chunk_size = chunk_size if fqdn and localname: raise ValueError( "cannot specify 'fqdn' and 'localname' arguments together") self.fqdn = fqdn self.localname = localname self.facility = facility self.level_names = level_names self.compress = compress def makePickle(self, record): """Convert a :class:`logging.LogRecord` into a bytes object representing a GELF log :param record: :class:`logging.LogRecord` to convert into a Graylog GELF log. :type record: logging.LogRecord :return: A bytes object representing a GELF log. :rtype: bytes """ gelf_dict = self._make_gelf_dict(record) packed = self._pack_gelf_dict(gelf_dict) pickle = zlib.compress(packed) if self.compress else packed return pickle def _make_gelf_dict(self, record): """Create a dictionary representing a Graylog GELF log from a python :class:`logging.LogRecord` :param record: :class:`logging.LogRecord` to create a Graylog GELF log from. :type record: logging.LogRecord :return: dictionary representing a Graylog GELF log. :rtype: dict """ # construct the base GELF format gelf_dict = { 'version': "1.0", 'host': BaseGELFHandler._resolve_host(self.fqdn, self.localname), 'short_message': self.formatter.format(record) if self.formatter else record.getMessage(), 'timestamp': record.created, 'level': SYSLOG_LEVELS.get(record.levelno, record.levelno), 'facility': self.facility or record.name, } # add in specified optional extras self._add_full_message(gelf_dict, record) if self.level_names: self._add_level_names(gelf_dict, record) if self.facility is not None: self._set_custom_facility(gelf_dict, self.facility, record) if self.debugging_fields: self._add_debugging_fields(gelf_dict, record) if self.extra_fields: self._add_extra_fields(gelf_dict, record) return gelf_dict @staticmethod def _add_level_names(gelf_dict, record): """Add the ``level_name`` field to the ``gelf_dict`` which notes the logging level via the string error level names instead of numerical values :param gelf_dict: dictionary representation of a GELF log. :type gelf_dict: dict :param record: :class:`logging.LogRecord` to extract a logging level from to insert into the given ``gelf_dict``. :type record: logging.LogRecord """ gelf_dict['level_name'] = logging.getLevelName(record.levelno) @staticmethod def _set_custom_facility(gelf_dict, facility_value, record): """Set the ``gelf_dict``'s ``facility`` field to the specified value also add the the extra ``_logger`` field containing the LogRecord.name :param gelf_dict: dictionary representation of a GELF log. :type gelf_dict: dict :param facility_value: Value to set as the ``gelf_dict``'s ``facility`` field. :type facility_value: str :param record: :class:`logging.LogRecord` to extract it's record name to insert into the given ``gelf_dict`` as the ``_logger`` field. :type record: logging.LogRecord """ gelf_dict.update({"facility": facility_value, '_logger': record.name}) @staticmethod def _add_full_message(gelf_dict, record): """Add the ``full_message`` field to the ``gelf_dict`` if any traceback information exists within the logging record :param gelf_dict: dictionary representation of a GELF log. :type gelf_dict: dict :param record: :class:`logging.LogRecord` to extract a full logging message from to insert into the given ``gelf_dict``. :type record: logging.LogRecord """ # if a traceback exists add it to the log as the full_message field full_message = None # format exception information if present if record.exc_info: full_message = '\n'.join( traceback.format_exception(*record.exc_info)) # use pre-formatted exception information in cases where the primary # exception information was removed, eg. for LogRecord serialization if record.exc_text: full_message = record.exc_text if full_message: gelf_dict["full_message"] = full_message @staticmethod def _resolve_host(fqdn, localname): """Resolve the ``host`` GELF field :param fqdn: Boolean indicating whether to use :meth:`socket.getfqdn` to obtain the ``host`` GELF field. :type fqdn: bool :param localname: Use specified hostname as the ``host`` GELF field. :type localname: str or None :return: String value representing the ``host`` GELF field. :rtype: str """ if fqdn: return socket.getfqdn() elif localname is not None: return localname return socket.gethostname() @staticmethod def _add_debugging_fields(gelf_dict, record): """Add debugging fields to the given ``gelf_dict`` :param gelf_dict: dictionary representation of a GELF log. :type gelf_dict: dict :param record: :class:`logging.LogRecord` to extract debugging fields from to insert into the given ``gelf_dict``. :type record: logging.LogRecord """ gelf_dict.update({ 'file': record.pathname, 'line': record.lineno, '_function': record.funcName, '_pid': record.process, '_thread_name': record.threadName, }) # record.processName was added in Python 2.6.2 pn = getattr(record, 'processName', None) if pn is not None: gelf_dict['_process_name'] = pn @staticmethod def _add_extra_fields(gelf_dict, record): """Add extra fields to the given ``gelf_dict`` However, this does not add additional fields in to ``message_dict`` that are either duplicated from standard :class:`logging.LogRecord` attributes, duplicated from the python logging module source (e.g. ``exc_text``), or violate GLEF format (i.e. ``id``). .. seealso:: The list of standard :class:`logging.LogRecord` attributes can be found at: http://docs.python.org/library/logging.html#logrecord-attributes :param gelf_dict: dictionary representation of a GELF log. :type gelf_dict: dict :param record: :class:`logging.LogRecord` to extract extra fields from to insert into the given ``gelf_dict``. :type record: logging.LogRecord """ # skip_list is used to filter additional fields in a log message. skip_list = ( 'args', 'asctime', 'created', 'exc_info', 'exc_text', 'filename', 'funcName', 'id', 'levelname', 'levelno', 'lineno', 'module', 'msecs', 'message', 'msg', 'name', 'pathname', 'process', 'processName', 'relativeCreated', 'thread', 'threadName') for key, value in record.__dict__.items(): if key not in skip_list and not key.startswith('_'): gelf_dict['_%s' % key] = value @staticmethod def _pack_gelf_dict(gelf_dict): """Convert a given ``gelf_dict`` to a JSON-encoded string, thus, creating an uncompressed GELF log ready for consumption by Graylog. Since we cannot be 100% sure of what is contained in the ``gelf_dict`` we have to do some sanitation. :param gelf_dict: dictionary representation of a GELF log. :type gelf_dict: dict :return: A prepped JSON-encoded GELF log as a bytes string encoded in UTF-8. :rtype: bytes """ gelf_dict = BaseGELFHandler._sanitize_to_unicode(gelf_dict) packed = json.dumps( gelf_dict, separators=',:', default=BaseGELFHandler._object_to_json ) return packed.encode('utf-8') @staticmethod def _sanitize_to_unicode(obj): """Convert all strings records of the object to unicode :param obj: object to sanitize to unicode. :type obj: object :return: Unicode string representation of the given object. :rtype: str """ if isinstance(obj, dict): return dict((BaseGELFHandler._sanitize_to_unicode(k), BaseGELFHandler._sanitize_to_unicode(v)) for k, v in obj.items()) if isinstance(obj, (list, tuple)): return obj.__class__([BaseGELFHandler._sanitize_to_unicode(i) for i in obj]) if isinstance(obj, data): obj = obj.decode('utf-8', errors='replace') return obj @staticmethod def _object_to_json(obj): """Convert objects that cannot be natively serialized into JSON into their string representation For datetime based objects convert them into their ISO formatted string as specified by :meth:`datetime.datetime.isoformat`. :param obj: object to convert into a JSON via getting its string representation. :type obj: object :return: String value representing the given object ready to be encoded into a JSON. :rtype: str """ if isinstance(obj, datetime.datetime): return obj.isoformat() return repr(obj) class GELFUDPHandler(BaseGELFHandler, DatagramHandler): """Graylog Extended Log Format UDP handler""" def __init__(self, host, port=12202, **kwargs): """Initialize the GELFUDPHandler :param host: The host of the Graylog server. :type host: str :param port: The port of the Graylog server (default ``12202``). :type port: int """ BaseGELFHandler.__init__(self, **kwargs) DatagramHandler.__init__(self, host, port) def send(self, s): if len(s) < self.chunk_size: DatagramHandler.send(self, s) else: for chunk in ChunkedGELF(s, self.chunk_size): DatagramHandler.send(self, chunk) class GELFTCPHandler(BaseGELFHandler, SocketHandler): """Graylog Extended Log Format TCP handler""" def __init__(self, host, port=12201, **kwargs): """Initialize the GELFTCPHandler :param host: The host of the Graylog server. :type host: str :param port: The port of the Graylog server (default ``12201``). :type port: int """ BaseGELFHandler.__init__(self, compress=False, **kwargs) SocketHandler.__init__(self, host, port) def makePickle(self, record): """Add a null terminator to a GELFTCPHandler's pickles as a TCP frame object needs to be null terminated :param record: :class:`logging.LogRecord` to create a null terminated GELF log. :type record: logging.LogRecord :return: A GELF log encoded as a null terminated bytes string. :rtype: bytes """ return BaseGELFHandler.makePickle(self, record) + b'\x00' class GELFTLSHandler(GELFTCPHandler): """Graylog Extended Log Format TCP handler with TLS support""" def __init__(self, host, port=12204, validate=False, ca_certs=None, certfile=None, keyfile=None, **kwargs): """Initialize the GELFTLSHandler :param host: The host of the Graylog server. :type host: str :param port: The port of the Graylog server (default ``12204``). :type port: int :param validate: If :obj:`True`, validate server certificate. In that case specifying ``ca_certs`` is required. :type validate: bool :param ca_certs: Path to CA bundle file. :type ca_certs: str :param certfile: Path to the client certificate file. :type certfile: str :param keyfile: Path to the client private key. If the private key is stored with the certificate, this parameter can be ignored. :type keyfile: str """ if validate and ca_certs is None: raise ValueError('CA bundle file path must be specified') if keyfile is not None and certfile is None: raise ValueError('certfile must be specified') GELFTCPHandler.__init__(self, host=host, port=port, **kwargs) self.ca_certs = ca_certs self.reqs = ssl.CERT_REQUIRED if validate else ssl.CERT_NONE self.certfile = certfile self.keyfile = keyfile if keyfile else certfile def makeSocket(self, timeout=1): """Override SocketHandler.makeSocket, to allow creating wrapped TLS sockets""" plain_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) if hasattr(plain_socket, 'settimeout'): plain_socket.settimeout(timeout) wrapped_socket = ssl.wrap_socket( plain_socket, ca_certs=self.ca_certs, cert_reqs=self.reqs, keyfile=self.keyfile, certfile=self.certfile ) wrapped_socket.connect((self.host, self.port)) return wrapped_socket # TODO: add https? class GELFHTTPHandler(BaseGELFHandler): """Graylog Extended Log Format HTTP handler""" def __init__(self, host, port=12203, compress=True, path='/gelf', timeout=5, **kwargs): """Initialize the GELFHTTPHandler :param host: GELF HTTP input host. :type host: str :param port: GELF HTTP input port. :type port: int :param compress: If :obj:`True` compress the GELF message before sending it to the server (:obj:`True` by default). :type compress: bool :param path: Path of the HTTP input. (see http://docs.graylog.org/en/latest/pages/sending_data.html#gelf-via-http) :type path: str :param timeout: Amount of seconds that HTTP client should wait before it discards the request if the server doesn't respond. :type timeout: int """ BaseGELFHandler.__init__(self, compress=compress, **kwargs) self.host = host self.port = port self.path = path self.timeout = timeout self.headers = {} if compress: self.headers['Content-Encoding'] = 'gzip,deflate' def emit(self, record): """Convert a :class:`logging.LogRecord` to GELF and emit it to Graylog via an HTTP POST request :param record: :class:`logging.LogRecord` to convert into a Graylog GELF log and emit to Graylog via HTTP POST. :type record: logging.LogRecord """ pickle = self.makePickle(record) connection = httplib.HTTPConnection( host=self.host, port=self.port, timeout=self.timeout ) connection.request('POST', self.path, pickle, self.headers) class ChunkedGELF(object): """Class that chunks a message into a GLEF compatible chunks""" def __init__(self, message, size): """Initialize the ChunkedGELF message class :param message: The message to chunk. :type message: bytes :param size: The size of the chunks. :type size: int """ self.message = message self.size = size self.pieces = \ struct.pack('B', int(math.ceil(len(message) * 1.0 / size))) self.id = struct.pack('Q', random.randint(0, 0xFFFFFFFFFFFFFFFF)) def message_chunks(self): return (self.message[i:i + self.size] for i in range(0, len(self.message), self.size)) def encode(self, sequence, chunk): return b''.join([ b'\x1e\x0f', self.id, struct.pack('B', sequence), self.pieces, chunk ]) def __iter__(self): for sequence, chunk in enumerate(self.message_chunks()): yield self.encode(sequence, chunk) graypy-1.1.3/MANIFEST.in0000664000372000037200000000007713513205276015437 0ustar travistravis00000000000000include LICENSE include README.rst recursive-include tests *.pygraypy-1.1.3/PKG-INFO0000664000372000037200000002060713513205725014776 0ustar travistravis00000000000000Metadata-Version: 2.1 Name: graypy Version: 1.1.3 Summary: Python logging handler that sends messages in Graylog Extended Log Format (GLEF). Home-page: https://github.com/severb/graypy Author: Sever Banesiu Author-email: banesiu.sever@gmail.com License: BSD License Description: |PyPI_Status| |Build_Status| |Coverage_Status| Description =========== Python logging handlers that send messages in the Graylog Extended Log Format (GELF_). Installing ========== Using pip --------- Install the basic graypy python logging handlers .. code-block:: bash pip install graypy Install with requirements for ``GELFRabbitHandler`` .. code-block:: bash pip install graypy[amqp] Using easy_install ------------------ Install the basic graypy python logging handlers .. code-block:: bash easy_install graypy Install with requirements for ``GELFRabbitHandler`` .. code-block:: bash easy_install graypy[amqp] Usage ===== Messages are sent to Graylog2 using a custom handler for the builtin logging library in GELF format .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_logger.debug('Hello Graylog2.') Alternately, use ``GELFRabbitHandler`` to send messages to RabbitMQ and configure your Graylog2 server to consume messages via AMQP. This prevents log messages from being lost due to dropped UDP packets (``GELFUDPHandler`` sends messages to Graylog2 using UDP). You will need to configure RabbitMQ with a 'gelf_log' queue and bind it to the 'logging.gelf' exchange so messages are properly routed to a queue that can be consumed by Graylog2 (the queue and exchange names may be customized to your liking) .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFRabbitHandler('amqp://guest:guest@localhost/', exchange='logging.gelf') my_logger.addHandler(handler) my_logger.debug('Hello Graylog2.') Tracebacks are added as full messages .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) try: puff_the_magic_dragon() except NameError: my_logger.debug('No dragons here.', exc_info=1) For more detailed usage information please see the documentation provided within graypy's handler's docstrings. Using with Django ================= It's easy to integrate ``graypy`` with Django's logging settings. Just add a new handler in your ``settings.py``: .. code-block:: python LOGGING = { ... 'handlers': { 'graypy': { 'level': 'WARNING', 'class': 'graypy.GELFUDPHandler', 'host': 'localhost', 'port': 12201, }, }, 'loggers': { 'django.request': { 'handlers': ['graypy'], 'level': 'ERROR', 'propagate': True, }, }, } Custom fields ============= A number of custom fields are automatically added if available: * function * pid * process_name * thread_name You can disable these additional fields if you don't want them by adding an the ``debugging_fields=False`` to the handler: .. code-block:: python handler = graypy.GELFUDPHandler('localhost', 12201, debugging_fields=False) graypy also supports additional fields to be included in the messages sent to Graylog2. This can be done by using Python's LoggerAdapter_ and Filter_. In general, LoggerAdapter makes it easy to add static information to your log messages and Filters give you more flexibility, for example to add additional information based on the message that is being logged. Example using LoggerAdapter_ .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_adapter = logging.LoggerAdapter(logging.getLogger('test_logger'), {'username': 'John'}) my_adapter.debug('Hello Graylog2 from John.') Example using Filter_ .. code-block:: python import logging import graypy class UsernameFilter(logging.Filter): def __init__(self): # In an actual use case would dynamically get this # (e.g. from memcache) self.username = "John" def filter(self, record): record.username = self.username return True my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_logger.addFilter(UsernameFilter()) my_logger.debug('Hello Graylog2 from John.') Contributors: * Sever Banesiu * Daniel Miller * Tushar Makkar * Nathan Klapstein .. _GELF: http://docs.graylog.org/en/latest/pages/gelf.html .. _LoggerAdapter: http://docs.python.org/howto/logging-cookbook.html#using-loggeradapters-to-impart-contextual-information .. _Filter: http://docs.python.org/howto/logging-cookbook.html#using-filters-to-impart-contextual-information .. |Build_Status| image:: https://travis-ci.org/severb/graypy.svg?branch=master :target: https://travis-ci.org/severb/graypy .. |Coverage_Status| image:: https://codecov.io/gh/severb/graypy/branch/master/graph/badge.svg :target: https://codecov.io/gh/severb/graypy .. |PyPI_Status| image:: https://img.shields.io/pypi/v/graypy.svg :target: https://pypi.python.org/pypi/graypy Keywords: logging gelf graylog2 graylog udp amqp Platform: UNKNOWN Classifier: License :: OSI Approved :: BSD License Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: System :: Logging Description-Content-Type: text/x-rst Provides-Extra: amqp graypy-1.1.3/graypy.egg-info/0000775000372000037200000000000013513205725016701 5ustar travistravis00000000000000graypy-1.1.3/graypy.egg-info/top_level.txt0000664000372000037200000000001513513205725021427 0ustar travistravis00000000000000graypy tests graypy-1.1.3/graypy.egg-info/not-zip-safe0000664000372000037200000000000113513205725021127 0ustar travistravis00000000000000 graypy-1.1.3/graypy.egg-info/requires.txt0000664000372000037200000000002713513205725021300 0ustar travistravis00000000000000 [amqp] amqplib==1.0.2 graypy-1.1.3/graypy.egg-info/PKG-INFO0000664000372000037200000002060713513205725020003 0ustar travistravis00000000000000Metadata-Version: 2.1 Name: graypy Version: 1.1.3 Summary: Python logging handler that sends messages in Graylog Extended Log Format (GLEF). Home-page: https://github.com/severb/graypy Author: Sever Banesiu Author-email: banesiu.sever@gmail.com License: BSD License Description: |PyPI_Status| |Build_Status| |Coverage_Status| Description =========== Python logging handlers that send messages in the Graylog Extended Log Format (GELF_). Installing ========== Using pip --------- Install the basic graypy python logging handlers .. code-block:: bash pip install graypy Install with requirements for ``GELFRabbitHandler`` .. code-block:: bash pip install graypy[amqp] Using easy_install ------------------ Install the basic graypy python logging handlers .. code-block:: bash easy_install graypy Install with requirements for ``GELFRabbitHandler`` .. code-block:: bash easy_install graypy[amqp] Usage ===== Messages are sent to Graylog2 using a custom handler for the builtin logging library in GELF format .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_logger.debug('Hello Graylog2.') Alternately, use ``GELFRabbitHandler`` to send messages to RabbitMQ and configure your Graylog2 server to consume messages via AMQP. This prevents log messages from being lost due to dropped UDP packets (``GELFUDPHandler`` sends messages to Graylog2 using UDP). You will need to configure RabbitMQ with a 'gelf_log' queue and bind it to the 'logging.gelf' exchange so messages are properly routed to a queue that can be consumed by Graylog2 (the queue and exchange names may be customized to your liking) .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFRabbitHandler('amqp://guest:guest@localhost/', exchange='logging.gelf') my_logger.addHandler(handler) my_logger.debug('Hello Graylog2.') Tracebacks are added as full messages .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) try: puff_the_magic_dragon() except NameError: my_logger.debug('No dragons here.', exc_info=1) For more detailed usage information please see the documentation provided within graypy's handler's docstrings. Using with Django ================= It's easy to integrate ``graypy`` with Django's logging settings. Just add a new handler in your ``settings.py``: .. code-block:: python LOGGING = { ... 'handlers': { 'graypy': { 'level': 'WARNING', 'class': 'graypy.GELFUDPHandler', 'host': 'localhost', 'port': 12201, }, }, 'loggers': { 'django.request': { 'handlers': ['graypy'], 'level': 'ERROR', 'propagate': True, }, }, } Custom fields ============= A number of custom fields are automatically added if available: * function * pid * process_name * thread_name You can disable these additional fields if you don't want them by adding an the ``debugging_fields=False`` to the handler: .. code-block:: python handler = graypy.GELFUDPHandler('localhost', 12201, debugging_fields=False) graypy also supports additional fields to be included in the messages sent to Graylog2. This can be done by using Python's LoggerAdapter_ and Filter_. In general, LoggerAdapter makes it easy to add static information to your log messages and Filters give you more flexibility, for example to add additional information based on the message that is being logged. Example using LoggerAdapter_ .. code-block:: python import logging import graypy my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_adapter = logging.LoggerAdapter(logging.getLogger('test_logger'), {'username': 'John'}) my_adapter.debug('Hello Graylog2 from John.') Example using Filter_ .. code-block:: python import logging import graypy class UsernameFilter(logging.Filter): def __init__(self): # In an actual use case would dynamically get this # (e.g. from memcache) self.username = "John" def filter(self, record): record.username = self.username return True my_logger = logging.getLogger('test_logger') my_logger.setLevel(logging.DEBUG) handler = graypy.GELFUDPHandler('localhost', 12201) my_logger.addHandler(handler) my_logger.addFilter(UsernameFilter()) my_logger.debug('Hello Graylog2 from John.') Contributors: * Sever Banesiu * Daniel Miller * Tushar Makkar * Nathan Klapstein .. _GELF: http://docs.graylog.org/en/latest/pages/gelf.html .. _LoggerAdapter: http://docs.python.org/howto/logging-cookbook.html#using-loggeradapters-to-impart-contextual-information .. _Filter: http://docs.python.org/howto/logging-cookbook.html#using-filters-to-impart-contextual-information .. |Build_Status| image:: https://travis-ci.org/severb/graypy.svg?branch=master :target: https://travis-ci.org/severb/graypy .. |Coverage_Status| image:: https://codecov.io/gh/severb/graypy/branch/master/graph/badge.svg :target: https://codecov.io/gh/severb/graypy .. |PyPI_Status| image:: https://img.shields.io/pypi/v/graypy.svg :target: https://pypi.python.org/pypi/graypy Keywords: logging gelf graylog2 graylog udp amqp Platform: UNKNOWN Classifier: License :: OSI Approved :: BSD License Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: System :: Logging Description-Content-Type: text/x-rst Provides-Extra: amqp graypy-1.1.3/graypy.egg-info/dependency_links.txt0000664000372000037200000000000113513205725022747 0ustar travistravis00000000000000 graypy-1.1.3/graypy.egg-info/SOURCES.txt0000664000372000037200000000132013513205725020561 0ustar travistravis00000000000000LICENSE MANIFEST.in README.rst setup.cfg setup.py graypy/__init__.py graypy/handler.py graypy/rabbitmq.py graypy.egg-info/PKG-INFO graypy.egg-info/SOURCES.txt graypy.egg-info/dependency_links.txt graypy.egg-info/not-zip-safe graypy.egg-info/requires.txt graypy.egg-info/top_level.txt tests/__init__.py tests/helper.py tests/integration/__init__.py tests/integration/helper.py tests/integration/test_chunked_logging.py tests/integration/test_common_logging.py tests/integration/test_debugging_fields.py tests/integration/test_extra_fields.py tests/integration/test_status_issue.py tests/unit/__init__.py tests/unit/helper.py tests/unit/test_ExcludeFilter.py tests/unit/test_GELFRabbitHandler.py tests/unit/test_handler.py