graphite-carbon-1.1.7/0000755000175000017500000000000013633714656013263 5ustar gringringraphite-carbon-1.1.7/.gitignore0000644000175000017500000000026013633714656015251 0ustar gringrinMANIFEST build dist *.log *.pyc .idea *.iml .venv .coverage coverage.xml .tox conf/*.conf storage _trial_temp htmlcov *.swp .eggs/ *.egg-info/ lib/twisted/plugins/dropin.cache graphite-carbon-1.1.7/.travis.yml0000644000175000017500000000256213633714656015401 0ustar gringrin# http://travis-ci.org/#!/graphite-project/carbon dist: xenial language: python python: 2.7 matrix: include: - python: pypy env: - TOXENV=pypy - python: 3.5 env: - TOXENV=py35 - python: 3.6 env: - TOXENV=py36 - python: 3.7 env: - TOXENV=py37 - python: 3.8 env: - TOXENV=py38-pyhash - python: 3.8 env: - TOXENV=py38 - python: 3.8 env: - TOXENV=lint - python: 2.7 arch: s390x env: - TOXENV=py27 - python: 2.7 arch: s390x env: - TOXENV=lint - python: 3.5 arch: s390x env: - TOXENV=py35 - python: 3.6 arch: s390x env: - TOXENV=py36 - python: 3.7 arch: s390x env: - TOXENV=py37 - python: 3.8 arch: s390x env: - TOXENV=py38 - python: 3.8 arch: s390x env: - TOXENV=lint env: - TOXENV=py27 - TOXENV=py27-pyhash - TOXENV=lint install: - if [[ $(uname -m) == 's390x' ]]; then sudo rm -rf $HOME/.cache/pip; fi - if echo "$TOXENV" | grep -q 'pyhash' ; then sudo apt-get -q install -y libboost-python-dev; fi - if echo "$TOXENV" | grep -q '^py2' ; then pip install --upgrade pip virtualenv; fi - pip install tox script: - tox -e $TOXENV after_success: - pip install codecov - codecov graphite-carbon-1.1.7/LICENSE0000644000175000017500000002613613633714656014300 0ustar gringrin Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. graphite-carbon-1.1.7/MANIFEST.in0000644000175000017500000000023713633714656015023 0ustar gringrinrecursive-include conf/ * recursive-include distro/ * exclude conf/*.conf include LICENSE include README.md include lib/carbon/amqp0-8.xml include MANIFEST.in graphite-carbon-1.1.7/README.md0000644000175000017500000000333113633714656014542 0ustar gringrin# Carbon [![Codacy Badge](https://api.codacy.com/project/badge/Grade/85221cd3bb6e49d7bbd6fed376a88264)](https://www.codacy.com/app/graphite-project/carbon?utm_source=github.com&utm_medium=referral&utm_content=graphite-project/carbon&utm_campaign=badger) [![Build Status](https://secure.travis-ci.org/graphite-project/carbon.png?branch=master)](http://travis-ci.org/graphite-project/carbon) [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Fgraphite-project%2Fcarbon.svg?type=shield)](https://app.fossa.io/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Fgraphite-project%2Fcarbon?ref=badge_shield) [![codecov](https://codecov.io/gh/graphite-project/carbon/branch/master/graph/badge.svg)](https://codecov.io/gh/graphite-project/carbon) ## Overview Carbon is one of three components within the Graphite project: 1. [Graphite-Web](https://github.com/graphite-project/graphite-web), a Django-based web application that renders graphs and dashboards 2. The Carbon metric processing daemons 3. The [Whisper](https://github.com/graphite-project/whisper) time-series database library ![Graphite Components](https://github.com/graphite-project/graphite-web/raw/master/webapp/content/img/overview.png "Graphite Components") Carbon is responsible for receiving metrics over the network, caching them in memory for "hot queries" from the Graphite-Web application, and persisting them to disk using the Whisper time-series library. ## Installation, Configuration and Usage Please refer to the instructions at [readthedocs](http://graphite.readthedocs.org/). ## License Carbon is licensed under version 2.0 of the Apache License. See the [LICENSE](https://github.com/graphite-project/carbon/blob/master/LICENSE) file for details. graphite-carbon-1.1.7/bin/0000755000175000017500000000000013633714656014033 5ustar gringringraphite-carbon-1.1.7/bin/carbon-aggregator-cache.py0000755000175000017500000000212313633714656021033 0ustar gringrin#!/usr/bin/env python """Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import sys import os.path # Figure out where we're installed BIN_DIR = os.path.dirname(os.path.abspath(__file__)) ROOT_DIR = os.path.dirname(BIN_DIR) # Make sure that carbon's 'lib' dir is in the $PYTHONPATH if we're running from # source. LIB_DIR = os.path.join(ROOT_DIR, "lib") sys.path.insert(0, LIB_DIR) from carbon.util import run_twistd_plugin # noqa from carbon.exceptions import CarbonConfigException # noqa try: run_twistd_plugin(__file__) except CarbonConfigException as exc: raise SystemExit(str(exc)) graphite-carbon-1.1.7/bin/carbon-aggregator.py0000755000175000017500000000212313633714656017772 0ustar gringrin#!/usr/bin/env python """Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import sys import os.path # Figure out where we're installed BIN_DIR = os.path.dirname(os.path.abspath(__file__)) ROOT_DIR = os.path.dirname(BIN_DIR) # Make sure that carbon's 'lib' dir is in the $PYTHONPATH if we're running from # source. LIB_DIR = os.path.join(ROOT_DIR, "lib") sys.path.insert(0, LIB_DIR) from carbon.util import run_twistd_plugin # noqa from carbon.exceptions import CarbonConfigException # noqa try: run_twistd_plugin(__file__) except CarbonConfigException as exc: raise SystemExit(str(exc)) graphite-carbon-1.1.7/bin/carbon-cache.py0000755000175000017500000000212313633714656016713 0ustar gringrin#!/usr/bin/env python """Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import sys import os.path # Figure out where we're installed BIN_DIR = os.path.dirname(os.path.abspath(__file__)) ROOT_DIR = os.path.dirname(BIN_DIR) # Make sure that carbon's 'lib' dir is in the $PYTHONPATH if we're running from # source. LIB_DIR = os.path.join(ROOT_DIR, "lib") sys.path.insert(0, LIB_DIR) from carbon.util import run_twistd_plugin # noqa from carbon.exceptions import CarbonConfigException # noqa try: run_twistd_plugin(__file__) except CarbonConfigException as exc: raise SystemExit(str(exc)) graphite-carbon-1.1.7/bin/carbon-client.py0000755000175000017500000001067113633714656017135 0ustar gringrin#!/usr/bin/env python """Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import sys from os.path import dirname, join, abspath, exists from optparse import OptionParser # Figure out where we're installed BIN_DIR = dirname(abspath(__file__)) ROOT_DIR = dirname(BIN_DIR) CONF_DIR = join(ROOT_DIR, 'conf') default_relayrules = join(CONF_DIR, 'relay-rules.conf') # Make sure that carbon's 'lib' dir is in the $PYTHONPATH if we're running from # source. LIB_DIR = join(ROOT_DIR, 'lib') sys.path.insert(0, LIB_DIR) try: from twisted.internet import epollreactor epollreactor.install() except ImportError: pass from twisted.internet import stdio, reactor, defer # noqa from twisted.protocols.basic import LineReceiver # noqa from carbon.routers import ConsistentHashingRouter, RelayRulesRouter # noqa from carbon.client import CarbonClientManager # noqa from carbon import log, events # noqa option_parser = OptionParser(usage="%prog [options] ...") option_parser.add_option('--debug', action='store_true', help="Log debug info to stdout") option_parser.add_option('--keyfunc', help="Use a custom key function (path/to/module.py:myFunc)") option_parser.add_option('--replication', type='int', default=1, help='Replication factor') option_parser.add_option( '--routing', default='consistent-hashing', help='Routing method: "consistent-hashing" (default) or "relay"') option_parser.add_option( '--diverse-replicas', action='store_true', help="Spread replicas across diff. servers") option_parser.add_option( '--relayrules', default=default_relayrules, help='relay-rules.conf file to use for relay routing') options, args = option_parser.parse_args() if not args: print('At least one host:port destination required\n') option_parser.print_usage() raise SystemExit(1) if options.routing not in ('consistent-hashing', 'relay'): print("Invalid --routing value, must be one of:") print(" consistent-hashing") print(" relay") raise SystemExit(1) destinations = [] for arg in args: parts = arg.split(':', 2) host = parts[0] port = int(parts[1]) if len(parts) > 2: instance = parts[2] else: instance = None destinations.append((host, port, instance)) if options.debug: log.logToStdout() log.setDebugEnabled(True) defer.setDebugging(True) if options.routing == 'consistent-hashing': router = ConsistentHashingRouter(options.replication, diverse_replicas=options.diverse_replicas) elif options.routing == 'relay': if exists(options.relayrules): router = RelayRulesRouter(options.relayrules) else: print("relay rules file %s does not exist" % options.relayrules) raise SystemExit(1) client_manager = CarbonClientManager(router) reactor.callWhenRunning(client_manager.startService) if options.keyfunc: router.setKeyFunctionFromModule(options.keyfunc) firstConnectAttempts = [client_manager.startClient(dest) for dest in destinations] firstConnectsAttempted = defer.DeferredList(firstConnectAttempts) class StdinMetricsReader(LineReceiver): delimiter = '\n' def lineReceived(self, line): # log.msg("[DEBUG] lineReceived(): %s" % line) try: (metric, value, timestamp) = line.split() datapoint = (float(timestamp), float(value)) assert datapoint[1] == datapoint[1] # filter out NaNs client_manager.sendDatapoint(metric, datapoint) except ValueError: log.err(None, 'Dropping invalid line: %s' % line) def connectionLost(self, reason): log.msg('stdin disconnected') def startShutdown(results): log.msg("startShutdown(%s)" % str(results)) allStopped = client_manager.stopAllClients() allStopped.addCallback(shutdown) firstConnectsAttempted.addCallback(startShutdown) stdio.StandardIO(StdinMetricsReader()) exitCode = 0 def shutdown(results): global exitCode for success, result in results: if not success: exitCode = 1 break if reactor.running: reactor.stop() reactor.run() raise SystemExit(exitCode) graphite-carbon-1.1.7/bin/carbon-relay.py0000755000175000017500000000212313633714656016764 0ustar gringrin#!/usr/bin/env python """Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import sys import os.path # Figure out where we're installed BIN_DIR = os.path.dirname(os.path.abspath(__file__)) ROOT_DIR = os.path.dirname(BIN_DIR) # Make sure that carbon's 'lib' dir is in the $PYTHONPATH if we're running from # source. LIB_DIR = os.path.join(ROOT_DIR, "lib") sys.path.insert(0, LIB_DIR) from carbon.util import run_twistd_plugin # noqa from carbon.exceptions import CarbonConfigException # noqa try: run_twistd_plugin(__file__) except CarbonConfigException as exc: raise SystemExit(str(exc)) graphite-carbon-1.1.7/bin/validate-storage-schemas.py0000755000175000017500000000442413633714656021270 0ustar gringrin#!/usr/bin/env python """Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import sys import whisper from os.path import dirname, join, realpath try: from ConfigParser import ConfigParser except ImportError: from configparser import ConfigParser if len(sys.argv) == 2: SCHEMAS_FILE = sys.argv[1] print("Loading storage-schemas configuration from: '%s'" % SCHEMAS_FILE) else: SCHEMAS_FILE = realpath(join(dirname(__file__), '..', 'conf', 'storage-schemas.conf')) print("Loading storage-schemas configuration from default location at: '%s'" % SCHEMAS_FILE) config_parser = ConfigParser() if not config_parser.read(SCHEMAS_FILE): raise SystemExit("Error: Couldn't read config file: %s" % SCHEMAS_FILE) errors_found = 0 for section in config_parser.sections(): print("Section '%s':" % section) options = dict(config_parser.items(section)) retentions = options['retentions'].split(',') archives = [] section_failed = False for retention in retentions: try: archives.append(whisper.parseRetentionDef(retention)) except ValueError as e: print( " - Error: Section '%s' contains an invalid item in its retention definition ('%s')" % (section, retention) ) print(" %s" % e) section_failed = True if not section_failed: try: whisper.validateArchiveList(archives) except whisper.InvalidConfiguration as e: print( " - Error: Section '%s' contains an invalid retention definition ('%s')" % (section, ','.join(retentions)) ) print(" %s" % e) if section_failed: errors_found += 1 else: print(" OK") if errors_found: raise SystemExit("Storage-schemas configuration '%s' failed validation" % SCHEMAS_FILE) print("Storage-schemas configuration '%s' is valid" % SCHEMAS_FILE) graphite-carbon-1.1.7/conf/0000755000175000017500000000000013633714656014210 5ustar gringringraphite-carbon-1.1.7/conf/aggregation-rules.conf.example0000644000175000017500000000340613633714656022133 0ustar gringrin# The form of each line in this file should be as follows: # # output_template (frequency) = method input_pattern # # This will capture any received metrics that match 'input_pattern' # for calculating an aggregate metric. The calculation will occur # every 'frequency' seconds and the 'method' can specify 'sum' or # 'avg'. The name of the aggregate metric will be derived from # 'output_template' filling in any captured fields from 'input_pattern'. # # For example, if you're metric naming scheme is: # # .applications... # # You could configure some aggregations like so: # # .applications..all.requests (60) = sum .applications..*.requests # .applications..all.latency (60) = avg .applications..*.latency # # As an example, if the following metrics are received: # # prod.applications.apache.www01.requests # prod.applications.apache.www01.requests # # They would all go into the same aggregation buffer and after 60 seconds the # aggregate metric 'prod.applications.apache.all.requests' would be calculated # by summing their values. # # Template components such as will match everything up to the next dot. # To match metric multiple components including the dots, use <> in the # input template: # # .applications..all. (60) = sum .applications..*.<> # # It is also possible to use regular expressions. Following the example above # when using: # # .applications...requests (60) = sum .applications..\d{2}.requests # # You will end up with 'prod.applications.apache.www.requests' instead of # 'prod.applications.apache.all.requests'. # # Note that any time this file is modified, it will be re-read automatically. graphite-carbon-1.1.7/conf/blacklist.conf.example0000644000175000017500000000075413633714656020467 0ustar gringrin# This file takes a single regular expression per line # If USE_WHITELIST is set to True in carbon.conf, any metrics received which # match one of these expressions will be dropped # This file is reloaded automatically when changes are made ^some\.noisy\.metric\.prefix\..* # Reject metrics with multiple or surrounding dots, since they lead to # counter intuitive behavior when read (they can be read from disk but not # from carbon-cache, at least with whisper data back-end) \.\. ^\. \.$ graphite-carbon-1.1.7/conf/carbon.amqp.conf.example0000644000175000017500000000504413633714656020715 0ustar gringrin# This is a configuration file with AMQP enabled [cache] LOCAL_DATA_DIR = # Specify the user to drop privileges to # If this is blank carbon runs as the user that invokes it # This user must have write access to the local data directory USER = # Limit the size of the cache to avoid swapping or becoming CPU bound. # Sorts and serving cache queries gets more expensive as the cache grows. # Use the value "inf" (infinity) for an unlimited cache size. MAX_CACHE_SIZE = inf # Limits the number of whisper update_many() calls per second, which effectively # means the number of write requests sent to the disk. This is intended to # prevent over-utilizing the disk and thus starving the rest of the system. # When the rate of required updates exceeds this, then carbon's caching will # take effect and increase the overall throughput accordingly. MAX_UPDATES_PER_SECOND = 1000 # Softly limits the number of whisper files that get created each minute. # Setting this value low (like at 50) is a good way to ensure your graphite # system will not be adversely impacted when a bunch of new metrics are # sent to it. The trade off is that it will take much longer for those metrics' # database files to all get created and thus longer until the data becomes usable. # Setting this value high (like "inf" for infinity) will cause graphite to create # the files quickly but at the risk of slowing I/O down considerably for a while. MAX_CREATES_PER_MINUTE = inf LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2003 UDP_RECEIVER_INTERFACE = 0.0.0.0 UDP_RECEIVER_PORT = 2003 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2004 CACHE_QUERY_INTERFACE = 0.0.0.0 CACHE_QUERY_PORT = 7002 # Enable AMQP if you want to receive metrics using your amqp broker ENABLE_AMQP = True # Verbose means a line will be logged for every metric received # useful for testing AMQP_VERBOSE = True # your credentials for the amqp server # AMQP_USER = guest # AMQP_PASSWORD = guest # the network settings for the amqp server # AMQP_HOST = localhost # AMQP_PORT = 5672 # if you want to include the metric name as part of the message body # instead of as the routing key, set this to True # AMQP_METRIC_NAME_IN_BODY = False # NOTE: you cannot run both a cache and a relay on the same server # with the default configuration, you have to specify a distinict # interfaces and ports for the listeners. [relay] LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2003 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2004 CACHE_SERVERS = server1, server2, server3 MAX_QUEUE_SIZE = 10000 graphite-carbon-1.1.7/conf/carbon.conf.example0000644000175000017500000007475213633714656017774 0ustar gringrin[cache] # Configure carbon directories. # # OS environment variables can be used to tell carbon where graphite is # installed, where to read configuration from and where to write data. # # GRAPHITE_ROOT - Root directory of the graphite installation. # Defaults to ../ # GRAPHITE_CONF_DIR - Configuration directory (where this file lives). # Defaults to $GRAPHITE_ROOT/conf/ # GRAPHITE_STORAGE_DIR - Storage directory for whisper/rrd/log/pid files. # Defaults to $GRAPHITE_ROOT/storage/ # # To change other directory paths, add settings to this file. The following # configuration variables are available with these default values: # # STORAGE_DIR = $GRAPHITE_STORAGE_DIR # LOCAL_DATA_DIR = %(STORAGE_DIR)s/whisper/ # WHITELISTS_DIR = %(STORAGE_DIR)s/lists/ # CONF_DIR = %(STORAGE_DIR)s/conf/ # LOG_DIR = %(STORAGE_DIR)s/log/ # PID_DIR = %(STORAGE_DIR)s/ # # For FHS style directory structures, use: # # STORAGE_DIR = /var/lib/carbon/ # CONF_DIR = /etc/carbon/ # LOG_DIR = /var/log/carbon/ # PID_DIR = /var/run/ # #LOCAL_DATA_DIR = /opt/graphite/storage/whisper/ # Specify the database library used to store metric data on disk. Each database # may have configurable options to change the behaviour of how it writes to # persistent storage. # # whisper - Fixed-size database, similar in design and purpose to RRD. This is # the default storage backend for carbon and the most rigorously tested. # # ceres - Experimental alternative database that supports storing data in sparse # files of arbitrary fixed-size resolutions. DATABASE = whisper # Enable daily log rotation. If disabled, a new file will be opened whenever the log file path no # longer exists (i.e. it is removed or renamed) ENABLE_LOGROTATION = True # Specify the user to drop privileges to # If this is blank carbon-cache runs as the user that invokes it # This user must have write access to the local data directory USER = # Limit the size of the cache to avoid swapping or becoming CPU bound. # Sorts and serving cache queries gets more expensive as the cache grows. # Use the value "inf" (infinity) for an unlimited cache size. # value should be an integer number of metric datapoints. MAX_CACHE_SIZE = inf # Limits the number of whisper update_many() calls per second, which effectively # means the number of write requests sent to the disk. This is intended to # prevent over-utilizing the disk and thus starving the rest of the system. # When the rate of required updates exceeds this, then carbon's caching will # take effect and increase the overall throughput accordingly. MAX_UPDATES_PER_SECOND = 500 # If defined, this changes the MAX_UPDATES_PER_SECOND in Carbon when a # stop/shutdown is initiated. This helps when MAX_UPDATES_PER_SECOND is # relatively low and carbon has cached a lot of updates; it enables the carbon # daemon to shutdown more quickly. # MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = 1000 # Softly limits the number of whisper files that get created each minute. # Setting this value low (e.g. 50) is a good way to ensure that your carbon # system will not be adversely impacted when a bunch of new metrics are # sent to it. The trade off is that any metrics received in excess of this # value will be silently dropped, and the whisper file will not be created # until such point as a subsequent metric is received and fits within the # defined rate limit. Setting this value high (like "inf" for infinity) will # cause carbon to create the files quickly but at the risk of increased I/O. MAX_CREATES_PER_MINUTE = 50 # Set the minimum timestamp resolution supported by this instance. This allows # internal optimisations by overwriting points with equal truncated timestamps # in order to limit the number of updates to the database. It defaults to one # second. MIN_TIMESTAMP_RESOLUTION = 1 # Set the minimum lag in seconds for a point to be written to the database # in order to optimize batching. This means that each point will wait at least # the duration of this lag before being written. Setting this to 0 disable the feature. # This currently only works when using the timesorted write strategy. # MIN_TIMESTAMP_LAG = 0 # Set the interface and port for the line (plain text) listener. Setting the # interface to 0.0.0.0 listens on all interfaces. Port can be set to 0 to # disable this listener if it is not required. LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2003 # Set this to True to enable the UDP listener. By default this is off # because it is very common to run multiple carbon daemons and managing # another (rarely used) port for every carbon instance is not fun. ENABLE_UDP_LISTENER = False UDP_RECEIVER_INTERFACE = 0.0.0.0 UDP_RECEIVER_PORT = 2003 # Set the interface and port for the pickle listener. Setting the interface to # 0.0.0.0 listens on all interfaces. Port can be set to 0 to disable this # listener if it is not required. PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2004 # Set the maximum length for a received pickle string (in bytes). # If the string length is bigger, the receiver will silently close connection. # Useful for big metric batches. # Default value is 1 Mb PICKLE_RECEIVER_MAX_LENGTH = 1048576 # Set the interface and port for the protobuf listener. Setting the interface to # 0.0.0.0 listens on all interfaces. Port can be set to 0 to disable this # listener if it is not required. # PROTOBUF_RECEIVER_INTERFACE = 0.0.0.0 # PROTOBUF_RECEIVER_PORT = 2005 # Limit the number of open connections the receiver can handle as any time. # Default is no limit. Setting up a limit for sites handling high volume # traffic may be recommended to avoid running out of TCP memory or having # thousands of TCP connections reduce the throughput of the service. #MAX_RECEIVER_CONNECTIONS = inf # Per security concerns outlined in Bug #817247 the pickle receiver # will use a more secure and slightly less efficient unpickler. # Set this to True to revert to the old-fashioned insecure unpickler. USE_INSECURE_UNPICKLER = False CACHE_QUERY_INTERFACE = 0.0.0.0 CACHE_QUERY_PORT = 7002 # Set this to False to drop datapoints received after the cache # reaches MAX_CACHE_SIZE. If this is True (the default) then sockets # over which metrics are received will temporarily stop accepting # data until the cache size falls below 95% MAX_CACHE_SIZE. USE_FLOW_CONTROL = True # If enabled this setting is used to timeout metric client connection if no # metrics have been sent in specified time in seconds #METRIC_CLIENT_IDLE_TIMEOUT = None # By default, carbon-cache will log every whisper update and cache hit. # This can be excessive and degrade performance if logging on the same # volume as the whisper data is stored. LOG_UPDATES = False LOG_CREATES = False LOG_CACHE_HITS = False LOG_CACHE_QUEUE_SORTS = False # The thread that writes metrics to disk can use one of the following strategies # determining the order in which metrics are removed from cache and flushed to # disk. The default option preserves the same behavior as has been historically # available in version 0.9.10. # # sorted - All metrics in the cache will be counted and an ordered list of # them will be sorted according to the number of datapoints in the cache at the # moment of the list's creation. Metrics will then be flushed from the cache to # disk in that order. # # timesorted - All metrics in the list will be looked at and sorted according # to the timestamp of there datapoints. The metric that were the least recently # written will be written first. This is an hybrid strategy between max and # sorted which is particularly adapted to sets of metrics with non-uniform # resolutions. # # max - The writer thread will always pop and flush the metric from cache # that has the most datapoints. This will give a strong flush preference to # frequently updated metrics and will also reduce random file-io. Infrequently # updated metrics may only ever be persisted to disk at daemon shutdown if # there are a large number of metrics which receive very frequent updates OR if # disk i/o is very slow. # # naive - Metrics will be flushed from the cache to disk in an unordered # fashion. This strategy may be desirable in situations where the storage for # whisper files is solid state, CPU resources are very limited or deference to # the OS's i/o scheduler is expected to compensate for the random write # pattern. # CACHE_WRITE_STRATEGY = sorted # On some systems it is desirable for whisper to write synchronously. # Set this option to True if you'd like to try this. Basically it will # shift the onus of buffering writes from the kernel into carbon's cache. WHISPER_AUTOFLUSH = False # By default new Whisper files are created pre-allocated with the data region # filled with zeros to prevent fragmentation and speed up contiguous reads and # writes (which are common). Enabling this option will cause Whisper to create # the file sparsely instead. Enabling this option may allow a large increase of # MAX_CREATES_PER_MINUTE but may have longer term performance implications # depending on the underlying storage configuration. # WHISPER_SPARSE_CREATE = False # Only beneficial on linux filesystems that support the fallocate system call. # It maintains the benefits of contiguous reads/writes, but with a potentially # much faster creation speed, by allowing the kernel to handle the block # allocation and zero-ing. Enabling this option may allow a large increase of # MAX_CREATES_PER_MINUTE. If enabled on an OS or filesystem that is unsupported # this option will gracefully fallback to standard POSIX file access methods. # If enabled, disables WHISPER_SPARSE_CREATE regardless of the value. WHISPER_FALLOCATE_CREATE = True # Enabling this option will cause Whisper to lock each Whisper file it writes # to with an exclusive lock (LOCK_EX, see: man 2 flock). This is useful when # multiple carbon-cache daemons are writing to the same files. # WHISPER_LOCK_WRITES = False # On systems which has a large number of metrics, an amount of Whisper write(2)'s # pageback sometimes cause disk thrashing due to memory shortage, so that abnormal # disk reads occur. Enabling this option makes it possible to decrease useless # page cache memory by posix_fadvise(2) with POSIX_FADVISE_RANDOM option. # WHISPER_FADVISE_RANDOM = False # By default all nodes stored in Ceres are cached in memory to improve the # throughput of reads and writes to underlying slices. Turning this off will # greatly reduce memory consumption for databases with millions of metrics, at # the cost of a steep increase in disk i/o, approximately an extra two os.stat # calls for every read and write. Reasons to do this are if the underlying # storage can handle stat() with practically zero cost (SSD, NVMe, zRAM). # Valid values are: # all - all nodes are cached # none - node caching is disabled # CERES_NODE_CACHING_BEHAVIOR = all # Ceres nodes can have many slices and caching the right ones can improve # performance dramatically. Note that there are many trade-offs to tinkering # with this, and unless you are a ceres developer you *really* should not # mess with this. Valid values are: # latest - only the most recent slice is cached # all - all slices are cached # none - slice caching is disabled # CERES_SLICE_CACHING_BEHAVIOR = latest # If a Ceres node accumulates too many slices, performance can suffer. # This can be caused by intermittently reported data. To mitigate # slice fragmentation there is a tolerance for how much space can be # wasted within a slice file to avoid creating a new one. That tolerance # level is determined by MAX_SLICE_GAP, which is the number of consecutive # null datapoints allowed in a slice file. # If you set this very low, you will waste less of the *tiny* bit disk space # that this feature wastes, and you will be prone to performance problems # caused by slice fragmentation, which can be pretty severe. # If you set this really high, you will waste a bit more disk space (each # null datapoint wastes 8 bytes, but keep in mind your filesystem's block # size). If you suffer slice fragmentation issues, you should increase this or # run the ceres-maintenance defrag plugin more often. However you should not # set it to be huge because then if a large but allowed gap occurs it has to # get filled in, which means instead of a simple 8-byte write to a new file we # could end up doing an (8 * MAX_SLICE_GAP)-byte write to the latest slice. # CERES_MAX_SLICE_GAP = 80 # Enabling this option will cause Ceres to lock each Ceres file it writes to # to with an exclusive lock (LOCK_EX, see: man 2 flock). This is useful when # multiple carbon-cache daemons are writing to the same files. # CERES_LOCK_WRITES = False # Set this to True to enable whitelisting and blacklisting of metrics in # CONF_DIR/whitelist.conf and CONF_DIR/blacklist.conf. If the whitelist is # missing or empty, all metrics will pass through # USE_WHITELIST = False # By default, carbon itself will log statistics (such as a count, # metricsReceived) with the top level prefix of 'carbon' at an interval of 60 # seconds. Set CARBON_METRIC_INTERVAL to 0 to disable instrumentation # CARBON_METRIC_PREFIX = carbon # CARBON_METRIC_INTERVAL = 60 # Enable AMQP if you want to receve metrics using an amqp broker # ENABLE_AMQP = False # Verbose means a line will be logged for every metric received # useful for testing # AMQP_VERBOSE = False # AMQP_HOST = localhost # AMQP_PORT = 5672 # AMQP_VHOST = / # AMQP_USER = guest # AMQP_PASSWORD = guest # AMQP_EXCHANGE = graphite # AMQP_METRIC_NAME_IN_BODY = False # The manhole interface allows you to SSH into the carbon daemon # and get a python interpreter. BE CAREFUL WITH THIS! If you do # something like time.sleep() in the interpreter, the whole process # will sleep! This is *extremely* helpful in debugging, assuming # you are familiar with the code. If you are not, please don't # mess with this, you are asking for trouble :) # # You need the bcrypt, cryptography and pyasn1 python modules installed for # manhole to work. # # Generate host keys with: # `ckeygen -t rsa -f /example/host_keys/ssh_host_key_rsa` # # ENABLE_MANHOLE = False # MANHOLE_INTERFACE = 127.0.0.1 # MANHOLE_PORT = 7222 # MANHOLE_USER = admin # MANHOLE_PUBLIC_KEY = ssh-rsa AAAAB3NzaC1yc2EAAAABiwAaAIEAoxN0sv/e4eZCPpi3N3KYvyzRaBaMeS2RsOQ/cDuKv11dlNzVeiyc3RFmCv5Rjwn/lQ79y0zyHxw67qLyhQ/kDzINc4cY41ivuQXm2tPmgvexdrBv5nsfEpjs3gLZfJnyvlcVyWK/lId8WUvEWSWHTzsbtmXAF2raJMdgLTbQ8wE= # MANHOLE_HOST_KEY_DIR = /example/host_keys # Patterns for all of the metrics this machine will store. Read more at # http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol#Bindings # # Example: store all sales, linux servers, and utilization metrics # BIND_PATTERNS = sales.#, servers.linux.#, #.utilization # # Example: store everything # BIND_PATTERNS = # # URL of graphite-web instance, this is used to add incoming series to the tag database # GRAPHITE_URL = http://127.0.0.1:80 # Tag support, when enabled carbon will make HTTP calls to graphite-web to update the tag index # ENABLE_TAGS = True # Tag update interval, this specifies how frequently updates to existing series will trigger # an update to the tag index, the default setting is once every 100 updates # TAG_UPDATE_INTERVAL = 100 # Tag hash filenames, this specifies whether tagged metric filenames should use the hash of the metric name # or a human-readable name, using hashed names avoids issues with path length when using a large number of tags # TAG_HASH_FILENAMES = True # Tag batch size, this specifies the maximum number of series to be sent to graphite-web in a single batch # TAG_BATCH_SIZE = 100 # Tag queue size, this specifies the maximum number of series to be queued for sending to graphite-web # There are separate queues for new series and for updates to existing series # TAG_QUEUE_SIZE = 10000 # Set to enable Sentry.io exception monitoring. # RAVEN_DSN='YOUR_DSN_HERE'. # To configure special settings for the carbon-cache instance 'b', uncomment this: #[cache:b] #LINE_RECEIVER_PORT = 2103 #PICKLE_RECEIVER_PORT = 2104 #CACHE_QUERY_PORT = 7102 # and any other settings you want to customize, defaults are inherited # from the [cache] section. # You can then specify the --instance=b option to manage this instance # # In order to turn off logging of successful connections for the line # receiver, set this to False # LOG_LISTENER_CONN_SUCCESS = True [relay] LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2013 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2014 PICKLE_RECEIVER_MAX_LENGTH = 1048576 # Carbon-relay has several options for metric routing controlled by RELAY_METHOD # # Use relay-rules.conf to route metrics to destinations based on pattern rules #RELAY_METHOD = rules # # Use consistent-hashing for even distribution of metrics between destinations #RELAY_METHOD = consistent-hashing # # Use consistent-hashing but take into account an aggregation-rules.conf shared # by downstream carbon-aggregator daemons. This will ensure that all metrics # that map to a given aggregation rule are sent to the same carbon-aggregator # instance. # Enable this for carbon-relays that send to a group of carbon-aggregators #RELAY_METHOD = aggregated-consistent-hashing # # You can also use fast-hashing and fast-aggregated-hashing which are in O(1) # and will always redirect the metrics to the same destination but do not try # to minimize rebalancing when the list of destinations is changing. RELAY_METHOD = rules # If you use consistent-hashing you can add redundancy by replicating every # datapoint to more than one machine. REPLICATION_FACTOR = 1 # When relaying tagged metrics, you should consider setting TAG_RELAY_NORMALIZED # to True so that metrics are normalized before they are relayed. Without # normalization, there might be inconsistencies between the destination selected # by the consistent-hashing method and where graphite-web actually expects the # metrics to end up (carbon-cache and thus graphite-web always normalize). In # this case, graphite-web query results will be missing some of the data that # hasn't been written to disk yet by carbon-cache. This increases CPU usage # though, so you might leave this off if you know that only normalized metrics # will be received by your relay, for example because an upstream relay already # does normalization. #TAG_RELAY_NORMALIZED = False # For REPLICATION_FACTOR >=2, set DIVERSE_REPLICAS to True to guarantee replicas # across distributed hosts. With this setting disabled, it's possible that replicas # may be sent to different caches on the same host. This has been the default # behavior since introduction of 'consistent-hashing' relay method. # Note that enabling this on an existing pre-0.9.14 cluster will require rebalancing # your metrics across the cluster nodes using a tool like Carbonate. #DIVERSE_REPLICAS = True # This is a list of carbon daemons we will send any relayed or # generated metrics to. The default provided would send to a single # carbon-cache instance on the default port. However if you # use multiple carbon-cache instances then it would look like this: # # DESTINATIONS = 127.0.0.1:2004:a, 127.0.0.1:2104:b # # The general form is IP:PORT:INSTANCE where the :INSTANCE part is # optional and refers to the "None" instance if omitted. # # Note that if the destinations are all carbon-caches then this should # exactly match the webapp's CARBONLINK_HOSTS setting in terms of # instances listed (order matters!). # # If using RELAY_METHOD = rules, all destinations used in relay-rules.conf # must be defined in this list DESTINATIONS = 127.0.0.1:2004 # This define the protocol to use to contact the destination. It can be # set to one of "line", "pickle", "udp" and "protobuf". This list can be # extended with CarbonClientFactory plugins and defaults to "pickle". # DESTINATION_PROTOCOL = pickle # This defines the wire transport, either none or ssl. # If SSL is used any TCP connection will be upgraded to TLS1. The system's # trust authority will be used unless DESTINATION_SSL_CA is specified in # which case an alternative certificate authority chain will be used for # verifying the remote certificate. # To use SSL you'll need the cryptography, service_identity, and twisted >= 14 # DESTINATION_TRANSPORT = none # DESTINATION_SSL_CA=/path/to/private-ca.crt # This allows to have multiple connections per destinations, this will # pool all the replicas of a single host in the same queue and distribute # points accross these replicas instead of replicating them. # The following example will balance the load between :0 and :1. ## DESTINATIONS = foo:2001:0, foo:2001:1 ## RELAY_METHOD = rules # Note: this is currently incompatible with USE_RATIO_RESET which gets # disabled if this option is enabled. # DESTINATIONS_POOL_REPLICAS = False # When using consistent hashing it sometime makes sense to make # the ring dynamic when you don't want to loose points when a # single destination is down. Replication is an answer to that # but it can be quite expensive. # DYNAMIC_ROUTER = False # Controls the number of connection attempts before marking a # destination as down. We usually do one connection attempt per # second. # DYNAMIC_ROUTER_MAX_RETRIES = 5 # This is the maximum number of datapoints that can be queued up # for a single destination. Once this limit is hit, we will # stop accepting new data if USE_FLOW_CONTROL is True, otherwise # we will drop any subsequently received datapoints. MAX_QUEUE_SIZE = 10000 # This defines the maximum "message size" between carbon daemons. If # your queue is large, setting this to a lower number will cause the # relay to forward smaller discrete chunks of stats, which may prevent # overloading on the receiving side after a disconnect. MAX_DATAPOINTS_PER_MESSAGE = 500 # Limit the number of open connections the receiver can handle as any time. # Default is no limit. Setting up a limit for sites handling high volume # traffic may be recommended to avoid running out of TCP memory or having # thousands of TCP connections reduce the throughput of the service. #MAX_RECEIVER_CONNECTIONS = inf # Specify the user to drop privileges to # If this is blank carbon-relay runs as the user that invokes it # USER = # This is the percentage that the queue must be empty before it will accept # more messages. For a larger site, if the queue is very large it makes sense # to tune this to allow for incoming stats. So if you have an average # flow of 100k stats/minute, and a MAX_QUEUE_SIZE of 3,000,000, it makes sense # to allow stats to start flowing when you've cleared the queue to 95% since # you should have space to accommodate the next minute's worth of stats # even before the relay incrementally clears more of the queue QUEUE_LOW_WATERMARK_PCT = 0.8 # To allow for batch efficiency from the pickle protocol and to benefit from # other batching advantages, all writes are deferred by putting them into a queue, # and then the queue is flushed and sent a small fraction of a second later. TIME_TO_DEFER_SENDING = 0.0001 # Set this to False to drop datapoints when any send queue (sending datapoints # to a downstream carbon daemon) hits MAX_QUEUE_SIZE. If this is True (the # default) then sockets over which metrics are received will temporarily stop accepting # data until the send queues fall below QUEUE_LOW_WATERMARK_PCT * MAX_QUEUE_SIZE. USE_FLOW_CONTROL = True # If enabled this setting is used to timeout metric client connection if no # metrics have been sent in specified time in seconds #METRIC_CLIENT_IDLE_TIMEOUT = None # Set this to True to enable whitelisting and blacklisting of metrics in # CONF_DIR/whitelist.conf and CONF_DIR/blacklist.conf. If the whitelist is # missing or empty, all metrics will pass through # USE_WHITELIST = False # By default, carbon itself will log statistics (such as a count, # metricsReceived) with the top level prefix of 'carbon' at an interval of 60 # seconds. Set CARBON_METRIC_INTERVAL to 0 to disable instrumentation # CARBON_METRIC_PREFIX = carbon # CARBON_METRIC_INTERVAL = 60 # # In order to turn off logging of successful connections for the line # receiver, set this to False # LOG_LISTENER_CONN_SUCCESS = True # If you're connecting from the relay to a destination that's over the # internet or similarly iffy connection, a backlog can develop because # of internet weather conditions, e.g. acks getting lost or similar issues. # To deal with that, you can enable USE_RATIO_RESET which will let you # re-set the connection to an individual destination. Defaults to being off. USE_RATIO_RESET=False # When there is a small number of stats flowing, it's not desirable to # perform any actions based on percentages - it's just too "twitchy". MIN_RESET_STAT_FLOW=1000 # When the ratio of stats being sent in a reporting interval is far # enough from 1.0, we will disconnect the socket and reconnecto to # clear out queued stats. The default ratio of 0.9 indicates that 10% # of stats aren't being delivered within one CARBON_METRIC_INTERVAL # (default of 60 seconds), which can lead to a queue backup. Under # some circumstances re-setting the connection can fix this, so # set this according to your tolerance, and look in the logs for # "resetConnectionForQualityReasons" to observe whether this is kicking # in when your sent queue is building up. MIN_RESET_RATIO=0.9 # The minimum time between resets. When a connection is re-set, we # need to wait before another reset is performed. # (2*CARBON_METRIC_INTERVAL) + 1 second is the minimum time needed # before stats for the new connection will be available. Setting this # below (2*CARBON_METRIC_INTERVAL) + 1 second will result in a lot of # reset connections for no good reason. MIN_RESET_INTERVAL=121 # Enable TCP Keep Alive (http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html). # Default settings will send a probe every 30s. Default is False. # TCP_KEEPALIVE=True # The interval between the last data packet sent (simple ACKs are not # considered data) and the first keepalive probe; after the connection is marked # to need keepalive, this counter is not used any further. # TCP_KEEPIDLE=10 # The interval between subsequential keepalive probes, regardless of what # the connection has exchanged in the meantime. # TCP_KEEPINTVL=30 # The number of unacknowledged probes to send before considering the connection # dead and notifying the application layer. # TCP_KEEPCNT=2 [aggregator] LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2023 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2024 # If set true, metric received will be forwarded to DESTINATIONS in addition to # the output of the aggregation rules. If set false the carbon-aggregator will # only ever send the output of aggregation. FORWARD_ALL = True # Filenames of the configuration files to use for this instance of aggregator. # Filenames are relative to CONF_DIR. # # AGGREGATION_RULES = aggregation-rules.conf # REWRITE_RULES = rewrite-rules.conf # This is a list of carbon daemons we will send any relayed or # generated metrics to. The default provided would send to a single # carbon-cache instance on the default port. However if you # use multiple carbon-cache instances then it would look like this: # # DESTINATIONS = 127.0.0.1:2004:a, 127.0.0.1:2104:b # # The format is comma-delimited IP:PORT:INSTANCE where the :INSTANCE part is # optional and refers to the "None" instance if omitted. # # Note that if the destinations are all carbon-caches then this should # exactly match the webapp's CARBONLINK_HOSTS setting in terms of # instances listed (order matters!). DESTINATIONS = 127.0.0.1:2004 # If you want to add redundancy to your data by replicating every # datapoint to more than one machine, increase this. REPLICATION_FACTOR = 1 # This is the maximum number of datapoints that can be queued up # for a single destination. Once this limit is hit, we will # stop accepting new data if USE_FLOW_CONTROL is True, otherwise # we will drop any subsequently received datapoints. MAX_QUEUE_SIZE = 10000 # Set this to False to drop datapoints when any send queue (sending datapoints # to a downstream carbon daemon) hits MAX_QUEUE_SIZE. If this is True (the # default) then sockets over which metrics are received will temporarily stop accepting # data until the send queues fall below 80% MAX_QUEUE_SIZE. USE_FLOW_CONTROL = True # If enabled this setting is used to timeout metric client connection if no # metrics have been sent in specified time in seconds #METRIC_CLIENT_IDLE_TIMEOUT = None # This defines the maximum "message size" between carbon daemons. # You shouldn't need to tune this unless you really know what you're doing. MAX_DATAPOINTS_PER_MESSAGE = 500 # This defines how many datapoints the aggregator remembers for # each metric. Aggregation only happens for datapoints that fall in # the past MAX_AGGREGATION_INTERVALS * intervalSize seconds. MAX_AGGREGATION_INTERVALS = 5 # Limit the number of open connections the receiver can handle as any time. # Default is no limit. Setting up a limit for sites handling high volume # traffic may be recommended to avoid running out of TCP memory or having # thousands of TCP connections reduce the throughput of the service. #MAX_RECEIVER_CONNECTIONS = inf # By default (WRITE_BACK_FREQUENCY = 0), carbon-aggregator will write back # aggregated data points once every rule.frequency seconds, on a per-rule basis. # Set this (WRITE_BACK_FREQUENCY = N) to write back all aggregated data points # every N seconds, independent of rule frequency. This is useful, for example, # to be able to query partially aggregated metrics from carbon-cache without # having to first wait rule.frequency seconds. # WRITE_BACK_FREQUENCY = 0 # Set this to True to enable whitelisting and blacklisting of metrics in # CONF_DIR/whitelist.conf and CONF_DIR/blacklist.conf. If the whitelist is # missing or empty, all metrics will pass through # USE_WHITELIST = False # By default, carbon itself will log statistics (such as a count, # metricsReceived) with the top level prefix of 'carbon' at an interval of 60 # seconds. Set CARBON_METRIC_INTERVAL to 0 to disable instrumentation # CARBON_METRIC_PREFIX = carbon # CARBON_METRIC_INTERVAL = 60 # In order to turn off logging of successful connections for the line # receiver, set this to False # LOG_LISTENER_CONN_SUCCESS = True # In order to turn off logging of metrics with no corresponding # aggregation rules receiver, set this to False # LOG_AGGREGATOR_MISSES = False # Specify the user to drop privileges to # If this is blank carbon-aggregator runs as the user that invokes it # USER = # Part of the code, and particularly aggregator rules, need # to cache metric names. To avoid leaking too much memory you # can tweak the size of this cache. The default allow for 1M # different metrics per rule (~200MiB). # CACHE_METRIC_NAMES_MAX=1000000 # You can optionally set a ttl to this cache. # CACHE_METRIC_NAMES_TTL=600 graphite-carbon-1.1.7/conf/relay-rules.conf.example0000644000175000017500000000157013633714656020760 0ustar gringrin# Relay destination rules for carbon-relay. Entries are scanned in order, # and the first pattern a metric matches will cause processing to cease after sending # unless `continue` is set to true # # [name] # pattern = # destinations = # continue = # default: False # # name: Arbitrary unique name to identify the rule # pattern: Regex pattern to match against the metric name # destinations: Comma-separated list of destinations. # ex: 127.0.0.1:2004:a, 10.1.2.4:2004, myserver.mydomain.com:2004 # continue: Continue processing rules if this rule matches (default: False) # You must have exactly one section with 'default = true' # Note that all destinations listed must also exist in carbon.conf # in the DESTINATIONS setting in the [relay] section [default] default = true destinations = 127.0.0.1:2004:a, 127.0.0.1:2104:b graphite-carbon-1.1.7/conf/rewrite-rules.conf.example0000644000175000017500000000105613633714656021324 0ustar gringrin# This file defines regular expression patterns that can be used to # rewrite metric names in a search & replace fashion. It consists of two # sections, [pre] and [post]. The rules in the pre section are applied to # metric names as soon as they are received. The post rules are applied # after aggregation has taken place. # # The general form of each rule is as follows: # # regex-pattern = replacement-text # # For example: # # [post] # _sum$ = # _avg$ = # # These rules would strip off a suffix of _sum or _avg from any metric names # after aggregation. graphite-carbon-1.1.7/conf/storage-aggregation.conf.example0000644000175000017500000000147313633714656022447 0ustar gringrin# Aggregation methods for whisper files. Entries are scanned in order, # and first match wins. This file is scanned for changes every 60 seconds # # [name] # pattern = # xFilesFactor = # aggregationMethod = # # name: Arbitrary unique name for the rule # pattern: Regex pattern to match against the metric name # xFilesFactor: Ratio of valid data points required for aggregation to the next retention to occur # aggregationMethod: function to apply to data points for aggregation # [min] pattern = \.min$ xFilesFactor = 0.1 aggregationMethod = min [max] pattern = \.max$ xFilesFactor = 0.1 aggregationMethod = max [sum] pattern = \.count$ xFilesFactor = 0 aggregationMethod = max [default_average] pattern = .* xFilesFactor = 0.5 aggregationMethod = average graphite-carbon-1.1.7/conf/storage-schemas.conf.example0000644000175000017500000000241113633714656021574 0ustar gringrin# Schema definitions for Whisper files. Entries are scanned in order, # and first match wins. This file is scanned for changes every 60 seconds. # # Definition Syntax: # # [name] # pattern = regex # retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ... # # name: Arbitrary unique name for the rule # pattern: Regular expression to match against the metric name. For syntax see: # https://docs.python.org/3/library/re.html#regular-expression-syntax # retentions: Retention schema for the metric # # Remember: To support accurate aggregation from higher to lower resolution # archives, the precision of a longer retention archive must be # cleanly divisible by precision of next lower retention archive. # # Valid: 60s:7d,300s:30d (300/60 = 5) # Invalid: 180s:7d,300s:30d (300/180 = 3.333) # # This retention is set at the time the first metric is sent. # Changing this file will not affect already-created .wsp files. # Use whisper-resize.py to change existing data files. # Carbon's internal metrics. This entry should match what is specified in # CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings [carbon] pattern = ^carbon\. retentions = 60:90d [default] pattern = .* retentions = 60s:1d,5m:30d,1h:3y graphite-carbon-1.1.7/conf/whitelist.conf.example0000644000175000017500000000047313633714656020531 0ustar gringrin# This file takes a single regular expression per line # If USE_WHITELIST is set to True in carbon.conf, only metrics received which # match one of these expressions will be persisted. If this file is empty or # missing, all metrics will pass through. # This file is reloaded automatically when changes are made .* graphite-carbon-1.1.7/distro/0000755000175000017500000000000013633714656014567 5ustar gringringraphite-carbon-1.1.7/distro/debian/0000755000175000017500000000000013633714656016011 5ustar gringringraphite-carbon-1.1.7/distro/debian/init.d/0000755000175000017500000000000013633714656017176 5ustar gringringraphite-carbon-1.1.7/distro/debian/init.d/carbon-aggregator0000644000175000017500000000436613633714656022516 0ustar gringrin#!/bin/bash # chkconfig: - 25 75 # description: carbon-aggregator # processname: carbon-aggregator export PYTHONPATH="$GRAPHITE_DIR/lib:$PYTHONPATH" # Source function library. if [ -e /lib/lsb/init-functions ]; then . /lib/lsb/init-functions fi; CARBON_DAEMON="aggregator" GRAPHITE_DIR="/opt/graphite" INSTANCES=`grep "^\[${CARBON_DAEMON}" ${GRAPHITE_DIR}/conf/carbon.conf | cut -d \[ -f 2 | cut -d \] -f 1 | cut -d : -f 2` function die { echo $1 exit 1 } start(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; log_action_begin_msg "Starting carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} start; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } stop(){ cd $GRAPHITE_DIR for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; log_action_begin_msg "Stopping carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} stop if [ `sleep 3; /usr/bin/pgrep -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}" | /usr/bin/wc -l` -gt 0 ]; then echo "Carbon did not stop yet. Sleeping longer, then force killing it..."; sleep 20; /usr/bin/pkill -9 -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}"; fi; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } status(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} status; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } case "$1" in start) start ;; stop) stop ;; status) status ;; restart|reload) stop start ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esacgraphite-carbon-1.1.7/distro/debian/init.d/carbon-cache0000644000175000017500000000434713633714656021436 0ustar gringrin#!/bin/bash # chkconfig: - 25 75 # description: carbon-cache # processname: carbon-cache export PYTHONPATH="$GRAPHITE_DIR/lib:$PYTHONPATH" # Source function library. if [ -e /lib/lsb/init-functions ]; then . /lib/lsb/init-functions fi; CARBON_DAEMON="cache" GRAPHITE_DIR="/opt/graphite" INSTANCES=`grep "^\[${CARBON_DAEMON}" ${GRAPHITE_DIR}/conf/carbon.conf | cut -d \[ -f 2 | cut -d \] -f 1 | cut -d : -f 2` function die { echo $1 exit 1 } start(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; log_action_begin_msg "Starting carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} start; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } stop(){ cd $GRAPHITE_DIR for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; log_action_begin_msg "Stopping carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} stop if [ `sleep 3; /usr/bin/pgrep -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}" | /usr/bin/wc -l` -gt 0 ]; then echo "Carbon did not stop yet. Sleeping longer, then force killing it..."; sleep 20; /usr/bin/pkill -9 -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}"; fi; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } status(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} status; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } case "$1" in start) start ;; stop) stop ;; status) status ;; restart|reload) stop start ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esac graphite-carbon-1.1.7/distro/debian/init.d/carbon-relay0000644000175000017500000000432713633714656021505 0ustar gringrin#!/bin/bash # chkconfig: - 25 75 # description: carbon-relay # processname: carbon-relay export PYTHONPATH="$GRAPHITE_DIR/lib:$PYTHONPATH" # Source function library. if [ -e /lib/lsb/init-functions ]; then . /lib/lsb/init-functions fi; CARBON_DAEMON="relay" GRAPHITE_DIR="/opt/graphite" INSTANCES=`grep "^\[${CARBON_DAEMON}" ${GRAPHITE_DIR}/conf/carbon.conf | cut -d \[ -f 2 | cut -d \] -f 1 | cut -d : -f 2` function die { echo $1 exit 1 } start(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; log_action_begin_msg "Starting carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} start; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } stop(){ cd $GRAPHITE_DIR for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Stopping carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} stop if [ `sleep 3; /usr/bin/pgrep -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}" | /usr/bin/wc -l` -gt 0 ]; then echo "Carbon did not stop yet. Sleeping longer, then force killing it..."; sleep 20; /usr/bin/pkill -9 -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}"; fi; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } status(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} status; if [ $? -eq 0 ]; then log_success_msg else log_failure_msg fi; echo "" done; } case "$1" in start) start ;; stop) stop ;; status) status ;; restart|reload) stop start ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esac graphite-carbon-1.1.7/distro/debian/misc/0000755000175000017500000000000013633714656016744 5ustar gringringraphite-carbon-1.1.7/distro/debian/misc/postinstall0000644000175000017500000000104313633714656021241 0ustar gringrin#!/bin/bash # Configure init scripts INIT_SCRIPTS="carbon-cache carbon-relay carbon-aggregator"; for s in $INIT_SCRIPTS; do /bin/chmod +x /etc/init.d/${s}; /usr/sbin/update-rc.d ${s} defaults done; GRAPHITE_PATH=/opt/graphite CONFFILES="carbon.conf relay-rules.conf storage-schemas.conf storage-aggregation.conf" for i in $CONFFILES; do if [ ! -e ${GRAPHITE_PATH}/conf/$i ]; then /bin/echo "No pre-existing $i - creating from example." /bin/cp ${GRAPHITE_PATH}/conf/$i.example ${GRAPHITE_PATH}/conf/$i; fi; done; graphite-carbon-1.1.7/distro/redhat/0000755000175000017500000000000013633714656016036 5ustar gringringraphite-carbon-1.1.7/distro/redhat/init.d/0000755000175000017500000000000013633714656017223 5ustar gringringraphite-carbon-1.1.7/distro/redhat/init.d/carbon-aggregator0000644000175000017500000000431313633714656022533 0ustar gringrin#!/bin/bash # chkconfig: - 25 75 # description: carbon-aggregator # processname: carbon-aggregator export PYTHONPATH="$GRAPHITE_DIR/lib:$PYTHONPATH" # Source function library. if [ -e /etc/rc.d/init.d/functions ]; then . /etc/rc.d/init.d/functions; fi; CARBON_DAEMON="aggregator" GRAPHITE_DIR="/opt/graphite" INSTANCES=`grep "^\[${CARBON_DAEMON}" ${GRAPHITE_DIR}/conf/carbon.conf | cut -d \[ -f 2 | cut -d \] -f 1 | cut -d : -f 2` function die { echo $1 exit 1 } start(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Starting carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} start; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } stop(){ cd $GRAPHITE_DIR for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Stopping carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} stop if [ `sleep 3; /usr/bin/pgrep -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}" | /usr/bin/wc -l` -gt 0 ]; then echo "Carbon did not stop yet. Sleeping longer, then force killing it..."; sleep 20; /usr/bin/pkill -9 -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}"; fi; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } status(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} status; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } case "$1" in start) start ;; stop) stop ;; status) status ;; restart|reload) stop start ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esac graphite-carbon-1.1.7/distro/redhat/init.d/carbon-cache0000644000175000017500000000427413633714656021462 0ustar gringrin#!/bin/bash # chkconfig: - 25 75 # description: carbon-cache # processname: carbon-cache # Source function library. if [ -e /etc/rc.d/init.d/functions ]; then . /etc/rc.d/init.d/functions; fi; CARBON_DAEMON="cache" GRAPHITE_DIR="/opt/graphite" INSTANCES=`grep "^\[${CARBON_DAEMON}" ${GRAPHITE_DIR}/conf/carbon.conf | cut -d \[ -f 2 | cut -d \] -f 1 | cut -d : -f 2` export PYTHONPATH="$GRAPHITE_DIR/lib:$PYTHONPATH" function die { echo $1 exit 1 } start(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Starting carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} start; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } stop(){ cd $GRAPHITE_DIR for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Stopping carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} stop if [ `sleep 3; /usr/bin/pgrep -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}" | /usr/bin/wc -l` -gt 0 ]; then echo "Carbon did not stop yet. Sleeping longer, then force killing it..."; sleep 20; /usr/bin/pkill -9 -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}"; fi; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } status(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} status; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } case "$1" in start) start ;; stop) stop ;; status) status ;; restart|reload) stop start ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esac graphite-carbon-1.1.7/distro/redhat/init.d/carbon-relay0000644000175000017500000000427413633714656021533 0ustar gringrin#!/bin/bash # chkconfig: - 25 75 # description: carbon-relay # processname: carbon-relay export PYTHONPATH="$GRAPHITE_DIR/lib:$PYTHONPATH" # Source function library. if [ -e /etc/rc.d/init.d/functions ]; then . /etc/rc.d/init.d/functions; fi; CARBON_DAEMON="relay" GRAPHITE_DIR="/opt/graphite" INSTANCES=`grep "^\[${CARBON_DAEMON}" ${GRAPHITE_DIR}/conf/carbon.conf | cut -d \[ -f 2 | cut -d \] -f 1 | cut -d : -f 2` function die { echo $1 exit 1 } start(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Starting carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} start; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } stop(){ cd $GRAPHITE_DIR for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; echo "Stopping carbon-${CARBON_DAEMON}:${INSTANCE}..." bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} stop if [ `sleep 3; /usr/bin/pgrep -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}" | /usr/bin/wc -l` -gt 0 ]; then echo "Carbon did not stop yet. Sleeping longer, then force killing it..."; sleep 20; /usr/bin/pkill -9 -f "carbon-${CARBON_DAEMON}.py --instance=${INSTANCE}"; fi; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } status(){ cd $GRAPHITE_DIR; for INSTANCE in ${INSTANCES}; do if [ "${INSTANCE}" == "${CARBON_DAEMON}" ]; then INSTANCE="a"; fi; bin/carbon-${CARBON_DAEMON}.py --instance=${INSTANCE} status; if [ $? -eq 0 ]; then echo_success else echo_failure fi; echo "" done; } case "$1" in start) start ;; stop) stop ;; status) status ;; restart|reload) stop start ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esac graphite-carbon-1.1.7/distro/redhat/misc/0000755000175000017500000000000013633714656016771 5ustar gringringraphite-carbon-1.1.7/distro/redhat/misc/postinstall0000644000175000017500000000107713633714656021275 0ustar gringrin# Configure init scripts INIT_SCRIPTS="carbon-cache carbon-relay carbon-aggregator"; for s in $INIT_SCRIPTS; do /bin/chmod +x /etc/init.d/${s}; if [ -x /sbin/chkconfig ]; then /sbin/chkconfig --add ${s}; fi; done; GRAPHITE_PATH=/opt/graphite CONFFILES="carbon.conf relay-rules.conf storage-schemas.conf storage-aggregation.conf" for i in $CONFFILES; do if [ ! -e ${GRAPHITE_PATH}/conf/$i ]; then /bin/echo "No pre-existing $i - creating from example." /bin/cp ${GRAPHITE_PATH}/conf/$i.example ${GRAPHITE_PATH}/conf/$i; fi; done; graphite-carbon-1.1.7/examples/0000755000175000017500000000000013633714656015101 5ustar gringringraphite-carbon-1.1.7/examples/example-client.py0000644000175000017500000000502413633714656020363 0ustar gringrin#!/usr/bin/python """Copyright 2008 Orbitz WorldWide Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import re import sys import time import socket import platform import subprocess CARBON_SERVER = '127.0.0.1' CARBON_PORT = 2003 DELAY = 60 def get_loadavg(): """ Get the load average for a unix-like system. For more details, "man proc" and "man uptime" """ if platform.system() == "Linux": return open('/proc/loadavg').read().split()[:3] else: command = "uptime" process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True) stdout = process.communicate()[0].strip() # Split on whitespace and commas output = re.split("[\s,]+", stdout) return output[-3:] def run(sock, delay): """Make the client go go go""" while True: now = int(time.time()) lines = [] #We're gonna report all three loadavg values loadavg = get_loadavg() lines.append("system.loadavg_1min %s %d" % (loadavg[0], now)) lines.append("system.loadavg_5min %s %d" % (loadavg[1], now)) lines.append("system.loadavg_15min %s %d" % (loadavg[2], now)) message = '\n'.join(lines) + '\n' #all lines must end in a newline print("sending message") print('-' * 80) print(message) sock.sendall(message) time.sleep(delay) def main(): """Wrap it all up together""" delay = DELAY if len(sys.argv) > 1: arg = sys.argv[1] if arg.isdigit(): delay = int(arg) else: sys.stderr.write("Ignoring non-integer argument. Using default: %ss\n" % delay) sock = socket.socket() try: sock.connect( (CARBON_SERVER, CARBON_PORT) ) except socket.error: raise SystemExit("Couldn't connect to %(server)s on port %(port)d, is carbon-cache.py running?" % { 'server':CARBON_SERVER, 'port':CARBON_PORT }) try: run(sock, delay) except KeyboardInterrupt: sys.stderr.write("\nExiting on CTRL-c\n") sys.exit(0) if __name__ == "__main__": main() graphite-carbon-1.1.7/examples/example-pickle-client.py0000644000175000017500000000561713633714656021640 0ustar gringrin#!/usr/bin/python """Copyright 2013 Bryan Irvine Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import re import sys import time import socket import platform import subprocess import pickle import struct CARBON_SERVER = '127.0.0.1' CARBON_PICKLE_PORT = 2004 DELAY = 60 def get_loadavg(): """ Get the load average for a unix-like system. For more details, "man proc" and "man uptime" """ if platform.system() == "Linux": return open('/proc/loadavg').read().split()[:3] else: command = "uptime" process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True) stdout = process.communicate()[0].strip() # Split on whitespace and commas output = re.split("[\s,]+", stdout) return output[-3:] def run(sock, delay): """Make the client go go go""" while True: now = int(time.time()) tuples = ([]) lines = [] #We're gonna report all three loadavg values loadavg = get_loadavg() tuples.append(('system.loadavg_1min', (now,loadavg[0]))) tuples.append(('system.loadavg_5min', (now,loadavg[1]))) tuples.append(('system.loadavg_15min', (now,loadavg[2]))) lines.append("system.loadavg_1min %s %d" % (loadavg[0], now)) lines.append("system.loadavg_5min %s %d" % (loadavg[1], now)) lines.append("system.loadavg_15min %s %d" % (loadavg[2], now)) message = '\n'.join(lines) + '\n' #all lines must end in a newline print("sending message") print('-' * 80) print(message) package = pickle.dumps(tuples, 1) size = struct.pack('!L', len(package)) sock.sendall(size) sock.sendall(package) time.sleep(delay) def main(): """Wrap it all up together""" delay = DELAY if len(sys.argv) > 1: arg = sys.argv[1] if arg.isdigit(): delay = int(arg) else: sys.stderr.write("Ignoring non-integer argument. Using default: %ss\n" % delay) sock = socket.socket() try: sock.connect( (CARBON_SERVER, CARBON_PICKLE_PORT) ) except socket.error: raise SystemExit("Couldn't connect to %(server)s on port %(port)d, is carbon-cache.py running?" % { 'server':CARBON_SERVER, 'port':CARBON_PICKLE_PORT }) try: run(sock, delay) except KeyboardInterrupt: sys.stderr.write("\nExiting on CTRL-c\n") sys.exit(0) if __name__ == "__main__": main() graphite-carbon-1.1.7/examples/example-protobuf-client.py0000644000175000017500000000512213633714656022220 0ustar gringrin#!/usr/bin/python # -*- coding: utf-8 -*- """ Copyright 2013 Bryan Irvine Copyright 2017 The Graphite Project Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" # Import the precompiled protobuffer. It can be recompiled with: # $ protoc --python_out=. carbon.proto from carbon.carbon_pb2 import Payload import os import sys import time import socket import struct CARBON_SERVER = '127.0.0.1' CARBON_PROTOBUF_PORT = 2005 DELAY = 60 def run(sock, delay): """Make the client go go go""" while True: # Epoch, timestamp in seconds since 1970 now = int(time.time()) # Initialize the protobuf payload payload_pb = Payload() labels = ['1min', '5min', '15min'] for name, value in zip(labels, os.getloadavg()): m = payload_pb.metrics.add() m.metric = 'system.loadavg_' + name p = m.points.add() p.timestamp = now p.value = value print("sending message") print(('-' * 80)) print(payload_pb) package = payload_pb.SerializeToString() # The message must be prepended with its size size = struct.pack('!L', len(package)) sock.sendall(size) # Then send the actual payload sock.sendall(package) time.sleep(delay) def main(): """Wrap it all up together""" delay = DELAY if len(sys.argv) > 1: arg = sys.argv[1] if arg.isdigit(): delay = int(arg) else: sys.stderr.write( "Ignoring non-integer argument. Using default: %ss\n" % delay) sock = socket.socket() try: sock.connect((CARBON_SERVER, CARBON_PROTOBUF_PORT)) except socket.error: raise SystemExit("Couldn't connect to %(server)s on port %(port)d, " "is carbon-cache.py running?" % {'server': CARBON_SERVER, 'port': CARBON_PROTOBUF_PORT}) try: run(sock, delay) except KeyboardInterrupt: sys.stderr.write("\nExiting on CTRL-c\n") sys.exit(0) if __name__ == "__main__": main() graphite-carbon-1.1.7/lib/0000755000175000017500000000000013633714656014031 5ustar gringringraphite-carbon-1.1.7/lib/carbon/0000755000175000017500000000000013633714656015275 5ustar gringringraphite-carbon-1.1.7/lib/carbon/__init__.py0000644000175000017500000000000013633714656017374 0ustar gringringraphite-carbon-1.1.7/lib/carbon/aggregator/0000755000175000017500000000000013633714656017417 5ustar gringringraphite-carbon-1.1.7/lib/carbon/aggregator/__init__.py0000644000175000017500000000000013633714656021516 0ustar gringringraphite-carbon-1.1.7/lib/carbon/aggregator/buffers.py0000644000175000017500000000625313633714656021433 0ustar gringrinimport time from twisted.internet.task import LoopingCall from carbon.conf import settings from carbon import log class _BufferManager: def __init__(self): self.buffers = {} def __len__(self): return len(self.buffers) def get_buffer(self, metric_path): if metric_path not in self.buffers: log.debug("Allocating new metric buffer for %s" % metric_path) self.buffers[metric_path] = MetricBuffer(metric_path) return self.buffers[metric_path] def clear(self): for buffer in self.buffers.values(): buffer.close() self.buffers.clear() class MetricBuffer: __slots__ = ('metric_path', 'interval_buffers', 'compute_task', 'configured', 'aggregation_frequency', 'aggregation_func') def __init__(self, metric_path): self.metric_path = metric_path self.interval_buffers = {} self.compute_task = None self.configured = False self.aggregation_frequency = None self.aggregation_func = None def input(self, datapoint): (timestamp, value) = datapoint interval = timestamp - (timestamp % self.aggregation_frequency) if interval in self.interval_buffers: buffer = self.interval_buffers[interval] else: buffer = self.interval_buffers[interval] = IntervalBuffer(interval) buffer.input(datapoint) def configure_aggregation(self, frequency, func): self.aggregation_frequency = int(frequency) self.aggregation_func = func self.compute_task = LoopingCall(self.compute_value) if settings['WRITE_BACK_FREQUENCY'] is not None: compute_frequency = min(settings['WRITE_BACK_FREQUENCY'], frequency) else: compute_frequency = frequency self.compute_task.start(compute_frequency, now=False) self.configured = True def compute_value(self): now = int(time.time()) current_interval = now - (now % self.aggregation_frequency) age_threshold = current_interval - ( settings['MAX_AGGREGATION_INTERVALS'] * self.aggregation_frequency) for buffer in list(self.interval_buffers.values()): if buffer.active: value = self.aggregation_func(buffer.values) datapoint = (buffer.interval, value) state.events.metricGenerated(self.metric_path, datapoint) state.instrumentation.increment('aggregateDatapointsSent') buffer.mark_inactive() if buffer.interval < age_threshold: del self.interval_buffers[buffer.interval] if not self.interval_buffers: self.close() self.configured = False del BufferManager.buffers[self.metric_path] def close(self): if self.compute_task and self.compute_task.running: self.compute_task.stop() @property def size(self): return sum([len(buf.values) for buf in self.interval_buffers.values()]) class IntervalBuffer: __slots__ = ('interval', 'values', 'active') def __init__(self, interval): self.interval = interval self.values = [] self.active = True def input(self, datapoint): self.values.append(datapoint[1]) self.active = True def mark_inactive(self): self.active = False # Shared importable singleton BufferManager = _BufferManager() # Avoid import circularity from carbon import state # NOQA graphite-carbon-1.1.7/lib/carbon/aggregator/processor.py0000644000175000017500000000216313633714656022012 0ustar gringrinfrom carbon.aggregator.rules import RuleManager from carbon.aggregator.buffers import BufferManager from carbon.instrumentation import increment from carbon.pipeline import Processor from carbon.conf import settings from carbon import log class AggregationProcessor(Processor): plugin_name = 'aggregate' def process(self, metric, datapoint): increment('datapointsReceived') aggregate_metrics = set() for rule in RuleManager.rules: aggregate_metric = rule.get_aggregate_metric(metric) if aggregate_metric is None: continue else: aggregate_metrics.add(aggregate_metric) values_buffer = BufferManager.get_buffer(aggregate_metric) if not values_buffer.configured: values_buffer.configure_aggregation(rule.frequency, rule.aggregation_func) values_buffer.input(datapoint) if settings.FORWARD_ALL and metric not in aggregate_metrics: if settings.LOG_AGGREGATOR_MISSES and len(aggregate_metrics) == 0: log.msg( "Couldn't match metric %s with any aggregation rule. Passing on un-aggregated." % metric) yield (metric, datapoint) graphite-carbon-1.1.7/lib/carbon/aggregator/rules.py0000644000175000017500000001213113633714656021121 0ustar gringrinimport re from math import floor, ceil from os.path import exists, getmtime from twisted.internet.task import LoopingCall from cachetools import TTLCache, LRUCache from carbon import log from carbon.conf import settings from carbon.aggregator.buffers import BufferManager def get_cache(): ttl = settings.CACHE_METRIC_NAMES_TTL size = settings.CACHE_METRIC_NAMES_MAX if ttl > 0 and size > 0: return TTLCache(size, ttl) elif size > 0: return LRUCache(size) else: return dict() class RuleManager(object): def __init__(self): self.rules = [] self.rules_file = None self.read_task = LoopingCall(self.read_rules) self.rules_last_read = 0.0 def clear(self): self.rules = [] def read_from(self, rules_file): self.rules_file = rules_file self.read_rules() self.read_task.start(10, now=False) def read_rules(self): if not exists(self.rules_file): self.clear() return # Only read if the rules file has been modified try: mtime = getmtime(self.rules_file) except OSError: log.err("Failed to get mtime of %s" % self.rules_file) return if mtime <= self.rules_last_read: return # Read new rules log.aggregator("reading new aggregation rules from %s" % self.rules_file) new_rules = [] for line in open(self.rules_file): line = line.strip() if line.startswith('#') or not line: continue rule = self.parse_definition(line) new_rules.append(rule) log.aggregator("clearing aggregation buffers") BufferManager.clear() self.rules = new_rules self.rules_last_read = mtime def parse_definition(self, line): try: left_side, right_side = line.split('=', 1) output_pattern, frequency = left_side.split() method, input_pattern = right_side.split() frequency = int(frequency.lstrip('(').rstrip(')')) return AggregationRule(input_pattern, output_pattern, method, frequency) except ValueError: log.err("Failed to parse rule in %s, line: %s" % (self.rules_file, line)) raise class AggregationRule(object): def __init__(self, input_pattern, output_pattern, method, frequency): self.input_pattern = input_pattern self.output_pattern = output_pattern self.method = method self.frequency = int(frequency) if method not in AGGREGATION_METHODS: raise ValueError("Invalid aggregation method '%s'" % method) self.aggregation_func = AGGREGATION_METHODS[method] self.build_regex() self.build_template() self.cache = get_cache() def get_aggregate_metric(self, metric_path): if metric_path in self.cache: try: return self.cache[metric_path] except KeyError: # The value can expire at any time, so we need to catch this. pass match = self.regex.match(metric_path) result = None if match: extracted_fields = match.groupdict() try: result = self.output_template % extracted_fields except TypeError: log.err("Failed to interpolate template %s with fields %s" % ( self.output_template, extracted_fields)) self.cache[metric_path] = result return result def build_regex(self): input_pattern_parts = self.input_pattern.split('.') regex_pattern_parts = [] for input_part in input_pattern_parts: if '<<' in input_part and '>>' in input_part: i = input_part.find('<<') j = input_part.find('>>') pre = input_part[:i] post = input_part[j + 2:] field_name = input_part[i + 2:j] regex_part = '%s(?P<%s>.+?)%s' % (pre, field_name, post) else: i = input_part.find('<') j = input_part.find('>') if i > -1 and j > i: pre = input_part[:i] post = input_part[j + 1:] field_name = input_part[i + 1:j] regex_part = '%s(?P<%s>[^.]+?)%s' % (pre, field_name, post) elif input_part == '*': regex_part = '[^.]+' else: regex_part = input_part.replace('*', '[^.]*') regex_pattern_parts.append(regex_part) regex_pattern = '\\.'.join(regex_pattern_parts) + '$' self.regex = re.compile(regex_pattern) def build_template(self): self.output_template = self.output_pattern.replace('<', '%(').replace('>', ')s') def avg(values): if values: return float(sum(values)) / len(values) def count(values): if values: return len(values) def percentile(factor): def func(values): if values: values = sorted(values) rank = factor * (len(values) - 1) rank_left = int(floor(rank)) rank_right = int(ceil(rank)) if rank_left == rank_right: return values[rank_left] else: return values[rank_left] * (rank_right - rank) + values[rank_right] * (rank - rank_left) return func AGGREGATION_METHODS = { 'sum': sum, 'avg': avg, 'min': min, 'max': max, 'p50': percentile(0.50), 'p75': percentile(0.75), 'p80': percentile(0.80), 'p90': percentile(0.90), 'p95': percentile(0.95), 'p99': percentile(0.99), 'p999': percentile(0.999), 'count': count, } # Importable singleton RuleManager = RuleManager() graphite-carbon-1.1.7/lib/carbon/amqp0-8.xml0000644000175000017500000007547713633714656017226 0ustar gringrin graphite-carbon-1.1.7/lib/carbon/amqp_listener.py0000644000175000017500000002114313633714656020513 0ustar gringrin#!/usr/bin/env python """ Copyright 2009 Lucio Torre Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This is an AMQP client that will connect to the specified broker and read messages, parse them, and post them as metrics. Each message's routing key should be a metric name. The message body should be one or more lines of the form: \n \n ... Where each is a real number and is a UNIX epoch time. This program can be started standalone for testing or using carbon-cache.py (see example config file provided) """ import sys import os import socket from optparse import OptionParser from twisted.internet.defer import inlineCallbacks from twisted.internet import reactor from twisted.internet.protocol import ReconnectingClientFactory from twisted.application.internet import TCPClient # txamqp is currently not ported to py3 try: from txamqp.protocol import AMQClient from txamqp.client import TwistedDelegate import txamqp.spec except ImportError: raise ImportError try: import carbon except ImportError: # this is being run directly, carbon is not installed LIB_DIR = os.path.dirname(os.path.dirname(__file__)) sys.path.insert(0, LIB_DIR) import carbon.protocols # NOQA satisfy import order requirements from carbon.protocols import CarbonServerProtocol from carbon.conf import settings from carbon import log, events HOSTNAME = socket.gethostname().split('.')[0] class AMQPProtocol(CarbonServerProtocol): plugin_name = "amqp" @classmethod def build(cls, root_service): if not settings.ENABLE_AMQP: return amqp_host = settings.AMQP_HOST amqp_port = settings.AMQP_PORT amqp_user = settings.AMQP_USER amqp_password = settings.AMQP_PASSWORD amqp_verbose = settings.AMQP_VERBOSE amqp_vhost = settings.AMQP_VHOST amqp_spec = settings.AMQP_SPEC amqp_exchange_name = settings.AMQP_EXCHANGE factory = createAMQPListener( amqp_user, amqp_password, vhost=amqp_vhost, spec=amqp_spec, exchange_name=amqp_exchange_name, verbose=amqp_verbose) service = TCPClient(amqp_host, amqp_port, factory) service.setServiceParent(root_service) class AMQPGraphiteProtocol(AMQClient): """This is the protocol instance that will receive and post metrics.""" consumer_tag = "graphite_consumer" @inlineCallbacks def connectionMade(self): yield AMQClient.connectionMade(self) log.listener("New AMQP connection made") yield self.setup() yield self.receive_loop() @inlineCallbacks def setup(self): exchange = self.factory.exchange_name yield self.authenticate(self.factory.username, self.factory.password) chan = yield self.channel(1) yield chan.channel_open() # declare the exchange and queue yield chan.exchange_declare(exchange=exchange, type="topic", durable=True, auto_delete=False) # we use a private queue to avoid conflicting with existing bindings reply = yield chan.queue_declare(exclusive=True) my_queue = reply.queue # bind each configured metric pattern for bind_pattern in settings.BIND_PATTERNS: log.listener("binding exchange '%s' to queue '%s' with pattern %s" % (exchange, my_queue, bind_pattern)) yield chan.queue_bind(exchange=exchange, queue=my_queue, routing_key=bind_pattern) yield chan.basic_consume(queue=my_queue, no_ack=True, consumer_tag=self.consumer_tag) @inlineCallbacks def receive_loop(self): queue = yield self.queue(self.consumer_tag) while True: msg = yield queue.get() self.processMessage(msg) def processMessage(self, message): """Parse a message and post it as a metric.""" if self.factory.verbose: log.listener("Message received: %s" % (message,)) metric = message.routing_key for line in message.content.body.split("\n"): line = line.strip() if not line: continue try: if settings.get("AMQP_METRIC_NAME_IN_BODY", False): metric, value, timestamp = line.split() else: value, timestamp = line.split() datapoint = (float(timestamp), float(value)) if datapoint[1] != datapoint[1]: # filter out NaN values continue except ValueError: log.listener("invalid message line: %s" % (line,)) continue events.metricReceived(metric, datapoint) if self.factory.verbose: log.listener("Metric posted: %s %s %s" % (metric, value, timestamp,)) class AMQPReconnectingFactory(ReconnectingClientFactory): """The reconnecting factory. Knows how to create the extended client and how to keep trying to connect in case of errors.""" protocol = AMQPGraphiteProtocol def __init__(self, username, password, delegate, vhost, spec, channel, exchange_name, verbose): self.username = username self.password = password self.delegate = delegate self.vhost = vhost self.spec = spec self.channel = channel self.exchange_name = exchange_name self.verbose = verbose def buildProtocol(self, addr): self.resetDelay() p = self.protocol(self.delegate, self.vhost, self.spec) p.factory = self return p def createAMQPListener(username, password, vhost, exchange_name, spec=None, channel=1, verbose=False): """ Create an C{AMQPReconnectingFactory} configured with the specified options. """ # use provided spec if not specified if not spec: spec = txamqp.spec.load(os.path.normpath( os.path.join(os.path.dirname(__file__), 'amqp0-8.xml'))) delegate = TwistedDelegate() factory = AMQPReconnectingFactory(username, password, delegate, vhost, spec, channel, exchange_name, verbose=verbose) return factory def startReceiver(host, port, username, password, vhost, exchange_name, spec=None, channel=1, verbose=False): """ Starts a twisted process that will read messages on the amqp broker and post them as metrics. """ factory = createAMQPListener(username, password, vhost, exchange_name, spec=spec, channel=channel, verbose=verbose) reactor.connectTCP(host, port, factory) def main(): parser = OptionParser() parser.add_option("-t", "--host", dest="host", help="host name", metavar="HOST", default="localhost") parser.add_option("-p", "--port", dest="port", type=int, help="port number", metavar="PORT", default=5672) parser.add_option("-u", "--user", dest="username", help="username", metavar="USERNAME", default="guest") parser.add_option("-w", "--password", dest="password", help="password", metavar="PASSWORD", default="guest") parser.add_option("-V", "--vhost", dest="vhost", help="vhost", metavar="VHOST", default="/") parser.add_option("-e", "--exchange", dest="exchange", help="exchange", metavar="EXCHANGE", default="graphite") parser.add_option("-v", "--verbose", dest="verbose", help="verbose", default=False, action="store_true") (options, args) = parser.parse_args() startReceiver(options.host, options.port, options.username, options.password, vhost=options.vhost, exchange_name=options.exchange, verbose=options.verbose) reactor.run() if __name__ == "__main__": main() graphite-carbon-1.1.7/lib/carbon/amqp_publisher.py0000644000175000017500000000753513633714656020674 0ustar gringrin#!/usr/bin/env python """ Copyright 2009 Lucio Torre Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Will publish metrics over AMQP """ import os import time from optparse import OptionParser from twisted.internet.defer import inlineCallbacks from twisted.internet import reactor from twisted.internet.protocol import ClientCreator from txamqp.protocol import AMQClient from txamqp.client import TwistedDelegate from txamqp.content import Content import txamqp.spec @inlineCallbacks def writeMetric(metric_path, value, timestamp, host, port, username, password, vhost, exchange, spec=None, channel_number=1, ssl=False): if not spec: spec = txamqp.spec.load(os.path.normpath( os.path.join(os.path.dirname(__file__), 'amqp0-8.xml'))) delegate = TwistedDelegate() connector = ClientCreator(reactor, AMQClient, delegate=delegate, vhost=vhost, spec=spec) if ssl: from twisted.internet.ssl import ClientContextFactory conn = yield connector.connectSSL(host, port, ClientContextFactory()) else: conn = yield connector.connectTCP(host, port) yield conn.authenticate(username, password) channel = yield conn.channel(channel_number) yield channel.channel_open() yield channel.exchange_declare(exchange=exchange, type="topic", durable=True, auto_delete=False) message = Content("%f %d" % (value, timestamp)) message["delivery mode"] = 2 channel.basic_publish(exchange=exchange, content=message, routing_key=metric_path) yield channel.channel_close() def main(): parser = OptionParser(usage="%prog [options] [timestamp]") parser.add_option("-t", "--host", dest="host", help="host name", metavar="HOST", default="localhost") parser.add_option("-p", "--port", dest="port", type=int, help="port number", metavar="PORT", default=5672) parser.add_option("-u", "--user", dest="username", help="username", metavar="USERNAME", default="guest") parser.add_option("-w", "--password", dest="password", help="password", metavar="PASSWORD", default="guest") parser.add_option("-v", "--vhost", dest="vhost", help="vhost", metavar="VHOST", default="/") parser.add_option("-s", "--ssl", dest="ssl", help="ssl", metavar="SSL", action="store_true", default=False) parser.add_option("-e", "--exchange", dest="exchange", help="exchange", metavar="EXCHANGE", default="graphite") (options, args) = parser.parse_args() try: metric_path = args[0] value = float(args[1]) if len(args) > 2: timestamp = int(args[2]) else: timestamp = time.time() except ValueError: parser.print_usage() raise SystemExit(1) d = writeMetric(metric_path, value, timestamp, options.host, options.port, options.username, options.password, vhost=options.vhost, exchange=options.exchange, ssl=options.ssl) d.addErrback(lambda f: f.printTraceback()) d.addBoth(lambda _: reactor.stop()) reactor.run() if __name__ == "__main__": main() graphite-carbon-1.1.7/lib/carbon/cache.py0000644000175000017500000001713013633714656016714 0ustar gringrin"""Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import time import threading from operator import itemgetter from random import choice from collections import defaultdict from carbon.conf import settings from carbon import events, log from carbon.pipeline import Processor from carbon.util import TaggedSeries def by_timestamp(t_v): # useful sort key function (timestamp, _) = t_v return timestamp class CacheFeedingProcessor(Processor): plugin_name = 'write' def __init__(self, *args, **kwargs): super(CacheFeedingProcessor, self).__init__(*args, **kwargs) self.cache = MetricCache() def process(self, metric, datapoint): # normalize metric name (reorder tags) try: metric = TaggedSeries.parse(metric).path except Exception as err: log.msg('Error parsing metric %s: %s' % (metric, err)) self.cache.store(metric, datapoint) return Processor.NO_OUTPUT class DrainStrategy(object): """Implements the strategy for writing metrics. The strategy chooses what order (if any) metrics will be popped from the backing cache""" def __init__(self, cache): self.cache = cache def choose_item(self): raise NotImplementedError() class NaiveStrategy(DrainStrategy): """Pop points in an unordered fashion.""" def __init__(self, cache): super(NaiveStrategy, self).__init__(cache) def _generate_queue(): while True: metric_names = list(self.cache.keys()) while metric_names: yield metric_names.pop() self.queue = _generate_queue() def choose_item(self): return next(self.queue) class MaxStrategy(DrainStrategy): """Always pop the metric with the greatest number of points stored. This method leads to less variance in pointsPerUpdate but may mean that infrequently or irregularly updated metrics may not be written until shutdown """ def choose_item(self): metric_name, _ = max(self.cache.items(), key=lambda x: len(itemgetter(1)(x))) return metric_name class RandomStrategy(DrainStrategy): """Pop points randomly""" def choose_item(self): return choice(list(self.cache.keys())) # nosec class SortedStrategy(DrainStrategy): """ The default strategy which prefers metrics with a greater number of cached points but guarantees every point gets written exactly once during a loop of the cache """ def __init__(self, cache): super(SortedStrategy, self).__init__(cache) def _generate_queue(): while True: t = time.time() metric_counts = sorted(self.cache.counts, key=lambda x: x[1]) size = len(metric_counts) if settings.LOG_CACHE_QUEUE_SORTS and size: log.msg("Sorted %d cache queues in %.6f seconds" % (size, time.time() - t)) while metric_counts: yield itemgetter(0)(metric_counts.pop()) if settings.LOG_CACHE_QUEUE_SORTS and size: log.msg("Queue consumed in %.6f seconds" % (time.time() - t)) self.queue = _generate_queue() def choose_item(self): return next(self.queue) class TimeSortedStrategy(DrainStrategy): """ This strategy prefers metrics wich are lagging behind guarantees every point gets written exactly once during a loop of the cache """ def __init__(self, cache): super(TimeSortedStrategy, self).__init__(cache) def _generate_queue(): while True: t = time.time() metric_lw = sorted(self.cache.watermarks, key=lambda x: x[1], reverse=True) if settings.MIN_TIMESTAMP_LAG: metric_lw = [x for x in metric_lw if t - x[1] > settings.MIN_TIMESTAMP_LAG] size = len(metric_lw) if settings.LOG_CACHE_QUEUE_SORTS and size: log.msg("Sorted %d cache queues in %.6f seconds" % (size, time.time() - t)) if not metric_lw: # If there is nothing to do give a chance to sleep to the reader. yield None while metric_lw: yield itemgetter(0)(metric_lw.pop()) if settings.LOG_CACHE_QUEUE_SORTS and size: log.msg("Queue consumed in %.6f seconds" % (time.time() - t)) self.queue = _generate_queue() def choose_item(self): return next(self.queue) class _MetricCache(defaultdict): """A Singleton dictionary of metric names and lists of their datapoints""" def __init__(self, strategy=None): self.lock = threading.Lock() self.size = 0 self.strategy = None if strategy: self.strategy = strategy(self) super(_MetricCache, self).__init__(dict) @property def counts(self): return [(metric, len(datapoints)) for (metric, datapoints) in self.items()] @property def watermarks(self): return [(metric, min(datapoints.keys()), max(datapoints.keys())) for (metric, datapoints) in self.items() if datapoints] @property def is_full(self): if settings.MAX_CACHE_SIZE == float('inf'): return False else: return self.size >= settings.MAX_CACHE_SIZE def _check_available_space(self): if state.cacheTooFull and self.size < settings.CACHE_SIZE_LOW_WATERMARK: log.msg("MetricCache below watermark: self.size=%d" % self.size) events.cacheSpaceAvailable() def drain_metric(self): """Returns a metric and it's datapoints in order determined by the `DrainStrategy`_""" if not self: return (None, []) if self.strategy: with self.lock: metric = self.strategy.choose_item() else: # Avoid .keys() as it dumps the whole list metric = next(iter(self)) if metric is None: return (None, []) return (metric, self.pop(metric)) def get_datapoints(self, metric): """Return a list of currently cached datapoints sorted by timestamp""" return sorted(self.get(metric, {}).items(), key=by_timestamp) def pop(self, metric): with self.lock: datapoint_index = defaultdict.pop(self, metric) self.size -= len(datapoint_index) self._check_available_space() return sorted(datapoint_index.items(), key=by_timestamp) def store(self, metric, datapoint): timestamp, value = datapoint with self.lock: if timestamp not in self[metric]: # Not a duplicate, hence process if cache is not full if self.is_full: log.msg("MetricCache is full: self.size=%d" % self.size) events.cacheFull() else: self.size += 1 self[metric][timestamp] = value else: # Updating a duplicate does not increase the cache size self[metric][timestamp] = value _Cache = None def MetricCache(): global _Cache if _Cache is not None: return _Cache # Initialize a singleton cache instance # TODO: use plugins. write_strategy = None if settings.CACHE_WRITE_STRATEGY == 'naive': write_strategy = NaiveStrategy if settings.CACHE_WRITE_STRATEGY == 'max': write_strategy = MaxStrategy if settings.CACHE_WRITE_STRATEGY == 'sorted': write_strategy = SortedStrategy if settings.CACHE_WRITE_STRATEGY == 'timesorted': write_strategy = TimeSortedStrategy if settings.CACHE_WRITE_STRATEGY == 'random': write_strategy = RandomStrategy _Cache = _MetricCache(write_strategy) return _Cache # Avoid import circularities from carbon import state # NOQA graphite-carbon-1.1.7/lib/carbon/carbon.proto0000644000175000017500000000040513633714656017625 0ustar gringrin// protoc --python_out=. carbon.proto syntax = "proto3"; package carbon; message Point { uint32 timestamp = 1; double value = 2; } message Metric { string metric = 1; repeated Point points = 2; } message Payload { repeated Metric metrics = 1; } graphite-carbon-1.1.7/lib/carbon/carbon_pb2.py0000644000175000017500000001126113633714656017657 0ustar gringrin# Generated by the protocol buffer compiler. DO NOT EDIT! # source: carbon.proto import sys from google.protobuf import descriptor as _descriptor from google.protobuf import message as _message from google.protobuf import reflection as _reflection from google.protobuf import symbol_database as _symbol_database # @@protoc_insertion_point(imports) _sym_db = _symbol_database.Default() _b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode('latin1')) DESCRIPTOR = _descriptor.FileDescriptor( name='carbon.proto', package='carbon', syntax='proto3', serialized_pb=_b('\n\x0c\x63\x61rbon.proto\x12\x06\x63\x61rbon\")\n\x05Point\x12\x11\n\ttimestamp' '\x18\x01 \x01(\r\x12\r\n\x05value\x18\x02 \x01(\x01\"7\n\x06Metric\x12\x0e\n' '\x06metric\x18\x01 \x01(\t\x12\x1d\n\x06points\x18\x02 \x03(\x0b\x32\r.carbon.' 'Point\"*\n\x07Payload\x12\x1f\n\x07metrics\x18\x01 \x03(\x0b\x32\x0e.carbon.' 'Metricb\x06proto3') ) _sym_db.RegisterFileDescriptor(DESCRIPTOR) _POINT = _descriptor.Descriptor( name='Point', full_name='carbon.Point', filename=None, file=DESCRIPTOR, containing_type=None, fields=[ _descriptor.FieldDescriptor( name='timestamp', full_name='carbon.Point.timestamp', index=0, number=1, type=13, cpp_type=3, label=1, has_default_value=False, default_value=0, message_type=None, enum_type=None, containing_type=None, is_extension=False, extension_scope=None, options=None), _descriptor.FieldDescriptor( name='value', full_name='carbon.Point.value', index=1, number=2, type=1, cpp_type=5, label=1, has_default_value=False, default_value=float(0), message_type=None, enum_type=None, containing_type=None, is_extension=False, extension_scope=None, options=None), ], extensions=[ ], nested_types=[], enum_types=[ ], options=None, is_extendable=False, syntax='proto3', extension_ranges=[], oneofs=[ ], serialized_start=24, serialized_end=65, ) _METRIC = _descriptor.Descriptor( name='Metric', full_name='carbon.Metric', filename=None, file=DESCRIPTOR, containing_type=None, fields=[ _descriptor.FieldDescriptor( name='metric', full_name='carbon.Metric.metric', index=0, number=1, type=9, cpp_type=9, label=1, has_default_value=False, default_value=_b("").decode('utf-8'), message_type=None, enum_type=None, containing_type=None, is_extension=False, extension_scope=None, options=None), _descriptor.FieldDescriptor( name='points', full_name='carbon.Metric.points', index=1, number=2, type=11, cpp_type=10, label=3, has_default_value=False, default_value=[], message_type=None, enum_type=None, containing_type=None, is_extension=False, extension_scope=None, options=None), ], extensions=[ ], nested_types=[], enum_types=[ ], options=None, is_extendable=False, syntax='proto3', extension_ranges=[], oneofs=[ ], serialized_start=67, serialized_end=122, ) _PAYLOAD = _descriptor.Descriptor( name='Payload', full_name='carbon.Payload', filename=None, file=DESCRIPTOR, containing_type=None, fields=[ _descriptor.FieldDescriptor( name='metrics', full_name='carbon.Payload.metrics', index=0, number=1, type=11, cpp_type=10, label=3, has_default_value=False, default_value=[], message_type=None, enum_type=None, containing_type=None, is_extension=False, extension_scope=None, options=None), ], extensions=[ ], nested_types=[], enum_types=[ ], options=None, is_extendable=False, syntax='proto3', extension_ranges=[], oneofs=[ ], serialized_start=124, serialized_end=166, ) _METRIC.fields_by_name['points'].message_type = _POINT _PAYLOAD.fields_by_name['metrics'].message_type = _METRIC DESCRIPTOR.message_types_by_name['Point'] = _POINT DESCRIPTOR.message_types_by_name['Metric'] = _METRIC DESCRIPTOR.message_types_by_name['Payload'] = _PAYLOAD Point = _reflection.GeneratedProtocolMessageType('Point', (_message.Message,), dict( DESCRIPTOR=_POINT, __module__='carbon_pb2' # @@protoc_insertion_point(class_scope:carbon.Point) )) _sym_db.RegisterMessage(Point) Metric = _reflection.GeneratedProtocolMessageType('Metric', (_message.Message,), dict( DESCRIPTOR=_METRIC, __module__='carbon_pb2' # @@protoc_insertion_point(class_scope:carbon.Metric) )) _sym_db.RegisterMessage(Metric) Payload = _reflection.GeneratedProtocolMessageType('Payload', (_message.Message,), dict( DESCRIPTOR=_PAYLOAD, __module__='carbon_pb2' # @@protoc_insertion_point(class_scope:carbon.Payload) )) _sym_db.RegisterMessage(Payload) # @@protoc_insertion_point(module_scope) graphite-carbon-1.1.7/lib/carbon/client.py0000644000175000017500000005764413633714656017145 0ustar gringrinfrom collections import deque, defaultdict from time import time from six import with_metaclass from twisted.application.service import Service from twisted.internet import reactor from twisted.internet.defer import Deferred, DeferredList from twisted.internet.protocol import ReconnectingClientFactory from twisted.protocols.basic import LineOnlyReceiver, Int32StringReceiver from carbon.conf import settings from carbon.util import pickle from carbon.util import PluginRegistrar, TaggedSeries from carbon.util import enableTcpKeepAlive from carbon import instrumentation, log, pipeline, state try: from OpenSSL import SSL except ImportError: SSL = None try: from twisted.internet import ssl except ImportError: ssl = None try: import signal except ImportError: log.debug("Couldn't import signal module") try: from carbon.resolver import setUpRandomResolver except ImportError: setUpRandomResolver = None SEND_QUEUE_LOW_WATERMARK = settings.MAX_QUEUE_SIZE * settings.QUEUE_LOW_WATERMARK_PCT class CarbonClientProtocol(object): def connectionMade(self): log.clients("%s::connectionMade" % self) self.paused = False self.connected = True self.transport.registerProducer(self, streaming=True) # Define internal metric names self.lastResetTime = time() self.destination = self.factory.destination self.destinationName = self.factory.destinationName self.queuedUntilReady = 'destinations.%s.queuedUntilReady' % self.destinationName self.sent = 'destinations.%s.sent' % self.destinationName self.batchesSent = 'destinations.%s.batchesSent' % self.destinationName self.slowConnectionReset = 'destinations.%s.slowConnectionReset' % self.destinationName enableTcpKeepAlive(self.transport, settings.TCP_KEEPALIVE, settings) d = self.factory.connectionMade # Setup a new deferred before calling the callback to allow callbacks # to re-register themselves. self.factory.connectionMade = Deferred() d.callback(self) self.sendQueued() def connectionLost(self, reason): log.clients("%s::connectionLost %s" % (self, reason.getErrorMessage())) self.connected = False def pauseProducing(self): self.paused = True def resumeProducing(self): self.paused = False self.sendQueued() def stopProducing(self): self.disconnect() def disconnect(self): if self.connected: self.transport.unregisterProducer() self.transport.loseConnection() self.connected = False def sendDatapoint(self, metric, datapoint): self.factory.enqueue(metric, datapoint) self.factory.scheduleSend() def _sendDatapointsNow(self, datapoints): """Implement this function to actually send datapoints.""" raise NotImplementedError() def sendDatapointsNow(self, datapoints): self._sendDatapointsNow(datapoints) instrumentation.increment(self.sent, len(datapoints)) instrumentation.increment(self.batchesSent) self.factory.checkQueue() def sendQueued(self): """This should be the only method that will be used to send stats. In order to not hold the event loop and prevent stats from flowing in while we send them out, this will process settings.MAX_DATAPOINTS_PER_MESSAGE stats, send them, and if there are still items in the queue, this will invoke reactor.callLater to schedule another run of sendQueued after a reasonable enough time for the destination to process what it has just received. Given a queue size of one million stats, and using a chained_invocation_delay of 0.0001 seconds, you'd get 1,000 sendQueued() invocations/second max. With a settings.MAX_DATAPOINTS_PER_MESSAGE of 100, the rate of stats being sent could theoretically be as high as 100,000 stats/sec, or 6,000,000 stats/minute. This is probably too high for a typical receiver to handle. In practice this theoretical max shouldn't be reached because network delays should add an extra delay - probably on the order of 10ms per send, so the queue should drain with an order of minutes, which seems more realistic. """ queueSize = self.factory.queueSize if self.paused: instrumentation.max(self.queuedUntilReady, queueSize) return if not self.factory.hasQueuedDatapoints(): return if not self.connectionQualityMonitor(): self.resetConnectionForQualityReasons("Sent: {0}, Received: {1}".format( instrumentation.prior_stats.get(self.sent, 0), instrumentation.prior_stats.get('metricsReceived', 0))) self.sendDatapointsNow(self.factory.takeSomeFromQueue()) if (self.factory.queueFull.called and queueSize < SEND_QUEUE_LOW_WATERMARK): if not self.factory.queueHasSpace.called: self.factory.queueHasSpace.callback(queueSize) if self.factory.hasQueuedDatapoints(): self.factory.scheduleSend() def connectionQualityMonitor(self): """Checks to see if the connection for this factory appears to be delivering stats at a speed close to what we're receiving them at. This is open to other measures of connection quality. Returns a Bool True means that quality is good, OR True means that the total received is less than settings.MIN_RESET_STAT_FLOW False means that quality is bad """ if not settings.USE_RATIO_RESET: return True if settings.DESTINATION_POOL_REPLICAS: received = self.factory.attemptedRelays else: received = 'metricsReceived' destination_sent = float(instrumentation.prior_stats.get(self.sent, 0)) total_received = float(instrumentation.prior_stats.get(received, 0)) instrumentation.increment(self.slowConnectionReset, 0) if total_received < settings.MIN_RESET_STAT_FLOW: return True if (destination_sent / total_received) < settings.MIN_RESET_RATIO: return False else: return True def resetConnectionForQualityReasons(self, reason): """Only re-sets the connection if it's been settings.MIN_RESET_INTERVAL seconds since the last re-set. Reason should be a string containing the quality info that led to a re-set. """ if (time() - self.lastResetTime) < float(settings.MIN_RESET_INTERVAL): return else: self.factory.connectedProtocol.disconnect() self.lastResetTime = time() instrumentation.increment(self.slowConnectionReset) log.clients("%s:: resetConnectionForQualityReasons: %s" % (self, reason)) def __str__(self): return 'CarbonClientProtocol(%s:%d:%s)' % (self.factory.destination) __repr__ = __str__ class CAReplaceClientContextFactory: """A context factory for SSL clients needing a different CA chain.""" isClient = 1 # SSLv23_METHOD allows SSLv2, SSLv3, and TLSv1. We disable SSLv2 below, # though. method = SSL.SSLv23_METHOD if SSL else None _cafile = None def __init__(self, file=None): self._cafile = file def getContext(self): ctx = SSL.Context(self.method) ctx.set_options(SSL.OP_NO_SSLv2) if self._cafile is not None: ctx.use_certificate_chain_file(self._cafile) return ctx class CarbonClientFactory(with_metaclass(PluginRegistrar, ReconnectingClientFactory, object)): plugins = {} maxDelay = 5 def __init__(self, destination, router): self.destination = destination self.router = router self.destinationName = ('%s:%d:%s' % destination).replace('.', '_') self.host, self.port, self.carbon_instance = destination self.addr = (self.host, self.port) self.started = False # This factory maintains protocol state across reconnects self.queue = deque() # Change to make this the sole source of metrics to be sent. self.connectedProtocol = None self.queueEmpty = Deferred() self.queueFull = Deferred() self.queueFull.addCallbacks(self.queueFullCallback, log.err) self.queueHasSpace = Deferred() self.queueHasSpace.addCallbacks(self.queueSpaceCallback, log.err) # Args: {'connector': connector, 'reason': reason} self.connectFailed = Deferred() # Args: {'connector': connector, 'reason': reason} self.connectionLost = Deferred() # Args: protocol instance self.connectionMade = Deferred() self.connectionMade.addCallbacks(self.clientConnectionMade, log.err) self.deferSendPending = None # Define internal metric names self.attemptedRelays = 'destinations.%s.attemptedRelays' % self.destinationName self.fullQueueDrops = 'destinations.%s.fullQueueDrops' % self.destinationName self.queuedUntilConnected = 'destinations.%s.queuedUntilConnected' % self.destinationName self.relayMaxQueueLength = 'destinations.%s.relayMaxQueueLength' % self.destinationName def clientProtocol(self): raise NotImplementedError() def scheduleSend(self): if self.deferSendPending and self.deferSendPending.active(): return self.deferSendPending = reactor.callLater(settings.TIME_TO_DEFER_SENDING, self.sendQueued) def sendQueued(self): if self.connectedProtocol: self.connectedProtocol.sendQueued() def queueFullCallback(self, result): state.events.cacheFull() log.clients('%s send queue is full (%d datapoints)' % (self, result)) def queueSpaceCallback(self, result): if self.queueFull.called: log.clients('%s send queue has space available' % self.connectedProtocol) self.queueFull = Deferred() self.queueFull.addCallbacks(self.queueFullCallback, log.err) state.events.cacheSpaceAvailable() self.queueHasSpace = Deferred() self.queueHasSpace.addCallbacks(self.queueSpaceCallback, log.err) def buildProtocol(self, addr): self.connectedProtocol = self.clientProtocol() self.connectedProtocol.factory = self return self.connectedProtocol def startConnecting(self): # calling this startFactory yields recursion problems self.started = True if settings['DESTINATION_TRANSPORT'] == "ssl": if not SSL or not ssl: print("SSL destination transport request, but no Python OpenSSL available.") raise SystemExit(1) authority = None if settings['DESTINATION_SSL_CA']: try: with open(settings['DESTINATION_SSL_CA']) as f: authority = ssl.Certificate.loadPEM(f.read()) except IOError: print("Failed to read CA chain: %s" % settings['DESTINATION_SSL_CA']) raise SystemExit(1) # Twisted 14 introduced this function, it might not be around on older installs. if hasattr(ssl, "optionsForClientTLS"): from six import u client = ssl.optionsForClientTLS(u(self.host), authority) else: client = CAReplaceClientContextFactory(settings['DESTINATION_SSL_CA']) self.connector = reactor.connectSSL(self.host, self.port, self, client) else: self.connector = reactor.connectTCP(self.host, self.port, self) def stopConnecting(self): self.started = False self.stopTrying() if self.connectedProtocol and self.connectedProtocol.connected: return self.connectedProtocol.disconnect() @property def queueSize(self): return len(self.queue) def hasQueuedDatapoints(self): return bool(self.queue) def takeSomeFromQueue(self): """Use self.queue, which is a collections.deque, to pop up to settings.MAX_DATAPOINTS_PER_MESSAGE items from the left of the queue. """ def yield_max_datapoints(): for _ in range(settings.MAX_DATAPOINTS_PER_MESSAGE): try: yield self.queue.popleft() except IndexError: return return list(yield_max_datapoints()) def checkQueue(self): """Check if the queue is empty. If the queue isn't empty or doesn't exist yet, then this will invoke the callback chain on the self.queryEmpty Deferred chain with the argument 0, and will re-set the queueEmpty callback chain with a new Deferred object. """ if not self.queue: self.queueEmpty.callback(0) self.queueEmpty = Deferred() def enqueue(self, metric, datapoint): self.queue.append((metric, datapoint)) def enqueue_from_left(self, metric, datapoint): self.queue.appendleft((metric, datapoint)) def sendDatapoint(self, metric, datapoint): instrumentation.increment(self.attemptedRelays) instrumentation.max(self.relayMaxQueueLength, self.queueSize) if self.queueSize >= settings.MAX_QUEUE_SIZE: if not self.queueFull.called: self.queueFull.callback(self.queueSize) instrumentation.increment(self.fullQueueDrops) else: self.enqueue(metric, datapoint) if self.connectedProtocol: self.scheduleSend() else: instrumentation.increment(self.queuedUntilConnected) def sendHighPriorityDatapoint(self, metric, datapoint): """The high priority datapoint is one relating to the carbon daemon itself. It puts the datapoint on the left of the deque, ahead of other stats, so that when the carbon-relay, specifically, is overwhelmed its stats are more likely to make it through and expose the issue at hand. In addition, these stats go on the deque even when the max stats capacity has been reached. This relies on not creating the deque with a fixed max size. """ instrumentation.increment(self.attemptedRelays) self.enqueue_from_left(metric, datapoint) if self.connectedProtocol: self.scheduleSend() else: instrumentation.increment(self.queuedUntilConnected) def startedConnecting(self, connector): log.clients("%s::startedConnecting (%s:%d)" % ( self, connector.host, connector.port)) def clientConnectionMade(self, client): log.clients("%s::connectionMade (%s)" % (self, client)) self.resetDelay() self.destinationUp(client.destination) self.connectionMade.addCallbacks(self.clientConnectionMade, log.err) return client def clientConnectionLost(self, connector, reason): ReconnectingClientFactory.clientConnectionLost(self, connector, reason) log.clients("%s::clientConnectionLost (%s:%d) %s" % ( self, connector.host, connector.port, reason.getErrorMessage())) self.connectedProtocol = None self.destinationDown(self.destination) args = dict(connector=connector, reason=reason) d = self.connectionLost self.connectionLost = Deferred() d.callback(args) def clientConnectionFailed(self, connector, reason): ReconnectingClientFactory.clientConnectionFailed(self, connector, reason) log.clients("%s::clientConnectionFailed (%s:%d) %s" % ( self, connector.host, connector.port, reason.getErrorMessage())) self.destinationDown(self.destination) args = dict(connector=connector, reason=reason) d = self.connectFailed self.connectFailed = Deferred() d.callback(args) def destinationUp(self, destination): log.clients("Destination is up: %s:%d:%s" % destination) if not self.router.hasDestination(destination): log.clients("Adding client %s:%d:%s to router" % destination) self.router.addDestination(destination) state.events.resumeReceivingMetrics() def destinationDown(self, destination): # Only blacklist the destination if we tried a lot. log.clients("Destination is down: %s:%d:%s (%d/%d)" % ( destination[0], destination[1], destination[2], self.retries, settings.DYNAMIC_ROUTER_MAX_RETRIES)) # Retries comes from the ReconnectingClientFactory. if self.retries < settings.DYNAMIC_ROUTER_MAX_RETRIES: return if settings.DYNAMIC_ROUTER and self.router.hasDestination(destination): log.clients("Removing client %s:%d:%s to router" % destination) self.router.removeDestination(destination) # Do not receive more metrics if we don't have any usable destinations. if not self.router.countDestinations(): state.events.pauseReceivingMetrics() # Re-inject queued metrics. metrics = list(self.queue) log.clients("Re-injecting %d metrics from %s" % (len(metrics), self)) for metric, datapoint in metrics: state.events.metricGenerated(metric, datapoint) self.queue.clear() def disconnect(self): self.queueEmpty.addCallbacks(lambda result: self.stopConnecting(), log.err) readyToStop = DeferredList( [self.connectionLost, self.connectFailed], fireOnOneCallback=True, fireOnOneErrback=True) self.checkQueue() # This can happen if the client is stopped before a connection is ever made if (not readyToStop.called) and (not self.started): readyToStop.callback(None) return readyToStop def __str__(self): return 'CarbonClientFactory(%s:%d:%s)' % self.destination __repr__ = __str__ # Basic clients and associated factories. class CarbonPickleClientProtocol(CarbonClientProtocol, Int32StringReceiver): def _sendDatapointsNow(self, datapoints): self.sendString(pickle.dumps(datapoints, protocol=2)) class CarbonPickleClientFactory(CarbonClientFactory): plugin_name = "pickle" def clientProtocol(self): return CarbonPickleClientProtocol() class CarbonLineClientProtocol(CarbonClientProtocol, LineOnlyReceiver): def _sendDatapointsNow(self, datapoints): for metric, datapoint in datapoints: if isinstance(datapoint[1], float): value = ("%.10f" % datapoint[1]).rstrip('0').rstrip('.') else: value = "%d" % datapoint[1] to_send = "%s %s %d" % (metric, value, datapoint[0]) self.sendLine(to_send.encode('utf-8')) class CarbonLineClientFactory(CarbonClientFactory): plugin_name = "line" def clientProtocol(self): return CarbonLineClientProtocol() class FakeClientFactory(object): """Fake client factory that buffers points This is used when all the destinations are down and before we pause the reception of metrics to avoid loosing points. """ def __init__(self): # This queue isn't explicitely bounded but will implicitely be. It receives # only metrics when no destinations are available, and as soon as we detect # that we don't have any destination we pause the producer: this mean that # it will contain only a few seconds of metrics. self.queue = deque() self.started = False def startConnecting(self): pass def sendDatapoint(self, metric, datapoint): self.queue.append((metric, datapoint)) def sendHighPriorityDatapoint(self, metric, datapoint): self.queue.append((metric, datapoint)) def reinjectDatapoints(self): metrics = list(self.queue) log.clients("Re-injecting %d metrics from %s" % (len(metrics), self)) for metric, datapoint in metrics: state.events.metricGenerated(metric, datapoint) self.queue.clear() class CarbonClientManager(Service): def __init__(self, router): if settings.DESTINATION_POOL_REPLICAS: # If we decide to open multiple TCP connection to a replica, we probably # want to try to also load-balance accross hosts. In this case we need # to make sure rfc3484 doesn't get in the way. if setUpRandomResolver: setUpRandomResolver(reactor) else: print("Import error, Twisted >= 17.1.0 needed for using DESTINATION_POOL_REPLICAS.") raise SystemExit(1) self.router = router self.client_factories = {} # { destination : CarbonClientFactory() } # { destination[0:2]: set(CarbonClientFactory()) } self.pooled_factories = defaultdict(set) # This fake factory will be used as a buffer when we did not manage # to connect to any destination. fake_factory = FakeClientFactory() self.client_factories[None] = fake_factory state.events.resumeReceivingMetrics.addHandler(fake_factory.reinjectDatapoints) def createFactory(self, destination): factory_name = settings["DESTINATION_PROTOCOL"] factory_class = CarbonClientFactory.plugins.get(factory_name) if not factory_class: print("In carbon.conf, DESTINATION_PROTOCOL must be one of %s. " "Invalid value: '%s'" % (', '.join(CarbonClientFactory.plugins), factory_name)) raise SystemExit(1) return factory_class(destination, self.router) def startService(self): if 'signal' in globals().keys(): log.debug("Installing SIG_IGN for SIGHUP") signal.signal(signal.SIGHUP, signal.SIG_IGN) Service.startService(self) for factory in self.client_factories.values(): if not factory.started: factory.startConnecting() def stopService(self): Service.stopService(self) return self.stopAllClients() def startClient(self, destination): if destination in self.client_factories: return log.clients("connecting to carbon daemon at %s:%d:%s" % destination) if not settings.DYNAMIC_ROUTER: # If not using a dynamic router we add the destination before # it's known to be working. self.router.addDestination(destination) factory = self.createFactory(destination) self.client_factories[destination] = factory self.pooled_factories[destination[0:2]].add(factory) connectAttempted = DeferredList( [factory.connectionMade, factory.connectFailed], fireOnOneCallback=True, fireOnOneErrback=True) if self.running: factory.startConnecting() # this can trigger & replace connectFailed return connectAttempted def stopClient(self, destination): factory = self.client_factories.get(destination) if factory is None or destination is None: return None self.router.removeDestination(destination) stopCompleted = factory.disconnect() stopCompleted.addCallbacks( lambda result: self.disconnectClient(destination), log.err ) return stopCompleted def disconnectClient(self, destination): factory = self.client_factories.pop(destination) self.pooled_factories[destination[0:2]].remove(factory) c = factory.connector if c and c.state == 'connecting' and not factory.hasQueuedDatapoints(): c.stopConnecting() def stopAllClients(self): deferreds = [] for destination in list(self.client_factories): deferred = self.stopClient(destination) if deferred: deferreds.append(deferred) return DeferredList(deferreds) def getDestinations(self, metric): destinations = list(self.router.getDestinations(metric)) # If we can't find any destination we just buffer the # points. We will also pause the socket on the receiving side. if not destinations: return [None] return destinations def getFactories(self, metric): destinations = self.getDestinations(metric) factories = set() if not settings.DESTINATION_POOL_REPLICAS: # Simple case, with only one replica per destination. for d in destinations: # If we can't find it, we add to the 'fake' factory / buffer. factories.add(self.client_factories.get(d)) else: # Here we might have multiple replicas per destination. for d in destinations: if d is None: # d == None means there are no destinations currently available, so # we just put the data into our fake factory / buffer. factories.add(self.client_factories[None]) else: # Else we take the replica with the smallest queue size. key = d[0:2] # Take only host:port, not instance. factories.add(min(self.pooled_factories[key], key=lambda f: f.queueSize)) return factories def sendDatapoint(self, metric, datapoint): for factory in self.getFactories(metric): factory.sendDatapoint(metric, datapoint) def sendHighPriorityDatapoint(self, metric, datapoint): for factory in self.getFactories(metric): factory.sendHighPriorityDatapoint(metric, datapoint) def __str__(self): return "<%s[%x]>" % (self.__class__.__name__, id(self)) class RelayProcessor(pipeline.Processor): plugin_name = 'relay' def process(self, metric, datapoint): if settings.TAG_RELAY_NORMALIZED: # normalize metric name try: metric = TaggedSeries.parse(metric).path except Exception as err: log.msg('Error parsing metric %s: %s' % (metric, err)) # continue anyway with processing the unnormalized metric for robustness state.client_manager.sendDatapoint(metric, datapoint) return pipeline.Processor.NO_OUTPUT graphite-carbon-1.1.7/lib/carbon/conf.py0000644000175000017500000005517113633714656016605 0ustar gringrin"""Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import os import sys import pwd import errno from os.path import join, dirname, normpath, exists, isdir from optparse import OptionParser try: from ConfigParser import ConfigParser # ConfigParser is renamed to configparser in py3 except ImportError: from configparser import ConfigParser from carbon import log, state from carbon.database import TimeSeriesDatabase from carbon.routers import DatapointRouter from carbon.exceptions import CarbonConfigException from twisted.python import usage defaults = dict( USER="", MAX_CACHE_SIZE=float('inf'), MAX_UPDATES_PER_SECOND=500, MAX_CREATES_PER_MINUTE=float('inf'), MIN_TIMESTAMP_RESOLUTION=0, MIN_TIMESTAMP_LAG=0, LINE_RECEIVER_INTERFACE='0.0.0.0', LINE_RECEIVER_PORT=2003, ENABLE_UDP_LISTENER=False, UDP_RECEIVER_INTERFACE='0.0.0.0', UDP_RECEIVER_PORT=2003, PICKLE_RECEIVER_INTERFACE='0.0.0.0', PICKLE_RECEIVER_PORT=2004, MAX_RECEIVER_CONNECTIONS=float('inf'), CACHE_QUERY_INTERFACE='0.0.0.0', CACHE_QUERY_PORT=7002, LOG_UPDATES=True, LOG_CREATES=True, LOG_CACHE_HITS=True, LOG_CACHE_QUEUE_SORTS=True, DATABASE='whisper', WHISPER_AUTOFLUSH=False, WHISPER_SPARSE_CREATE=False, WHISPER_FALLOCATE_CREATE=False, WHISPER_LOCK_WRITES=False, WHISPER_FADVISE_RANDOM=False, CERES_MAX_SLICE_GAP=80, CERES_NODE_CACHING_BEHAVIOR='all', CERES_SLICE_CACHING_BEHAVIOR='latest', CERES_LOCK_WRITES=False, MAX_DATAPOINTS_PER_MESSAGE=500, MAX_AGGREGATION_INTERVALS=5, FORWARD_ALL=True, MAX_QUEUE_SIZE=1000, QUEUE_LOW_WATERMARK_PCT=0.8, TIME_TO_DEFER_SENDING=0.0001, ENABLE_AMQP=False, AMQP_METRIC_NAME_IN_BODY=False, AMQP_VERBOSE=False, AMQP_SPEC=None, BIND_PATTERNS=['#'], GRAPHITE_URL='http://127.0.0.1:80', ENABLE_TAGS=True, TAG_UPDATE_INTERVAL=100, TAG_BATCH_SIZE=100, TAG_QUEUE_SIZE=10000, TAG_HASH_FILENAMES=True, TAG_RELAY_NORMALIZED=False, ENABLE_MANHOLE=False, MANHOLE_INTERFACE='127.0.0.1', MANHOLE_PORT=7222, MANHOLE_USER="", MANHOLE_PUBLIC_KEY="", MANHOLE_HOST_KEY_DIR="", RELAY_METHOD='rules', DYNAMIC_ROUTER=False, DYNAMIC_ROUTER_MAX_RETRIES=5, ROUTER_HASH_TYPE=None, REPLICATION_FACTOR=1, DIVERSE_REPLICAS=True, DESTINATIONS=[], DESTINATION_PROTOCOL="pickle", DESTINATION_TRANSPORT="none", DESTINATION_SSL_CA=None, DESTINATION_POOL_REPLICAS=False, USE_FLOW_CONTROL=True, USE_INSECURE_UNPICKLER=False, USE_WHITELIST=False, CARBON_METRIC_PREFIX='carbon', CARBON_METRIC_INTERVAL=60, CACHE_WRITE_STRATEGY='sorted', WRITE_BACK_FREQUENCY=None, MIN_RESET_STAT_FLOW=1000, MIN_RESET_RATIO=0.9, MIN_RESET_INTERVAL=121, TCP_KEEPALIVE=True, TCP_KEEPIDLE=10, TCP_KEEPINTVL=30, TCP_KEEPCNT=2, USE_RATIO_RESET=False, LOG_LISTENER_CONN_SUCCESS=True, LOG_AGGREGATOR_MISSES=True, AGGREGATION_RULES='aggregation-rules.conf', REWRITE_RULES='rewrite-rules.conf', RELAY_RULES='relay-rules.conf', ENABLE_LOGROTATION=True, METRIC_CLIENT_IDLE_TIMEOUT=None, CACHE_METRIC_NAMES_MAX=0, CACHE_METRIC_NAMES_TTL=0, RAVEN_DSN=None, PICKLE_RECEIVER_MAX_LENGTH=2**20, ) def _process_alive(pid): if exists("/proc"): return exists("/proc/%d" % pid) else: try: os.kill(int(pid), 0) return True except OSError as err: return err.errno == errno.EPERM class OrderedConfigParser(ConfigParser): """Hacky workaround to ensure sections are always returned in the order they are defined in. Note that this does *not* make any guarantees about the order of options within a section or the order in which sections get written back to disk on write().""" _ordered_sections = [] def read(self, path): # Verifies a file exists *and* is readable if not os.access(path, os.R_OK): raise CarbonConfigException("Error: Missing config file or wrong perms on %s" % path) result = ConfigParser.read(self, path) sections = [] with open(path) as f: for line in f: line = line.strip() if line.startswith('[') and line.endswith(']'): sections.append(line[1:-1]) self._ordered_sections = sections return result def sections(self): return list(self._ordered_sections) # return a copy for safety class Settings(dict): __getattr__ = dict.__getitem__ def __init__(self): dict.__init__(self) self.update(defaults) def readFrom(self, path, section): parser = ConfigParser() if not parser.read(path): raise CarbonConfigException("Failed to read config file %s" % path) if not parser.has_section(section): return for key, value in parser.items(section): key = key.upper() # Detect type from defaults dict if key in defaults: valueType = type(defaults[key]) else: valueType = str if valueType is list: value = [v.strip() for v in value.split(',')] elif valueType is bool: value = parser.getboolean(section, key) else: # Attempt to figure out numeric types automatically try: value = int(value) except ValueError: try: value = float(value) except ValueError: pass self[key] = value settings = Settings() settings.update(defaults) class CarbonCacheOptions(usage.Options): optFlags = [ ["debug", "", "Run in debug mode."], ] optParameters = [ ["config", "c", None, "Use the given config file."], ["instance", "", "a", "Manage a specific carbon instance."], ["logdir", "", None, "Write logs to the given directory."], ["whitelist", "", None, "List of metric patterns to allow."], ["blacklist", "", None, "List of metric patterns to disallow."], ] def postOptions(self): global settings program = self.parent.subCommand # Use provided pidfile (if any) as default for configuration. If it's # set to 'twistd.pid', that means no value was provided and the default # was used. pidfile = self.parent["pidfile"] if pidfile.endswith("twistd.pid"): pidfile = None self["pidfile"] = pidfile # Enforce a default umask of '022' if none was set. if "umask" not in self.parent or self.parent["umask"] is None: self.parent["umask"] = 0o022 # Read extra settings from the configuration file. program_settings = read_config(program, self) settings.update(program_settings) settings["program"] = program # Normalize and expand paths def cleanpath(path): return os.path.normpath(os.path.expanduser(path)) settings["STORAGE_DIR"] = cleanpath(settings["STORAGE_DIR"]) settings["LOCAL_DATA_DIR"] = cleanpath(settings["LOCAL_DATA_DIR"]) settings["WHITELISTS_DIR"] = cleanpath(settings["WHITELISTS_DIR"]) settings["PID_DIR"] = cleanpath(settings["PID_DIR"]) settings["LOG_DIR"] = cleanpath(settings["LOG_DIR"]) settings["pidfile"] = cleanpath(settings["pidfile"]) # Set process uid/gid by changing the parent config, if a user was # provided in the configuration file. if settings.USER: self.parent["uid"], self.parent["gid"] = ( pwd.getpwnam(settings.USER)[2:4]) # Set the pidfile in parent config to the value that was computed by # C{read_config}. self.parent["pidfile"] = settings["pidfile"] storage_schemas = join(settings["CONF_DIR"], "storage-schemas.conf") if not exists(storage_schemas): print("Error: missing required config %s" % storage_schemas) sys.exit(1) if settings.CACHE_WRITE_STRATEGY not in ('timesorted', 'sorted', 'max', 'naive'): log.err("%s is not a valid value for CACHE_WRITE_STRATEGY, defaulting to %s" % (settings.CACHE_WRITE_STRATEGY, defaults['CACHE_WRITE_STRATEGY'])) else: log.msg("Using %s write strategy for cache" % settings.CACHE_WRITE_STRATEGY) # Database-specific settings database = settings.DATABASE if database not in TimeSeriesDatabase.plugins: print("No database plugin implemented for '%s'" % database) raise SystemExit(1) database_class = TimeSeriesDatabase.plugins[database] state.database = database_class(settings) settings.CACHE_SIZE_LOW_WATERMARK = settings.MAX_CACHE_SIZE * 0.95 if "action" not in self: self["action"] = "start" self.handleAction() # If we are not running in debug mode or non-daemon mode, then log to a # directory, otherwise log output will go to stdout. If parent options # are set to log to syslog, then use that instead. if not self["debug"]: if self.parent.get("syslog", None): prefix = "%s-%s[%d]" % (program, self["instance"], os.getpid()) log.logToSyslog(prefix) elif not self.parent["nodaemon"]: logdir = settings.LOG_DIR if not isdir(logdir): os.makedirs(logdir) if settings.USER: # We have not yet switched to the specified user, # but that user must be able to create files in this # directory. os.chown(logdir, self.parent["uid"], self.parent["gid"]) log.logToDir(logdir) if self["whitelist"] is None: self["whitelist"] = join(settings["CONF_DIR"], "whitelist.conf") settings["whitelist"] = self["whitelist"] if self["blacklist"] is None: self["blacklist"] = join(settings["CONF_DIR"], "blacklist.conf") settings["blacklist"] = self["blacklist"] def parseArgs(self, *action): """If an action was provided, store it for further processing.""" if len(action) == 1: self["action"] = action[0] def handleAction(self): """Handle extra argument for backwards-compatibility. * C{start} will simply do minimal pid checking and otherwise let twistd take over. * C{stop} will kill an existing running process if it matches the C{pidfile} contents. * C{status} will simply report if the process is up or not. """ action = self["action"] pidfile = self.parent["pidfile"] program = settings["program"] instance = self["instance"] if action == "stop": if not exists(pidfile): print("Pidfile %s does not exist" % pidfile) raise SystemExit(0) pf = open(pidfile, 'r') try: pid = int(pf.read().strip()) pf.close() except ValueError: print("Failed to parse pid from pidfile %s" % pidfile) pf.close() try: print("removing corrupted pidfile %s" % pidfile) os.unlink(pidfile) except IOError: print("Could not remove pidfile %s" % pidfile) raise SystemExit(1) except IOError: print("Could not read pidfile %s" % pidfile) raise SystemExit(1) print("Sending kill signal to pid %d" % pid) try: os.kill(pid, 15) except OSError as e: if e.errno == errno.ESRCH: print("No process with pid %d running" % pid) else: raise raise SystemExit(0) elif action == "status": if not exists(pidfile): print("%s (instance %s) is not running" % (program, instance)) raise SystemExit(1) pf = open(pidfile, "r") try: pid = int(pf.read().strip()) pf.close() except ValueError: print("Failed to parse pid from pidfile %s" % pidfile) pf.close() try: print("removing corrupted pidfile %s" % pidfile) os.unlink(pidfile) except IOError: print("Could not remove pidfile %s" % pidfile) raise SystemExit(1) except IOError: print("Failed to read pid from %s" % pidfile) raise SystemExit(1) if _process_alive(pid): print("%s (instance %s) is running with pid %d" % (program, instance, pid)) raise SystemExit(0) else: print("%s (instance %s) is not running" % (program, instance)) raise SystemExit(1) elif action == "start": if exists(pidfile): pf = open(pidfile, 'r') try: pid = int(pf.read().strip()) pf.close() except ValueError: print("Failed to parse pid from pidfile %s" % pidfile) pf.close() try: print("removing corrupted pidfile %s" % pidfile) os.unlink(pidfile) except IOError: print("Could not remove pidfile %s" % pidfile) raise SystemExit(1) except IOError: print("Could not read pidfile %s" % pidfile) raise SystemExit(1) if _process_alive(pid): print("%s (instance %s) is already running with pid %d" % (program, instance, pid)) raise SystemExit(1) else: print("Removing stale pidfile %s" % pidfile) try: os.unlink(pidfile) except IOError: print("Could not remove pidfile %s" % pidfile) # Try to create the PID directory else: if not os.path.exists(settings["PID_DIR"]): try: os.makedirs(settings["PID_DIR"]) except OSError as exc: # Python >2.5 if exc.errno == errno.EEXIST and os.path.isdir(settings["PID_DIR"]): pass else: raise print("Starting %s (instance %s)" % (program, instance)) else: print("Invalid action '%s'" % action) print("Valid actions: start stop status") raise SystemExit(1) class CarbonAggregatorOptions(CarbonCacheOptions): optParameters = [ ["rules", "", None, "Use the given aggregation rules file."], ["rewrite-rules", "", None, "Use the given rewrite rules file."], ] + CarbonCacheOptions.optParameters def postOptions(self): CarbonCacheOptions.postOptions(self) if self["rules"] is None: self["rules"] = join(settings["CONF_DIR"], settings['AGGREGATION_RULES']) settings["aggregation-rules"] = self["rules"] if self["rewrite-rules"] is None: self["rewrite-rules"] = join(settings["CONF_DIR"], settings['REWRITE_RULES']) settings["rewrite-rules"] = self["rewrite-rules"] class CarbonRelayOptions(CarbonCacheOptions): optParameters = [ ["rules", "", None, "Use the given relay rules file."], ["aggregation-rules", "", None, "Use the given aggregation rules file."], ] + CarbonCacheOptions.optParameters def postOptions(self): CarbonCacheOptions.postOptions(self) if self["rules"] is None: self["rules"] = join(settings["CONF_DIR"], settings['RELAY_RULES']) settings["relay-rules"] = self["rules"] if self["aggregation-rules"] is None: self["aggregation-rules"] = join(settings["CONF_DIR"], settings['AGGREGATION_RULES']) settings["aggregation-rules"] = self["aggregation-rules"] router = settings["RELAY_METHOD"] if router not in DatapointRouter.plugins: print("In carbon.conf, RELAY_METHOD must be one of %s. " "Invalid value: '%s'" % (', '.join(DatapointRouter.plugins), router)) raise SystemExit(1) def get_default_parser(usage="%prog [options] "): """Create a parser for command line options.""" parser = OptionParser(usage=usage) parser.add_option( "--debug", action="store_true", help="Run in the foreground, log to stdout") parser.add_option( "--syslog", action="store_true", help="Write logs to syslog") parser.add_option( "--nodaemon", action="store_true", help="Run in the foreground") parser.add_option( "--profile", help="Record performance profile data to the given file") parser.add_option( "--profiler", help="Specify the profiler to use") parser.add_option( "--pidfile", default=None, help="Write pid to the given file") parser.add_option( "--umask", default=None, help="Use the given umask when creating files") parser.add_option( "--config", default=None, help="Use the given config file") parser.add_option( "--whitelist", default=None, help="Use the given whitelist file") parser.add_option( "--blacklist", default=None, help="Use the given blacklist file") parser.add_option( "--logdir", default=None, help="Write logs in the given directory") parser.add_option( "--instance", default='a', help="Manage a specific carbon instance") parser.add_option( "--logfile", default=None, help="Log to a specified file, - for stdout") parser.add_option( "--logger", default=None, help="A fully-qualified name to a log observer factory to use for the initial log " "observer. Takes precedence over --logfile and --syslog (when available).") return parser def get_parser(name): parser = get_default_parser() if "carbon-aggregator" in name: parser.add_option( "--rules", default=None, help="Use the given aggregation rules file.") parser.add_option( "--rewrite-rules", default=None, help="Use the given rewrite rules file.") elif name == "carbon-relay": parser.add_option( "--rules", default=None, help="Use the given relay rules file.") return parser def parse_options(parser, args): """ Parse command line options and print usage message if no arguments were provided for the command. """ (options, args) = parser.parse_args(args) if not args: parser.print_usage() raise SystemExit(1) if args[0] not in ("start", "stop", "status"): parser.print_usage() raise SystemExit(1) return options, args def read_config(program, options, **kwargs): """ Read settings for 'program' from configuration file specified by 'options["config"]', with missing values provided by 'defaults'. """ settings = Settings() settings.update(defaults) # Initialize default values if not set yet. for name, value in kwargs.items(): settings.setdefault(name, value) graphite_root = kwargs.get("ROOT_DIR") if graphite_root is None: graphite_root = os.environ.get('GRAPHITE_ROOT') if graphite_root is None: raise CarbonConfigException("Either ROOT_DIR or GRAPHITE_ROOT " "needs to be provided.") # Default config directory to root-relative, unless overriden by the # 'GRAPHITE_CONF_DIR' environment variable. settings.setdefault("CONF_DIR", os.environ.get("GRAPHITE_CONF_DIR", join(graphite_root, "conf"))) if options["config"] is None: options["config"] = join(settings["CONF_DIR"], "carbon.conf") else: # Set 'CONF_DIR' to the parent directory of the 'carbon.conf' config # file. settings["CONF_DIR"] = dirname(normpath(options["config"])) # Storage directory can be overriden by the 'GRAPHITE_STORAGE_DIR' # environment variable. It defaults to a path relative to GRAPHITE_ROOT # for backwards compatibility though. settings.setdefault("STORAGE_DIR", os.environ.get("GRAPHITE_STORAGE_DIR", join(graphite_root, "storage"))) def update_STORAGE_DIR_deps(): # By default, everything is written to subdirectories of the storage dir. settings.setdefault( "PID_DIR", settings["STORAGE_DIR"]) settings.setdefault( "LOG_DIR", join(settings["STORAGE_DIR"], "log", program)) settings.setdefault( "LOCAL_DATA_DIR", join(settings["STORAGE_DIR"], "whisper")) settings.setdefault( "WHITELISTS_DIR", join(settings["STORAGE_DIR"], "lists")) # Read configuration options from program-specific section. section = program[len("carbon-"):] config = options["config"] if not exists(config): raise CarbonConfigException("Error: missing required config %r" % config) settings.readFrom(config, section) settings.setdefault("instance", options["instance"]) update_STORAGE_DIR_deps() # If a specific instance of the program is specified, augment the settings # with the instance-specific settings and provide sane defaults for # optional settings. if options["instance"]: settings.readFrom(config, "%s:%s" % (section, options["instance"])) settings["pidfile"] = ( options["pidfile"] or join(settings["PID_DIR"], "%s-%s.pid" % (program, options["instance"]))) settings["LOG_DIR"] = ( options["logdir"] or join(settings["LOG_DIR"], "%s-%s" % (program, options["instance"]))) else: settings["pidfile"] = ( options["pidfile"] or join(settings["PID_DIR"], '%s.pid' % program)) settings["LOG_DIR"] = (options["logdir"] or settings["LOG_DIR"]) update_STORAGE_DIR_deps() return settings graphite-carbon-1.1.7/lib/carbon/database.py0000644000175000017500000002045013633714656017414 0ustar gringrin"""Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import os import time from os.path import exists, dirname, join, sep from carbon.util import PluginRegistrar, TaggedSeries from carbon import log from six import with_metaclass class TimeSeriesDatabase(with_metaclass(PluginRegistrar, object)): "Abstract base class for Carbon database backends." plugins = {} "List of supported aggregation methods for the database." aggregationMethods = [] def __init__(self, settings): self.graphite_url = settings.GRAPHITE_URL def write(self, metric, datapoints): "Persist datapoints in the database for metric." raise NotImplementedError() def exists(self, metric): "Return True if the given metric path exists, False otherwise." raise NotImplementedError() def create(self, metric, retentions, xfilesfactor, aggregation_method): "Create an entry in the database for metric using options." raise NotImplementedError() def getMetadata(self, metric, key): "Lookup metric metadata." raise NotImplementedError() def setMetadata(self, metric, key, value): "Modify metric metadata." raise NotImplementedError() def getFilesystemPath(self, metric): "Return filesystem path for metric, defaults to None." pass def validateArchiveList(self, archiveList): "Validate that the database can handle the given archiveList." pass def tag(self, *metrics): from carbon.http import httpRequest log.debug("Tagging %s" % ', '.join(metrics), type='tagdb') t = time.time() try: httpRequest( self.graphite_url + '/tags/tagMultiSeries', [('path', metric) for metric in metrics] ) log.debug("Tagged %s in %s" % (', '.join(metrics), time.time() - t), type='tagdb') except Exception as err: log.msg("Error tagging %s: %s" % (', '.join(metrics), err), type='tagdb') try: import whisper except ImportError: pass else: class WhisperDatabase(TimeSeriesDatabase): plugin_name = 'whisper' aggregationMethods = whisper.aggregationMethods def __init__(self, settings): super(WhisperDatabase, self).__init__(settings) self.data_dir = settings.LOCAL_DATA_DIR self.tag_hash_filenames = settings.TAG_HASH_FILENAMES self.sparse_create = settings.WHISPER_SPARSE_CREATE self.fallocate_create = settings.WHISPER_FALLOCATE_CREATE if settings.WHISPER_AUTOFLUSH: log.msg("Enabling Whisper autoflush") whisper.AUTOFLUSH = True if settings.WHISPER_FALLOCATE_CREATE: if whisper.CAN_FALLOCATE: log.msg("Enabling Whisper fallocate support") else: log.err("WHISPER_FALLOCATE_CREATE is enabled but linking failed.") if settings.WHISPER_LOCK_WRITES: if whisper.CAN_LOCK: log.msg("Enabling Whisper file locking") whisper.LOCK = True else: log.err("WHISPER_LOCK_WRITES is enabled but import of fcntl module failed.") if settings.WHISPER_FADVISE_RANDOM: try: if whisper.CAN_FADVISE: log.msg("Enabling Whisper fadvise_random support") whisper.FADVISE_RANDOM = True else: log.err("WHISPER_FADVISE_RANDOM is enabled but import of ftools module failed.") except AttributeError: log.err("WHISPER_FADVISE_RANDOM is enabled but skipped because it is not compatible " + "with the version of Whisper.") def write(self, metric, datapoints): path = self.getFilesystemPath(metric) whisper.update_many(path, datapoints) def exists(self, metric): if exists(self.getFilesystemPath(metric)): return True # if we're using hashed filenames and a non-hashed file exists then move it to the new name if self.tag_hash_filenames and exists(self._getFilesystemPath(metric, False)): os.rename(self._getFilesystemPath(metric, False), self.getFilesystemPath(metric)) return True return False def create(self, metric, retentions, xfilesfactor, aggregation_method): path = self.getFilesystemPath(metric) directory = dirname(path) try: if not exists(directory): os.makedirs(directory) except OSError as e: log.err("%s" % e) whisper.create(path, retentions, xfilesfactor, aggregation_method, self.sparse_create, self.fallocate_create) def getMetadata(self, metric, key): if key != 'aggregationMethod': raise ValueError("Unsupported metadata key \"%s\"" % key) wsp_path = self.getFilesystemPath(metric) return whisper.info(wsp_path)['aggregationMethod'] def setMetadata(self, metric, key, value): if key != 'aggregationMethod': raise ValueError("Unsupported metadata key \"%s\"" % key) wsp_path = self.getFilesystemPath(metric) return whisper.setAggregationMethod(wsp_path, value) def getFilesystemPath(self, metric): return self._getFilesystemPath(metric, self.tag_hash_filenames) def _getFilesystemPath(self, metric, tag_hash_filenames): return join( self.data_dir, TaggedSeries.encode(metric, sep, hash_only=tag_hash_filenames) + '.wsp' ) def validateArchiveList(self, archiveList): try: whisper.validateArchiveList(archiveList) except whisper.InvalidConfiguration as e: raise ValueError("%s" % e) try: import ceres except ImportError: pass else: class CeresDatabase(TimeSeriesDatabase): plugin_name = 'ceres' aggregationMethods = ['average', 'sum', 'last', 'max', 'min'] def __init__(self, settings): super(CeresDatabase, self).__init__(settings) self.data_dir = settings.LOCAL_DATA_DIR self.tag_hash_filenames = settings.TAG_HASH_FILENAMES ceres.setDefaultNodeCachingBehavior(settings.CERES_NODE_CACHING_BEHAVIOR) ceres.setDefaultSliceCachingBehavior(settings.CERES_SLICE_CACHING_BEHAVIOR) ceres.MAX_SLICE_GAP = int(settings.CERES_MAX_SLICE_GAP) if settings.CERES_LOCK_WRITES: if ceres.CAN_LOCK: log.msg("Enabling Ceres file locking") ceres.LOCK_WRITES = True else: log.err("CERES_LOCK_WRITES is enabled but import of fcntl module failed.") self.tree = ceres.CeresTree(self.data_dir) def encode(self, metric, tag_hash_filenames=None): if tag_hash_filenames is None: tag_hash_filenames = self.tag_hash_filenames return TaggedSeries.encode(metric, hash_only=tag_hash_filenames) def write(self, metric, datapoints): self.tree.store(self.encode(metric), datapoints) def exists(self, metric): if self.tree.hasNode(self.encode(metric)): return True # if we're using hashed filenames and a non-hashed file exists then move it to the new name if self.tag_hash_filenames and self.tree.hasNode(self.encode(metric, False)): os.rename(self._getFilesystemPath(metric, False), self.getFilesystemPath(metric)) return True return False def create(self, metric, retentions, xfilesfactor, aggregation_method): self.tree.createNode(self.encode(metric), retentions=retentions, timeStep=retentions[0][0], xFilesFactor=xfilesfactor, aggregationMethod=aggregation_method) def getMetadata(self, metric, key): return self.tree.getNode(self.encode(metric)).readMetadata()[key] def setMetadata(self, metric, key, value): node = self.tree.getNode(self.encode(metric)) metadata = node.readMetadata() metadata[key] = value node.writeMetadata(metadata) def getFilesystemPath(self, metric): return self._getFilesystemPath(metric, self.tag_hash_filenames) def _getFilesystemPath(self, metric, tag_hash_filenames): return self.tree.getFilesystemPath(self.encode(metric, tag_hash_filenames)) graphite-carbon-1.1.7/lib/carbon/events.py0000644000175000017500000000263313633714656017157 0ustar gringrinclass Event: def __init__(self, name): self.name = name self.handlers = [] def addHandler(self, handler): if handler not in self.handlers: self.handlers.append(handler) def removeHandler(self, handler): if handler in self.handlers: self.handlers.remove(handler) def __call__(self, *args, **kwargs): for handler in self.handlers: try: handler(*args, **kwargs) except Exception: log.err( None, "Exception in %s event handler: args=%s kwargs=%s" % (self.name, args, kwargs)) metricReceived = Event('metricReceived') metricGenerated = Event('metricGenerated') cacheFull = Event('cacheFull') cacheSpaceAvailable = Event('cacheSpaceAvailable') pauseReceivingMetrics = Event('pauseReceivingMetrics') resumeReceivingMetrics = Event('resumeReceivingMetrics') # Default handlers metricReceived.addHandler( lambda metric, datapoint: state.instrumentation.increment('metricsReceived')) cacheFull.addHandler(lambda: state.instrumentation.increment('cache.overflow')) cacheFull.addHandler(lambda: setattr(state, 'cacheTooFull', True)) cacheSpaceAvailable.addHandler(lambda: setattr(state, 'cacheTooFull', False)) pauseReceivingMetrics.addHandler(lambda: setattr(state, 'metricReceiversPaused', True)) resumeReceivingMetrics.addHandler(lambda: setattr(state, 'metricReceiversPaused', False)) # Avoid import circularities from carbon import log, state # NOQA graphite-carbon-1.1.7/lib/carbon/exceptions.py0000644000175000017500000000014713633714656020032 0ustar gringrinclass CarbonConfigException(Exception): """Raised when a carbon daemon is improperly configured""" graphite-carbon-1.1.7/lib/carbon/hashing.py0000644000175000017500000000651013633714656017272 0ustar gringrinfrom hashlib import md5 import bisect import sys try: import mmh3 except ImportError: mmh3 = None try: import pyhash hasher = pyhash.fnv1a_32() def fnv32a(data, seed=0x811c9dc5): return hasher(data, seed=seed) except ImportError: def fnv32a(data, seed=0x811c9dc5): """ FNV-1a Hash (http://isthe.com/chongo/tech/comp/fnv/) in Python. Taken from https://gist.github.com/vaiorabbit/5670985 """ hval = seed fnv_32_prime = 0x01000193 uint32_max = 2 ** 32 if sys.version_info >= (3, 0): # data is a bytes object, s is an integer for s in data: hval = hval ^ s hval = (hval * fnv_32_prime) % uint32_max else: # data is an str object, s is a single character for s in data: hval = hval ^ ord(s) hval = (hval * fnv_32_prime) % uint32_max return hval def compactHash(string): return md5(string.encode('utf-8')).hexdigest() def carbonHash(key, hash_type): if hash_type == 'fnv1a_ch': big_hash = int(fnv32a(key.encode('utf-8'))) small_hash = (big_hash >> 16) ^ (big_hash & 0xffff) elif hash_type == 'mmh3_ch': if mmh3 is None: raise Exception('Install "mmh3" to use this hashing function.') small_hash = mmh3.hash(key) else: big_hash = compactHash(key) small_hash = int(big_hash[:4], 16) return small_hash class ConsistentHashRing: def __init__(self, nodes, replica_count=100, hash_type='carbon_ch'): self.ring = [] self.ring_len = len(self.ring) self.nodes = set() self.nodes_len = len(self.nodes) self.replica_count = replica_count self.hash_type = hash_type for node in nodes: self.add_node(node) def compute_ring_position(self, key): return carbonHash(key, self.hash_type) def add_node(self, key): self.nodes.add(key) self.nodes_len = len(self.nodes) for i in range(self.replica_count): if self.hash_type == 'fnv1a_ch': replica_key = "%d-%s" % (i, key[1]) else: replica_key = "%s:%d" % (key, i) position = self.compute_ring_position(replica_key) while position in [r[0] for r in self.ring]: position = position + 1 entry = (position, key) bisect.insort(self.ring, entry) self.ring_len = len(self.ring) def remove_node(self, key): self.nodes.discard(key) self.nodes_len = len(self.nodes) self.ring = [entry for entry in self.ring if entry[1] != key] self.ring_len = len(self.ring) def get_node(self, key): assert self.ring position = self.compute_ring_position(key) search_entry = (position, ()) index = bisect.bisect_left(self.ring, search_entry) % self.ring_len entry = self.ring[index] return entry[1] def get_nodes(self, key): nodes = set() if not self.ring: return if self.nodes_len == 1: for node in self.nodes: yield node position = self.compute_ring_position(key) search_entry = (position, ()) index = bisect.bisect_left(self.ring, search_entry) % self.ring_len last_index = (index - 1) % self.ring_len nodes_len = len(nodes) while nodes_len < self.nodes_len and index != last_index: next_entry = self.ring[index] (position, next_node) = next_entry if next_node not in nodes: nodes.add(next_node) nodes_len += 1 yield next_node index = (index + 1) % self.ring_len graphite-carbon-1.1.7/lib/carbon/http.py0000644000175000017500000000076113633714656016632 0ustar gringrinimport urllib3 # async http client connection pool http = urllib3.PoolManager() def httpRequest(url, values=None, headers=None, method='POST', timeout=5): try: result = http.request( method, url, fields=values, headers=headers, timeout=timeout) except BaseException as err: raise Exception("Error requesting %s: %s" % (url, err)) if result.status != 200: raise Exception("Error response %d from %s" % (result.status, url)) return result.data graphite-carbon-1.1.7/lib/carbon/instrumentation.py0000644000175000017500000001521213633714656021113 0ustar gringrinimport os import time import socket from resource import getrusage, RUSAGE_SELF from twisted.application.service import Service from twisted.internet.task import LoopingCall from carbon.conf import settings stats = {} prior_stats = {} HOSTNAME = socket.gethostname().replace('.', '_') PAGESIZE = os.sysconf('SC_PAGESIZE') rusage = getrusage(RUSAGE_SELF) lastUsage = rusage.ru_utime + rusage.ru_stime lastUsageTime = time.time() # NOTE: Referencing settings in this *top level scope* will # give you *defaults* only. Probably not what you wanted. # TODO(chrismd) refactor the graphite metrics hierarchy to be cleaner, # more consistent, and make room for frontend metrics. # metric_prefix = "Graphite.backend.%(program)s.%(instance)s." % settings def increment(stat, increase=1): try: stats[stat] += increase except KeyError: stats[stat] = increase def max(stat, newval): try: if stats[stat] < newval: stats[stat] = newval except KeyError: stats[stat] = newval def append(stat, value): try: stats[stat].append(value) except KeyError: stats[stat] = [value] def getCpuUsage(): global lastUsage, lastUsageTime rusage = getrusage(RUSAGE_SELF) currentUsage = rusage.ru_utime + rusage.ru_stime currentTime = time.time() usageDiff = currentUsage - lastUsage timeDiff = currentTime - lastUsageTime if timeDiff == 0: # shouldn't be possible, but I've actually seen a ZeroDivisionError from this timeDiff = 0.000001 cpuUsagePercent = (usageDiff / timeDiff) * 100.0 lastUsage = currentUsage lastUsageTime = currentTime return cpuUsagePercent def getMemUsage(): with open('/proc/self/statm') as statm: rss_pages = int(statm.read().split()[1]) return rss_pages * PAGESIZE def recordMetrics(): global lastUsage global prior_stats myStats = stats.copy() myPriorStats = {} stats.clear() # cache metrics if 'cache' in settings.program: record = cache_record updateTimes = myStats.get('updateTimes', []) committedPoints = myStats.get('committedPoints', 0) creates = myStats.get('creates', 0) droppedCreates = myStats.get('droppedCreates', 0) errors = myStats.get('errors', 0) cacheQueries = myStats.get('cacheQueries', 0) cacheBulkQueries = myStats.get('cacheBulkQueries', 0) cacheOverflow = myStats.get('cache.overflow', 0) cacheBulkQuerySizes = myStats.get('cacheBulkQuerySize', []) # Calculate cache-data-structure-derived metrics prior to storing anything # in the cache itself -- which would otherwise affect said metrics. cache_size = cache.MetricCache().size cache_queues = len(cache.MetricCache()) record('cache.size', cache_size) record('cache.queues', cache_queues) if updateTimes: avgUpdateTime = sum(updateTimes) / len(updateTimes) record('avgUpdateTime', avgUpdateTime) if committedPoints: pointsPerUpdate = float(committedPoints) / len(updateTimes) record('pointsPerUpdate', pointsPerUpdate) if cacheBulkQuerySizes: avgBulkSize = sum(cacheBulkQuerySizes) / len(cacheBulkQuerySizes) record('cache.bulk_queries_average_size', avgBulkSize) record('updateOperations', len(updateTimes)) record('committedPoints', committedPoints) record('creates', creates) record('droppedCreates', droppedCreates) record('errors', errors) record('cache.queries', cacheQueries) record('cache.bulk_queries', cacheBulkQueries) record('cache.overflow', cacheOverflow) # aggregator metrics elif 'aggregator' in settings.program: record = aggregator_record record('allocatedBuffers', len(BufferManager)) record('bufferedDatapoints', sum([b.size for b in BufferManager.buffers.values()])) record('aggregateDatapointsSent', myStats.get('aggregateDatapointsSent', 0)) # relay metrics else: record = relay_record # shared relay stats for relays & aggregators if settings.program in ['carbon-aggregator', 'carbon-relay']: prefix = 'destinations.' relay_stats = [(k, v) for (k, v) in myStats.items() if k.startswith(prefix)] for stat_name, stat_value in relay_stats: record(stat_name, stat_value) # Preserve the count of sent metrics so that the ratio of # received : sent can be checked per-relay to determine the # health of the destination. if stat_name.endswith('.sent') or stat_name.endswith('.attemptedRelays'): myPriorStats[stat_name] = stat_value # common metrics record('activeConnections', len(state.connectedMetricReceiverProtocols)) record('metricsReceived', myStats.get('metricsReceived', 0)) record('blacklistMatches', myStats.get('blacklistMatches', 0)) record('whitelistRejects', myStats.get('whitelistRejects', 0)) record('cpuUsage', getCpuUsage()) # And here preserve count of messages received in the prior periiod myPriorStats['metricsReceived'] = myStats.get('metricsReceived', 0) prior_stats.clear() prior_stats.update(myPriorStats) try: # This only works on Linux record('memUsage', getMemUsage()) except Exception: pass def cache_record(metric, value): prefix = settings.CARBON_METRIC_PREFIX if settings.instance is None: fullMetric = '%s.agents.%s.%s' % (prefix, HOSTNAME, metric) else: fullMetric = '%s.agents.%s-%s.%s' % (prefix, HOSTNAME, settings.instance, metric) datapoint = (time.time(), value) cache.MetricCache().store(fullMetric, datapoint) def relay_record(metric, value): prefix = settings.CARBON_METRIC_PREFIX if settings.instance is None: fullMetric = '%s.relays.%s.%s' % (prefix, HOSTNAME, metric) else: fullMetric = '%s.relays.%s-%s.%s' % (prefix, HOSTNAME, settings.instance, metric) datapoint = (time.time(), value) events.metricGenerated(fullMetric, datapoint) def aggregator_record(metric, value): prefix = settings.CARBON_METRIC_PREFIX if settings.instance is None: fullMetric = '%s.aggregator.%s.%s' % (prefix, HOSTNAME, metric) else: fullMetric = '%s.aggregator.%s-%s.%s' % (prefix, HOSTNAME, settings.instance, metric) datapoint = (time.time(), value) events.metricGenerated(fullMetric, datapoint) class InstrumentationService(Service): def __init__(self): self.record_task = LoopingCall(recordMetrics) def startService(self): if settings.CARBON_METRIC_INTERVAL > 0: self.record_task.start(settings.CARBON_METRIC_INTERVAL, False) Service.startService(self) def stopService(self): if settings.CARBON_METRIC_INTERVAL > 0: self.record_task.stop() Service.stopService(self) # Avoid import circularities from carbon import state, events, cache # NOQA from carbon.aggregator.buffers import BufferManager # NOQA graphite-carbon-1.1.7/lib/carbon/log.py0000644000175000017500000001206113633714656016430 0ustar gringrinimport os import time from sys import stdout from zope.interface import implementer from twisted.python.log import startLoggingWithObserver, textFromEventDict, msg, err, ILogObserver # NOQA from twisted.python.syslog import SyslogObserver from twisted.python.logfile import DailyLogFile class CarbonLogFile(DailyLogFile): """Overridden to support logrotate.d""" def __init__(self, *args, **kwargs): DailyLogFile.__init__(self, *args, **kwargs) # avoid circular dependencies from carbon.conf import settings self.enableRotation = settings.ENABLE_LOGROTATION def _openFile(self): """ Fix Umask Issue https://twistedmatrix.com/trac/ticket/7026 """ openMode = self.defaultMode or 0o777 self._file = os.fdopen(os.open( self.path, os.O_CREAT | os.O_RDWR, openMode), 'rb+', 1) self.closed = False # Try our best to update permissions for files which already exist. if self.defaultMode: try: os.chmod(self.path, self.defaultMode) except OSError: pass # Seek is needed for uniformity of stream positioning # for read and write between Linux and BSD systems due # to differences in fopen() between operating systems. self._file.seek(0, os.SEEK_END) self.lastDate = self.toDate(os.stat(self.path)[8]) def shouldRotate(self): if self.enableRotation: return DailyLogFile.shouldRotate(self) else: return False def write(self, data): if not self.enableRotation: if not os.path.exists(self.path): self.reopen() else: path_stat = os.stat(self.path) fd_stat = os.fstat(self._file.fileno()) if not (path_stat.st_ino == fd_stat.st_ino and path_stat.st_dev == fd_stat.st_dev): self.reopen() DailyLogFile.write(self, data) # Backport from twisted >= 10 def reopen(self): self.close() self._openFile() @implementer(ILogObserver) class CarbonLogObserver(object): def __init__(self): self._raven_client = None def raven_client(self): if self._raven_client is not None: return self._raven_client # Import here to avoid dependency hell. try: import raven except ImportError: return None from carbon.conf import settings if settings.RAVEN_DSN is None: return None self._raven_client = raven.Client(dsn=settings.RAVEN_DSN) return self._raven_client def log_to_raven(self, event): if not event.get('isError') or 'failure' not in event: return client = self.raven_client() if client is None: return f = event['failure'] client.captureException( (f.type, f.value, f.getTracebackObject()) ) def log_to_dir(self, logdir): self.logdir = logdir self.console_logfile = CarbonLogFile('console.log', logdir) self.custom_logs = {} self.observer = self.logdir_observer def log_to_syslog(self, prefix): observer = SyslogObserver(prefix).emit def syslog_observer(event): event["system"] = event.get("type", "console") observer(event) self.observer = syslog_observer def __call__(self, event): self.log_to_raven(event) return self.observer(event) @staticmethod def stdout_observer(event): stdout.write(formatEvent(event, includeType=True) + '\n') stdout.flush() def logdir_observer(self, event): message = formatEvent(event) log_type = event.get('type') if log_type is not None and log_type not in self.custom_logs: self.custom_logs[log_type] = CarbonLogFile(log_type + '.log', self.logdir) logfile = self.custom_logs.get(log_type, self.console_logfile) logfile.write(message + '\n') logfile.flush() # Default to stdout observer = stdout_observer carbonLogObserver = CarbonLogObserver() def formatEvent(event, includeType=False): event['isError'] = 'failure' in event message = textFromEventDict(event) if includeType: typeTag = '[%s] ' % event.get('type', 'console') else: typeTag = '' timestamp = time.strftime("%d/%m/%Y %H:%M:%S") return "%s :: %s%s" % (timestamp, typeTag, message) logToDir = carbonLogObserver.log_to_dir logToSyslog = carbonLogObserver.log_to_syslog def logToStdout(): startLoggingWithObserver(carbonLogObserver) def cache(message, **context): context['type'] = 'cache' msg(message, **context) def clients(message, **context): context['type'] = 'clients' msg(message, **context) def creates(message, **context): context['type'] = 'creates' msg(message, **context) def updates(message, **context): context['type'] = 'updates' msg(message, **context) def listener(message, **context): context['type'] = 'listener' msg(message, **context) def relay(message, **context): context['type'] = 'relay' msg(message, **context) def aggregator(message, **context): context['type'] = 'aggregator' msg(message, **context) def query(message, **context): context['type'] = 'query' msg(message, **context) def debug(message, **context): if debugEnabled: msg(message, **context) debugEnabled = False def setDebugEnabled(enabled): global debugEnabled debugEnabled = enabled graphite-carbon-1.1.7/lib/carbon/management.py0000644000175000017500000000075213633714656017767 0ustar gringrinimport traceback from carbon import log, state def getMetadata(metric, key): try: value = state.database.getMetadata(metric, key) return dict(value=value) except Exception: log.err() return dict(error=traceback.format_exc()) def setMetadata(metric, key, value): try: old_value = state.database.setMetadata(metric, key, value) return dict(old_value=old_value, new_value=value) except Exception: log.err() return dict(error=traceback.format_exc()) graphite-carbon-1.1.7/lib/carbon/manhole.py0000644000175000017500000000475313633714656017303 0ustar gringrinfrom twisted.cred import portal, checkers from twisted.conch.ssh import keys from twisted.conch.checkers import SSHPublicKeyDatabase from twisted.conch.manhole import Manhole from twisted.conch.manhole_ssh import TerminalRealm, ConchFactory from twisted.conch.openssh_compat.factory import OpenSSHFactory from twisted.internet import reactor from twisted.application.internet import TCPServer from carbon.protocols import CarbonServerProtocol from carbon.conf import settings import carbon from carbon.exceptions import CarbonConfigException namespace = {'carbon': carbon} class PublicKeyChecker(SSHPublicKeyDatabase): def __init__(self, userKeys): self.userKeys = {} for username, keyData in userKeys.items(): self.userKeys[username] = keys.Key.fromString(data=keyData).blob() def checkKey(self, credentials): if credentials.username in self.userKeys: keyBlob = self.userKeys[credentials.username] return keyBlob == credentials.blob def createManholeListener(): sshRealm = TerminalRealm() sshRealm.chainedProtocolFactory.protocolFactory = lambda _: Manhole(namespace) if settings.MANHOLE_PUBLIC_KEY == 'None': credChecker = checkers.InMemoryUsernamePasswordDatabaseDontUse() credChecker.addUser(settings.MANHOLE_USER.encode('utf-8'), ''.encode('utf-8')) else: userKeys = { settings.MANHOLE_USER.encode('utf-8'): settings.MANHOLE_PUBLIC_KEY.encode('utf-8'), } credChecker = PublicKeyChecker(userKeys) sshPortal = portal.Portal(sshRealm) sshPortal.registerChecker(credChecker) sessionFactory = ConchFactory(sshPortal) # set ssh host keys if settings.MANHOLE_HOST_KEY_DIR == "": raise CarbonConfigException("MANHOLE_HOST_KEY_DIR not defined") openSSHFactory = OpenSSHFactory() openSSHFactory.dataRoot = settings.MANHOLE_HOST_KEY_DIR sessionFactory.publicKeys = openSSHFactory.getPublicKeys() sessionFactory.privateKeys = openSSHFactory.getPrivateKeys() return sessionFactory def start(): sessionFactory = createManholeListener() reactor.listenTCP(settings.MANHOLE_PORT, sessionFactory, interface=settings.MANHOLE_INTERFACE) class ManholeProtocol(CarbonServerProtocol): plugin_name = "manhole" @classmethod def build(cls, root_service): if not settings.ENABLE_MANHOLE: return factory = createManholeListener() service = TCPServer( settings.MANHOLE_PORT, factory, interface=settings.MANHOLE_INTERFACE) service.setServiceParent(root_service) graphite-carbon-1.1.7/lib/carbon/pipeline.py0000644000175000017500000000170013633714656017452 0ustar gringrinfrom carbon.util import PluginRegistrar from carbon import state, log from six import with_metaclass class Processor(with_metaclass(PluginRegistrar, object)): plugins = {} NO_OUTPUT = () def pipeline_ready(self): "override me if you want" def process(self, metric, datapoint): raise NotImplementedError() def run_pipeline_generated(metric, datapoint): # For generated points, use a special pipeline to avoid points # infinitely being trapped. run_pipeline(metric, datapoint, state.pipeline_processors_generated) def run_pipeline(metric, datapoint, processors=None): if processors is None: processors = state.pipeline_processors elif not processors: return processor = processors[0] try: for out_metric, out_datapoint in processor.process(metric, datapoint): try: run_pipeline(out_metric, out_datapoint, processors[1:]) except Exception: log.err() except Exception: log.err() graphite-carbon-1.1.7/lib/carbon/protobuf.py0000644000175000017500000000320113633714656017503 0ustar gringrinfrom twisted.protocols.basic import Int32StringReceiver from carbon import log from carbon.protocols import MetricReceiver from carbon.client import CarbonClientProtocol, CarbonClientFactory from carbon.carbon_pb2 import Payload from google.protobuf.message import DecodeError class MetricProtobufReceiver(MetricReceiver, Int32StringReceiver): plugin_name = "protobuf" MAX_LENGTH = 2 ** 20 def stringReceived(self, data): try: payload_pb = Payload.FromString(data) except DecodeError: log.listener('invalid protobuf received from %s, ignoring' % self.peerName) return for metric_pb in payload_pb.metrics: for point_pb in metric_pb.points: self.metricReceived( metric_pb.metric, (point_pb.timestamp, point_pb.value)) class CarbonProtobufClientProtocol(CarbonClientProtocol, Int32StringReceiver): def _sendDatapointsNow(self, datapoints): metrics = {} payload_pb = Payload() for metric, datapoint in datapoints: if metric not in metrics: metric_pb = payload_pb.metrics.add() metric_pb.metric = metric metrics[metric] = metric_pb else: metric_pb = metrics[metric] point_pb = metric_pb.points.add() point_pb.timestamp = int(datapoint[0]) point_pb.value = datapoint[1] self.sendString(payload_pb.SerializeToString()) class CarbonProtobufClientFactory(CarbonClientFactory): plugin_name = "protobuf" def clientProtocol(self): return CarbonProtobufClientProtocol() graphite-carbon-1.1.7/lib/carbon/protocols.py0000644000175000017500000002570313633714656017702 0ustar gringrinimport time import socket import sys from twisted.internet.protocol import ServerFactory, DatagramProtocol # from twisted.application.internet import TCPServer, UDPServer from twisted.application import service from twisted.internet.error import ConnectionDone from twisted.internet import reactor, tcp, udp from twisted.protocols.basic import LineOnlyReceiver, Int32StringReceiver from twisted.protocols.policies import TimeoutMixin from carbon import log, events, state, management from carbon.conf import settings from carbon.regexlist import WhiteList, BlackList from carbon.util import pickle, get_unpickler from carbon.util import PluginRegistrar from six import with_metaclass from carbon.util import enableTcpKeepAlive def checkIfAcceptingConnections(): clients = len(state.connectedMetricReceiverProtocols) max_clients = settings.MAX_RECEIVER_CONNECTIONS if clients < max_clients: for port in state.listeningPorts: if port.paused: log.listener( "Resuming %s (%d/%d connections)" % (port, clients, max_clients)) port.resumeProducing() port.paused = False else: for port in state.listeningPorts: if not port.paused: log.listener( "Pausing %s (%d/%d connections)" % (port, clients, max_clients)) port.pauseProducing() port.paused = True class CarbonReceiverFactory(ServerFactory): def buildProtocol(self, addr): clients = len(state.connectedMetricReceiverProtocols) max_clients = settings.MAX_RECEIVER_CONNECTIONS if clients < max_clients: return ServerFactory.buildProtocol(self, addr) else: return None class CarbonService(service.Service): """Create our own socket to support SO_REUSEPORT. To be removed when twisted supports it natively See: https://github.com/twisted/twisted/pull/759. """ factory = None protocol = None def __init__(self, interface, port, protocol, factory): self.protocol = protocol self.factory = factory self.interface = interface self.port = port def startService(self): # use socket creation from twisted to use the same options as before if hasattr(self.protocol, 'datagramReceived'): tmp_port = udp.Port(None, None, interface=self.interface) else: tmp_port = tcp.Port(None, None, interface=self.interface) carbon_sock = tmp_port.createInternetSocket() if hasattr(socket, 'SO_REUSEPORT'): carbon_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) carbon_sock.bind((self.interface, self.port)) if hasattr(self.protocol, 'datagramReceived'): self._port = reactor.adoptDatagramPort( carbon_sock.fileno(), socket.AF_INET, self.protocol()) else: carbon_sock.listen(tmp_port.backlog) self._port = reactor.adoptStreamPort( carbon_sock.fileno(), socket.AF_INET, self.factory) state.listeningPorts.append(self._port) self._port.paused = False carbon_sock.close() def stopService(self): self._port.stopListening() class CarbonServerProtocol(with_metaclass(PluginRegistrar, object)): plugins = {} @classmethod def build(cls, root_service): plugin_up = cls.plugin_name.upper() interface = settings.get('%s_RECEIVER_INTERFACE' % plugin_up, None) port = int(settings.get('%s_RECEIVER_PORT' % plugin_up, 0)) protocol = cls if not port: return if hasattr(protocol, 'datagramReceived'): service = CarbonService(interface, port, protocol, None) else: factory = CarbonReceiverFactory() factory.protocol = protocol service = CarbonService(interface, port, protocol, factory) service.setServiceParent(root_service) class MetricReceiver(CarbonServerProtocol, TimeoutMixin): """ Base class for all metric receiving protocols, handles flow control events and connection state logging. """ def connectionMade(self): self.setTimeout(settings.METRIC_CLIENT_IDLE_TIMEOUT) enableTcpKeepAlive(self.transport, settings.TCP_KEEPALIVE, settings) self.peerName = self.getPeerName() if settings.LOG_LISTENER_CONN_SUCCESS: log.listener("%s connection with %s established" % ( self.__class__.__name__, self.peerName)) if state.metricReceiversPaused: self.pauseReceiving() state.connectedMetricReceiverProtocols.add(self) checkIfAcceptingConnections() if settings.USE_FLOW_CONTROL: events.pauseReceivingMetrics.addHandler(self.pauseReceiving) events.resumeReceivingMetrics.addHandler(self.resumeReceiving) def getPeerName(self): if hasattr(self.transport, 'getPeer'): peer = self.transport.getPeer() return "%s:%d" % (peer.host, peer.port) else: return "peer" def pauseReceiving(self): self.transport.pauseProducing() def resumeReceiving(self): self.transport.resumeProducing() def connectionLost(self, reason): if reason.check(ConnectionDone): if settings.LOG_LISTENER_CONN_SUCCESS: log.listener( "%s connection with %s closed cleanly" % (self.__class__.__name__, self.peerName)) else: log.listener( "%s connection with %s lost: %s" % (self.__class__.__name__, self.peerName, reason.value)) state.connectedMetricReceiverProtocols.remove(self) checkIfAcceptingConnections() if settings.USE_FLOW_CONTROL: events.pauseReceivingMetrics.removeHandler(self.pauseReceiving) events.resumeReceivingMetrics.removeHandler(self.resumeReceiving) def metricReceived(self, metric, datapoint): if BlackList and metric in BlackList: instrumentation.increment('blacklistMatches') return if WhiteList and metric not in WhiteList: instrumentation.increment('whitelistRejects') return if datapoint[1] != datapoint[1]: # filter out NaN values return # use current time if none given: https://github.com/graphite-project/carbon/issues/54 if int(datapoint[0]) == -1: datapoint = (time.time(), datapoint[1]) res = settings.MIN_TIMESTAMP_RESOLUTION if res: datapoint = (int(datapoint[0]) // res * res, datapoint[1]) events.metricReceived(metric, datapoint) self.resetTimeout() class MetricLineReceiver(MetricReceiver, LineOnlyReceiver): plugin_name = "line" delimiter = b'\n' def lineReceived(self, line): if sys.version_info >= (3, 0): line = line.decode('utf-8') try: metric, value, timestamp = line.strip().split() datapoint = (float(timestamp), float(value)) except ValueError: if len(line) > 400: line = line[:400] + '...' log.listener('invalid line received from client %s, ignoring [%s]' % (self.peerName, repr(line.strip())[1:-1])) return self.metricReceived(metric, datapoint) class MetricDatagramReceiver(MetricReceiver, DatagramProtocol): plugin_name = "udp" @classmethod def build(cls, root_service): if not settings.ENABLE_UDP_LISTENER: return super(MetricDatagramReceiver, cls).build(root_service) def datagramReceived(self, data, addr): (host, _) = addr if sys.version_info >= (3, 0): data = data.decode('utf-8') for line in data.splitlines(): try: metric, value, timestamp = line.strip().split() datapoint = (float(timestamp), float(value)) self.metricReceived(metric, datapoint) except ValueError: if len(line) > 400: line = line[:400] + '...' log.listener('invalid line received from %s, ignoring [%s]' % (host, repr(line.strip())[1:-1])) class MetricPickleReceiver(MetricReceiver, Int32StringReceiver): plugin_name = "pickle" def __init__(self): super(MetricPickleReceiver, self).__init__() self.MAX_LENGTH = settings.PICKLE_RECEIVER_MAX_LENGTH def connectionMade(self): MetricReceiver.connectionMade(self) self.unpickler = get_unpickler(insecure=settings.USE_INSECURE_UNPICKLER) def stringReceived(self, data): try: datapoints = self.unpickler.loads(data) # Pickle can throw a wide range of exceptions except (pickle.UnpicklingError, ValueError, IndexError, ImportError, KeyError, EOFError) as exc: log.listener('invalid pickle received from %s, error: "%s", ignoring' % ( self.peerName, exc)) return for raw in datapoints: try: (metric, (value, timestamp)) = raw except Exception as e: log.listener('Error decoding pickle: %s' % e) continue try: datapoint = (float(value), float(timestamp)) # force proper types except (ValueError, TypeError): continue # convert python2 unicode objects to str/bytes if not isinstance(metric, str): metric = metric.encode('utf-8') self.metricReceived(metric, datapoint) class CacheManagementHandler(Int32StringReceiver): MAX_LENGTH = 1024 ** 3 # 1mb def connectionMade(self): peer = self.transport.getPeer() self.peerAddr = "%s:%d" % (peer.host, peer.port) log.query("%s connected" % self.peerAddr) self.unpickler = get_unpickler(insecure=settings.USE_INSECURE_UNPICKLER) def connectionLost(self, reason): if reason.check(ConnectionDone): log.query("%s disconnected" % self.peerAddr) else: log.query("%s connection lost: %s" % (self.peerAddr, reason.value)) def stringReceived(self, rawRequest): request = self.unpickler.loads(rawRequest) cache = MetricCache() if request['type'] == 'cache-query': metric = request['metric'] datapoints = list(cache.get(metric, {}).items()) result = dict(datapoints=datapoints) if settings.LOG_CACHE_HITS: log.query('[%s] cache query for \"%s\" returned %d values' % ( self.peerAddr, metric, len(datapoints) )) instrumentation.increment('cacheQueries') elif request['type'] == 'cache-query-bulk': datapointsByMetric = {} metrics = request['metrics'] for metric in metrics: datapointsByMetric[metric] = list(cache.get(metric, {}).items()) result = dict(datapointsByMetric=datapointsByMetric) if settings.LOG_CACHE_HITS: log.query('[%s] cache query bulk for \"%d\" metrics returned %d values' % ( self.peerAddr, len(metrics), sum([len(datapoints) for datapoints in datapointsByMetric.values()]) )) instrumentation.increment('cacheBulkQueries') instrumentation.append('cacheBulkQuerySize', len(metrics)) elif request['type'] == 'get-metadata': result = management.getMetadata(request['metric'], request['key']) elif request['type'] == 'set-metadata': result = management.setMetadata(request['metric'], request['key'], request['value']) else: result = dict(error="Invalid request type \"%s\"" % request['type']) response = pickle.dumps(result, protocol=2) self.sendString(response) # Avoid import circularities from carbon.cache import MetricCache # NOQA from carbon import instrumentation # NOQA graphite-carbon-1.1.7/lib/carbon/regexlist.py0000644000175000017500000000300113633714656017647 0ustar gringrinimport re import os.path from carbon import log from twisted.internet.task import LoopingCall class RegexList: """ Maintain a list of regex for matching whitelist and blacklist """ def __init__(self): self.regex_list = [] self.list_file = None self.read_task = LoopingCall(self.read_list) self.rules_last_read = 0.0 def read_from(self, list_file): self.list_file = list_file self.read_list() self.read_task.start(10, now=False) def read_list(self): # Clear rules and move on if file isn't there if not os.path.exists(self.list_file): self.regex_list = [] return try: mtime = os.path.getmtime(self.list_file) except OSError: log.err("Failed to get mtime of %s" % self.list_file) return if mtime <= self.rules_last_read: return # Begin read new_regex_list = [] for line in open(self.list_file): pattern = line.strip() if line.startswith('#') or not pattern: continue try: new_regex_list.append(re.compile(pattern)) except re.error: log.err("Failed to parse '%s' in '%s'. Ignoring line" % (pattern, self.list_file)) self.regex_list = new_regex_list self.rules_last_read = mtime def __contains__(self, value): for regex in self.regex_list: if regex.search(value): return True return False def __nonzero__(self): return bool(self.regex_list) __bool__ = __nonzero__ # py2/3 compatibility WhiteList = RegexList() BlackList = RegexList() graphite-carbon-1.1.7/lib/carbon/relayrules.py0000644000175000017500000000432713633714656020044 0ustar gringrinimport re from carbon.conf import OrderedConfigParser from carbon.util import parseDestinations from carbon.exceptions import CarbonConfigException class RelayRule: def __init__(self, condition, destinations, continue_matching=False): self.condition = condition self.destinations = destinations self.continue_matching = continue_matching def matches(self, metric): return bool(self.condition(metric)) def loadRelayRules(path): rules = [] parser = OrderedConfigParser() if not parser.read(path): raise CarbonConfigException("Could not read rules file %s" % path) defaultRule = None for section in parser.sections(): if not parser.has_option(section, 'destinations'): raise CarbonConfigException("Rules file %s section %s does not define a " "'destinations' list" % (path, section)) destination_strings = parser.get(section, 'destinations').split(',') destinations = parseDestinations(destination_strings) if parser.has_option(section, 'pattern'): if parser.has_option(section, 'default'): raise CarbonConfigException("Section %s contains both 'pattern' and " "'default'. You must use one or the other." % section) pattern = parser.get(section, 'pattern') regex = re.compile(pattern, re.I) continue_matching = False if parser.has_option(section, 'continue'): continue_matching = parser.getboolean(section, 'continue') rule = RelayRule( condition=regex.search, destinations=destinations, continue_matching=continue_matching) rules.append(rule) continue if parser.has_option(section, 'default'): if not parser.getboolean(section, 'default'): continue # just ignore default = false if defaultRule: raise CarbonConfigException("Only one default rule can be specified") defaultRule = RelayRule(condition=lambda metric: True, destinations=destinations) if not defaultRule: raise CarbonConfigException("No default rule defined. You must specify exactly one " "rule with 'default = true' instead of a pattern.") rules.append(defaultRule) return rules graphite-carbon-1.1.7/lib/carbon/resolver.py0000644000175000017500000000476513633714656017524 0ustar gringrinimport random from zope.interface import implementer from twisted.internet._resolver import GAIResolver from twisted.internet.defer import Deferred from twisted.internet.address import IPv4Address from twisted.internet.interfaces import IResolverSimple, IResolutionReceiver from twisted.internet.error import DNSLookupError # Inspired from /twisted/internet/_resolver.py @implementer(IResolutionReceiver) class RandomWins(object): """ An L{IResolutionReceiver} which fires a L{Deferred} with a random result. """ def __init__(self, deferred): """ @param deferred: The L{Deferred} to fire with one resolution result arrives. """ self._deferred = deferred self._results = [] def resolutionBegan(self, resolution): """ See L{IResolutionReceiver.resolutionBegan} @param resolution: See L{IResolutionReceiver.resolutionBegan} """ self._resolution = resolution def addressResolved(self, address): """ See L{IResolutionReceiver.addressResolved} @param address: See L{IResolutionReceiver.addressResolved} """ self._results.append(address.host) def resolutionComplete(self): """ See L{IResolutionReceiver.resolutionComplete} """ if self._results: random.shuffle(self._results) self._deferred.callback(self._results[0]) else: self._deferred.errback(DNSLookupError(self._resolution.name)) @implementer(IResolverSimple) class ComplexResolverSimplifier(object): """ A converter from L{IHostnameResolver} to L{IResolverSimple} """ def __init__(self, nameResolver): """ Create a L{ComplexResolverSimplifier} with an L{IHostnameResolver}. @param nameResolver: The L{IHostnameResolver} to use. """ self._nameResolver = nameResolver def getHostByName(self, name, timeouts=()): """ See L{IResolverSimple.getHostByName} @param name: see L{IResolverSimple.getHostByName} @param timeouts: see L{IResolverSimple.getHostByName} @return: see L{IResolverSimple.getHostByName} """ result = Deferred() self._nameResolver.resolveHostName(RandomWins(result), name, 0, [IPv4Address]) return result def setUpRandomResolver(reactor): resolver = GAIResolver(reactor, reactor.getThreadPool) reactor.installResolver(ComplexResolverSimplifier(resolver)) graphite-carbon-1.1.7/lib/carbon/rewrite.py0000644000175000017500000000470213633714656017333 0ustar gringrinimport re from collections import defaultdict from os.path import exists, getmtime from twisted.internet.task import LoopingCall from carbon.pipeline import Processor from carbon import log # rulesets PRE = 'pre' POST = 'post' class RewriteProcessor(Processor): plugin_name = 'rewrite' def __init__(self, ruleset): self.ruleset = ruleset def process(self, metric, datapoint): for rule in RewriteRuleManager.rules(self.ruleset): metric = rule.apply(metric) yield (metric, datapoint) class _RewriteRuleManager: def __init__(self): self.rulesets = defaultdict(list) self.rules_file = None self.read_task = LoopingCall(self.read_rules) self.rules_last_read = 0.0 def clear(self, ruleset=None): if ruleset: self.rulesets[ruleset] = [] else: self.rulesets.clear() def rules(self, ruleset): return self.rulesets[ruleset] def read_from(self, rules_file): self.rules_file = rules_file self.read_rules() if not self.read_task.running: self.read_task.start(10, now=False) def read_rules(self): if not exists(self.rules_file): self.clear() return # Only read if the rules file has been modified try: mtime = getmtime(self.rules_file) except (OSError, IOError): log.err("Failed to get mtime of %s" % self.rules_file) return if mtime <= self.rules_last_read: return section = None for line in open(self.rules_file): line = line.strip() if line.startswith('#') or not line: continue if line.startswith('[') and line.endswith(']'): section = line[1:-1].lower() self.clear(section) elif '=' in line: pattern, replacement = line.split('=', 1) pattern, replacement = pattern.strip(), replacement.strip() try: rule = RewriteRule(pattern, replacement) except re.error: log.err("Invalid regular expression in rewrite rule: '{0}'".format(pattern)) continue self.rulesets[section].append(rule) else: log.err("Invalid syntax: not a section heading or rule: '{0}'".format(line)) self.rules_last_read = mtime class RewriteRule: def __init__(self, pattern, replacement): self.pattern = pattern self.replacement = replacement self.regex = re.compile(pattern) def apply(self, metric): return self.regex.sub(self.replacement, metric) # Ghetto singleton RewriteRuleManager = _RewriteRuleManager() graphite-carbon-1.1.7/lib/carbon/routers.py0000644000175000017500000001726213633714656017362 0ustar gringrinimport imp from carbon.hashing import ConsistentHashRing, carbonHash from carbon.util import PluginRegistrar from six import with_metaclass from six.moves import xrange class DatapointRouter(with_metaclass(PluginRegistrar, object)): "Abstract base class for datapoint routing logic implementations" plugins = {} def addDestination(self, destination): "destination is a (host, port, instance) triple" raise NotImplementedError() def removeDestination(self, destination): "destination is a (host, port, instance) triple" raise NotImplementedError() def hasDestination(self, destination): "destination is a (host, port, instance) triple" raise NotImplementedError() def countDestinations(self): "return number of configured destinations" raise NotImplementedError() def getDestinations(self, key): """Generate the destinations where the given routing key should map to. Only destinations which are configured (addDestination has been called for it) may be generated by this method.""" raise NotImplementedError() class RelayRulesRouter(DatapointRouter): plugin_name = 'rules' def __init__(self, settings): # We need to import relayrules here to avoid circular dependencies. from carbon.relayrules import loadRelayRules rules_path = settings["relay-rules"] self.rules_path = rules_path self.rules = loadRelayRules(rules_path) self.destinations = set() def addDestination(self, destination): self.destinations.add(destination) def removeDestination(self, destination): self.destinations.discard(destination) def hasDestination(self, destination): return destination in self.destinations def countDestinations(self): return len(self.destinations) def getDestinations(self, key): for rule in self.rules: if rule.matches(key): for destination in rule.destinations: if destination in self.destinations: yield destination if not rule.continue_matching: return class ConsistentHashingRouter(DatapointRouter): plugin_name = 'consistent-hashing' def __init__(self, settings): replication_factor = settings.REPLICATION_FACTOR diverse_replicas = settings.DIVERSE_REPLICAS self.replication_factor = int(replication_factor) self.diverse_replicas = diverse_replicas self.instance_ports = {} # { (server, instance) : port } hash_type = settings.ROUTER_HASH_TYPE or 'carbon_ch' self.ring = ConsistentHashRing([], hash_type=hash_type) def addDestination(self, destination): (server, port, instance) = destination if self.hasDestination(destination): raise Exception("destination instance (%s, %s) already configured" % (server, instance)) self.instance_ports[(server, instance)] = port self.ring.add_node((server, instance)) def removeDestination(self, destination): (server, port, instance) = destination if not self.hasDestination(destination): raise Exception("destination instance (%s, %s) not configured" % (server, instance)) del self.instance_ports[(server, instance)] self.ring.remove_node((server, instance)) def hasDestination(self, destination): (server, _, instance) = destination return (server, instance) in self.instance_ports def countDestinations(self): return len(self.instance_ports) def getDestinations(self, metric): key = self.getKey(metric) if self.diverse_replicas: used_servers = set() for (server, instance) in self.ring.get_nodes(key): if server in used_servers: continue else: used_servers.add(server) port = self.instance_ports[(server, instance)] yield (server, port, instance) if len(used_servers) >= self.replication_factor: return else: for (count, node) in enumerate(self.ring.get_nodes(key)): if count == self.replication_factor: return (server, instance) = node port = self.instance_ports[(server, instance)] yield (server, port, instance) def getKey(self, metric): return metric def setKeyFunction(self, func): self.getKey = func def setKeyFunctionFromModule(self, keyfunc_spec): module_path, func_name = keyfunc_spec.rsplit(':', 1) module_file = open(module_path, 'U') description = ('.py', 'U', imp.PY_SOURCE) module = imp.load_module('keyfunc_module', module_file, module_path, description) keyfunc = getattr(module, func_name) self.setKeyFunction(keyfunc) class AggregatedConsistentHashingRouter(DatapointRouter): plugin_name = 'aggregated-consistent-hashing' def __init__(self, settings): from carbon.aggregator.rules import RuleManager aggregation_rules_path = settings["aggregation-rules"] if aggregation_rules_path: RuleManager.read_from(aggregation_rules_path) self.hash_router = ConsistentHashingRouter(settings) self.agg_rules_manager = RuleManager def addDestination(self, destination): self.hash_router.addDestination(destination) def removeDestination(self, destination): self.hash_router.removeDestination(destination) def hasDestination(self, destination): return self.hash_router.hasDestination(destination) def countDestinations(self): return self.hash_router.countDestinations() def getDestinations(self, key): # resolve metric to aggregate forms resolved_metrics = [] for rule in self.agg_rules_manager.rules: aggregate_metric = rule.get_aggregate_metric(key) if aggregate_metric is None: continue else: resolved_metrics.append(aggregate_metric) # if the metric will not be aggregated, send it raw # (will pass through aggregation) if len(resolved_metrics) == 0: resolved_metrics.append(key) # get consistent hashing destinations based on aggregate forms destinations = set() for resolved_metric in resolved_metrics: for destination in self.hash_router.getDestinations(resolved_metric): destinations.add(destination) for destination in destinations: yield destination class FastHashRing(object): """A very fast hash 'ring'. Instead of trying to avoid rebalancing data when changing the list of nodes we try to making routing as fast as we can. It's good enough because the current rebalancing tools performances depend on the total number of metrics and not the number of metrics to rebalance. """ def __init__(self, settings): self.nodes = set() self.sorted_nodes = [] self.hash_type = settings.ROUTER_HASH_TYPE or 'mmh3_ch' def _hash(self, key): return carbonHash(key, self.hash_type) def _update_nodes(self): self.sorted_nodes = sorted( [(self._hash(str(n)), n) for n in self.nodes], key=lambda v: v[0] ) def add_node(self, node): self.nodes.add(node) self._update_nodes() def remove_node(self, node): self.nodes.discard(node) self._update_nodes() def get_nodes(self, key): if not self.nodes: return seed = self._hash(key) % len(self.nodes) for n in xrange(seed, seed + len(self.nodes)): yield self.sorted_nodes[n % len(self.sorted_nodes)][1] class FastHashingRouter(ConsistentHashingRouter): """Same as ConsistentHashingRouter but using FastHashRing.""" plugin_name = 'fast-hashing' def __init__(self, settings): super(FastHashingRouter, self).__init__(settings) self.ring = FastHashRing(settings) class FastAggregatedHashingRouter(AggregatedConsistentHashingRouter): """Same as AggregatedConsistentHashingRouter but using FastHashRing.""" plugin_name = 'fast-aggregated-hashing' def __init__(self, settings): super(FastAggregatedHashingRouter, self).__init__(settings) self.hash_router.ring = FastHashRing(settings) graphite-carbon-1.1.7/lib/carbon/service.py0000644000175000017500000001476213633714656017321 0ustar gringrin"""Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" from os.path import exists from twisted.application.service import MultiService from twisted.application.internet import TCPServer from twisted.internet.protocol import ServerFactory from twisted.python.components import Componentized from twisted.python.log import ILogObserver # Attaching modules to the global state module simplifies import order hassles from carbon import state, events, instrumentation, util from carbon.exceptions import CarbonConfigException from carbon.log import carbonLogObserver from carbon.pipeline import Processor, run_pipeline, run_pipeline_generated state.events = events state.instrumentation = instrumentation # Import plugins. try: import carbon.manhole except ImportError: pass try: import carbon.amqp_listener except ImportError: pass try: import carbon.protobuf # NOQA except ImportError: pass class CarbonRootService(MultiService): """Root Service that properly configures twistd logging""" def setServiceParent(self, parent): MultiService.setServiceParent(self, parent) if isinstance(parent, Componentized): parent.setComponent(ILogObserver, carbonLogObserver) def createBaseService(config, settings): root_service = CarbonRootService() root_service.setName(settings.program) if settings.USE_WHITELIST: from carbon.regexlist import WhiteList, BlackList WhiteList.read_from(settings.whitelist) BlackList.read_from(settings.blacklist) # Instantiate an instrumentation service that will record metrics about # this service. from carbon.instrumentation import InstrumentationService service = InstrumentationService() service.setServiceParent(root_service) return root_service def setupPipeline(pipeline, root_service, settings): state.pipeline_processors = [] for processor in pipeline: args = [] if ':' in processor: processor, arglist = processor.split(':', 1) args = arglist.split(',') if processor == 'aggregate': setupAggregatorProcessor(root_service, settings) elif processor == 'rewrite': setupRewriterProcessor(root_service, settings) elif processor == 'relay': setupRelayProcessor(root_service, settings) elif processor == 'write': setupWriterProcessor(root_service, settings) else: raise ValueError("Invalid pipeline processor '%s'" % processor) plugin_class = Processor.plugins[processor] state.pipeline_processors.append(plugin_class(*args)) if processor in ['relay', 'write']: state.pipeline_processors_generated.append(plugin_class(*args)) events.metricReceived.addHandler(run_pipeline) events.metricGenerated.addHandler(run_pipeline_generated) def activate_processors(): for processor in state.pipeline_processors: processor.pipeline_ready() from twisted.internet import reactor reactor.callWhenRunning(activate_processors) def createCacheService(config): from carbon.conf import settings root_service = createBaseService(config, settings) setupPipeline(['write'], root_service, settings) setupReceivers(root_service, settings) return root_service def createAggregatorService(config): from carbon.conf import settings settings.RELAY_METHOD = 'consistent-hashing' root_service = createBaseService(config, settings) setupPipeline( ['rewrite:pre', 'aggregate', 'rewrite:post', 'relay'], root_service, settings) setupReceivers(root_service, settings) return root_service def createAggregatorCacheService(config): from carbon.conf import settings settings.RELAY_METHOD = 'consistent-hashing' root_service = createBaseService(config, settings) setupPipeline( ['rewrite:pre', 'aggregate', 'rewrite:post', 'write'], root_service, settings) setupReceivers(root_service, settings) return root_service def createRelayService(config): from carbon.conf import settings root_service = createBaseService(config, settings) setupPipeline(['relay'], root_service, settings) setupReceivers(root_service, settings) return root_service def setupReceivers(root_service, settings): from carbon.protocols import MetricReceiver for _, plugin_class in MetricReceiver.plugins.items(): plugin_class.build(root_service) def setupAggregatorProcessor(root_service, settings): from carbon.aggregator.processor import AggregationProcessor # NOQA Register the plugin class from carbon.aggregator.rules import RuleManager aggregation_rules_path = settings["aggregation-rules"] if not exists(aggregation_rules_path): raise CarbonConfigException( "aggregation processor: file does not exist {0}".format(aggregation_rules_path)) RuleManager.read_from(aggregation_rules_path) def setupRewriterProcessor(root_service, settings): from carbon.rewrite import RewriteRuleManager rewrite_rules_path = settings["rewrite-rules"] RewriteRuleManager.read_from(rewrite_rules_path) def setupRelayProcessor(root_service, settings): from carbon.routers import DatapointRouter from carbon.client import CarbonClientManager router_class = DatapointRouter.plugins[settings.RELAY_METHOD] router = router_class(settings) state.client_manager = CarbonClientManager(router) state.client_manager.setServiceParent(root_service) for destination in util.parseDestinations(settings.DESTINATIONS): state.client_manager.startClient(destination) def setupWriterProcessor(root_service, settings): from carbon import cache # NOQA Register CacheFeedingProcessor from carbon.protocols import CacheManagementHandler from carbon.writer import WriterService factory = ServerFactory() factory.protocol = CacheManagementHandler service = TCPServer( settings.CACHE_QUERY_PORT, factory, interface=settings.CACHE_QUERY_INTERFACE) service.setServiceParent(root_service) writer_service = WriterService() writer_service.setServiceParent(root_service) if settings.USE_FLOW_CONTROL: events.cacheFull.addHandler(events.pauseReceivingMetrics) events.cacheSpaceAvailable.addHandler(events.resumeReceivingMetrics) graphite-carbon-1.1.7/lib/carbon/state.py0000644000175000017500000000047613633714656016776 0ustar gringrin__doc__ = """ This module exists for the purpose of tracking global state used across several modules. """ metricReceiversPaused = False cacheTooFull = False client_manager = None connectedMetricReceiverProtocols = set() pipeline_processors = [] pipeline_processors_generated = [] database = None listeningPorts = [] graphite-carbon-1.1.7/lib/carbon/storage.py0000644000175000017500000001110513633714656017311 0ustar gringrin"""Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import re from os.path import join from carbon.conf import OrderedConfigParser, settings from carbon.exceptions import CarbonConfigException from carbon.util import parseRetentionDef from carbon import log, state STORAGE_SCHEMAS_CONFIG = join(settings.CONF_DIR, 'storage-schemas.conf') STORAGE_AGGREGATION_CONFIG = join(settings.CONF_DIR, 'storage-aggregation.conf') STORAGE_LISTS_DIR = join(settings.CONF_DIR, 'lists') class Schema: def test(self, metric): raise NotImplementedError() def matches(self, metric): return bool(self.test(metric)) class DefaultSchema(Schema): def __init__(self, name, archives): self.name = name self.archives = archives def test(self, metric): return True class PatternSchema(Schema): def __init__(self, name, pattern, archives): self.name = name self.pattern = pattern self.regex = re.compile(pattern) self.archives = archives def test(self, metric): return self.regex.search(metric) class Archive: def __init__(self, secondsPerPoint, points): self.secondsPerPoint = int(secondsPerPoint) self.points = int(points) def __str__(self): return "Archive = (Seconds per point: %d, Datapoints to save: %d)" % ( self.secondsPerPoint, self.points) def getTuple(self): return (self.secondsPerPoint, self.points) @staticmethod def fromString(retentionDef): (secondsPerPoint, points) = parseRetentionDef(retentionDef) return Archive(secondsPerPoint, points) def loadStorageSchemas(): schemaList = [] config = OrderedConfigParser() config.read(STORAGE_SCHEMAS_CONFIG) for section in config.sections(): options = dict(config.items(section)) pattern = options.get('pattern') try: retentions = options['retentions'].split(',') except KeyError: log.err("Schema %s missing 'retentions', skipping" % section) continue try: archives = [Archive.fromString(s) for s in retentions] except ValueError as exc: log.err("{msg} in section [{section}] in {fn}".format( msg=exc, section=section.title(), fn=STORAGE_SCHEMAS_CONFIG)) raise SystemExit(1) if pattern: mySchema = PatternSchema(section, pattern, archives) else: log.err("Schema %s missing 'pattern', skipping" % section) continue archiveList = [a.getTuple() for a in archives] try: if state.database is not None: state.database.validateArchiveList(archiveList) schemaList.append(mySchema) except ValueError as e: log.msg("Invalid schemas found in %s: %s" % (section, e)) schemaList.append(defaultSchema) return schemaList def loadAggregationSchemas(): # NOTE: This abuses the Schema classes above, and should probably be refactored. schemaList = [] config = OrderedConfigParser() try: config.read(STORAGE_AGGREGATION_CONFIG) except (IOError, CarbonConfigException): log.msg("%s not found or wrong perms, ignoring." % STORAGE_AGGREGATION_CONFIG) for section in config.sections(): options = dict(config.items(section)) pattern = options.get('pattern') xFilesFactor = options.get('xfilesfactor') aggregationMethod = options.get('aggregationmethod') try: if xFilesFactor is not None: xFilesFactor = float(xFilesFactor) assert 0 <= xFilesFactor <= 1 if aggregationMethod is not None: if state.database is not None: assert aggregationMethod in state.database.aggregationMethods except ValueError: log.msg("Invalid schemas found in %s." % section) continue archives = (xFilesFactor, aggregationMethod) if pattern: mySchema = PatternSchema(section, pattern, archives) else: log.err("Section missing 'pattern': %s" % section) continue schemaList.append(mySchema) schemaList.append(defaultAggregation) return schemaList # default retention for unclassified data (7 days of minutely data) defaultArchive = Archive(60, 60 * 24 * 7) defaultSchema = DefaultSchema('default', [defaultArchive]) defaultAggregation = DefaultSchema('default', (None, None)) graphite-carbon-1.1.7/lib/carbon/tests/0000755000175000017500000000000013633714656016437 5ustar gringringraphite-carbon-1.1.7/lib/carbon/tests/__init__.py0000644000175000017500000000000013633714656020536 0ustar gringringraphite-carbon-1.1.7/lib/carbon/tests/benchmark_aggregator.py0000644000175000017500000000407413633714656023152 0ustar gringrinimport timeit import time from carbon.aggregator.processor import AggregationProcessor, RuleManager from carbon.aggregator.buffers import BufferManager from carbon.tests.util import print_stats from carbon.conf import settings from carbon import state METRIC = 'prod.applications.foo.1.requests' METRIC_AGGR = 'prod.applications.foo.all.requests' FREQUENCY = 1000 def bench_aggregator_noop(): RuleManager.clear() _bench_aggregator("noop") def bench_aggregator_sum(): RuleManager.clear() RuleManager.rules = [ RuleManager.parse_definition( ('.applications..all.requests (%d) =' % FREQUENCY) + 'sum .applications..*.requests'), ] _bench_aggregator("sum") def bench_aggregator_fake(): RuleManager.clear() RuleManager.rules = [ RuleManager.parse_definition('foo (60) = sum bar'), ] _bench_aggregator("fake") def _bench_aggregator(name): print("== %s ==" % name) max_intervals = settings['MAX_AGGREGATION_INTERVALS'] now = time.time() - (max_intervals * FREQUENCY) buf = None for n in [1, 1000, 10000, 100000, 1000000, 10000000]: processor = AggregationProcessor() processor.process(METRIC, (now, 1)) def _process(): processor.process(METRIC, (now + _process.i, 1)) if (_process.i % FREQUENCY) == 0 and buf is not None: buf.compute_values() _process.i += 1 _process.i = 0 if buf is None: buf = BufferManager.get_buffer(METRIC_AGGR, 1, None) t = timeit.timeit(_process, number=n) buf.close() print_stats(n, t) print("") def main(): settings.LOG_AGGREGATOR_MISSES = False class _Fake(object): def metricGenerated(self, metric, datapoint): pass def increment(self, metric): pass state.events = _Fake() state.instrumentation = _Fake() _bench_aggregator("warmup") bench_aggregator_noop() bench_aggregator_sum() bench_aggregator_fake() if __name__ == '__main__': main() graphite-carbon-1.1.7/lib/carbon/tests/benchmark_cache.py0000644000175000017500000000523513633714656022073 0ustar gringrinimport timeit from carbon.cache import _MetricCache, DrainStrategy, \ NaiveStrategy, MaxStrategy, RandomStrategy, SortedStrategy, TimeSortedStrategy metric_cache = _MetricCache(DrainStrategy) count = 0 strategies = { 'naive': NaiveStrategy, 'max': MaxStrategy, 'random': RandomStrategy, 'sorted': SortedStrategy, 'timesorted': TimeSortedStrategy, } def command_store_foo(): global count count = count + 1 return metric_cache.store('foo', (count, 1.0)) def command_store_foo_n(): global count count = count + 1 return metric_cache.store("foo.%d" % count, (count, 1.0)) def command_drain(): while metric_cache: metric_cache.drain_metric() return metric_cache.size def print_stats(n, t): usec = t * 1e6 if usec < 1000: print(" datapoints: %-10d usecs: %d" % (n, int(usec))) else: msec = usec / 1000 if msec < 1000: print(" datapoints: %-10d msecs: %d" % (n, int(msec))) else: sec = msec / 1000 print(" datapoints: %-10d secs: %3g" % (n, sec)) if __name__ == '__main__': print("Benchmarking single metric MetricCache store...") for n in [1000, 10000, 100000, 1000000]: count = 0 metric_cache = _MetricCache(DrainStrategy) t = timeit.timeit(command_store_foo, number=n) print_stats(n, t) print("Benchmarking unique metric MetricCache store...") for n in [1000, 10000, 100000, 1000000]: count = 0 metric_cache = _MetricCache(DrainStrategy) t = timeit.timeit(command_store_foo_n, number=n) print_stats(n, t) print("Benchmarking single metric MetricCache drain...") for name, strategy in sorted(strategies.items()): print("CACHE_WRITE_STRATEGY: %s" % name) for n in [1000, 10000, 100000, 1000000]: count = 0 metric_cache = _MetricCache(strategy) timeit.timeit(command_store_foo, number=n) t = timeit.timeit(command_drain, number=1) print_stats(n, t) print("Benchmarking unique metric MetricCache drain...") for name, strategy in sorted(strategies.items()): print("CACHE_WRITE_STRATEGY: %s" % name) for n in [1000, 10000, 100000, 1000000]: # remove me when strategy is fast if (name == 'max' and n > 10000) or (name == 'random' and n > 10000): print(" datapoints: %-10d [skipped]" % n) continue count = 0 metric_cache = _MetricCache(strategy) timeit.timeit(command_store_foo_n, number=n) t = timeit.timeit(command_drain, number=1) print_stats(n, t) graphite-carbon-1.1.7/lib/carbon/tests/benchmark_routers.py0000644000175000017500000000526513633714656022536 0ustar gringrinimport timeit from carbon.routers import DatapointRouter from test_routers import createSettings from six.moves import xrange REPLICATION_FACTORS = [1, 4] DIVERSE_REPLICAS = [True, False] N_DESTINATIONS = [1, 16, 32, 48] HASH_TYPES = [None, 'carbon_ch', 'fnv1a_ch', 'mmh3_ch'] def print_stats(r, t): usec = t * 1e6 msec = usec / 1000 text = " %s %s datapoints: %d" % (r.plugin_name, r.__id, r.__count) if usec < 1000: text += " usecs: %d" % int(usec) elif msec < 1000: text += " msecs: %d" % int(msec) else: sec = msec / 1000 text += " secs: %3g" % sec print(text) def generateDestinations(n): for i in xrange(n): host_id = i % 10 instance_id = i port = 2000 + i yield ('carbon%d' % host_id, port, instance_id) def benchmark(router_class): for hash_type in HASH_TYPES: for replication_factor in REPLICATION_FACTORS: for diverse_replicas in DIVERSE_REPLICAS: for n_destinations in N_DESTINATIONS: _benchmark( router_class, replication_factor, diverse_replicas, n_destinations, hash_type ) def _benchmark(router_class, replication_factor, diverse_replicas, n_destinations, hash_type): destinations = list(generateDestinations(n_destinations)) settings = createSettings() settings['REPLICATION_FACTOR'] = replication_factor settings['DIVERSE_REPLICAS'] = diverse_replicas settings['DESTINATIONS'] = destinations settings['ROUTER_HASH_TYPE'] = hash_type router = router_class(settings) router.__count = 0 # Ugly hack for timeit ! router.__id = ( ' replication_factor: %d' % replication_factor + ' diverse_replicas: %d' % diverse_replicas + ' n_destinations: %-5d' % n_destinations + ' hash_type: %s' % hash_type) settings.DESTINATIONS = [] for destination in destinations: router.addDestination(destination) settings.DESTINATIONS.append( '%s:%s:%s' % ( destination[0], destination[1], destination[2])) benchmark_router(router) def benchmark_router(router): def router_getDestinations(): router.__count += 1 dst = list(router.getDestinations('foo.%d' % router.__count)) assert(len(dst) != 0) n = 100000 t = timeit.timeit(router_getDestinations, number=n) print_stats(router, t) def main(): for router_class in DatapointRouter.plugins.values(): # Skip 'rules' because it's hard to mock. if router_class.plugin_name == 'rules': continue benchmark(router_class) if __name__ == '__main__': main() graphite-carbon-1.1.7/lib/carbon/tests/data/0000755000175000017500000000000013633714656017350 5ustar gringringraphite-carbon-1.1.7/lib/carbon/tests/data/conf-directory/0000755000175000017500000000000013633714656022277 5ustar gringringraphite-carbon-1.1.7/lib/carbon/tests/data/conf-directory/storage-aggregation.conf0000644000175000017500000000147313633714656027104 0ustar gringrin# Aggregation methods for whisper files. Entries are scanned in order, # and first match wins. This file is scanned for changes every 60 seconds # # [name] # pattern = # xFilesFactor = # aggregationMethod = # # name: Arbitrary unique name for the rule # pattern: Regex pattern to match against the metric name # xFilesFactor: Ratio of valid data points required for aggregation to the next retention to occur # aggregationMethod: function to apply to data points for aggregation # [min] pattern = \.min$ xFilesFactor = 0.1 aggregationMethod = min [max] pattern = \.max$ xFilesFactor = 0.1 aggregationMethod = max [sum] pattern = \.count$ xFilesFactor = 0 aggregationMethod = sum [default_average] pattern = .* xFilesFactor = 0.5 aggregationMethod = average graphite-carbon-1.1.7/lib/carbon/tests/data/conf-directory/storage-schemas.conf0000644000175000017500000000015313633714656026232 0ustar gringrin[carbon] pattern = ^carbon\. retentions = 60:90d [default_1min_for_1day] pattern = .* retentions = 60s:1d graphite-carbon-1.1.7/lib/carbon/tests/relay-rules.conf0000644000175000017500000000006213633714656021550 0ustar gringrin[default] destinations = foo:124:a default = true graphite-carbon-1.1.7/lib/carbon/tests/test_aggregator_buffers.py0000644000175000017500000002301213633714656023704 0ustar gringrinfrom mock import call, Mock, patch from unittest import TestCase from twisted.internet.task import LoopingCall from carbon import instrumentation from carbon.aggregator.buffers import BufferManager, IntervalBuffer, MetricBuffer from carbon.tests.util import TestSettings class AggregationBufferManagerTest(TestCase): def tearDown(self): BufferManager.clear() @patch("carbon.aggregator.buffers.MetricBuffer") def test_get_nonexistent_buffer_creates_new(self, metric_buffer_mock): BufferManager.get_buffer("carbon.foo") metric_buffer_mock.assert_called_once_with("carbon.foo") @patch("carbon.aggregator.buffers.MetricBuffer", new=Mock()) def test_get_nonexistent_buffer_creates_and_saves_it(self): new_buffer = BufferManager.get_buffer("carbon.foo") existing_buffer = BufferManager.get_buffer("carbon.foo") self.assertTrue(new_buffer is existing_buffer) @patch("carbon.aggregator.buffers.MetricBuffer", new=Mock(spec=MetricBuffer)) def test_clear_closes_buffers(self): metric_buffer_mock = BufferManager.get_buffer("carbon.foo") BufferManager.clear() metric_buffer_mock.close.assert_called_once_with() class AggregationMetricBufferTest(TestCase): def setUp(self): self.new_metric_buffer = MetricBuffer("carbon.foo") with patch("carbon.aggregator.buffers.LoopingCall", new=Mock()): self.metric_buffer = MetricBuffer("carbon.foo.bar") self.metric_buffer.configure_aggregation(60, sum) def tearDown(self): instrumentation.stats.clear() def test_new_buffer_is_unconfigured(self): self.assertFalse(self.new_metric_buffer.configured) @patch("carbon.aggregator.buffers.LoopingCall", new=Mock()) def test_configure_buffer_marks_configured(self): self.new_metric_buffer.configure_aggregation(60, sum) self.assertTrue(self.new_metric_buffer.configured) @patch("carbon.aggregator.buffers.LoopingCall", spec=LoopingCall) def test_configure_buffer_creates_looping_call(self, looping_call_mock): self.new_metric_buffer.configure_aggregation(60, sum) looping_call_mock.assert_called_once_with(self.new_metric_buffer.compute_value) @patch("carbon.aggregator.buffers.LoopingCall", spec=LoopingCall) def test_configure_buffer_starts_looping_call(self, looping_call_mock): self.new_metric_buffer.configure_aggregation(60, sum) looping_call_mock.return_value.start.assert_called_once_with(60, now=False) @patch("carbon.aggregator.buffers.LoopingCall", spec=LoopingCall) def test_configure_buffer_uses_freq_if_less_than_writeback_freq(self, looping_call_mock): settings = TestSettings() settings['WRITE_BACK_FREQUENCY'] = 300 with patch('carbon.aggregator.buffers.settings', new=settings): self.new_metric_buffer.configure_aggregation(60, sum) looping_call_mock.return_value.start.assert_called_once_with(60, now=False) @patch("carbon.aggregator.buffers.LoopingCall", spec=LoopingCall) def test_configure_buffer_uses_writeback_freq_if_less_than_freq(self, looping_call_mock): settings = TestSettings() settings['WRITE_BACK_FREQUENCY'] = 30 with patch('carbon.aggregator.buffers.settings', new=settings): self.new_metric_buffer.configure_aggregation(60, sum) looping_call_mock.return_value.start.assert_called_once_with(30, now=False) @patch("carbon.aggregator.buffers.IntervalBuffer", new=Mock()) def test_input_rounds_down_to_interval(self): # Interval of 60 self.metric_buffer.input((125, 1.0)) self.assertTrue(120 in self.metric_buffer.interval_buffers) @patch("carbon.aggregator.buffers.IntervalBuffer", spec=IntervalBuffer) def test_input_passes_datapoint_to_interval_buffer(self, interval_buffer_mock): self.metric_buffer.input((120, 1.0)) interval_buffer_mock.return_value.input.assert_called_once_with((120, 1.0)) @patch("time.time", new=Mock(return_value=600)) @patch("carbon.state.events.metricGenerated") def test_compute_value_flushes_active_buffer(self, metric_generated_mock): self.metric_buffer.input((600, 1.0)) self.metric_buffer.compute_value() metric_generated_mock.assert_called_once_with("carbon.foo.bar", (600, 1.0)) @patch("time.time", new=Mock(return_value=600)) @patch("carbon.state.events.metricGenerated") def test_compute_value_uses_interval_for_flushed_datapoint(self, metric_generated_mock): self.metric_buffer.input((630, 1.0)) self.metric_buffer.compute_value() metric_generated_mock.assert_called_once_with("carbon.foo.bar", (600, 1.0)) @patch("time.time", new=Mock(return_value=600)) @patch("carbon.state.events.metricGenerated", new=Mock()) def test_compute_value_marks_buffer_inactive(self): interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) self.metric_buffer.interval_buffers[600] = interval_buffer with patch.object(IntervalBuffer, 'mark_inactive') as mark_inactive_mock: self.metric_buffer.compute_value() mark_inactive_mock.assert_called_once_with() @patch("time.time", new=Mock(return_value=600)) @patch("carbon.state.events.metricGenerated", new=Mock()) def test_compute_value_computes_aggregate(self): interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) interval_buffer.input((601, 2.0)) interval_buffer.input((602, 3.0)) self.metric_buffer.interval_buffers[600] = interval_buffer with patch.object(self.metric_buffer, 'aggregation_func') as aggregation_func_mock: self.metric_buffer.compute_value() aggregation_func_mock.assert_called_once_with([1.0, 2.0, 3.0]) @patch("time.time", new=Mock(return_value=600)) @patch("carbon.state.events.metricGenerated") def test_compute_value_skips_inactive_buffers(self, metric_generated_mock): interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) interval_buffer.mark_inactive() self.metric_buffer.interval_buffers[600] = interval_buffer self.metric_buffer.compute_value() self.assertFalse(metric_generated_mock.called) @patch("carbon.state.events.metricGenerated") def test_compute_value_can_flush_interval_multiple_times(self, metric_generated_mock): interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) interval_buffer.input((601, 2.0)) interval_buffer.input((602, 3.0)) self.metric_buffer.interval_buffers[600] = interval_buffer with patch("time.time") as time_mock: time_mock.return_value = 600 self.metric_buffer.compute_value() calls = [call("carbon.foo.bar", (600, 6.0))] # say WRITE_BACK_FREQUENCY is 30, we flush again if another point came in time_mock.return_value = 630 interval_buffer.input((604, 4.0)) self.metric_buffer.compute_value() calls.append(call("carbon.foo.bar", (600, 10.0))) metric_generated_mock.assert_has_calls(calls) @patch("carbon.state.events.metricGenerated") def test_compute_value_doesnt_flush_unchanged_interval_many_times(self, metric_generated_mock): interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) self.metric_buffer.interval_buffers[600] = interval_buffer with patch("time.time") as time_mock: time_mock.return_value = 600 self.metric_buffer.compute_value() calls = [call("carbon.foo.bar", (600, 1.0))] # say WRITE_BACK_FREQUENCY is 30, we flush again but no point came in time_mock.return_value = 630 self.metric_buffer.compute_value() metric_generated_mock.assert_has_calls(calls) def test_compute_value_deletes_expired_buffers(self): from carbon.conf import settings current_interval = 600 + 60 * settings['MAX_AGGREGATION_INTERVALS'] interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) interval_buffer.mark_inactive() self.metric_buffer.interval_buffers[600] = interval_buffer # 2nd interval for current time interval_buffer = IntervalBuffer(current_interval) interval_buffer.input((current_interval, 1.0)) interval_buffer.mark_inactive() self.metric_buffer.interval_buffers[current_interval] = interval_buffer with patch("time.time", new=Mock(return_value=current_interval + 60)): self.metric_buffer.compute_value() self.assertFalse(600 in self.metric_buffer.interval_buffers) def test_compute_value_closes_metric_if_last_buffer_deleted(self): from carbon.conf import settings current_interval = 600 + 60 * settings['MAX_AGGREGATION_INTERVALS'] interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) interval_buffer.mark_inactive() self.metric_buffer.interval_buffers[600] = interval_buffer BufferManager.buffers['carbon.foo.bar'] = self.metric_buffer with patch("time.time", new=Mock(return_value=current_interval + 60)): with patch.object(MetricBuffer, 'close') as close_mock: self.metric_buffer.compute_value() close_mock.assert_called_once_with() def test_compute_value_unregisters_metric_if_last_buffer_deleted(self): from carbon.conf import settings current_interval = 600 + 60 * settings['MAX_AGGREGATION_INTERVALS'] interval_buffer = IntervalBuffer(600) interval_buffer.input((600, 1.0)) interval_buffer.mark_inactive() self.metric_buffer.interval_buffers[600] = interval_buffer BufferManager.buffers['carbon.foo.bar'] = self.metric_buffer with patch("time.time", new=Mock(return_value=current_interval + 60)): self.metric_buffer.compute_value() self.assertFalse('carbon.foo.bar' in BufferManager.buffers) def test_close_stops_looping_call(self): with patch.object(MetricBuffer, 'close') as close_mock: self.metric_buffer.close() close_mock.assert_called_once_with() graphite-carbon-1.1.7/lib/carbon/tests/test_aggregator_methods.py0000644000175000017500000000212313633714656023713 0ustar gringrinimport unittest from carbon.aggregator.rules import AGGREGATION_METHODS PERCENTILE_METHODS = ['p999', 'p99', 'p95', 'p90', 'p80', 'p75', 'p50'] VALUES = [4, 8, 15, 16, 23, 42] def almost_equal(a, b): return abs(a - b) < 0.0000000001 class AggregationMethodTest(unittest.TestCase): def test_percentile_simple(self): for method in PERCENTILE_METHODS: self.assertTrue(almost_equal(AGGREGATION_METHODS[method]([1]), 1)) def test_percentile_order(self): for method in PERCENTILE_METHODS: a = AGGREGATION_METHODS[method]([1, 2, 3, 4, 5]) b = AGGREGATION_METHODS[method]([3, 2, 1, 4, 5]) self.assertTrue(almost_equal(a, b)) def test_percentile_values(self): examples = [ ('p999', 41.905, ), ('p99', 41.05, ), ('p95', 37.25, ), ('p90', 32.5, ), ('p80', 23, ), ('p75', 21.25, ), ('p50', 15.5, ), ] for (method, result) in examples: self.assertTrue(almost_equal(AGGREGATION_METHODS[method](VALUES), result)) graphite-carbon-1.1.7/lib/carbon/tests/test_aggregator_processor.py0000644000175000017500000000511613633714656024274 0ustar gringrinfrom mock import patch from unittest import TestCase from carbon import instrumentation from carbon.pipeline import Processor from carbon.aggregator.buffers import BufferManager from carbon.aggregator.rules import AggregationRule, RuleManager from carbon.aggregator.processor import AggregationProcessor class AggregationProcessorTest(TestCase): def setUp(self): self.sample_aggregation_rule = AggregationRule(r'^carbon.foo', r'carbon.foo.sum', 'sum', 1) self.sample_overwriting_aggregation_rule = \ AggregationRule(r'^carbon.foo', r'carbon.foo', 'sum', 1) self.processor = AggregationProcessor() def tearDown(self): instrumentation.stats.clear() BufferManager.clear() RuleManager.clear() def test_registers_plugin(self): self.assertTrue('aggregate' in Processor.plugins) def test_process_increments_datapoints_metric(self): list(self.processor.process('carbon.foo', (0, 0))) self.assertEqual(1, instrumentation.stats['datapointsReceived']) def test_unaggregated_metrics_pass_through_when_no_rules(self): result = list(self.processor.process('carbon.foo', (0, 0))) self.assertEqual([('carbon.foo', (0, 0))], result) def test_unaggregated_metrics_pass_through(self): RuleManager.rules = [self.sample_aggregation_rule] result = list(self.processor.process('carbon.foo', (0, 0))) self.assertEqual([('carbon.foo', (0, 0))], result) def test_aggregation_rule_checked(self): RuleManager.rules = [self.sample_aggregation_rule] with patch.object(self.sample_aggregation_rule, 'get_aggregate_metric'): list(self.processor.process('carbon.foo', (0, 0))) self.sample_aggregation_rule.get_aggregate_metric.assert_called_once_with('carbon.foo') def test_new_buffer_configured(self): RuleManager.rules = [self.sample_aggregation_rule] list(self.processor.process('carbon.foo', (0, 0))) values_buffer = BufferManager.get_buffer('carbon.foo.sum') self.assertTrue(values_buffer.configured) self.assertEqual(1, values_buffer.aggregation_frequency) self.assertEqual(sum, values_buffer.aggregation_func) def test_buffer_receives_value(self): RuleManager.rules = [self.sample_aggregation_rule] list(self.processor.process('carbon.foo', (0, 0))) values_buffer = BufferManager.get_buffer('carbon.foo.sum') self.assertEqual([0], values_buffer.interval_buffers[0].values) def test_metric_not_passed_through_when_aggregate_overwrites(self): RuleManager.rules = [self.sample_overwriting_aggregation_rule] result = list(self.processor.process('carbon.foo', (0, 0))) self.assertEqual([], result) graphite-carbon-1.1.7/lib/carbon/tests/test_aggregator_rules.py0000644000175000017500000000312013633714656023400 0ustar gringrinimport unittest from carbon.aggregator.rules import AggregationRule class AggregationRuleTest(unittest.TestCase): def test_inclusive_regexes(self): """ Test case for https://github.com/graphite-project/carbon/pull/120 Consider the two rules: aggregated.hist.p99 (10) = avg hosts.*.hist.p99 aggregated.hist.p999 (10) = avg hosts.*.hist.p999 Before the abovementioned patch the second rule would be treated as expected but the first rule would lead to an aggregated metric aggregated.hist.p99 which would in fact be equivalent to avgSeries(hosts.*.hist.p99,hosts.*.hist.p999). """ method = 'avg' frequency = 10 input_pattern = 'hosts.*.hist.p99' output_pattern = 'aggregated.hist.p99' rule99 = AggregationRule(input_pattern, output_pattern, method, frequency) input_pattern = 'hosts.*.hist.p999' output_pattern = 'aggregated.hist.p999' rule999 = AggregationRule(input_pattern, output_pattern, method, frequency) self.assertEqual(rule99.get_aggregate_metric('hosts.abc.hist.p99'), 'aggregated.hist.p99') self.assertEqual(rule99.get_aggregate_metric('hosts.abc.hist.p999'), None) self.assertEqual(rule999.get_aggregate_metric('hosts.abc.hist.p99'), None) self.assertEqual(rule999.get_aggregate_metric('hosts.abc.hist.p999'), 'aggregated.hist.p999') graphite-carbon-1.1.7/lib/carbon/tests/test_cache.py0000644000175000017500000002653113633714656021122 0ustar gringrinimport time from unittest import TestCase from mock import Mock, PropertyMock, patch from carbon.cache import ( MetricCache, _MetricCache, DrainStrategy, MaxStrategy, RandomStrategy, SortedStrategy, TimeSortedStrategy ) class MetricCacheTest(TestCase): def setUp(self): settings = { 'MAX_CACHE_SIZE': float('inf'), 'CACHE_SIZE_LOW_WATERMARK': float('inf') } self._settings_patch = patch.dict('carbon.conf.settings', settings) self._settings_patch.start() self.strategy_mock = Mock(spec=DrainStrategy) self.metric_cache = _MetricCache(self.strategy_mock) def tearDown(self): self._settings_patch.stop() def test_constructor(self): settings = { 'CACHE_WRITE_STRATEGY': 'max', } settings_patch = patch.dict('carbon.conf.settings', settings) settings_patch.start() cache = MetricCache() self.assertNotEqual(cache, None) self.assertTrue(isinstance(cache.strategy, MaxStrategy)) def test_cache_is_a_dict(self): self.assertTrue(issubclass(_MetricCache, dict)) def test_initial_size(self): self.assertEqual(0, self.metric_cache.size) def test_store_new_metric(self): self.metric_cache.store('foo', (123456, 1.0)) self.assertEqual(1, self.metric_cache.size) self.assertEqual([(123456, 1.0)], list(self.metric_cache['foo'].items())) def test_store_multiple_datapoints(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.assertEqual(2, self.metric_cache.size) result = self.metric_cache['foo'].items() self.assertTrue((123456, 1.0) in result) self.assertTrue((123457, 2.0) in result) def test_store_duplicate_timestamp(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123456, 2.0)) self.assertEqual(1, self.metric_cache.size) self.assertEqual([(123456, 2.0)], list(self.metric_cache['foo'].items())) def test_store_checks_fullness(self): is_full_mock = PropertyMock() with patch.object(_MetricCache, 'is_full', is_full_mock): with patch('carbon.cache.events'): metric_cache = _MetricCache() metric_cache.store('foo', (123456, 1.0)) self.assertEqual(1, is_full_mock.call_count) def test_store_on_full_triggers_events(self): is_full_mock = PropertyMock(return_value=True) with patch.object(_MetricCache, 'is_full', is_full_mock): with patch('carbon.cache.events') as events_mock: self.metric_cache.store('foo', (123456, 1.0)) events_mock.cacheFull.assert_called_with() def test_pop_multiple_datapoints(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) result = self.metric_cache.pop('foo') self.assertTrue((123456, 1.0) in result) self.assertTrue((123457, 2.0) in result) def test_pop_reduces_size(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.pop('foo') self.assertEqual(0, self.metric_cache.size) def test_pop_triggers_space_check(self): with patch.object(self.metric_cache, '_check_available_space') as check_space_mock: self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.pop('foo') self.assertEqual(1, check_space_mock.call_count) def test_pop_triggers_space_event(self): with patch('carbon.state.cacheTooFull', new=Mock(return_value=True)): with patch('carbon.cache.events') as events_mock: self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.pop('foo') events_mock.cacheSpaceAvailable.assert_called_with() def test_pop_returns_sorted_timestamps(self): self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('foo', (123458, 3.0)) self.metric_cache.store('foo', (123456, 1.0)) result = self.metric_cache.pop('foo') expected = [(123456, 1.0), (123457, 2.0), (123458, 3.0)] self.assertEqual(expected, result) def test_pop_raises_on_missing(self): self.assertRaises(KeyError, self.metric_cache.pop, 'foo') def test_get_datapoints(self): self.metric_cache.store('foo', (123456, 1.0)) self.assertEqual([(123456, 1.0)], self.metric_cache.get_datapoints('foo')) def test_get_datapoints_doesnt_pop(self): self.metric_cache.store('foo', (123456, 1.0)) self.assertEqual([(123456, 1.0)], self.metric_cache.get_datapoints('foo')) self.assertEqual(1, self.metric_cache.size) self.assertEqual([(123456, 1.0)], self.metric_cache.get_datapoints('foo')) def test_get_datapoints_returns_empty_on_missing(self): self.assertEqual([], self.metric_cache.get_datapoints('foo')) def test_get_datapoints_returns_sorted_timestamps(self): self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('foo', (123458, 3.0)) self.metric_cache.store('foo', (123456, 1.0)) result = self.metric_cache.get_datapoints('foo') expected = [(123456, 1.0), (123457, 2.0), (123458, 3.0)] self.assertEqual(expected, result) def test_drain_metric_respects_strategy(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('bar', (123456, 1.0)) self.metric_cache.store('baz', (123456, 1.0)) self.strategy_mock.return_value.choose_item.side_effect = ['bar', 'baz', 'foo'] self.assertEqual('bar', self.metric_cache.drain_metric()[0]) self.assertEqual('baz', self.metric_cache.drain_metric()[0]) self.assertEqual('foo', self.metric_cache.drain_metric()[0]) def test_drain_metric_works_without_strategy(self): metric_cache = _MetricCache() # No strategy metric_cache.store('foo', (123456, 1.0)) self.assertEqual('foo', metric_cache.drain_metric()[0]) def test_is_full_short_circuits_on_inf(self): with patch.object(self.metric_cache, 'size') as size_mock: self.metric_cache.is_full size_mock.assert_not_called() def test_is_full(self): self._settings_patch.values['MAX_CACHE_SIZE'] = 2.0 self._settings_patch.start() with patch('carbon.cache.events'): self.assertFalse(self.metric_cache.is_full) self.metric_cache.store('foo', (123456, 1.0)) self.assertFalse(self.metric_cache.is_full) self.metric_cache.store('foo', (123457, 1.0)) self.assertTrue(self.metric_cache.is_full) def test_counts_one_datapoint(self): self.metric_cache.store('foo', (123456, 1.0)) self.assertEqual([('foo', 1)], self.metric_cache.counts) def test_counts_two_datapoints(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.assertEqual([('foo', 2)], self.metric_cache.counts) def test_counts_multiple_datapoints(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('bar', (123458, 3.0)) self.assertTrue(('foo', 2) in self.metric_cache.counts) self.assertTrue(('bar', 1) in self.metric_cache.counts) class DrainStrategyTest(TestCase): def setUp(self): self.metric_cache = _MetricCache() def test_max_strategy(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('foo', (123458, 3.0)) self.metric_cache.store('bar', (123459, 4.0)) self.metric_cache.store('bar', (123460, 5.0)) self.metric_cache.store('baz', (123461, 6.0)) max_strategy = MaxStrategy(self.metric_cache) # foo has 3 self.assertEqual('foo', max_strategy.choose_item()) # add 2 more 'bar' for 4 total self.metric_cache.store('bar', (123462, 8.0)) self.metric_cache.store('bar', (123463, 9.0)) self.assertEqual('bar', max_strategy.choose_item()) self.metric_cache.pop('foo') self.metric_cache.pop('bar') self.assertEqual('baz', max_strategy.choose_item()) def test_sorted_strategy_static_cache(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('foo', (123458, 3.0)) self.metric_cache.store('bar', (123459, 4.0)) self.metric_cache.store('bar', (123460, 5.0)) self.metric_cache.store('baz', (123461, 6.0)) sorted_strategy = SortedStrategy(self.metric_cache) # In order from most to least self.assertEqual('foo', sorted_strategy.choose_item()) self.assertEqual('bar', sorted_strategy.choose_item()) self.assertEqual('baz', sorted_strategy.choose_item()) def test_sorted_strategy_changing_sizes(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('foo', (123458, 3.0)) self.metric_cache.store('bar', (123459, 4.0)) self.metric_cache.store('bar', (123460, 5.0)) self.metric_cache.store('baz', (123461, 6.0)) sorted_strategy = SortedStrategy(self.metric_cache) # In order from most to least foo, bar, baz self.assertEqual('foo', sorted_strategy.choose_item()) # 'baz' gets 2 more, now greater than 'bar' self.metric_cache.store('baz', (123461, 6.0)) self.metric_cache.store('baz', (123461, 6.0)) # But 'bar' is popped anyway, because sort has already happened self.assertEqual('bar', sorted_strategy.choose_item()) self.assertEqual('baz', sorted_strategy.choose_item()) # Sort happens again self.assertEqual('foo', sorted_strategy.choose_item()) self.assertEqual('bar', sorted_strategy.choose_item()) self.assertEqual('baz', sorted_strategy.choose_item()) def test_time_sorted_strategy(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('foo', (123457, 2.0)) self.metric_cache.store('foo', (123458, 3.0)) self.metric_cache.store('bar', (123459, 4.0)) self.metric_cache.store('bar', (123460, 5.0)) self.metric_cache.store('baz', (123461, 6.0)) time_sorted_strategy = TimeSortedStrategy(self.metric_cache) # In order: foo, bar, baz self.assertEqual('foo', time_sorted_strategy.choose_item()) # 'baz' gets older points. self.metric_cache.store('baz', (123450, 6.0)) self.metric_cache.store('baz', (123451, 6.0)) # But 'bar' is popped anyway, because sort has already happened self.assertEqual('bar', time_sorted_strategy.choose_item()) self.assertEqual('baz', time_sorted_strategy.choose_item()) # Sort happens again self.assertEqual('baz', time_sorted_strategy.choose_item()) self.assertEqual('foo', time_sorted_strategy.choose_item()) self.assertEqual('bar', time_sorted_strategy.choose_item()) def test_time_sorted_strategy_min_lag(self): settings = { 'MIN_TIMESTAMP_LAG': 5, } settings_patch = patch.dict('carbon.conf.settings', settings) settings_patch.start() now = time.time() self.metric_cache.store('old', (now - 10, 1.0)) self.metric_cache.store('new', (now, 2.0)) time_sorted_strategy = TimeSortedStrategy(self.metric_cache) self.assertEqual('old', time_sorted_strategy.choose_item()) self.metric_cache.pop('old') self.assertEqual(None, time_sorted_strategy.choose_item()) class RandomStrategyTest(TestCase): def setUp(self): self.metric_cache = _MetricCache() def test_random_strategy(self): self.metric_cache.store('foo', (123456, 1.0)) self.metric_cache.store('bar', (123457, 2.0)) self.metric_cache.store('baz', (123458, 3.0)) strategy = RandomStrategy(self.metric_cache) for _i in range(3): item = strategy.choose_item() self.assertTrue(item in self.metric_cache) self.metric_cache.pop(item) graphite-carbon-1.1.7/lib/carbon/tests/test_client.py0000644000175000017500000001736713633714656021344 0ustar gringrinimport carbon.client as carbon_client from carbon.client import ( CarbonPickleClientFactory, CarbonPickleClientProtocol, CarbonLineClientProtocol, CarbonClientManager, RelayProcessor ) from carbon.routers import DatapointRouter from carbon.tests.util import TestSettings import carbon.service # NOQA from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.base import DelayedCall from twisted.internet.task import deferLater from twisted.trial.unittest import TestCase from twisted.test.proto_helpers import StringTransport from mock import Mock, patch from pickle import loads as pickle_loads from struct import unpack, calcsize INT32_FORMAT = '!I' INT32_SIZE = calcsize(INT32_FORMAT) def decode_sent(data): pickle_size = unpack(INT32_FORMAT, data[:INT32_SIZE])[0] return pickle_loads(data[INT32_SIZE:INT32_SIZE + pickle_size]) class BroadcastRouter(DatapointRouter): def __init__(self, destinations=[]): self.destinations = set(destinations) def addDestination(self, destination): self.destinations.append(destination) def removeDestination(self, destination): self.destinations.discard(destination) def getDestinations(self, key): for destination in self.destinations: yield destination class ConnectedCarbonClientProtocolTest(TestCase): def setUp(self): self.router_mock = Mock(spec=DatapointRouter) carbon_client.settings = TestSettings() # reset to defaults factory = CarbonPickleClientFactory(('127.0.0.1', 2003, 'a'), self.router_mock) self.protocol = factory.buildProtocol(('127.0.0.1', 2003)) self.transport = StringTransport() self.protocol.makeConnection(self.transport) def test_send_datapoint(self): def assert_sent(): sent_data = self.transport.value() sent_datapoints = decode_sent(sent_data) self.assertEqual([datapoint], sent_datapoints) datapoint = ('foo.bar', (1000000000, 1.0)) self.protocol.sendDatapoint(*datapoint) return deferLater(reactor, 0.1, assert_sent) class CarbonLineClientProtocolTest(TestCase): def setUp(self): self.protocol = CarbonLineClientProtocol() self.protocol.sendLine = Mock() def test_send_datapoints(self): calls = [ (('foo.bar', (1000000000, 1.0)), b'foo.bar 1 1000000000'), (('foo.bar', (1000000000, 1.1)), b'foo.bar 1.1 1000000000'), (('foo.bar', (1000000000, 1.123456789123)), b'foo.bar 1.1234567891 1000000000'), (('foo.bar', (1000000000, 1)), b'foo.bar 1 1000000000'), (('foo.bar', (1000000000, 1.498566361088E12)), b'foo.bar 1498566361088 1000000000'), ] i = 0 for (datapoint, expected_line_to_send) in calls: i += 1 self.protocol._sendDatapointsNow([datapoint]) self.assertEqual(self.protocol.sendLine.call_count, i) self.protocol.sendLine.assert_called_with(expected_line_to_send) class CarbonClientFactoryTest(TestCase): def setUp(self): self.router_mock = Mock(spec=DatapointRouter) self.protocol_mock = Mock(spec=CarbonPickleClientProtocol) self.protocol_patch = patch( 'carbon.client.CarbonPickleClientProtocol', new=Mock(return_value=self.protocol_mock)) self.protocol_patch.start() carbon_client.settings = TestSettings() self.factory = CarbonPickleClientFactory(('127.0.0.1', 2003, 'a'), self.router_mock) self.connected_factory = CarbonPickleClientFactory(('127.0.0.1', 2003, 'a'), self.router_mock) self.connected_factory.buildProtocol(None) self.connected_factory.started = True def tearDown(self): if self.factory.deferSendPending and self.factory.deferSendPending.active(): self.factory.deferSendPending.cancel() self.protocol_patch.stop() def test_schedule_send_schedules_call_to_send_queued(self): self.factory.scheduleSend() self.assertIsInstance(self.factory.deferSendPending, DelayedCall) self.assertTrue(self.factory.deferSendPending.active()) def test_schedule_send_ignores_already_scheduled(self): self.factory.scheduleSend() expected_fire_time = self.factory.deferSendPending.getTime() self.factory.scheduleSend() self.assertTrue(expected_fire_time, self.factory.deferSendPending.getTime()) def test_send_queued_should_noop_if_not_connected(self): self.factory.scheduleSend() self.assertFalse(self.protocol_mock.sendQueued.called) def test_send_queued_should_call_protocol_send_queued(self): self.connected_factory.sendQueued() self.protocol_mock.sendQueued.assert_called_once_with() class CarbonClientManagerTest(TestCase): timeout = 1.0 def setUp(self): self.router_mock = Mock(spec=DatapointRouter) self.factory_mock = Mock(spec=CarbonPickleClientFactory) self.client_mgr = CarbonClientManager(self.router_mock) self.client_mgr.createFactory = lambda dest: self.factory_mock(dest, self.router_mock) def test_start_service_installs_sig_ignore(self): from signal import SIGHUP, SIG_IGN with patch('signal.signal', new=Mock()) as signal_mock: self.client_mgr.startService() signal_mock.assert_called_once_with(SIGHUP, SIG_IGN) def test_start_service_starts_factory_connect(self): factory_mock = Mock(spec=CarbonPickleClientFactory) factory_mock.started = False self.client_mgr.client_factories[('127.0.0.1', 2003, 'a')] = factory_mock self.client_mgr.startService() factory_mock.startConnecting.assert_called_once_with() def test_stop_service_waits_for_clients_to_disconnect(self): dest = ('127.0.0.1', 2003, 'a') self.client_mgr.startService() self.client_mgr.startClient(dest) disconnect_deferred = Deferred() reactor.callLater(0.1, disconnect_deferred.callback, 0) self.factory_mock.return_value.disconnect.return_value = disconnect_deferred return self.client_mgr.stopService() def test_start_client_instantiates_client_factory(self): dest = ('127.0.0.1', 2003, 'a') self.client_mgr.startClient(dest) self.factory_mock.assert_called_once_with(dest, self.router_mock) def test_start_client_ignores_duplicate(self): dest = ('127.0.0.1', 2003, 'a') self.client_mgr.startClient(dest) self.client_mgr.startClient(dest) self.factory_mock.assert_called_once_with(dest, self.router_mock) def test_start_client_starts_factory_if_running(self): dest = ('127.0.0.1', 2003, 'a') self.client_mgr.startService() self.client_mgr.startClient(dest) self.factory_mock.return_value.startConnecting.assert_called_once_with() def test_start_client_adds_destination_to_router(self): dest = ('127.0.0.1', 2003, 'a') self.client_mgr.startClient(dest) self.router_mock.addDestination.assert_called_once_with(dest) def test_stop_client_removes_destination_from_router(self): dest = ('127.0.0.1', 2003, 'a') self.client_mgr.startClient(dest) self.client_mgr.stopClient(dest) self.router_mock.removeDestination.assert_called_once_with(dest) class RelayProcessorTest(TestCase): timeout = 1.0 def setUp(self): carbon_client.settings = TestSettings() # reset to defaults self.client_mgr_mock = Mock(spec=CarbonClientManager) self.client_mgr_patch = patch( 'carbon.state.client_manager', new=self.client_mgr_mock) self.client_mgr_patch.start() def tearDown(self): self.client_mgr_patch.stop() def test_relay_normalized(self): carbon_client.settings.TAG_RELAY_NORMALIZED = True relayProcessor = RelayProcessor() relayProcessor.process('my.metric;foo=a;bar=b', (0.0, 0.0)) self.client_mgr_mock.sendDatapoint.assert_called_once_with('my.metric;bar=b;foo=a', (0.0, 0.0)) def test_relay_unnormalized(self): carbon_client.settings.TAG_RELAY_NORMALIZED = False relayProcessor = RelayProcessor() relayProcessor.process('my.metric;foo=a;bar=b', (0.0, 0.0)) self.client_mgr_mock.sendDatapoint.assert_called_once_with('my.metric;foo=a;bar=b', (0.0, 0.0)) graphite-carbon-1.1.7/lib/carbon/tests/test_conf.py0000644000175000017500000003717313633714656021010 0ustar gringrinimport os from tempfile import mkdtemp, mkstemp from shutil import rmtree from os import makedirs from os.path import dirname, join from unittest import TestCase from carbon.conf import get_default_parser, parse_options, read_config from carbon.exceptions import CarbonConfigException class FakeParser(object): def __init__(self): self.called = [] def parse_args(self, args): return object(), args def print_usage(self): self.called.append("print_usage") class FakeOptions(object): def __init__(self, **kwargs): self.__dict__.update(kwargs) def __getitem__(self, name): return self.__dict__[name] def __setitem__(self, name, value): self.__dict__[name] = value class DefaultParserTest(TestCase): def test_default_parser(self): """Check default parser settings.""" parser = get_default_parser() self.assertTrue(parser.has_option("--debug")) self.assertEqual(None, parser.defaults["debug"]) self.assertTrue(parser.has_option("--profile")) self.assertEqual(None, parser.defaults["profile"]) self.assertTrue(parser.has_option("--pidfile")) self.assertEqual(None, parser.defaults["pidfile"]) self.assertTrue(parser.has_option("--config")) self.assertEqual(None, parser.defaults["config"]) self.assertTrue(parser.has_option("--logdir")) self.assertEqual(None, parser.defaults["logdir"]) self.assertTrue(parser.has_option("--instance")) self.assertEqual("a", parser.defaults["instance"]) class ParseOptionsTest(TestCase): def test_no_args_prints_usage_and_exit(self): """ If no arguments are provided, the usage help will be printed and a SystemExit exception will be raised. """ parser = FakeParser() self.assertRaises(SystemExit, parse_options, parser, ()) self.assertEqual(["print_usage"], parser.called) def test_no_valid_args_prints_usage_and_exit(self): """ If an argument which isn't a valid command was provided, 'print_usage' will be called and a SystemExit exception will be raised. """ parser = FakeParser() self.assertRaises(SystemExit, parse_options, parser, ("bazinga!",)) self.assertEqual(["print_usage"], parser.called) def test_valid_args(self): """ If a valid argument is provided, it will be returned along with options. """ parser = FakeParser() options, args = parser.parse_args(("start",)) self.assertEqual(("start",), args) class ReadConfigTest(TestCase): def makeFile(self, content=None, basename=None, dirname=None): """ Create a temporary file with content Deletes the file after tests """ if basename is not None: path = join(dirname, basename) else: fd, path = mkstemp(dir=dirname) os.close(fd) self.addCleanup(os.unlink, path) if content is not None: with open(path, "w") as f: f.write(content) return path def test_root_dir_is_required(self): """ At minimum, the caller must provide a 'ROOT_DIR' setting. """ try: read_config("carbon-foo", FakeOptions(config=None)) except CarbonConfigException as e: self.assertEqual("Either ROOT_DIR or GRAPHITE_ROOT " "needs to be provided.", str(e)) else: self.fail("Did not raise exception.") def test_config_is_not_required(self): """ If the '--config' option is not provided, it defaults to ROOT_DIR/conf/carbon.conf. """ root_dir = mkdtemp() self.addCleanup(rmtree, root_dir) conf_dir = join(root_dir, "conf") makedirs(conf_dir) self.makeFile(content="[foo]", basename="carbon.conf", dirname=conf_dir) options = FakeOptions(config=None, instance=None, pidfile=None, logdir=None) read_config("carbon-foo", options, ROOT_DIR=root_dir) self.assertEqual(join(root_dir, "conf", "carbon.conf"), options["config"]) def test_config_dir_from_environment(self): """ If the 'GRAPHITE_CONFIG_DIR' variable is set in the environment, then 'CONFIG_DIR' will be set to that directory. """ root_dir = mkdtemp() self.addCleanup(rmtree, root_dir) conf_dir = join(root_dir, "configs", "production") makedirs(conf_dir) self.makeFile(content="[foo]", basename="carbon.conf", dirname=conf_dir) orig_value = os.environ.get("GRAPHITE_CONF_DIR", None) if orig_value is not None: self.addCleanup(os.environ.__setitem__, "GRAPHITE_CONF_DIR", orig_value) else: self.addCleanup(os.environ.__delitem__, "GRAPHITE_CONF_DIR") os.environ["GRAPHITE_CONF_DIR"] = conf_dir settings = read_config("carbon-foo", FakeOptions(config=None, instance=None, pidfile=None, logdir=None), ROOT_DIR=root_dir) self.assertEqual(conf_dir, settings.CONF_DIR) def test_conf_dir_defaults_to_config_dirname(self): """ The 'CONF_DIR' setting defaults to the parent directory of the provided configuration file. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(dirname(config), settings.CONF_DIR) def test_storage_dir_relative_to_root_dir(self): """ The 'STORAGE_DIR' setting defaults to the 'storage' directory relative to the 'ROOT_DIR' setting. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("foo", "storage"), settings.STORAGE_DIR) def test_log_dir_relative_to_storage_dir(self): """ The 'LOG_DIR' setting defaults to a program-specific directory relative to the 'STORAGE_DIR' setting. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("foo", "storage", "log", "carbon-foo"), settings.LOG_DIR) def test_log_dir_relative_to_provided_storage_dir(self): """ Providing a different 'STORAGE_DIR' in defaults overrides the default of being relative to 'ROOT_DIR'. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo", STORAGE_DIR="bar") self.assertEqual(join("bar", "log", "carbon-foo"), settings.LOG_DIR) def test_log_dir_for_instance_relative_to_storage_dir(self): """ The 'LOG_DIR' setting defaults to a program-specific directory relative to the 'STORAGE_DIR' setting. In the case of an instance, the instance name is appended to the directory. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance="x", pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("foo", "storage", "log", "carbon-foo", "carbon-foo-x"), settings.LOG_DIR) def test_log_dir_for_instance_relative_to_provided_storage_dir(self): """ Providing a different 'STORAGE_DIR' in defaults overrides the default of being relative to 'ROOT_DIR'. In the case of an instance, the instance name is appended to the directory. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance="x", pidfile=None, logdir=None), ROOT_DIR="foo", STORAGE_DIR="bar") self.assertEqual(join("bar", "log", "carbon-foo", "carbon-foo-x"), settings.LOG_DIR) def test_pidfile_relative_to_storage_dir(self): """ The 'pidfile' setting defaults to a program-specific filename relative to the 'STORAGE_DIR' setting. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("foo", "storage", "carbon-foo.pid"), settings.pidfile) def test_pidfile_in_options_has_precedence(self): """ The 'pidfile' option from command line overrides the default setting. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile="foo.pid", logdir=None), ROOT_DIR="foo") self.assertEqual("foo.pid", settings.pidfile) def test_pidfile_for_instance_in_options_has_precedence(self): """ The 'pidfile' option from command line overrides the default setting for the instance, if one is specified. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance="x", pidfile="foo.pid", logdir=None), ROOT_DIR="foo") self.assertEqual("foo.pid", settings.pidfile) def test_storage_dir_as_provided(self): """ Providing a 'STORAGE_DIR' in defaults overrides the root-relative default. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo", STORAGE_DIR="bar") self.assertEqual("bar", settings.STORAGE_DIR) def test_log_dir_as_provided(self): """ Providing a 'LOG_DIR' in defaults overrides the storage-relative default. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo", STORAGE_DIR="bar", LOG_DIR='baz') self.assertEqual("baz", settings.LOG_DIR) def test_log_dir_from_options(self): """ Providing a 'LOG_DIR' in the command line overrides the storage-relative default. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir="baz"), ROOT_DIR="foo") self.assertEqual("baz", settings.LOG_DIR) def test_log_dir_for_instance_from_options(self): """ Providing a 'LOG_DIR' in the command line overrides the storage-relative default for the instance. """ config = self.makeFile(content="[foo]") settings = read_config( "carbon-foo", FakeOptions(config=config, instance="x", pidfile=None, logdir="baz"), ROOT_DIR="foo") self.assertEqual("baz", settings.LOG_DIR) def test_storage_dir_from_config(self): """ Providing a 'STORAGE_DIR' in the configuration file overrides the root-relative default. """ config = self.makeFile(content="[foo]\nSTORAGE_DIR = bar") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual("bar", settings.STORAGE_DIR) def test_log_dir_from_config(self): """ Providing a 'LOG_DIR' in the configuration file overrides the storage-relative default. """ config = self.makeFile(content="[foo]\nLOG_DIR = baz") settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual("baz", settings.LOG_DIR) def test_log_dir_from_instance_config(self): """ Providing a 'LOG_DIR' for the specific instance in the configuration file overrides the storage-relative default. The actual value will have the instance name appended to it and ends with a forward slash. """ config = self.makeFile( content=("[foo]\nLOG_DIR = baz\n" "[foo:x]\nLOG_DIR = boo")) settings = read_config( "carbon-foo", FakeOptions(config=config, instance="x", pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual("boo/carbon-foo-x", settings.LOG_DIR) def test_pid_dir_depends_on_storage_dir(self): """ Tests 'STORAGE_DIR' dependency 'PID_DIR' """ config = self.makeFile( content=("[foo]\n" "STORAGE_DIR = bar")) settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual("bar", settings.PID_DIR) def test_log_dir_depends_on_storage_dir(self): """ Tests 'STORAGE_DIR' dependency 'LOG_DIR' """ config = self.makeFile( content=("[foo]\n" "STORAGE_DIR = bar")) settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("bar", "log", "carbon-foo"), settings.LOG_DIR) def test_local_data_dir_depends_on_storage_dir(self): """ Tests 'STORAGE_DIR' dependency 'LOCAL_DATA_DIR' """ config = self.makeFile( content=("[foo]\n" "STORAGE_DIR = bar")) settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("bar", "whisper"), settings.LOCAL_DATA_DIR) def test_whitelists_dir_depends_on_storage_dir(self): """ Tests 'STORAGE_DIR' dependency 'WHITELISTS_DIR' """ config = self.makeFile( content=("[foo]\n" "STORAGE_DIR = bar")) settings = read_config( "carbon-foo", FakeOptions(config=config, instance=None, pidfile=None, logdir=None), ROOT_DIR="foo") self.assertEqual(join("bar", "lists"), settings.WHITELISTS_DIR) graphite-carbon-1.1.7/lib/carbon/tests/test_database.py0000644000175000017500000001412513633714656021617 0ustar gringrinimport os from unittest import TestCase from mock import patch from os.path import exists import shutil from carbon.tests.util import TestSettings from carbon.database import WhisperDatabase, CeresDatabase class WhisperDatabaseTest(TestCase): def setUp(self): self._sep_patch = patch.object(os.path, 'sep', "/") self._sep_patch.start() def tearDown(self): self._sep_patch.stop() def test_getFilesystemPath(self): settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' database = WhisperDatabase(settings) result = database.getFilesystemPath('stats.example.counts') self.assertEqual(result, '/tmp/stats/example/counts.wsp') # nosec def test_getTaggedFilesystemPath(self): metric = 'stats.example.counts;tag1=value1' settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' settings['TAG_HASH_FILENAMES'] = False database = WhisperDatabase(settings) result = database.getFilesystemPath(metric) self.assertEqual( result, '/tmp/_tagged/872/252/stats_DOT_example_DOT_counts;tag1=value1.wsp') # nosec result = database.exists(metric) self.assertEqual(result, False) def test_getTaggedFilesystemPathHashed(self): metric = 'stats.example.counts;tag1=value1' settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' settings['TAG_HASH_FILENAMES'] = True database = WhisperDatabase(settings) result = database.getFilesystemPath(metric) self.assertEqual( result, '/tmp/_tagged/872/252/' + # nosec '872252dcead671982862f82a3b440f02aa8f525dd6d0f2921de0dc2b3e874ad0.wsp') result = database.exists(metric) self.assertEqual(result, False) def test_migrateTaggedFilesystemPathHashed(self): metric = 'stats.example.counts;tag1=value1' settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' settings['TAG_HASH_FILENAMES'] = False database = WhisperDatabase(settings) result = database.exists(metric) self.assertEqual(result, False) old_path = database.getFilesystemPath(metric) self.assertEqual( old_path, '/tmp/_tagged/872/252/stats_DOT_example_DOT_counts;tag1=value1.wsp') # nosec self.assertEqual(exists(old_path), False) result = database.create(metric, [(60, 60)], 0.5, 'average') self.assertEqual(exists(old_path), True) result = database.exists(metric) self.assertEqual(result, True) settings['TAG_HASH_FILENAMES'] = True database = WhisperDatabase(settings) hashed_path = database.getFilesystemPath(metric) self.assertEqual( hashed_path, '/tmp/_tagged/872/252/' + # nosec '872252dcead671982862f82a3b440f02aa8f525dd6d0f2921de0dc2b3e874ad0.wsp') self.assertEqual(exists(hashed_path), False) result = database.exists(metric) self.assertEqual(result, True) self.assertEqual(exists(old_path), False) self.assertEqual(exists(hashed_path), True) os.remove(hashed_path) class CeresDatabaseTest(TestCase): def setUp(self): self._sep_patch = patch.object(os.path, 'sep', "/") self._sep_patch.start() def tearDown(self): self._sep_patch.stop() def test_getFilesystemPath(self): settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' database = CeresDatabase(settings) result = database.getFilesystemPath('stats.example.counts') self.assertEqual(result, '/tmp/stats/example/counts') # nosec def test_getTaggedFilesystemPath(self): metric = 'stats.example.counts;tag1=value1' settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' settings['TAG_HASH_FILENAMES'] = False database = CeresDatabase(settings) result = database.getFilesystemPath(metric) self.assertEqual( result, '/tmp/_tagged/872/252/stats_DOT_example_DOT_counts;tag1=value1') # nosec result = database.exists(metric) self.assertEqual(result, False) def test_getTaggedFilesystemPathHashed(self): metric = 'stats.example.counts;tag1=value1' settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' settings['TAG_HASH_FILENAMES'] = True database = CeresDatabase(settings) result = database.getFilesystemPath(metric) self.assertEqual( result, '/tmp/_tagged/872/252/' + # nosec '872252dcead671982862f82a3b440f02aa8f525dd6d0f2921de0dc2b3e874ad0') result = database.exists(metric) self.assertEqual(result, False) def test_migrateTaggedFilesystemPathHashed(self): metric = 'stats.example.counts;tag1=value1' settings = TestSettings() settings['LOCAL_DATA_DIR'] = '/tmp/' settings['TAG_HASH_FILENAMES'] = False database = CeresDatabase(settings) result = database.exists(metric) self.assertEqual(result, False) old_path = database.getFilesystemPath(metric) self.assertEqual( old_path, '/tmp/_tagged/872/252/stats_DOT_example_DOT_counts;tag1=value1') # nosec self.assertEqual(exists(old_path), False) result = database.create(metric, [(60, 60)], 0.5, 'average') self.assertEqual(exists(old_path), True) result = database.exists(metric) self.assertEqual(result, True) settings['TAG_HASH_FILENAMES'] = True database = CeresDatabase(settings) hashed_path = database.getFilesystemPath(metric) self.assertEqual( hashed_path, '/tmp/_tagged/872/252/' + # nosec '872252dcead671982862f82a3b440f02aa8f525dd6d0f2921de0dc2b3e874ad0') self.assertEqual(exists(hashed_path), False) result = database.exists(metric) self.assertEqual(result, True) self.assertEqual(exists(old_path), False) self.assertEqual(exists(hashed_path), True) shutil.rmtree(hashed_path) graphite-carbon-1.1.7/lib/carbon/tests/test_hashing.py0000644000175000017500000002127013633714656021473 0ustar gringrinimport unittest from carbon.hashing import ConsistentHashRing class HashIntegrityTest(unittest.TestCase): def test_2_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(2): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_3_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(3): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_4_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(4): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_5_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(5): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_6_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(6): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_7_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(7): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_8_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(8): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_9_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([]) for n in range(9): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_10_get_node(self): """Trigger bisect on identical first key, see: issues/766""" ring = ConsistentHashRing([], replica_count=1) ring.add_node(("1", "1")) n = ring.get_node("('1', '1'):0") self.assertEqual(('1', '1'), n) def test_11_get_nodes(self): """Trigger bisect on identical first key, see: issues/766""" ring = ConsistentHashRing([], replica_count=1) ring.add_node(("1", "1")) n = ring.get_nodes("('1', '1'):0") self.assertEqual([('1', '1')], list(n)) class FNVHashIntegrityTest(unittest.TestCase): def test_2_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(2): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_3_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(3): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_4_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(4): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_5_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(5): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_6_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(6): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_7_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(7): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_8_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(8): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) def test_9_node_positional_itegrity(self): """Make a cluster, verify we don't have positional collisions""" ring = ConsistentHashRing([], hash_type='fnv1a_ch') for n in range(9): ring.add_node(("192.168.10.%s" % str(10 + n), "%s" % str(10 + n))) self.assertEqual( len([n[0] for n in ring.ring]), len(set([n[0] for n in ring.ring]))) class ConsistentHashRingTestFNV1A(unittest.TestCase): def test_chr_compute_ring_position_fnv1a(self): hosts = [("127.0.0.1", "ba603c36342304ed77953f84ac4d357b"), ("127.0.0.2", "5dd63865534f84899c6e5594dba6749a"), ("127.0.0.3", "866a18b81f2dc4649517a1df13e26f28")] hashring = ConsistentHashRing(hosts, hash_type='fnv1a_ch') self.assertEqual(hashring.compute_ring_position('hosts.worker1.cpu'), 59573) self.assertEqual(hashring.compute_ring_position('hosts.worker1.load'), 57163) self.assertEqual(hashring.compute_ring_position('hosts.worker2.cpu'), 35749) self.assertEqual(hashring.compute_ring_position('hosts.worker2.network'), 43584) self.assertEqual(hashring.compute_ring_position('hosts.worker3.cpu'), 12600) self.assertEqual(hashring.compute_ring_position('hosts.worker3.irq'), 10052) def test_chr_get_node_fnv1a(self): hosts = [("127.0.0.1", "ba603c36342304ed77953f84ac4d357b"), ("127.0.0.2", "5dd63865534f84899c6e5594dba6749a"), ("127.0.0.3", "866a18b81f2dc4649517a1df13e26f28")] hashring = ConsistentHashRing(hosts, hash_type='fnv1a_ch') self.assertEqual(hashring.get_node('hosts.worker1.cpu'), ('127.0.0.1', 'ba603c36342304ed77953f84ac4d357b')) self.assertEqual(hashring.get_node('hosts.worker2.cpu'), ('127.0.0.3', '866a18b81f2dc4649517a1df13e26f28')) self.assertEqual(hashring.get_node( 'stats.checkout.cluster.padamski-wro.api.v1.payment-initialize.count'), ('127.0.0.3', '866a18b81f2dc4649517a1df13e26f28')) graphite-carbon-1.1.7/lib/carbon/tests/test_instrumentation.py0000644000175000017500000000126713633714656023321 0ustar gringrin# -*- coding: utf-8 -*- import os import sys from unittest import TestCase from mock import mock_open, patch, call from carbon.instrumentation import getMemUsage class TestInstrumentation(TestCase): def test_getMemUsage(self): if sys.version_info[0] >= 3: open_call = 'builtins.open' else: open_call = '__builtin__.open' with patch(open_call, mock_open(read_data='1 2 1 1 0 1 0')) as m: page_size = os.sysconf('SC_PAGESIZE') usage = getMemUsage() m.assert_has_calls([call().__exit__(None, None, None)], any_order=True) self.assertEqual(usage, 2 * page_size) graphite-carbon-1.1.7/lib/carbon/tests/test_log.py0000644000175000017500000000142113633714656020627 0ustar gringrin# -*- coding: utf-8 -*- from unittest import TestCase from os import path from twisted.python.log import addObserver, removeObserver from carbon import log try: from tempfile import TemporaryDirectory except ImportError: from backports.tempfile import TemporaryDirectory class CarbonLogFileTest(TestCase): def test_write_to_logfile(self): with TemporaryDirectory() as tmpdir: o = log.CarbonLogObserver() o.log_to_dir(tmpdir) addObserver(o) log.creates('😀😀😀😀 test !!!!') removeObserver(o) with open(path.join(tmpdir, 'creates.log')) as logfile: read_line = logfile.readline() self.assertRegexpMatches(read_line, '.*😀😀😀😀 test !!!!') graphite-carbon-1.1.7/lib/carbon/tests/test_pipeline.py0000644000175000017500000000201113633714656021647 0ustar gringrinfrom unittest import TestCase from mock import MagicMock, patch from carbon.pipeline import Processor, run_pipeline class ProcessorTest(TestCase): def test_processor_registers(self): class DummyProcessor(Processor): plugin_name = "dummy_processor" self.assertTrue("dummy_processor" in Processor.plugins) del Processor.plugins["dummy_processor"] def test_run_pipeline_empty_processors(self): self.assertEqual(None, run_pipeline("carbon.metric", (0, 0), [])) def test_run_pipeline(self): processor_mock = MagicMock(Processor) run_pipeline("carbon.metric", (0, 0), [processor_mock]) processor_mock.process.assert_called_once_with("carbon.metric", (0, 0)) def test_run_pipeline_no_processors_uses_state(self): processor_mock = MagicMock(Processor) import carbon.pipeline with patch.object(carbon.pipeline.state, 'pipeline_processors', [processor_mock]): run_pipeline("carbon.metric", (0, 0)) processor_mock.process.assert_called_once_with("carbon.metric", (0, 0)) graphite-carbon-1.1.7/lib/carbon/tests/test_protobuf.py0000644000175000017500000000336313633714656021715 0ustar gringrinimport carbon.client as carbon_client from carbon.routers import DatapointRouter from carbon.tests.util import TestSettings import carbon.service # NOQA from carbon.carbon_pb2 import Payload from carbon.protobuf import CarbonProtobufClientFactory from twisted.internet import reactor # from twisted.internet.defer import Deferred # from twisted.internet.base import DelayedCall from twisted.internet.task import deferLater from twisted.trial.unittest import TestCase from twisted.test.proto_helpers import StringTransport from mock import Mock from struct import unpack, calcsize INT32_FORMAT = '!I' INT32_SIZE = calcsize(INT32_FORMAT) def decode_sent(data): pb_size = unpack(INT32_FORMAT, data[:INT32_SIZE])[0] data = data[INT32_SIZE:INT32_SIZE + pb_size] datapoints = [] payload_pb = Payload.FromString(data) for metric_pb in payload_pb.metrics: for point_pb in metric_pb.points: datapoints.append( (metric_pb.metric, (point_pb.timestamp, point_pb.value))) return datapoints class ConnectedCarbonClientProtocolTest(TestCase): def setUp(self): self.router_mock = Mock(spec=DatapointRouter) carbon_client.settings = TestSettings() # reset to defaults factory = CarbonProtobufClientFactory(('127.0.0.1', 2003, 'a'), self.router_mock) self.protocol = factory.buildProtocol(('127.0.0.1', 2003)) self.transport = StringTransport() self.protocol.makeConnection(self.transport) def test_send_datapoint(self): def assert_sent(): sent_data = self.transport.value() sent_datapoints = decode_sent(sent_data) self.assertEqual([datapoint], sent_datapoints) datapoint = ('foo.bar', (1000000000, 1.0)) self.protocol.sendDatapoint(*datapoint) return deferLater(reactor, 0.1, assert_sent) graphite-carbon-1.1.7/lib/carbon/tests/test_protocols.py0000644000175000017500000003312213633714656022075 0ustar gringrin# -*- coding: utf-8 -*- from carbon.protocols import MetricReceiver, MetricLineReceiver, \ MetricDatagramReceiver, MetricPickleReceiver from carbon import events from unittest import TestCase from mock import Mock, patch, call from carbon.cache import _MetricCache import os.path import pickle import re import time class TestMetricReceiversHandler(TestCase): def test_build(self): expected_plugins = sorted(['line', 'udp', 'pickle', 'protobuf']) # amqp not supported with py3 try: import carbon.amqp_listener expected_plugins.append('amqp') except ImportError: pass # Can't always test manhole because 'cryptography' can # be a pain to install and we don't want to make the CI # flaky because of that. try: import carbon.manhole # NOQA expected_plugins.append('manhole') except ImportError: pass expected_plugins = sorted(expected_plugins) plugins = sorted(MetricReceiver.plugins.keys()) self.assertEquals(expected_plugins, plugins) class _FakeService(object): def addService(_, __): pass fake_service = _FakeService() for plugin_name, plugin_class in MetricReceiver.plugins.items(): plugin_class.build(fake_service) class TestMetricReceiver(TestCase): def setUp(self): self.receiver = MetricReceiver() self.event_mock = Mock() self._event_patch = patch.object(events, 'metricReceived', self.event_mock) self._event_patch.start() self.time_mock = Mock() self.time_mock.return_value = 123456 def tearDown(self): self._event_patch.stop() def test_valid_metricReceived(self): """ Valid metric should call events.metricReceived """ metric = ('carbon.foo', (1, 2)) self.receiver.metricReceived(*metric) events.metricReceived.assert_called_once_with(*metric) def test_min_timestamp_metricReceived(self): """ Should round the timestamp down to whole interval """ settings = {'MIN_TIMESTAMP_RESOLUTION': 10} metric = ('carbon.foo', (1005, 0)) with patch.dict('carbon.conf.settings', settings): self.receiver.metricReceived(*metric) events.metricReceived.assert_called_once_with('carbon.foo', (1000, 0)) def test_nan_metricReceived(self): """ NaN value should not call events.metricReceived """ metric = ('carbon.foo', (1, float('NaN'))) self.receiver.metricReceived(*metric) events.metricReceived.assert_not_called() def test_notime_metricReceived(self): """ metric with timestamp -1 Should call events.metricReceived with current (mocked) time """ with patch.object(time, 'time', self.time_mock): metric = ('carbon.foo', (-1, 2)) self.receiver.metricReceived(*metric) events.metricReceived.assert_called_once_with('carbon.foo', (time.time(), 2)) def test_allowlist_metricReceived(self): """ metrics which don't match should be dropped """ regexes = [re.compile(r'.*\.is\.allowed\..*'), re.compile(r'^жопа\.驢\.γάιδαρος$')] metrics = [('this.metric.is.allowed.a', (1, 2)), ('this.metric.is.not_allowed.a', (3, 4)), ('osioł.الاغ.नितंब$', (5, 6)), ('жопа.驢.γάιδαρος', (7, 8))] with patch('carbon.regexlist.WhiteList.regex_list', regexes): for m in metrics: self.receiver.metricReceived(*m) events.metricReceived.assert_has_calls([call(*metrics[0]), call(*metrics[3])]) def test_disallowlist_metricReceived(self): """ metrics which match should be dropped """ regexes = [re.compile(r'.*\.invalid\.metric\..*'), re.compile('^osioł.الاغ.नितंब$')] metrics = [('some.invalid.metric.a', (1, 2)), ('a.valid.metric.b', (3, 4)), ('osioł.الاغ.नितंब', (5, 6)), ('жопа.驢.γάιδαρος', (7, 8))] with patch('carbon.regexlist.BlackList.regex_list', regexes): for m in metrics: self.receiver.metricReceived(*m) events.metricReceived.assert_has_calls([call(*metrics[1]), call(*metrics[3])]) class TestMetricLineReceiver(TestCase): def setUp(self): self.receiver = MetricLineReceiver() self.receiver.peerName = 'localhost' self._receiver_mock = Mock() self._receiver_patch = patch.object(MetricReceiver, 'metricReceived', self._receiver_mock) self._receiver_patch.start() def tearDown(self): self._receiver_patch.stop() def test_invalid_line_received(self): """ Metric with no timestamp and value should not call metricReceived """ self.receiver.lineReceived(b'\xd0\xb6' * 401) MetricReceiver.metricReceived.assert_not_called() def test_valid_line_received(self): """ Should call metricReceived with str object """ self.receiver.lineReceived( b'\xd0\xb6\xd0\xbe\xd0\xbf\xd0\xb0 42 -1') MetricReceiver.metricReceived.assert_called_once_with( 'жопа', (-1, 42)) def test_get_peer_name(self): """ getPeerName without transport info should return 'peer' """ self.assertEqual(self.receiver.getPeerName(), 'peer') class TestMetricDatagramReceiver(TestCase): def setUp(self): self.receiver = MetricDatagramReceiver() self.receiver.peerName = 'localhost' self.addr = ('127.0.0.1', 9999) self._receiver_mock = Mock() self._receiver_patch = patch.object(MetricReceiver, 'metricReceived', self._receiver_mock) self._receiver_patch.start() def tearDown(self): self._receiver_patch.stop() def test_valid_datagramReceived(self): """ metricReceived should be called with valid metric """ metric = b'carbon.foo 1 2' self.receiver.datagramReceived(metric, self.addr) MetricReceiver.metricReceived.assert_called_once_with( 'carbon.foo', (2, 1)) def test_invalid_datagramReceived(self): """ metricReceived should not be called with missing timestamp """ metric = b'carbon.foo 1' self.receiver.datagramReceived(metric, self.addr) metric = b'c' * 401 self.receiver.datagramReceived(metric, self.addr) MetricReceiver.metricReceived.assert_not_called() def test_utf8_datagramReceived(self): """ metricReceived should be called with UTF-8 metricname """ metric = b'\xd0\xb6\xd0\xbe\xd0\xbf\xd0\xb0 42 -1' self.receiver.datagramReceived(metric, self.addr) MetricReceiver.metricReceived.assert_called_once_with( 'жопа', (-1, 42)) def test_multiple_datagramReceived(self): """ metricReceived should only be called with valid lines """ metric = b'lines 1 2\nare 3 4\nnot\nvalid 5 6\n' self.receiver.datagramReceived(metric, self.addr) MetricReceiver.metricReceived.assert_has_calls([ call('lines', (2, 1)), call('are', (4, 3)), call('valid', (6, 5))]) class TestMetricPickleReceiver(TestCase): def setUp(self): self.receiver = MetricPickleReceiver() self.receiver.unpickler = pickle self.receiver.peerName = 'localhost' self._receiver_mock = Mock() self._receiver_patch = patch.object(MetricReceiver, 'metricReceived', self._receiver_mock) self._receiver_patch.start() def tearDown(self): self._receiver_patch.stop() @patch('carbon.protocols.get_unpickler') def test_pickler_configured_on_connect(self, get_unpickler_mock): """ connectionMade should configure a pickler """ from twisted.internet.address import IPv4Address address = IPv4Address('TCP', 'localhost', 2004) self.receiver.transport = Mock() self.receiver.transport.getPeer = Mock(return_value=address) self.receiver.connectionMade() get_unpickler_mock.assert_called_once_with(insecure=False) def test_string_received(self): """ Valid received metrics should call metricReceived """ metrics = [('foo.bar', (1, 1.5)), ('bar.foo', (2, 2.5))] self.receiver.stringReceived(pickle.dumps(metrics)) MetricReceiver.metricReceived.assert_has_calls( [call('foo.bar', (1, 1.5)), call('bar.foo', (2, 2.5))]) def test_invalid_pickle(self): """ Invalid formatted pickle should not call metricReceived """ # IndexError self.receiver.stringReceived(b"1") # ValueError self.receiver.stringReceived(b"i") # ImportError self.receiver.stringReceived(b"iii") MetricReceiver.metricReceived.not_called() def test_decode_pickle(self): """ Missing timestamp/value should not call metricReceived """ metrics = [('foo.bar', 1)] self.receiver.stringReceived(pickle.dumps(metrics)) MetricReceiver.metricReceived.not_called() def test_invalid_types(self): """ Timestamp/value in wrong type should not call metricReceived """ metrics = [('foo.bar', ('a', 'b'))] self.receiver.stringReceived(pickle.dumps(metrics)) MetricReceiver.metricReceived.not_called() def test_py2_unicode_to_string_conversion(self): """ Metricname in python2 unicode type should be transformed to str """ metrics = [(u'foo.bar.中文', (1, 2))] self.receiver.stringReceived(pickle.dumps(metrics)) MetricReceiver.metricReceived.assert_called_once_with( 'foo.bar.中文', (1, 2)) # assert_called_once does not verify type args, _ = MetricReceiver.metricReceived.call_args self.assertIsInstance(args[0], str) class TestCacheManagementHandler(TestCase): def setUp(self): test_directory = os.path.dirname(os.path.realpath(__file__)) settings = { 'CONF_DIR': os.path.join(test_directory, 'data', 'conf-directory'), 'USE_INSECURE_PICKLER': False } self._settings_patch = patch.dict('carbon.conf.settings', settings) self._settings_patch.start() from carbon.protocols import CacheManagementHandler self.handler = CacheManagementHandler() self.cache = _MetricCache() def _get_cache(): return self.cache self._metriccache_patch = patch('carbon.protocols.MetricCache', _get_cache) self._metriccache_patch.start() self.handler.unpickler = pickle self.handler.peerAddr = 'localhost:7002' self.send_string_mock = Mock(side_effect=self._save_response) self._send_string_patch = patch.object(self.handler, 'sendString', self.send_string_mock) self._send_string_patch.start() def tearDown(self): self._settings_patch.stop() self._send_string_patch.stop() self._metriccache_patch.stop() del self.cache def _save_response(self, arg): self.response = None if arg: raw_response = arg self.response = pickle.loads(raw_response) def send_request(self, request_type, **kwargs): request = {} request['type'] = request_type request.update(kwargs) self.handler.stringReceived(pickle.dumps(request)) @patch('carbon.protocols.get_unpickler') def test_pickler_configured_on_connect(self, get_unpickler_mock): from twisted.internet.address import IPv4Address address = IPv4Address('TCP', 'localhost', 7002) self.handler.transport = Mock() self.handler.transport.getPeer = Mock(return_value=address) self.handler.connectionMade() get_unpickler_mock.assert_called_once_with(insecure=False) def test_invalid_request_type_returns_error(self): self.send_request('foo') self.assertIn('error', self.response) def test_cache_query_returns_response_dict(self): self.send_request('cache-query', metric='carbon.foo') self.assertIsInstance(self.response, dict) def test_cache_query_response_has_datapoints(self): self.send_request('cache-query', metric='carbon.foo') self.assertIn('datapoints', self.response) def test_cache_query_returns_empty_if_no_match(self): self.send_request('cache-query', metric='carbon.foo') self.assertEquals({'datapoints': []}, self.response) def test_cache_query_returns_cached_datapoints_if_matches(self): self.cache.store('carbon.foo', (600, 1.0)) self.send_request('cache-query', metric='carbon.foo') self.assertEqual([(600, 1.0)], self.response['datapoints']) def test_cache_bulk_query_returns_response_dict(self): self.send_request('cache-query-bulk', metrics=[]) self.assertIsInstance(self.response, dict) def test_cache_bulk_query_response_has_datapointsByMetric(self): self.send_request('cache-query-bulk', metrics=[]) self.assertIn('datapointsByMetric', self.response) def test_cache_bulk_query_response_returns_empty_if_no_match(self): self.send_request('cache-query-bulk', metrics=[]) self.assertEquals({'datapointsByMetric': {}}, self.response) def test_cache_bulk_query_response(self): self.cache.store('carbon.foo', (600, 1.0)) self.cache.store('carbon.bar', (600, 2.0)) expected_response = {'carbon.foo': [(600, 1.0)], 'carbon.bar': [(600, 2.0)]} self.send_request('cache-query-bulk', metrics=['carbon.foo', 'carbon.bar']) self.assertEquals({'datapointsByMetric': expected_response}, self.response) graphite-carbon-1.1.7/lib/carbon/tests/test_retentions.py0000644000175000017500000000264113633714656022245 0ustar gringrinfrom unittest import TestCase from carbon.util import parseRetentionDef class TestParseRetentionDef(TestCase): def test_valid_retentions(self): retention_map = ( ('60:10', (60, 10)), ('10:60', (10, 60)), ('10s:10h', (10, 3600)), ) for retention, expected in retention_map: results = parseRetentionDef(retention) self.assertEqual(results, expected) def test_invalid_retentions(self): retention_map = ( # From getUnitString ('10x:10', ValueError("Invalid unit 'x'")), ('60:10x', ValueError("Invalid unit 'x'")), # From parseRetentionDef ('10X:10', ValueError("Invalid precision specification '10X'")), ('10:10$', ValueError("Invalid retention specification '10$'")), ('60:10', (60, 10)), ) for retention, expected_exc in retention_map: try: results = parseRetentionDef(retention) except expected_exc.__class__ as exc: self.assertEqual( str(expected_exc), str(exc), ) self.assertEqual( expected_exc.__class__, exc.__class__, ) else: # When there isn't an exception raised self.assertEqual(results, expected_exc) graphite-carbon-1.1.7/lib/carbon/tests/test_rewrite.py0000644000175000017500000001715613633714656021543 0ustar gringrinfrom mock import Mock, mock_open, patch from unittest import TestCase from carbon.pipeline import Processor from carbon.rewrite import PRE, RewriteProcessor, RewriteRule, RewriteRuleManager class RewriteProcessorTest(TestCase): def tearDown(self): RewriteRuleManager.clear() def test_registers_plugin(self): self.assertTrue('rewrite' in Processor.plugins) def test_applies_rule(self): mock_rule = Mock(spec=RewriteRule) RewriteRuleManager.rulesets[PRE] = [mock_rule] list(RewriteProcessor(PRE).process('carbon.foo', (0, 0))) mock_rule.apply.assert_called_once_with('carbon.foo') def test_applies_rule_and_returns_metric(self): mock_rule = Mock(spec=RewriteRule) mock_rule.apply.return_value = 'carbon.foo.bar' RewriteRuleManager.rulesets[PRE] = [mock_rule] result = list(RewriteProcessor(PRE).process('carbon.foo', (0, 0))) self.assertEqual(('carbon.foo.bar', (0, 0)), result[0]) def test_passes_through_with_no_rules(self): result = list(RewriteProcessor(PRE).process('carbon.foo', (0, 0))) self.assertEqual(('carbon.foo', (0, 0)), result[0]) class TestRewriteRule(TestCase): def setUp(self): self.sample_rule = RewriteRule('^carbon[.]foo[.]', 'carbon_foo.') def test_instantiation_compiles_pattern(self): self.assertTrue(hasattr(self.sample_rule.regex, 'sub')) def test_apply_substitutes(self): result = self.sample_rule.apply('carbon.foo.bar') self.assertEqual('carbon_foo.bar', result) class TestRewriteRuleManager(TestCase): def setUp(self): self.sample_config = """ [pre] ^carbon.foo = carbon.foo.bar ^carbon.bar = carbon.bar.baz """ self.sample_multi_config = """ [pre] ^carbon.foo = carbon.foo.bar ^carbon.bar = carbon.bar.baz [post] ^carbon.baz = carbon.foo.bar """ self.broken_pattern_config = """ [pre] ^carbon.foo = carbon.foo.bar ^carbon.(bar = carbon.bar.baz """ self.commented_config = """ [pre] ^carbon.foo = carbon.foo.bar #^carbon.bar = carbon.bar.baz """ def tearDown(self): RewriteRuleManager.rules_file = None RewriteRuleManager.rules_last_read = 0.0 RewriteRuleManager.clear() def test_looping_call_reads_rules(self): self.assertEqual(RewriteRuleManager.read_rules, RewriteRuleManager.read_task.f) def test_request_for_nonexistent_rules_returns_iterable(self): try: iter(RewriteRuleManager.rules('foo')) except TypeError: self.fail("RewriteRuleManager.rules() returned a non-iterable type") def test_read_from_starts_task(self): with patch.object(RewriteRuleManager, 'read_rules'): with patch.object(RewriteRuleManager.read_task, 'start') as task_start_mock: RewriteRuleManager.read_from('foo.conf') self.assertEqual(1, task_start_mock.call_count) def test_read_records_mtime(self): import carbon.rewrite RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', mock_open(), create=True): with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertEqual(1234, RewriteRuleManager.rules_last_read) def test_read_clears_if_no_file(self): import carbon.rewrite RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'exists', Mock(return_value=False)): with patch.object(RewriteRuleManager, 'clear') as clear_mock: RewriteRuleManager.read_rules() clear_mock.assert_called_once_with() def test_rules_unchanged_if_mtime_unchanged(self): import carbon.rewrite mtime = 1234 rulesets = {'pre': [Mock(RewriteRule)]} RewriteRuleManager.rules_last_read = mtime RewriteRuleManager.rulesets.update(rulesets) RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=mtime)): RewriteRuleManager.read_rules() self.assertEqual(rulesets, RewriteRuleManager.rulesets) def test_read_doesnt_open_file_if_mtime_unchanged(self): import carbon.rewrite mtime = 1234 RewriteRuleManager.rules_last_read = mtime RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', mock_open(), create=True) as open_mock: with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertFalse(open_mock.called) def test_read_opens_file_if_mtime_newer(self): import carbon.rewrite RewriteRuleManager.rules_last_read = 1234 RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', mock_open(), create=True) as open_mock: with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=5678)): RewriteRuleManager.read_rules() self.assertTrue(open_mock.called) def test_section_parsed_into_ruleset(self): import carbon.rewrite open_mock = Mock(return_value=iter(self.sample_config.splitlines())) RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', open_mock, create=True): with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertTrue('pre' in RewriteRuleManager.rulesets) def test_multiple_section_parsed_into_ruleset(self): import carbon.rewrite open_mock = Mock(return_value=iter(self.sample_multi_config.splitlines())) RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', open_mock, create=True): with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertTrue('pre' in RewriteRuleManager.rulesets) self.assertTrue('post' in RewriteRuleManager.rulesets) def test_rules_parsed(self): import carbon.rewrite open_mock = Mock(return_value=iter(self.sample_config.splitlines())) RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', open_mock, create=True): with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertEqual(2, len(RewriteRuleManager.rules('pre'))) def test_broken_patterns_ignored(self): import carbon.rewrite open_mock = Mock(return_value=iter(self.broken_pattern_config.splitlines())) RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', open_mock, create=True): with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertEqual(1, len(RewriteRuleManager.rules('pre'))) def test_comments_ignored(self): import carbon.rewrite open_mock = Mock(return_value=iter(self.commented_config.splitlines())) RewriteRuleManager.rules_file = 'foo.conf' with patch.object(carbon.rewrite, 'open', open_mock, create=True): with patch.object(carbon.rewrite, 'exists', Mock(return_value=True)): with patch.object(carbon.rewrite, 'getmtime', Mock(return_value=1234)): RewriteRuleManager.read_rules() self.assertEqual(1, len(RewriteRuleManager.rules('pre'))) graphite-carbon-1.1.7/lib/carbon/tests/test_routers.py0000644000175000017500000000321513633714656021554 0ustar gringrinimport os import unittest from carbon import routers from carbon.util import parseDestinations from carbon.tests import util DESTINATIONS = ( 'foo:124:a', 'foo:125:b', 'foo:126:c', 'bar:423:a', 'bar:424:b', 'bar:425:c', ) def createSettings(): settings = util.TestSettings() settings['DIVERSE_REPLICAS'] = True, settings['REPLICATION_FACTOR'] = 2 settings['DESTINATIONS'] = DESTINATIONS settings['relay-rules'] = os.path.join( os.path.dirname(__file__), 'relay-rules.conf') settings['aggregation-rules'] = None return settings def parseDestination(destination): return parseDestinations([destination])[0] class TestRelayRulesRouter(unittest.TestCase): def testBasic(self): router = routers.RelayRulesRouter(createSettings()) for destination in DESTINATIONS: router.addDestination(parseDestination(destination)) self.assertEquals(len(list(router.getDestinations('foo.bar'))), 1) class TestOtherRouters(unittest.TestCase): def testBasic(self): settings = createSettings() for plugin in routers.DatapointRouter.plugins: # Test everything except 'rules' which is special if plugin == 'rules': continue router = routers.DatapointRouter.plugins[plugin](settings) self.assertEquals(len(list(router.getDestinations('foo.bar'))), 0) for destination in DESTINATIONS: router.addDestination(parseDestination(destination)) self.assertEquals(len(list(router.getDestinations('foo.bar'))), settings['REPLICATION_FACTOR']) graphite-carbon-1.1.7/lib/carbon/tests/test_service.py0000644000175000017500000000541213633714656021512 0ustar gringrinfrom mock import Mock, patch from unittest import TestCase from carbon import events, state from carbon.pipeline import Processor, run_pipeline, run_pipeline_generated from carbon.service import CarbonRootService, setupPipeline from carbon.tests.util import TestSettings class TestSetupPipeline(TestCase): def setUp(self): self.settings = TestSettings() self.root_service_mock = Mock(CarbonRootService) self.call_when_running_patch = patch('twisted.internet.reactor.callWhenRunning') self.call_when_running_mock = self.call_when_running_patch.start() def tearDown(self): self.call_when_running_patch.stop() state.pipeline_processors = [] events.metricReceived.handlers = [] events.metricGenerated.handlers = [] def test_run_pipeline_chained_to_metric_received(self): setupPipeline([], self.root_service_mock, self.settings) self.assertTrue(run_pipeline in events.metricReceived.handlers) def test_run_pipeline_chained_to_metric_generated(self): setupPipeline([], self.root_service_mock, self.settings) self.assertTrue(run_pipeline_generated in events.metricGenerated.handlers) @patch('carbon.service.setupAggregatorProcessor') def test_aggregate_processor_set_up(self, setup_mock): setupPipeline(['aggregate'], self.root_service_mock, self.settings) setup_mock.assert_called_once_with(self.root_service_mock, self.settings) @patch('carbon.service.setupRewriterProcessor') def test_rewrite_processor_set_up(self, setup_mock): setupPipeline(['rewrite:pre'], self.root_service_mock, self.settings) setup_mock.assert_called_once_with(self.root_service_mock, self.settings) @patch('carbon.service.setupRelayProcessor') def test_relay_processor_set_up(self, setup_mock): setupPipeline(['relay'], self.root_service_mock, self.settings) setup_mock.assert_called_once_with(self.root_service_mock, self.settings) @patch('carbon.service.setupWriterProcessor') def test_write_processor_set_up(self, setup_mock): setupPipeline(['write'], self.root_service_mock, self.settings) setup_mock.assert_called_once_with(self.root_service_mock, self.settings) def test_unknown_processor_raises_value_error(self): self.assertRaises( ValueError, setupPipeline, ['foo'], self.root_service_mock, self.settings) @patch('carbon.service.setupRewriterProcessor', new=Mock()) def test_parses_processor_args(self): # XXX Patch doesnt work on this import directly rewrite_mock = Mock() Processor.plugins['rewrite'] = rewrite_mock setupPipeline(['rewrite:pre'], self.root_service_mock, self.settings) rewrite_mock.assert_called_once_with('pre') def test_schedules_pipeline_ready(self): setupPipeline([], self.root_service_mock, self.settings) self.assertTrue(self.call_when_running_mock.called) graphite-carbon-1.1.7/lib/carbon/tests/test_storage.py0000644000175000017500000000760113633714656021520 0ustar gringrinimport os from unittest import TestCase from mock import patch from carbon.tests.util import TestSettings from carbon.database import WhisperDatabase # class NoConfigSchemaLoadingTest(TestCase): # def setUp(self): # settings = { # 'CONF_DIR': '', # } # self._settings_patch = patch.dict('carbon.conf.settings', settings) # self._settings_patch.start() # def tearDown(self): # self._settings_patch.stop() # def test_loadAggregationSchemas_load_default_schema(self): # from carbon.storage import loadAggregationSchemas, defaultAggregation # schema_list = loadAggregationSchemas() # self.assertEquals(len(schema_list), 1) # schema = schema_list[0] # self.assertEquals(schema, defaultAggregation) # def test_loadStorageSchemas_raise_CarbonConfigException(self): # from carbon.storage import loadStorageSchemas # from carbon.exceptions import CarbonConfigException # with self.assertRaises(CarbonConfigException): # loadStorageSchemas() class ExistingConfigSchemaLoadingTest(TestCase): def setUp(self): test_directory = os.path.dirname(os.path.realpath(__file__)) settings = TestSettings() settings['CONF_DIR'] = os.path.join(test_directory, 'data', 'conf-directory') settings['LOCAL_DATA_DIR'] = '' self._settings_patch = patch('carbon.conf.settings', settings) self._settings_patch.start() self._database_patch = patch('carbon.state.database', new=WhisperDatabase(settings)) self._database_patch.start() def tearDown(self): self._database_patch.stop() self._settings_patch.stop() def test_loadStorageSchemas_return_schemas(self): from carbon.storage import loadStorageSchemas, PatternSchema, Archive schema_list = loadStorageSchemas() self.assertEquals(len(schema_list), 3) expected = [ PatternSchema('carbon', r'^carbon\.', [Archive.fromString('60:90d')]), PatternSchema('default_1min_for_1day', '.*', [Archive.fromString('60s:1d')]) ] for schema, expected_schema in zip(schema_list[:-1], expected): self.assertEquals(schema.name, expected_schema.name) self.assertEquals(schema.pattern, expected_schema.pattern) for (archive, expected_archive) in zip(schema.archives, expected_schema.archives): self.assertEquals(archive.getTuple(), expected_archive.getTuple()) def test_loadStorageSchemas_return_the_default_schema_last(self): from carbon.storage import loadStorageSchemas, defaultSchema schema_list = loadStorageSchemas() last_schema = schema_list[-1] self.assertEquals(last_schema.name, defaultSchema.name) self.assertEquals(last_schema.archives, defaultSchema.archives) def test_loadAggregationSchemas_return_schemas(self): from carbon.storage import loadAggregationSchemas, PatternSchema schema_list = loadAggregationSchemas() self.assertEquals(len(schema_list), 5) expected = [ PatternSchema('min', r'\.min$', (0.1, 'min')), PatternSchema('max', r'\.max$', (0.1, 'max')), PatternSchema('sum', r'\.count$', (0, 'sum')), PatternSchema('default_average', '.*', (0.5, 'average')) ] for schema, expected_schema in zip(schema_list[:-1], expected): self.assertEquals(schema.name, expected_schema.name) self.assertEquals(schema.pattern, expected_schema.pattern) self.assertEquals(schema.archives, expected_schema.archives) def test_loadAggregationSchema_return_the_default_schema_last(self): from carbon.storage import loadAggregationSchemas, defaultAggregation schema_list = loadAggregationSchemas() last_schema = schema_list[-1] self.assertEquals(last_schema, defaultAggregation) graphite-carbon-1.1.7/lib/carbon/tests/test_util.py0000644000175000017500000001437513633714656021037 0ustar gringrinimport socket from unittest import TestCase from carbon.util import parseDestinations from carbon.util import enableTcpKeepAlive from carbon.util import TaggedSeries class UtilTest(TestCase): def test_enable_tcp_keep_alive(self): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) class _Transport(): def getHandle(self): return s def setTcpKeepAlive(self, value): s.setsockopt(socket.SOL_TCP, socket.SO_KEEPALIVE, value) enableTcpKeepAlive(_Transport(), True, None) self.assertEquals(s.getsockopt(socket.SOL_TCP, socket.SO_KEEPALIVE), 1) def test_sanitizing_name_as_tag_value(self): test_cases = [ { 'original': "my~.test.abc", 'expected': "my~.test.abc", }, { 'original': "a.b.c", 'expected': "a.b.c", }, { 'original': "~~a~~.~~~b~~~.~~~c~~~", 'expected': "a~~.~~~b~~~.~~~c~~~", }, { 'original': "a.b.c~", 'expected': "a.b.c~", }, { 'original': "~a.b.c", 'expected': "a.b.c", }, { 'original': "~a~", 'expected': "a~", }, { 'original': "~~~", 'raises': True, }, { 'original': "~", 'raises': True, }, ] for test_case in test_cases: if test_case.get('raises', False): self.assertRaises( Exception, TaggedSeries.sanitize_name_as_tag_value, test_case['original'], ) else: result = TaggedSeries.sanitize_name_as_tag_value(test_case['original']) self.assertEquals(result, test_case['expected']) def test_validate_tag_key_and_value(self): # assert that it raises exception when sanitized name is still not valid with self.assertRaises(Exception): # sanitized name is going to be '', which is not a valid tag value TaggedSeries.sanitize_name_as_tag_value('~~~~') with self.assertRaises(Exception): # given tag value is invalid because it has length 0 TaggedSeries.validateTagAndValue('metric.name;tag=') with self.assertRaises(Exception): # given tag key is invalid because it has length 0 TaggedSeries.validateTagAndValue('metric.name;=value') with self.assertRaises(Exception): # given tag is missing = TaggedSeries.validateTagAndValue('metric.name;tagvalue') with self.assertRaises(Exception): # given tag value is invalid because it starts with ~ TaggedSeries.validateTagAndValue('metric.name;tag=~value') with self.assertRaises(Exception): # given tag key is invalid because it contains ! TaggedSeries.validateTagAndValue('metric.name;ta!g=value') # Destinations have the form: # ::= | "[" "]" # ::= # ::= # ::= ":" | ":" ":" class ParseDestinationsTest(TestCase): def test_valid_dest_unbracketed(self): # Tests valid destinations in the unbracketed form of . dests = [ "127.0.0.1:1234:alpha", # Full IPv4 address "127.1:1234:beta", # 'Short' IPv4 address "localhost:987:epsilon", # Relative domain name "foo.bar.baz.uk.:890:sigma" # Absolute domain name ] expected = [ ("127.0.0.1", 1234, "alpha"), ("127.1", 1234, "beta"), ("localhost", 987, "epsilon"), ("foo.bar.baz.uk.", 890, "sigma") ] actual = parseDestinations(dests) self.assertEquals(len(expected), len(actual)) for exp, act in zip(expected, actual): self.assertEquals(exp, act) def test_valid_dest_bracketed(self): # Tests valid destinations in the bracketed form of . dests = [ "[fe80:dead:beef:cafe:0007:0007:0007:0001]:123:gamma", # Full IPv6 address "[fe80:1234::7]:456:theta", # Compact IPv6 address "[::]:1:o", # Very compact IPv6 address "[ffff::127.0.0.1]:789:omicron" # IPv6 mapped IPv4 address ] expected = [ ("fe80:dead:beef:cafe:0007:0007:0007:0001", 123, "gamma"), ("fe80:1234::7", 456, "theta"), ("::", 1, "o"), ("ffff::127.0.0.1", 789, "omicron"), ] actual = parseDestinations(dests) self.assertEquals(len(expected), len(actual)) for exp, act in zip(expected, actual): self.assertEquals(exp, act) def test_valid_dest_without_instance(self): # Tests destinations without instance specified. dests = [ "1.2.3.4:5678", "[::1]:2", "stats.example.co.uk:8125", "[127.0.0.1]:78", # Odd use of the bracket feature, but why not? "[why.not.this.com]:89", ] expected = [ ("1.2.3.4", 5678, None), ("::1", 2, None), ("stats.example.co.uk", 8125, None), ("127.0.0.1", 78, None), ("why.not.this.com", 89, None) ] actual = parseDestinations(dests) self.assertEquals(len(expected), len(actual)) for exp, act in zip(expected, actual): self.assertEquals(exp, act) def test_wrong_dest(self): # Some cases of invalid input, e.g. invalid/missing port. dests = [ "1.2.3.4", # No port "1.2.3.4:huh", # Invalid port (must be int) "[fe80::3285:a9ff:fe91:e287]", # No port "[ffff::1.2.3.4]:notaport" # Invalid port ] for dest in dests: try: parseDestinations([dest]) except ValueError: continue raise AssertionError("Invalid input was accepted.") graphite-carbon-1.1.7/lib/carbon/tests/util.py0000644000175000017500000000015313633714656017765 0ustar gringrinfrom carbon.conf import Settings class TestSettings(Settings): def readFrom(*args, **kwargs): pass graphite-carbon-1.1.7/lib/carbon/util.py0000644000175000017500000003672013633714656016634 0ustar gringrinimport sys import os import pwd import re try: import builtins as __builtin__ except ImportError: import __builtin__ from hashlib import sha256 from os.path import abspath, basename, dirname import socket from time import sleep, time from twisted.python.util import initgroups from twisted.scripts.twistd import runApp from carbon.log import setDebugEnabled try: from OpenSSL import SSL except ImportError: SSL = None # BytesIO is needed on py3 as StringIO does not operate on byte input anymore # We could use BytesIO on py2 as well but it is slower than StringIO if sys.version_info >= (3, 0): from io import BytesIO as StringIO else: try: from cStringIO import StringIO except ImportError: from StringIO import StringIO try: import cPickle as pickle USING_CPICKLE = True except ImportError: import pickle USING_CPICKLE = False def dropprivs(user): uid, gid = pwd.getpwnam(user)[2:4] initgroups(uid, gid) os.setregid(gid, gid) os.setreuid(uid, uid) return (uid, gid) def enableTcpKeepAlive(transport, enable, settings): if not enable or not hasattr(transport, 'getHandle'): return fd = transport.getHandle() if SSL: if isinstance(fd, SSL.Connection): return if fd.type != socket.SOCK_STREAM: return transport.setTcpKeepAlive(1) for attr in ['TCP_KEEPIDLE', 'TCP_KEEPINTVL', 'TCP_KEEPCNT']: flag = getattr(socket, attr, None) value = getattr(settings, attr, None) if not flag or value is None: continue fd.setsockopt(socket.SOL_TCP, flag, value) def run_twistd_plugin(filename): from carbon.conf import get_parser from twisted.scripts.twistd import ServerOptions bin_dir = dirname(abspath(filename)) root_dir = dirname(bin_dir) os.environ.setdefault('GRAPHITE_ROOT', root_dir) program = basename(filename).split('.')[0] # First, parse command line options as the legacy carbon scripts used to # do. parser = get_parser(program) (options, args) = parser.parse_args() if not args: parser.print_usage() return # This isn't as evil as you might think __builtin__.instance = options.instance __builtin__.program = program # Then forward applicable options to either twistd or to the plugin itself. twistd_options = ["--no_save"] # If no reactor was selected yet, try to use the epoll reactor if # available. try: from twisted.internet import epollreactor # noqa: F401 twistd_options.append("--reactor=epoll") except ImportError: pass if options.debug or options.nodaemon: twistd_options.extend(["--nodaemon"]) if options.profile: twistd_options.extend(["--profile", options.profile]) if options.profiler: twistd_options.extend(["--profiler", options.profiler]) if options.pidfile: twistd_options.extend(["--pidfile", options.pidfile]) if options.umask: twistd_options.extend(["--umask", options.umask]) if options.logger: twistd_options.extend(["--logger", options.logger]) if options.logger: twistd_options.extend(["--logfile", options.logfile]) if options.syslog: twistd_options.append("--syslog") # Now for the plugin-specific options. twistd_options.append(program) if options.debug: twistd_options.append("--debug") setDebugEnabled(True) for option_name, option_value in vars(options).items(): if (option_value is not None and option_name not in ( "debug", "profile", "profiler", "pidfile", "umask", "nodaemon", "syslog", "logger", "logfile")): twistd_options.extend(["--%s" % option_name.replace("_", "-"), option_value]) # Finally, append extra args so that twistd has a chance to process them. twistd_options.extend(args) config = ServerOptions() config.parseOptions(twistd_options) runApp(config) def parseDestination(dest_string): s = dest_string.strip() bidx = s.rfind(']:') # find closing bracket and following colon. cidx = s.find(':') if s.startswith('[') and bidx is not None: server = s[1:bidx] port = s[bidx + 2:] elif cidx is not None: server = s[:cidx] port = s[cidx + 1:] else: raise ValueError("Invalid destination string \"%s\"" % dest_string) if ':' in port: port, _, instance = port.partition(':') else: instance = None return server, int(port), instance def parseDestinations(destination_strings): return [parseDestination(dest_string) for dest_string in destination_strings] # Yes this is duplicated in whisper. Yes, duplication is bad. # But the code is needed in both places and we do not want to create # a dependency on whisper especiaily as carbon moves toward being a more # generic storage service that can use various backends. UnitMultipliers = { 's': 1, 'm': 60, 'h': 60 * 60, 'd': 60 * 60 * 24, 'w': 60 * 60 * 24 * 7, 'y': 60 * 60 * 24 * 365, } def getUnitString(s): if s not in UnitMultipliers: raise ValueError("Invalid unit '%s'" % s) return s def parseRetentionDef(retentionDef): import re (precision, points) = retentionDef.strip().split(':') if precision.isdigit(): precision = int(precision) * UnitMultipliers[getUnitString('s')] else: precision_re = re.compile(r'^(\d+)([a-z]+)$') match = precision_re.match(precision) if match: precision = int(match.group(1)) * UnitMultipliers[getUnitString(match.group(2))] else: raise ValueError("Invalid precision specification '%s'" % precision) if points.isdigit(): points = int(points) else: points_re = re.compile(r'^(\d+)([a-z]+)$') match = points_re.match(points) if match: points = int(match.group(1)) * UnitMultipliers[getUnitString(match.group(2))] / precision else: raise ValueError("Invalid retention specification '%s'" % points) return (precision, points) # This whole song & dance is due to pickle being insecure # yet performance critical for carbon. We leave the insecure # mode (which is faster) as an option (USE_INSECURE_UNPICKLER). # The SafeUnpickler classes were largely derived from # http://nadiana.com/python-pickle-insecure if USING_CPICKLE: class SafeUnpickler(object): PICKLE_SAFE = { 'copy_reg': set(['_reconstructor']), '__builtin__': set(['object']), } @classmethod def find_class(cls, module, name): if module not in cls.PICKLE_SAFE: raise pickle.UnpicklingError('Attempting to unpickle unsafe module %s' % module) __import__(module) mod = sys.modules[module] if name not in cls.PICKLE_SAFE[module]: raise pickle.UnpicklingError('Attempting to unpickle unsafe class %s' % name) return getattr(mod, name) @classmethod def loads(cls, pickle_string): pickle_obj = pickle.Unpickler(StringIO(pickle_string)) pickle_obj.find_global = cls.find_class return pickle_obj.load() else: class SafeUnpickler(pickle.Unpickler): PICKLE_SAFE = { 'copy_reg': set(['_reconstructor']), '__builtin__': set(['object']), } def find_class(self, module, name): if module not in self.PICKLE_SAFE: raise pickle.UnpicklingError('Attempting to unpickle unsafe module %s' % module) __import__(module) mod = sys.modules[module] if name not in self.PICKLE_SAFE[module]: raise pickle.UnpicklingError('Attempting to unpickle unsafe class %s' % name) return getattr(mod, name) @classmethod def loads(cls, pickle_string): if sys.version_info >= (3, 0): return cls(StringIO(pickle_string), encoding='utf-8').load() else: return cls(StringIO(pickle_string)).load() def get_unpickler(insecure=False): if insecure: return pickle else: return SafeUnpickler class TokenBucket(object): '''This is a basic tokenbucket rate limiter implementation for use in enforcing various configurable rate limits''' def __init__(self, capacity, fill_rate): '''Capacity is the total number of tokens the bucket can hold, fill rate is the rate in tokens (or fractional tokens) to be added to the bucket per second.''' self.capacity = float(capacity) self._tokens = float(capacity) self.fill_rate = float(fill_rate) self.timestamp = time() def drain(self, cost, blocking=False): '''Given a number of tokens (or fractions) drain will return True and drain the number of tokens from the bucket if the capacity allows, otherwise we return false and leave the contents of the bucket.''' if cost <= self.tokens: self._tokens -= cost return True if not blocking: return False tokens_needed = cost - self._tokens seconds_per_token = 1 / self.fill_rate seconds_left = seconds_per_token * tokens_needed time_to_sleep = self.timestamp + seconds_left - time() if time_to_sleep > 0: sleep(time_to_sleep) self._tokens -= cost return True def setCapacityAndFillRate(self, new_capacity, new_fill_rate): delta = float(new_capacity) - self.capacity self.capacity = float(new_capacity) self.fill_rate = float(new_fill_rate) self._tokens = delta + self._tokens @property def tokens(self): '''The tokens property will return the current number of tokens in the bucket.''' if self._tokens < self.capacity: now = time() delta = self.fill_rate * (now - self.timestamp) self._tokens = min(self.capacity, self._tokens + delta) self.timestamp = now return self._tokens class PluginRegistrar(type): """Clever subclass detection hack that makes plugin loading trivial. To use this, define an abstract base class for plugin implementations that defines the plugin API. Give that base class a __metaclass__ of PluginRegistrar, and define a 'plugins = {}' class member. Subclasses defining a 'plugin_name' member will then appear in the plugins dict. """ def __init__(classObj, name, bases, members): super(PluginRegistrar, classObj).__init__(name, bases, members) if hasattr(classObj, 'plugin_name'): classObj.plugins[classObj.plugin_name] = classObj class TaggedSeries(object): prohibitedTagChars = ';!^=' @classmethod def validateTagAndValue(cls, tag, value): """validate the given tag / value based on the specs in the documentation""" if len(tag) == 0: raise Exception('Tag may not be empty') if len(value) == 0: raise Exception('Value for tag "{tag}" may not be empty'.format(tag=tag)) for char in cls.prohibitedTagChars: if char in tag: raise Exception( 'Character "{char}" is not allowed in tag "{tag}"'.format(char=char, tag=tag)) if ';' in value: raise Exception( 'Character ";" is not allowed in value "{value}" of tag {tag}'.format(value=value, tag=tag)) if value[0] == '~': raise Exception('Tag values are not allowed to start with "~" in tag "{tag}"'.format(tag=tag)) @classmethod def parse(cls, path): # if path is in openmetrics format: metric{tag="value",...} if path[-2:] == '"}' and '{' in path: return cls.parse_openmetrics(path) # path is a carbon path with optional tags: metric;tag=value;... return cls.parse_carbon(path) @classmethod def parse_openmetrics(cls, path): """parse a path in openmetrics format: metric{tag="value",...} https://github.com/RichiH/OpenMetrics """ (metric, rawtags) = path[0:-1].split('{', 2) if not metric: raise Exception('Cannot parse path %s, no metric found' % path) tags = {} while len(rawtags) > 0: m = re.match(r'([^=]+)="((?:[\\]["\\]|[^"\\])+)"(:?,|$)', rawtags) if not m: raise Exception('Cannot parse path %s, invalid segment %s' % (path, rawtags)) tag = m.group(1) value = m.group(2).replace(r'\"', '"').replace(r'\\', '\\') cls.validateTagAndValue(tag, value) tags[tag] = value rawtags = rawtags[len(m.group(0)):] tags['name'] = cls.sanitize_name_as_tag_value(metric) return cls(metric, tags) @classmethod def parse_carbon(cls, path): """parse a carbon path with optional tags: metric;tag=value;...""" segments = path.split(';') metric = segments[0] if not metric: raise Exception('Cannot parse path %s, no metric found' % path) tags = {} for segment in segments[1:]: tag = segment.split('=', 1) if len(tag) != 2 or not tag[0]: raise Exception('Cannot parse path %s, invalid segment %s' % (path, segment)) cls.validateTagAndValue(*tag) tags[tag[0]] = tag[1] tags['name'] = cls.sanitize_name_as_tag_value(metric) return cls(metric, tags) @staticmethod def sanitize_name_as_tag_value(name): """take a metric name and sanitize it so it is guaranteed to be a valid tag value""" sanitized = name.lstrip('~') if len(sanitized) == 0: raise Exception('Cannot use metric name %s as tag value, results in emptry string' % (name)) return sanitized @staticmethod def format(tags): return tags.get('name', '') + ''.join(sorted([ ';%s=%s' % (tag, value) for tag, value in tags.items() if tag != 'name' ])) @staticmethod def encode(metric, sep='.', hash_only=False): """ Helper function to encode tagged series for storage in whisper etc When tagged series are detected, they are stored in a separate hierarchy of folders under a top-level _tagged folder, where subfolders are created by using the first 3 hex digits of the sha256 hash of the tagged metric path (4096 possible folders), and second-level subfolders are based on the following 3 hex digits (another 4096 possible folders) for a total of 4096^2 possible subfolders. The metric files themselves are created with any . in the metric path replaced with -, to avoid any issues where metrics, tags or values containing a '.' would end up creating further subfolders. This helper is used by both whisper and ceres, but by design each carbon database and graphite-web finder is responsible for handling its own encoding so that different backends can create their own schemes if desired. The hash_only parameter can be set to True to use the hash as the filename instead of a human-readable name. This avoids issues with filename length restrictions, at the expense of being unable to decode the filename and determine the original metric name. A concrete example: .. code-block:: none some.metric;tag1=value2;tag2=value.2 with sha256 hash starting effaae would be stored in: _tagged/eff/aae/some-metric;tag1=value2;tag2=value-2.wsp (whisper) _tagged/eff/aae/some-metric;tag1=value2;tag2=value-2 (ceres) """ if ';' in metric: metric_hash = sha256(metric.encode('utf8')).hexdigest() return sep.join([ '_tagged', metric_hash[0:3], metric_hash[3:6], metric_hash if hash_only else metric.replace('.', '_DOT_') ]) # metric isn't tagged, just replace dots with the separator and trim any leading separator return metric.replace('.', sep).lstrip(sep) @staticmethod def decode(path, sep='.'): """ Helper function to decode tagged series from storage in whisper etc """ if path.startswith('_tagged'): return path.split(sep, 3)[-1].replace('_DOT_', '.') # metric isn't tagged, just replace the separator with dots return path.replace(sep, '.') def __init__(self, metric, tags, series_id=None): self.metric = metric self.tags = tags self.id = series_id @property def path(self): return self.__class__.format(self.tags) graphite-carbon-1.1.7/lib/carbon/writer.py0000644000175000017500000002115213633714656017164 0ustar gringrin"""Copyright 2009 Chris Davis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.""" import time from six.moves import queue from carbon import state from carbon.cache import MetricCache from carbon.storage import loadStorageSchemas, loadAggregationSchemas from carbon.conf import settings from carbon import log, instrumentation from carbon.util import TokenBucket from twisted.internet import reactor from twisted.internet.task import LoopingCall from twisted.application.service import Service try: import signal except ImportError: log.msg("Couldn't import signal module") SCHEMAS = loadStorageSchemas() AGGREGATION_SCHEMAS = loadAggregationSchemas() # Inititalize token buckets so that we can enforce rate limits on creates and # updates if the config wants them. CREATE_BUCKET = None UPDATE_BUCKET = None if settings.MAX_CREATES_PER_MINUTE != float('inf'): capacity = settings.MAX_CREATES_PER_MINUTE fill_rate = float(settings.MAX_CREATES_PER_MINUTE) / 60 CREATE_BUCKET = TokenBucket(capacity, fill_rate) if settings.MAX_UPDATES_PER_SECOND != float('inf'): capacity = settings.MAX_UPDATES_PER_SECOND fill_rate = settings.MAX_UPDATES_PER_SECOND UPDATE_BUCKET = TokenBucket(capacity, fill_rate) class TagQueue(object): def __init__(self, maxsize=0, update_interval=1): self.add_queue = queue.Queue(maxsize) self.update_queue = queue.Queue(maxsize) self.update_interval = update_interval self.update_counter = 0 def add(self, metric): try: self.add_queue.put_nowait(metric) except queue.Full: pass def update(self, metric): self.update_counter = self.update_counter % self.update_interval + 1 if self.update_counter == 1: try: self.update_queue.put_nowait(metric) except queue.Full: pass def getbatch(self, maxsize=1): batch = [] while len(batch) < maxsize: try: batch.append(self.add_queue.get_nowait()) except queue.Empty: break while len(batch) < maxsize: try: batch.append(self.update_queue.get_nowait()) except queue.Empty: break return batch tagQueue = TagQueue(maxsize=settings.TAG_QUEUE_SIZE, update_interval=settings.TAG_UPDATE_INTERVAL) def writeCachedDataPoints(): "Write datapoints until the MetricCache is completely empty" cache = MetricCache() while cache: (metric, datapoints) = cache.drain_metric() if metric is None: # end the loop break dbFileExists = state.database.exists(metric) if not dbFileExists: if CREATE_BUCKET and not CREATE_BUCKET.drain(1): # If our tokenbucket doesn't have enough tokens available to create a new metric # file then we'll just drop the metric on the ground and move on to the next # metric. # XXX This behavior should probably be configurable to no tdrop metrics # when rate limitng unless our cache is too big or some other legit # reason. instrumentation.increment('droppedCreates') continue archiveConfig = None xFilesFactor, aggregationMethod = None, None for schema in SCHEMAS: if schema.matches(metric): if settings.LOG_CREATES: log.creates('new metric %s matched schema %s' % (metric, schema.name)) archiveConfig = [archive.getTuple() for archive in schema.archives] break for schema in AGGREGATION_SCHEMAS: if schema.matches(metric): if settings.LOG_CREATES: log.creates('new metric %s matched aggregation schema %s' % (metric, schema.name)) xFilesFactor, aggregationMethod = schema.archives break if not archiveConfig: raise Exception(("No storage schema matched the metric '%s'," " check your storage-schemas.conf file.") % metric) if settings.LOG_CREATES: log.creates("creating database metric %s (archive=%s xff=%s agg=%s)" % (metric, archiveConfig, xFilesFactor, aggregationMethod)) try: state.database.create(metric, archiveConfig, xFilesFactor, aggregationMethod) if settings.ENABLE_TAGS: tagQueue.add(metric) instrumentation.increment('creates') except Exception as e: log.err() log.msg("Error creating %s: %s" % (metric, e)) instrumentation.increment('errors') continue # If we've got a rate limit configured lets makes sure we enforce it waitTime = 0 if UPDATE_BUCKET: t1 = time.time() UPDATE_BUCKET.drain(1, blocking=True) waitTime = time.time() - t1 try: t1 = time.time() # If we have duplicated points, always pick the last. update_many() # has no guaranted behavior for that, and in fact the current implementation # will keep the first point in the list. datapoints = dict(datapoints).items() state.database.write(metric, datapoints) if settings.ENABLE_TAGS: tagQueue.update(metric) updateTime = time.time() - t1 except Exception as e: log.err() log.msg("Error writing to %s: %s" % (metric, e)) instrumentation.increment('errors') else: pointCount = len(datapoints) instrumentation.increment('committedPoints', pointCount) instrumentation.append('updateTimes', updateTime) if settings.LOG_UPDATES: if waitTime > 0.001: log.updates("wrote %d datapoints for %s in %.5f seconds after waiting %.5f seconds" % ( pointCount, metric, updateTime, waitTime)) else: log.updates("wrote %d datapoints for %s in %.5f seconds" % ( pointCount, metric, updateTime)) def writeForever(): while reactor.running: try: writeCachedDataPoints() except Exception: log.err() # Back-off on error to give the backend time to recover. time.sleep(0.1) else: # Avoid churning CPU when there are no metrics are in the cache time.sleep(1) def writeTags(): while True: tags = tagQueue.getbatch(settings.TAG_BATCH_SIZE) if not tags: break state.database.tag(*tags) def writeTagsForever(): while reactor.running: try: writeTags() except Exception: log.err() # Back-off on error to give the backend time to recover. time.sleep(0.1) else: # Avoid churning CPU when there are no series in the queue time.sleep(0.2) def reloadStorageSchemas(): global SCHEMAS try: SCHEMAS = loadStorageSchemas() except Exception as e: log.msg("Failed to reload storage SCHEMAS: %s" % (e)) def reloadAggregationSchemas(): global AGGREGATION_SCHEMAS try: AGGREGATION_SCHEMAS = loadAggregationSchemas() except Exception as e: log.msg("Failed to reload aggregation SCHEMAS: %s" % (e)) def shutdownModifyUpdateSpeed(): try: shut = settings.MAX_UPDATES_PER_SECOND_ON_SHUTDOWN if UPDATE_BUCKET: UPDATE_BUCKET.setCapacityAndFillRate(shut, shut) if CREATE_BUCKET: CREATE_BUCKET.setCapacityAndFillRate(shut, shut) log.msg("Carbon shutting down. Changed the update rate to: " + str(settings.MAX_UPDATES_PER_SECOND_ON_SHUTDOWN)) except KeyError: log.msg("Carbon shutting down. Update rate not changed") # Also set MIN_TIMESTAMP_LAG to 0 to avoid waiting for nothing. settings.MIN_TIMESTAMP_LAG = 0 class WriterService(Service): def __init__(self): self.storage_reload_task = LoopingCall(reloadStorageSchemas) self.aggregation_reload_task = LoopingCall(reloadAggregationSchemas) def startService(self): if 'signal' in globals().keys(): log.msg("Installing SIG_IGN for SIGHUP") signal.signal(signal.SIGHUP, signal.SIG_IGN) self.storage_reload_task.start(60, False) self.aggregation_reload_task.start(60, False) reactor.addSystemEventTrigger('before', 'shutdown', shutdownModifyUpdateSpeed) reactor.callInThread(writeForever) if settings.ENABLE_TAGS: reactor.callInThread(writeTagsForever) Service.startService(self) def stopService(self): self.storage_reload_task.stop() self.aggregation_reload_task.stop() Service.stopService(self) graphite-carbon-1.1.7/lib/twisted/0000755000175000017500000000000013633714656015514 5ustar gringringraphite-carbon-1.1.7/lib/twisted/plugins/0000755000175000017500000000000013633714656017175 5ustar gringringraphite-carbon-1.1.7/lib/twisted/plugins/carbon_aggregator_cache_plugin.py0000644000175000017500000000131713633714656025720 0ustar gringrinfrom zope.interface import implementer from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker from carbon import conf @implementer(IServiceMaker, IPlugin) class CarbonAggregatorCacheServiceMaker(object): tapname = "carbon-aggregator-cache" description = "Aggregate and write stats for graphite." options = conf.CarbonAggregatorOptions def makeService(self, options): """ Construct a C{carbon-aggregator-cache} service. """ from carbon import service return service.createAggregatorCacheService(options) # Now construct an object which *provides* the relevant interfaces serviceMaker = CarbonAggregatorCacheServiceMaker() graphite-carbon-1.1.7/lib/twisted/plugins/carbon_aggregator_plugin.py0000644000175000017500000000125213633714656024573 0ustar gringrinfrom zope.interface import implementer from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker from carbon import conf @implementer(IServiceMaker, IPlugin) class CarbonAggregatorServiceMaker(object): tapname = "carbon-aggregator" description = "Aggregate stats for graphite." options = conf.CarbonAggregatorOptions def makeService(self, options): """ Construct a C{carbon-aggregator} service. """ from carbon import service return service.createAggregatorService(options) # Now construct an object which *provides* the relevant interfaces serviceMaker = CarbonAggregatorServiceMaker() graphite-carbon-1.1.7/lib/twisted/plugins/carbon_cache_plugin.py0000644000175000017500000000121213633714656023510 0ustar gringrinfrom zope.interface import implementer from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker from carbon import conf @implementer(IServiceMaker, IPlugin) class CarbonCacheServiceMaker(object): tapname = "carbon-cache" description = "Collect stats for graphite." options = conf.CarbonCacheOptions def makeService(self, options): """ Construct a C{carbon-cache} service. """ from carbon import service return service.createCacheService(options) # Now construct an object which *provides* the relevant interfaces serviceMaker = CarbonCacheServiceMaker() graphite-carbon-1.1.7/lib/twisted/plugins/carbon_relay_plugin.py0000644000175000017500000000121013633714656023557 0ustar gringrinfrom zope.interface import implementer from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker from carbon import conf @implementer(IServiceMaker, IPlugin) class CarbonRelayServiceMaker(object): tapname = "carbon-relay" description = "Relay stats for graphite." options = conf.CarbonRelayOptions def makeService(self, options): """ Construct a C{carbon-relay} service. """ from carbon import service return service.createRelayService(options) # Now construct an object which *provides* the relevant interfaces serviceMaker = CarbonRelayServiceMaker() graphite-carbon-1.1.7/requirements.txt0000644000175000017500000000015013633714656016543 0ustar gringrinTwisted>=13.2.0 git+git://github.com/graphite-project/whisper.git#egg=whisper txAMQP cachetools urllib3 graphite-carbon-1.1.7/setup.cfg0000644000175000017500000000015013633714656015100 0ustar gringrin[bdist_rpm] requires = python-twisted whisper post-install = distro/redhat/misc/postinstall graphite-carbon-1.1.7/setup.py0000644000175000017500000001012013633714656014767 0ustar gringrin#!/usr/bin/env python from __future__ import with_statement import os from glob import glob try: from ConfigParser import ConfigParser, DuplicateSectionError # Python 2 except ImportError: from configparser import ConfigParser, DuplicateSectionError # Python 3 # Graphite historically has an install prefix set in setup.cfg. Being in a # configuration file, it's not easy to override it or unset it (for installing # graphite in a virtualenv for instance). # The prefix is now set by ``setup.py`` and *unset* if an environment variable # named ``GRAPHITE_NO_PREFIX`` is present. # While ``setup.cfg`` doesn't contain the prefix anymore, the *unset* step is # required for installations from a source tarball because running # ``python setup.py sdist`` will re-add the prefix to the tarball's # ``setup.cfg``. cf = ConfigParser() with open('setup.cfg', 'r') as f: orig_setup_cfg = f.read() f.seek(0) cf.readfp(f, 'setup.cfg') if os.environ.get('GRAPHITE_NO_PREFIX'): cf.remove_section('install') else: print('#' * 80) print('') print('Carbon\'s default installation prefix is "/opt/graphite".') print('') print('To install Carbon in the Python\'s default location run:') print('$ GRAPHITE_NO_PREFIX=True python setup.py install') print('') print('#' * 80) try: cf.add_section('install') except DuplicateSectionError: pass if not cf.has_option('install', 'prefix'): cf.set('install', 'prefix', '/opt/graphite') if not cf.has_option('install', 'install-lib'): cf.set('install', 'install-lib', '%(prefix)s/lib') with open('setup.cfg', 'w') as f: cf.write(f) if os.environ.get('USE_SETUPTOOLS'): from setuptools import setup setup_kwargs = dict(zip_safe=0) else: from distutils.core import setup setup_kwargs = dict() storage_dirs = [ ('storage/ceres/dummy.txt', []), ('storage/whisper/dummy.txt',[]), ('storage/lists',[]), ('storage/log/dummy.txt',[]), ('storage/rrd/dummy.txt',[]) ] conf_files = [ ('conf', glob('conf/*.example')) ] install_files = storage_dirs + conf_files # Let's include redhat init scripts, despite build platform # but won't put them in /etc/init.d/ automatically anymore init_scripts = [ ('examples/init.d', ['distro/redhat/init.d/carbon-cache', 'distro/redhat/init.d/carbon-relay', 'distro/redhat/init.d/carbon-aggregator']) ] install_files += init_scripts def read(fname): with open(os.path.join(os.path.dirname(__file__), fname)) as f: return f.read() try: setup( name='carbon', version='1.1.7', url='http://graphiteapp.org/', author='Chris Davis', author_email='chrismd@gmail.com', license='Apache Software License 2.0', description='Backend data caching and persistence daemon for Graphite', long_description=read('README.md'), long_description_content_type='text/markdown', packages=['carbon', 'carbon.aggregator', 'twisted.plugins'], package_dir={'' : 'lib'}, scripts=glob('bin/*'), package_data={ 'carbon' : ['*.xml'] }, data_files=install_files, install_requires=['Twisted', 'txAMQP', 'cachetools', 'urllib3'], classifiers=( 'Intended Audience :: Developers', 'Natural Language :: English', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', ), **setup_kwargs ) finally: with open('setup.cfg', 'w') as f: f.write(orig_setup_cfg) graphite-carbon-1.1.7/stdeb.cfg0000644000175000017500000000010013633714656015034 0ustar gringrin[DEFAULT] Package: carbon Depends: python-twisted-bin, whisper graphite-carbon-1.1.7/tests-requirements.txt0000644000175000017500000000017713633714656017714 0ustar gringrincoverage mock mocker nose protobuf mmh3 pyOpenSSL git+git://github.com/graphite-project/ceres.git#egg=ceres backports.tempfile graphite-carbon-1.1.7/tox.ini0000644000175000017500000000123213633714656014574 0ustar gringrin[tox] envlist = py{27,35,36,37,38,py}{,-pyhash}, lint, benchmark [testenv] setenv = GRAPHITE_NO_PREFIX=true PYTHONPATH={toxinidir}/lib commands = coverage run --branch --source=lib,bin --omit=lib/carbon/tests/* "{envbindir}/trial" carbon coverage xml coverage report deps = -rrequirements.txt -rtests-requirements.txt pyhash: pyhash [testenv:lint] deps = flake8 six commands = flake8 {toxinidir}/lib {toxinidir}/bin [testenv:benchmark] voting = False commands = python {toxinidir}/lib/carbon/tests/benchmark_cache.py python {toxinidir}/lib/carbon/tests/benchmark_routers.py [flake8] max-line-length=100 ignore=E111,E114,E121,W504