pax_global_header00006660000000000000000000000064126567322650014530gustar00rootroot0000000000000052 comment=72885efea46ed1357f13730fd2c340e910a108ce curator-3.4.1/000077500000000000000000000000001265673226500132145ustar00rootroot00000000000000curator-3.4.1/.gitignore000066400000000000000000000003451265673226500152060ustar00rootroot00000000000000.*.swp *~ *.py[co] build dist scratch docs/_build *.egg .eggs elasticsearch_curator.egg-info elasticsearch_curator_dev.egg-info .coverage .idea coverage.xml nosetests.xml index.html docs/asciidoc/html_docs wheelhouse Elastic.ico curator-3.4.1/.travis.yml000066400000000000000000000007471265673226500153350ustar00rootroot00000000000000language: python python: - "2.7" - "3.3" install: - mkdir /tmp/elasticsearch - wget -O - https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.0.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - pip install . before_script: - /tmp/elasticsearch/bin/elasticsearch -d -D es.path.data=/tmp -D es.gateway.type=none -D es.index.store.type=memory -D es.discovery.zen.ping.multicast.enabled=false script: - python setup.py test curator-3.4.1/CONTRIBUTING.md000066400000000000000000000073101265673226500154460ustar00rootroot00000000000000# Contributing to Curator All contributions are welcome: ideas, patches, documentation, bug reports, complaints, etc! Programming is not a required skill, and there are many ways to help out! It is more important to us that you are able to contribute. That said, some basic guidelines, which you are free to ignore :) ## Want to learn? Want to write your own code to do something Curator doesn't do out of the box? * [Curator API Documentation](http://curator.readthedocs.org/) Since version 2.0, Curator ships with both an API and wrapper scripts (which are actually defined as entry points). This allows you to write your own scripts to accomplish similar goals, or even new and different things with the [Curator API](http://curator.readthedocs.org/), and the [Elasticsearch Python API](http://elasticsearch-py.readthedocs.org/). Want to know how to use the command-line interface (CLI)? * [Curator CLI Documentation](http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) The Curator CLI Documentation is now a part of the document repository at http://elastic.co/guide at http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html Want to lurk about and see what others are doing with Curator? * The irc channels (#logstash and #elasticsearch on irc.freenode.org) are good places for this ## Got Questions? Have a problem you want Curator to solve for you? * You are welcome to join the IRC channel #logstash (or #elasticsearch) on irc.freenode.org and ask for help there! ## Have an Idea or Feature Request? * File a ticket on [github](https://github.com/elastic/curator/issues) ## Something Not Working? Found a Bug? If you think you found a bug, it probably is a bug. * File it on [github](https://github.com/elastic/logstash/issues) # Contributing Documentation and Code Changes If you have a bugfix or new feature that you would like to contribute to Curator, and you think it will take more than a few minutes to produce the fix (ie; write code), it is worth discussing the change with the Curator users and developers first! You can reach us via [github](https://github.com/elastic/logstash/issues), or via IRC (#logstash or #elasticsearch on freenode irc) Documentation is in two parts: API and CLI documentation. API documentation is generated from comments inside the classes and methods within the code. This documentation is rendered and hosted at http://curator.readthedocs.org CLI documentation is in Asciidoc format in the GitHub repository at https://github.com/elastic/curator/tree/master/docs/asciidoc. This documentation can be changed via a pull request as with any other code change. ## Contribution Steps 1. Test your changes! Run the test suite ('python setup.py test'). Please note that this requires an Elasticsearch instance. The tests will try to connect to your local elasticsearch instance and run integration tests against it. **This will delete all the data stored there!** You can use the env variable `TEST_ES_SERVER` to point to a different instance (for example 'otherhost:9203'). 2. Please make sure you have signed our [Contributor License Agreement](http://www.elastic.co/contributor-agreement/). We are not asking you to assign copyright to us, but to give us the right to distribute your code without restriction. We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. You only need to sign the CLA once. 3. Send a pull request! Push your changes to your fork of the repository and [submit a pull request](https://help.github.com/articles/using-pull-requests). In the pull request, describe what your changes do and mention any bugs/issues related to the pull request. curator-3.4.1/CONTRIBUTORS000066400000000000000000000035451265673226500151030ustar00rootroot00000000000000The following is a list of people who have contributed ideas, code, bug reports, or in general have helped curator along its way. Contributors: * Jordan Sissel (jordansissel) (For Logstash, first and foremost) * Shay Banon (kimchy) (For Elasticsearch, of course!) * Aaron Mildenstein (untergeek) * Njal Karevoll * François Deppierraz * Honza Kral (HonzaKral) * Benjamin Smith (benjaminws) * Colin Moller (LeftyBC) * Elliot (edgeofnite) * Ram Viswanadha (ramv) * Chris Meisinger (cmeisinger) * Stuart Warren (stuart-warren) * (gitshaw) * (sfritz) * (sjoelsam) * Jose Diaz-Gonzalez (josegonzalez) * Arie Bro (arieb) * David Harrigan (dharrigan) * Mathieu Geli (gelim) * Nick Ethier (nickethier) * Mohab Usama (mohabusama) * (gitshaw) * Stuart Warren (stuart-warren) * Xavier Calland (xavier-calland) * Chad Schellenger (cschellenger) * Kamil Essekkat (ekamil) * (gbutt) * Ben Buchacher (bbuchacher) * Ehtesh Choudhury (shurane) * Markus Fischer (mfn) * Fabien Wernli (faxm0dem) * Michael Weiser (michaelweiser) * (digital-wonderland) * cassiano (cassianoleal) * Matt Dainty (bodgit) * Alex Philipp (alex-sf) * (krzaczek) * Justin Lintz (jlintz) * Jeremy Falling (jjfalling) * Ian Babrou (bobrik) * Ferenc Erki (ferki) * George Heppner (gheppner) * Matt Hughes (matthughes) * Brian Lalor (blalor) * Paweł Krzaczkowski (krzaczek) * Ben Tse (bt5e) * Tom Hendrikx (whyscream) * Christian Vozar (christianvozar) * Magnus Baeck (magnusbaeck) * Robin Kearney (rk295) * (cfeio) * (malagoli) * Dan Sheridan (djs52) * Michael-Keith Bernard (SegFaultAX) * Simon Lundström (simmel) * (pkr1234) * Mark Feltner (feltnerm) * William Jimenez (wjimenez5271) * Jeremy Canady (jrmycanady) * Steven Ottenhoff (steffo) * Ole Rößner (Basster) * Jack (univerio) * Tomáš Mózes (hydrapolic) * Gary Gao (garyelephant) * Panagiotis Moustafellos (pmoust) * (pbamba) * Pavel Strashkin (xaka) * Wadim Kruse (wkruse) curator-3.4.1/Changelog.rst000077700000000000000000000000001265673226500212012docs/Changelog.rstustar00rootroot00000000000000curator-3.4.1/Dockerfile000066400000000000000000000003271265673226500152100ustar00rootroot00000000000000# Docker Definition for ElasticSearch Curator FROM python:2.7.8-slim MAINTAINER Christian R. Vozar RUN pip install --quiet elasticsearch-curator ENTRYPOINT [ "/usr/local/bin/curator" ] curator-3.4.1/LICENSE.txt000066400000000000000000000011261265673226500150370ustar00rootroot00000000000000Copyright 2011–2016 Elasticsearch and contributors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. curator-3.4.1/MANIFEST.in000066400000000000000000000006101265673226500147470ustar00rootroot00000000000000include Changelog.rst include CONTRIBUTORS CONTRIBUTING.md include LICENSE.txt include README.rst include Dockerfile recursive-exclude * __pycache__ recursive-exclude * *.py[co] recursive-include test * recursive-include docs * recursive-exclude docs *.pyc recursive-exclude docs *.pyo recursive-exclude test *.pyc recursive-exclude test *.pyo prune docs/_build prune docs/asciidoc/html_docs curator-3.4.1/NOTICE000066400000000000000000000077631265673226500141350ustar00rootroot00000000000000In accordance with section 4d of the Apache 2.0 license (http://www.apache.org/licenses/LICENSE-2.0), this NOTICE file is included. All users mentioned in the CONTRIBUTORS file at https://github.com/elastic/curator/blob/master/CONTRIBUTORS must be included in any derivative work. All conditions of section 4 of the Apache 2.0 license will be enforced: 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: a. You must give any other recipients of the Work or Derivative Works a copy of this License; and b. You must cause any modified files to carry prominent notices stating that You changed the files; and c. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and d. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. Contributors: * Jordan Sissel (jordansissel) (For Logstash, first and foremost) * Shay Banon (kimchy) (For Elasticsearch, of course!) * Aaron Mildenstein (untergeek) * Njal Karevoll * François Deppierraz * Honza Kral (HonzaKral) * Benjamin Smith (benjaminws) * Colin Moller (LeftyBC) * Elliot (edgeofnite) * Ram Viswanadha (ramv) * Chris Meisinger (cmeisinger) * Stuart Warren (stuart-warren) * (gitshaw) * (sfritz) * (sjoelsam) * Jose Diaz-Gonzalez (josegonzalez) * Arie Bro (arieb) * David Harrigan (dharrigan) * Mathieu Geli (gelim) * Nick Ethier (nickethier) * Mohab Usama (mohabusama) * (gitshaw) * Stuart Warren (stuart-warren) * Xavier Calland (xavier-calland) * Chad Schellenger (cschellenger) * Kamil Essekkat (ekamil) * (gbutt) * Ben Buchacher (bbuchacher) * Ehtesh Choudhury (shurane) * Markus Fischer (mfn) * Fabien Wernli (faxm0dem) * Michael Weiser (michaelweiser) * (digital-wonderland) * cassiano (cassianoleal) * Matt Dainty (bodgit) * Alex Philipp (alex-sf) * (krzaczek) * Justin Lintz (jlintz) * Jeremy Falling (jjfalling) * Ian Babrou (bobrik) * Ferenc Erki (ferki) * George Heppner (gheppner) * Matt Hughes (matthughes) * Brian Lalor (blalor) * Paweł Krzaczkowski (krzaczek) * Ben Tse (bt5e) * Tom Hendrikx (whyscream) * Christian Vozar (christianvozar) * Magnus Baeck (magnusbaeck) * Robin Kearney (rk295) * (cfeio) * (malagoli) * Dan Sheridan (djs52) * Michael-Keith Bernard (SegFaultAX) * Simon Lundström (simmel) * (pkr1234) * Mark Feltner (feltnerm) * William Jimenez (wjimenez5271) * Jeremy Canady (jrmycanady) * Steven Ottenhoff (steffo) * Ole Rößner (Basster) * Jack (univerio) curator-3.4.1/README.rst000066400000000000000000000107221265673226500147050ustar00rootroot00000000000000.. _readme: Curator |buildstatus| ===================== .. |buildstatus| image:: http://build-eu-00.elastic.co/job/es-curator_core/badge/icon :target: http://build-eu-00.elastic.co/job/es-curator_core/ Have indices in Elasticsearch? This is the tool for you! Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your indices. `Curator API Documentation`_ ---------------------------- Since version 2.0, Curator ships with both an API and wrapper scripts (which are actually defined as entry points). This allows you to write your own scripts to accomplish similar goals, or even new and different things with the `Curator API`_, and the `Elasticsearch Python API`_. .. _Curator API: http://curator.readthedocs.org/ .. _Curator API Documentation: `Curator API`_ .. _Elasticsearch Python API: http://elasticsearch-py.readthedocs.org/ `Curator CLI Documentation`_ ---------------------------- The `Curator CLI Documentation`_ is now a part of the document repository at http://elastic.co/guide at http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html .. _Curator CLI Documentation: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html `Getting Started`_ ------------------ .. _Getting Started: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/getting-started.html See the `Installation guide `_ and the `command-line usage guide `_ Running ``curator --help`` will also show usage information. `Frequently Asked Questions`_ ----------------------------- .. _Frequently Asked Questions: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/faq.html Encountering issues like ``DistributionNotFound``? See the FAQ_ for that issue, and more. .. _FAQ: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/entrypoint-fix.html `Documentation & Examples`_ --------------------------- .. _Documentation & Examples: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html The documentation for the CLI is now part of the document repository at http://elastic.co/guide at http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html The `Curator Wiki `_ on Github is now a place to add your own examples and ideas. Contributing ------------ * fork the repo * make changes in your fork * add tests to cover your changes (if necessary) * run tests * sign the `CLA `_ * send a pull request! To run from source, use the ``run_curator.py`` and ``run_es_repo_mgr.py`` scripts in the root directory of the project. Running Tests ------------- To run the test suite just run ``python setup.py test`` When changing code, contributing new code or fixing a bug please make sure you include tests in your PR (or mark it as without tests so that someone else can pick it up to add the tests). When fixing a bug please make sure the test actually tests the bug - it should fail without the code changes and pass after they're applied (it can still be one commit of course). The tests will try to connect to your local elasticsearch instance and run integration tests against it. This will delete all the data stored there! You can use the env variable ``TEST_ES_SERVER`` to point to a different instance (for example 'otherhost:9203'). Versioning ---------- There are two branches for development - ``master`` and ``0.6``. Master branch is used to track all the changes for Elasticsearch 1.0 and beyond whereas 0.6 tracks Elasticsearch 0.90 and the corresponding ``elasticsearch-py`` version. Releases with major versions greater than 1 (X.Y.Z, where X is > 1) are to be used with Elasticsearch 1.0 and later, 0.6 releases are meant to work with Elasticsearch 0.90.X. Origins ------- Curator was first called ``clearESindices.py`` [1] and was almost immediately renamed to ``logstash_index_cleaner.py`` [1]. After a time it was migrated under the [logstash](https://github.com/elastic/logstash) repository as ``expire_logs``. Soon thereafter, Jordan Sissel was hired by Elasticsearch, as was the original author of this tool. It became Elasticsearch Curator after that and is now hosted at [1] curator-3.4.1/binary_release.py000066400000000000000000000070741265673226500165620ustar00rootroot00000000000000import os import re import sys import shutil import hashlib # This script simply takes the output of `python setup.py build_exe` and makes # a compressed archive (zip for windows, tar.gz for Linux) for distribution. # Utility function to read from file. def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION archive_format = 'gztar' enviro = dict(os.environ) platform = sys.platform pyver = str(sys.version_info[0]) + '.' + str(sys.version_info[1]) if platform == 'win32': # Win32 stuff sys_string = platform archive_format = 'zip' elif platform == 'linux' or platform == 'linux2': sys_type = enviro['_system_type'].lower() sys_arch = enviro['_system_arch'].lower() sys_string = sys_type + '-' + sys_arch else: # Unsupported platform? print('Your platform ({0}) is not yet supported for binary build/distribution.'.format(platform)) sys.exit(1) build_name = 'exe.' + sys_string + '-' + pyver #print('Expected build directory: {0}'.format(build_name)) build_path = os.path.join('build', build_name) if os.path.exists(build_path): #print("I found the path: {0}".format(build_path)) target_name = "curator-" + str(get_version()) + "-" + sys_string target_path = os.path.join('.', target_name) # Check to see if an older directory exists... if os.path.exists(target_path): print('An older build exists at {0}. Please delete this before continuing.'.format(target_path)) sys.exit(1) else: os.rename(build_path, target_path) # Ensure the rename went smoothly, then continue if os.path.exists(target_path): #print("Build successfully renamed") if float(pyver) >= 2.7: shutil.make_archive(target_name, archive_format, '.', target_path) if platform == 'win32': fname = target_name + '.zip' else: fname = target_name + '.tar.gz' # Clean up directory if we made a viable archive. if os.path.exists(fname): shutil.rmtree(target_path) else: print('Something went wrong creating the archive {0}'.format(fname)) sys.exit(1) md5sum = hashlib.md5(open(fname, 'rb').read()).hexdigest() sha1sum = hashlib.sha1(open(fname, 'rb').read()).hexdigest() with open(fname + ".md5.txt", "w") as md5_file: md5_file.write("{0}".format(md5sum)) with open(fname + ".sha1.txt", "w") as sha1_file: sha1_file.write("{0}".format(sha1sum)) print('Archive: {0}'.format(fname)) print('{0} = {1}'.format(fname + ".md5.txt", md5sum)) print('{0} = {1}'.format(fname + ".sha1.txt", sha1sum)) else: print('Your python version ({0}) is too old to use with shutil.make_archive.'.format(pyver)) print('You can manually compress the {0} directory to achieve the same result.'.format(target_name)) else: # We couldn't find a build_path print("Build not found. Please run 'python setup.py build_exe' to create the build directory.") sys.exit(1) curator-3.4.1/curator/000077500000000000000000000000001265673226500146735ustar00rootroot00000000000000curator-3.4.1/curator/__init__.py000066400000000000000000000001101265673226500167740ustar00rootroot00000000000000from ._version import __version__ from .api import * from .cli import * curator-3.4.1/curator/__main__.py000066400000000000000000000001371265673226500167660ustar00rootroot00000000000000 """Executed when package directory is called as a script""" from .curator import main main() curator-3.4.1/curator/_version.py000066400000000000000000000000261265673226500170670ustar00rootroot00000000000000__version__ = '3.4.1' curator-3.4.1/curator/api/000077500000000000000000000000001265673226500154445ustar00rootroot00000000000000curator-3.4.1/curator/api/__init__.py000066400000000000000000000004401265673226500175530ustar00rootroot00000000000000from .utils import * from .filter import * from .alias import * from .allocation import * from .bloom import * from .close import * from .delete import * from .opener import * from .optimize import * from .replicas import * from .seal import * from .show import * from .snapshot import * curator-3.4.1/curator/api/alias.py000066400000000000000000000073331265673226500171150ustar00rootroot00000000000000from .utils import * import elasticsearch import logging logger = logging.getLogger(__name__) def add_to_alias(client, index_name, alias=None): """ Add indicated index to the specified alias. :arg client: The Elasticsearch client connection :arg index_name: The index name :arg alias: Alias name to operate on. :rtype: bool """ if check_csv(index_name): logger.error("Must specify only a single index as an argument.") return False if not client.indices.exists(index_name): logger.error("Index {0} not found.".format(index_name)) return False if not alias: # This prevents _all from being aliased by accident... logger.error('No alias provided.') return False if not client.indices.exists_alias(name=alias): indices_in_alias = [] logger.info('Alias {0} not found. Creating...'.format(alias)) else: indices_in_alias = get_alias(client, alias) if not index_name in indices_in_alias: if index_closed(client, index_name): logger.error('Failed to add index {0} to alias {1} because it is closed.'.format(index_name, alias)) return False else: try: client.indices.update_aliases(body={'actions': [{ 'add': { 'index': index_name, 'alias': alias}}]}) logger.info('Adding index {0} to alias {1}...'.format(index_name, alias)) return True except Exception as e: logger.error("Error adding index {0} to alias {1}. Exception: {2} Run with --debug flag and/or check Elasticsearch logs for more information.".format(index_name, alias, e)) return False else: logger.debug('Skipping index {0}: Index already exists in alias {1}...'.format(index_name, alias)) return True def remove_from_alias(client, index_name, alias=None): """ Remove the indicated index from the specified alias. :arg client: The Elasticsearch client connection :arg index_name: The index name :arg alias: Alias name to operate on. :rtype: bool """ if check_csv(index_name): logger.error("Must specify only a single index as an argument.") return False if not alias: logger.error('No alias provided.') return False indices_in_alias = get_alias(client, alias) if not indices_in_alias: logger.error("Index {0} not found in alias {1}".format(index_name, alias)) return False if index_name in indices_in_alias: try: client.indices.update_aliases(body={'actions': [{ 'remove': { 'index': index_name, 'alias': alias}}]}) logger.info("Removing index {0} from alias {1}.".format(index_name, alias)) return True except Exception as e: logger.error("Error removing index {0} from alias {1}. Exception: {2} Run with --debug flag and/or check Elasticsearch logs for more information.".format(index_name, alias, e)) return False else: logger.warn('Index {0} does not exist in alias {1}; skipping.'.format(index_name, alias)) return False def alias(client, indices, alias=None, remove=False): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg alias: Alias name to operate on. :arg remove: If true, remove the alias. :rtype: bool """ retval = True for i in ensure_list(indices): if remove: success = remove_from_alias(client, i, alias=alias) else: success = add_to_alias(client, i, alias=alias) # if we fail once, we fail completely if not success: retval = False return retval curator-3.4.1/curator/api/allocation.py000066400000000000000000000046141265673226500201500ustar00rootroot00000000000000from .utils import * import elasticsearch import logging logger = logging.getLogger(__name__) def apply_allocation_rule(client, indices, rule=None, allocation_type='require' ): """ Apply a required allocation rule to a list of indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg rule: The routing allocation rule to apply, e.g. ``tag=ssd``. Must be in the format of ``key=value``, and should match values declared on the correlating nodes in your cluster. :arg allocation_type: Type of allocation to apply :rtype: bool .. note:: See: http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-allocation.html#index-modules-allocation """ if not rule: logger.error('Missing rule parameter') return False key = rule.split('=')[0] value = rule.split('=')[1] indices = prune_allocated(client, indices, key, value, allocation_type) if allocation_type not in ['require', 'include', 'exclude']: logger.error("{0} is an invalid allocation_type. Must be one of 'require', 'include', 'exclude'.".format(allocation_type)) return False if not indices: logger.warn("No indices to act on.") return True # Send successful execution on empty list #531 logger.info('Updating index setting index.routing.allocation.{0}.{1}={2}'.format(allocation_type,key,value)) try: client.indices.put_settings(index=to_csv(indices), body='index.routing.allocation.{0}.{1}={2}'.format(allocation_type,key,value), ) return True except: logger.error("Error in updating index settings with allocation rule. Run with --debug flag and/or check Elasticsearch logs for more information.") return False def allocation(client, indices, rule=None, allocation_type='require' ): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg rule: The routing allocation rule to apply, e.g. ``tag=ssd``. Must be in the format of ``key=value``, and should match values declared on the correlating nodes in your cluster. :arg allocation_type: Type of allocation to apply :rtype: bool """ return apply_allocation_rule(client, indices, rule=rule, allocation_type=allocation_type ) curator-3.4.1/curator/api/bloom.py000066400000000000000000000041461265673226500171330ustar00rootroot00000000000000from .utils import * import elasticsearch import time import logging logger = logging.getLogger(__name__) def disable_bloom_filter(client, indices, delay=None): """ Disable the bloom filter cache for the list of indices. This method will ignore closed indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg delay: Pause *n* seconds after operating on each index :rtype: bool """ indices = prune_closed(client, indices) no_more_bloom = (1, 4, 0) version_number = get_version(client) if version_number >= no_more_bloom: logger.warn('Bloom filters no longer exist for search in Elasticsearch since v1.4.0') return True else: try: if delay: if delay > 0: return loop_bloom(client, indices, delay) else: client.indices.put_settings(index=to_csv(indices), body='index.codec.bloom.load=false') return True except Exception: logger.error("Error disabling bloom filters. Run with --debug flag and/or check Elasticsearch logs for more information.") return False def loop_bloom(client, indices, delay): """ Iterate over list of indices. Only called from within :py:func:`curator.api.disable_bloom_filter` :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg delay: Pause *n* seconds after operating on each index :rtype: bool """ retval = True for i in indices: success = disable_bloom_filter(client, i) # If fail on even one iteration, we fail period if not success: retval = False time.sleep(delay) return retval def bloom(client, indices, delay=None): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg delay: Pause *n* seconds after operating on each index :rtype: bool """ return disable_bloom_filter(client, indices, delay=delay) curator-3.4.1/curator/api/close.py000066400000000000000000000022561265673226500171300ustar00rootroot00000000000000from .utils import * from .seal import * import elasticsearch import logging logger = logging.getLogger(__name__) def close_indices(client, indices): """ Close the indicated indices. Perform a flush (a synced flush, if your Elasticsearch version supports it) before closing. This method will ignore unavailable (including closed) indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: bool """ indices = ensure_list(indices) if get_version(client) >= (1, 6, 0): seal_indices(client, indices) else: client.indices.flush(index=to_csv(indices), ignore_unavailable=True) try: client.indices.close(index=to_csv(indices), ignore_unavailable=True) return True except Exception: logger.error("Error closing indices. Run with --debug flag and/or check Elasticsearch logs for more information.") return False def close(client, indices): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: bool """ return close_indices(client, indices) curator-3.4.1/curator/api/delete.py000066400000000000000000000046541265673226500172710ustar00rootroot00000000000000from .utils import * from .filter import * import elasticsearch import logging logger = logging.getLogger(__name__) def delete_indices(client, indices, master_timeout=30000): """ Delete the indicated indices, including closed indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg master_timeout: Number of milliseconds to wait for master node response :rtype bool: """ indices = ensure_list(indices) if get_version(client) >= (2,0,0): if type(master_timeout) == type(int()): master_timeout = str(master_timeout/1000) + 's' try: logger.info("Deleting indices as a batch operation:") for i in indices: logger.info("---deleting index {0}".format(i)) client.indices.delete(index=to_csv(indices), master_timeout=master_timeout) return True except Exception as e: logger.error("Error deleting one or more indices. Run with --debug flag and/or check Elasticsearch logs for more information.") try: logger.error('Got a {0} response from Elasticsearch. Error message: {1}'.format(e.status_code, e.error)) except AttributeError: logger.error('Error message: {0}'.format(e)) return False def delete(client, indices, master_timeout=30000): """ Helper method called by the CLI. Tries up to 3x to delete indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg master_timeout: Number of milliseconds to wait for master node response :rtype bool: """ if get_version(client) >= (2,0,0): if type(master_timeout) == type(int()): master_timeout = str(master_timeout/1000) + 's' for count in range(1, 4): # Try 3 times logger.debug("master_timeout value: {0}".format(master_timeout)) delete_indices(client, indices, master_timeout) result = [ i for i in indices if i in get_indices(client) ] if len(result) > 0: logger.error("Indices failed to delete:") for idx in result: logger.error("---{0}".format(idx)) else: # break return True # We leave the loop here, if everything deleted. indices = result logger.error("Unable to delete indices after 3 attempts: {0}".format(result)) return False # If we make it here, indices didn't delete after 3 tries. curator-3.4.1/curator/api/filter.py000066400000000000000000000335541265673226500173150ustar00rootroot00000000000000from .utils import * from datetime import timedelta, datetime, date import time import re import logging logger = logging.getLogger(__name__) REGEX_MAP = { 'timestring': r'^.*{0}.*$', 'newer_than': r'(?P{0})', 'older_than': r'(?P{0})', 'prefix': r'^{0}.*$', 'suffix': r'^.*{0}$', } DATE_REGEX = { 'Y' : '4', 'y' : '2', 'm' : '2', 'W' : '2', 'U' : '2', 'd' : '2', 'H' : '2', 'M' : '2', 'S' : '2', 'j' : '3', } def build_filter( kindOf=None, time_unit=None, timestring=None, groupname='date', value=None, exclude=False, ): """ Return a filter object based on the arguments. :arg kindOf: Can be one of: [older_than|newer_than|suffix|prefix|regex|timestring]. This option defines what kind of filter you will be building. :arg exclude: If `True`, exclude matches rather than include :arg groupname: The name of a named capture in pattern. Currently only acts on 'date' :arg timestring: An strftime string to match the datestamp in an index name. Only used for time-based filtering. :arg time_unit: One of ``hours``, ``days``, ``weeks``, ``months``. (default: ``days``). Only used for time-based filtering. :arg value: Depends on `kindOf`. It's a time-unit multiplier for `older_than` and `newer_than`. It is the strftime string if `kindOf` is `timestring`. It's used to build the regular expression for other kinds. """ if kindOf not in [ 'older_than', 'newer_than', 'regex', 'exclude', 'prefix', 'suffix', 'timestring' ]: logger.error('{0}: Invalid value for kindOf'.format(kindOf)) return {} # Stop here if None or empty value, but zero is okay if value == 0: logger.info("I found a zero value") argdict = {} elif not value: return {} else: argdict = {} if kindOf in ['older_than', 'newer_than']: if not time_unit: logger.error("older_than and newer_than require time_unit parameter") return {} if not timestring: logger.error("older_than and newer_than require timestring parameter") return {} argdict = { "groupname":groupname, "time_unit":time_unit, "timestring": timestring, "value": value, "method": kindOf } date_regex = get_date_regex(timestring) regex = REGEX_MAP[kindOf].format(date_regex) elif kindOf == 'timestring': regex = r'^.*{0}.*$'.format(get_date_regex(value)) elif kindOf == 'regex': regex = r'{0}'.format(value) elif kindOf in ['prefix', 'suffix']: regex = REGEX_MAP[kindOf].format(value) if kindOf == 'exclude': regex = r'{0}'.format(value) argdict['exclude'] = True logger.debug("REGEX = {0}".format(regex)) argdict['pattern'] = regex logger.debug("Added filter: {0}".format(argdict)) return argdict def apply_filter( items, pattern=None, exclude=False, groupname=None, timestring=None, time_unit=None, method=None, value=None, utc_now=None): """Iterate over all items in the list and return a list of matches :arg items: A list of indices or snapshots to act on :arg pattern: A regular expression to iterate all indices against. :arg exclude: If `True`, exclude matches rather than include :arg groupname: The name of a named capture in pattern. Currently only acts on 'date' :arg timestring: An strftime string to match the datestamp in an index name. Only used for time-based filtering. :arg time_unit: One of ``hours``, ``days``, ``weeks``, ``months``. (default: ``days``). Only used for time-based filtering. :arg method: Either ``older_than`` or ``newer_than``. Only used for time-based filtering. :arg value: `time_unit` multiplier used to calculate time window. Only used for time-based filtering. :arg utc_now: Used for testing. Overrides current time with specified time. """ if not pattern: logger.error("Missing required pattern parameter.") return None p = re.compile(pattern) if exclude: return list(filter(lambda x: not p.search(x), items)) result = [] items = ensure_list(items) for item in items: match = False if groupname: m = p.search(item) if m: if m.group(groupname): if groupname == "date": timestamp = m.group(groupname) # Get a boolean result match = timestamp_check( timestamp, timestring=timestring, time_unit=time_unit, method=method, value=value, utc_now=utc_now, ) else: m = p.match(item) if m: match = True if match == True: result.append(item) return result def get_date_regex(timestring): """ Return a regex string based on a provided strftime timestring. :arg timestring: An strftime pattern :rtype: str """ prev = ''; curr = ''; regex = '' for s in range(0, len(timestring)): curr = timestring[s] if curr == '%': pass elif curr in DATE_REGEX and prev == '%': regex += '\d{' + DATE_REGEX[curr] + '}' else: regex += "\\" + curr prev = curr logger.debug("regex = {0}".format(regex)) return regex def get_datetime(index_timestamp, timestring): """ Return the datetime extracted from the index name, which is the index creation time. :arg index_timestamp: The timestamp extracted from an index name :arg timestring: An strftime pattern :rtype: Datetime object """ # Compensate for week of year by appending '%w' to the timestring # and '1' (Monday) to index_timestamp if '%W' in timestring: timestring += '%w' index_timestamp += '1' elif '%U' in timestring: timestring += '%w' index_timestamp += '1' elif '%m' in timestring: if not '%d' in timestring: timestring += '%d' index_timestamp += '1' #logger.debug("index_timestamp: {0}, timestring: {1}, return value: {2}".format(index_timestamp, timestring, datetime.strptime(index_timestamp, timestring))) return datetime.strptime(index_timestamp, timestring) def month_bump(datestamp, sign='positive'): """ This method returns either the next or previous month based on the value of sign. :arg datestamp: A datetime datestamp :arg sign: Either `positive` or `negative` :rtype: Datetime object """ if sign is not 'positive' and sign is not 'negative': raise ValueError if sign == 'positive': if datestamp.month == 1: return date(datestamp.year-1, 12, 1) else: return date(datestamp.year, datestamp.month-1, 1) else: if datestamp.month == 12: return date(datestamp.year+1, 1, 1) else: return date(datestamp.year, datestamp.month+1, 1) def get_target_month(month_count, utc_now=None): """ Return datetime object for number of *full* months older than `month_count` from now, or `utc_now`, if provided. :arg month_count: Number of *full* months :arg utc_now: Used for testing. Overrides current time with specified time. :rtype: Datetime object """ utc_now = date(utc_now.year, utc_now.month, 1) if utc_now else date.today() target_date = date(utc_now.year, utc_now.month, 1) if month_count < 0: for i in range(0, month_count, -1): target_date = month_bump(target_date, sign='negative') elif month_count > 0: for i in range(0, month_count): target_date = month_bump(target_date) return datetime(target_date.year, target_date.month, target_date.day) def get_cutoff(unit_count=None, time_unit='days', utc_now=None): """ Find the cutoff time based on `unit_count` and `time_unit`. :arg unit_count: `time_unit` multiplier :arg time_unit: One of ``hours``, ``days``, ``weeks``, ``months``. (default: ``days``) :arg utc_now: Used for testing. Overrides current time with specified time. :rtype: Datetime object """ if unit_count == None: logger.error("Missing value for unit_count.") return False if type(unit_count) != type(int()): logger.error("Non-integer value for unit_count.") return False # time-injection for test purposes only utc_now = utc_now if utc_now else datetime.utcnow() # reset to start of the period to be sure we are not retiring a human by mistake utc_now = utc_now.replace(minute=0, second=0, microsecond=0) if time_unit == 'days': utc_now = utc_now.replace(hour=0) if time_unit == 'weeks': # Since week math always uses Monday as the start of the week, # this work-around resets utc_now to be Monday of the current week. weeknow = utc_now.strftime('%Y-%W') utc_now = get_datetime(weeknow, '%Y-%W') if time_unit == 'months': utc_now = utc_now.replace(hour=0) cutoff = get_target_month(unit_count, utc_now=utc_now) else: # This cutoff must be a multiple of time_units if unit_count < 0: cutoff = utc_now - timedelta(**{time_unit: (unit_count)}) else: cutoff = utc_now - timedelta(**{time_unit: (unit_count - 1)}) #logger.debug("time_cutoff: {0}".format(cutoff)) return cutoff def timestamp_check(timestamp, timestring=None, time_unit=None, method='older_than', value=None, utc_now=None): """ Check `timestamp` to see if it is `value` * `time_unit` `method` (``older_than`` or ``newer_than``) the calculated cutoff. :arg timestamp: An strftime parsable date string. :arg timestring: An strftime string to match against ``timestamp``. :arg time_unit: One of ``hours``, ``days``, ``weeks``, ``months``. :arg method: ``older_than`` or ``newer_than``. :arg value: `time_unit` multiplier used to calculate time window. :arg utc_now: Used for testing. Overrides current time with specified time. :rtype: bool """ cutoff = get_cutoff(unit_count=value, time_unit=time_unit, utc_now=utc_now) if not cutoff: logger.error('No cutoff value.') return False try: object_time = get_datetime(timestamp, timestring) except ValueError: logger.error('Could not extract a timestamp matching {0} from timestring {1}'.format(timestamp, timestring)) return False if method == "older_than": if object_time < cutoff: return True elif method == "newer_than": if object_time > cutoff: return True logger.debug('Timestamp "{0}" is outside the cutoff period ({1} {2} {3}).'.format( timestamp, method.replace('_', ' '), value, time_unit)) return False def filter_by_space(client, indices, disk_space=None, reverse=True): """ Remove indices from the provided list of indices based on space consumed, sorted reverse-alphabetically by default. If you set `reverse` to `False`, it will be sorted alphabetically. The default is usually what you will want. If only one kind of index is provided--for example, indices matching ``logstash-%Y.%m.%d``--then reverse alphabetical sorting will mean the oldest get removed first, because lower numbers in the dates mean older indices. By setting reverse to `False`, then ``index3`` will be deleted before ``index2``, which will be deleted before ``index1`` :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg disk_space: Filter indices over *n* gigabytes :arg reverse: The filtering direction. (default: `True`) :rtype: list """ def get_stat_list(stats): retval = [] for index_name in stats['indices']: size = stats['indices'][index_name]['total']['store']['size_in_bytes'] logger.debug('Index: {0} Size: {1}'.format(index_name, size)) retval.append((index_name, size)) return retval # Ensure that disk_space is a float if disk_space: disk_space = float(disk_space) else: logger.error("Mising value for disk_space.") return False disk_usage = 0.0 disk_limit = disk_space * 2**30 delete_list = [] not_closed = [i for i in indices if not index_closed(client, i)] # Because we're building a csv list of indices to pass, we need to ensure # that we actually have at least one index before calling # client.indices.status, otherwise the call will match _all indices, which # is very bad. # See https://github.com/elastic/curator/issues/254 logger.debug('List of open indices: {0}'.format(sorted(not_closed))) if not_closed: if len(to_csv(not_closed)) > 3072: logger.warn('Very large list of indices. Breaking it up into smaller chunks.') index_lists = chunk_index_list(not_closed) statlist = [] for l in index_lists: statlist.extend(get_stat_list(client.indices.stats(index=to_csv(l)))) else: statlist = get_stat_list(client.indices.stats(index=to_csv(not_closed))) sorted_indices = sorted(statlist, reverse=reverse) for index_name, index_size in sorted_indices: disk_usage += index_size if disk_usage > disk_limit: delete_list.append(index_name) logger.info('Deleting {0}, summed disk usage is {1:.3f} GB and disk limit is {2:.3f} GB.'.format(index_name, disk_usage/2**30, disk_limit/2**30)) else: logger.info('skipping {0}, summed disk usage is {1:.3f} GB and disk limit is {2:.3f} GB.'.format(index_name, disk_usage/2**30, disk_limit/2**30)) return delete_list curator-3.4.1/curator/api/opener.py000066400000000000000000000015511265673226500173100ustar00rootroot00000000000000from .utils import * import elasticsearch import logging logger = logging.getLogger(__name__) def open_indices(client, indices): """ Open the indicated indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: bool """ indices = ensure_list(indices) try: # Opening an already open index has no effect. client.indices.open(index=to_csv(indices)) return True except Exception: logger.error("Error opening indices. Run with --debug flag and/or check Elasticsearch logs for more information.") return False def opener(client, indices): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: bool """ return open_indices(client, indices) curator-3.4.1/curator/api/optimize.py000066400000000000000000000043041265673226500176570ustar00rootroot00000000000000from .utils import * import elasticsearch import time import logging logger = logging.getLogger(__name__) def optimize_index(client, index_name, max_num_segments=None, delay=0, request_timeout=21600): """ Optimize (Lucene forceMerge) index to `max_num_segments` per shard :arg client: The Elasticsearch client connection :arg index_name: The index name :arg max_num_segments: Merge to this number of segments per shard. :arg delay: A number of seconds to delay after successfully optimizing :arg request_timeout: Number of seconds before the connection should timeout :rtype: bool """ if optimized(client, index_name, max_num_segments): return True else: logger.info('Optimizing index {0} to {1} segments per shard. Please wait...'.format(index_name, max_num_segments)) try: client.indices.optimize( index=index_name, max_num_segments=max_num_segments, request_timeout=request_timeout ) if delay > 0: logger.info("Pausing for {0} seconds before continuing...".format(delay)) time.sleep(delay) return True except Exception: logger.error("Error optimizing index {0}. Run with --debug flag and/or check Elasticsearch logs for more information.".format(index_name)) return False def optimize(client, indices, max_num_segments=None, delay=0, request_timeout=21600): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg max_num_segments: Merge to this number of segments per shard. :arg delay: A number of seconds to delay after successfully optimizing :arg request_timeout: Number of seconds before the connection should timeout :rtype: bool """ retval = True for i in ensure_list(indices): # If we fail once, we fail completely success = optimize_index( client, i, max_num_segments=max_num_segments, delay=delay, request_timeout=request_timeout ) if not success: retval = False return retval curator-3.4.1/curator/api/replicas.py000066400000000000000000000032041265673226500176170ustar00rootroot00000000000000from .utils import * import elasticsearch import logging logger = logging.getLogger(__name__) def change_replicas(client, indices, replicas=None): """ Change the number of replicas, more or less, for the indicated indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg replicas: The number of replicas the indices should have :rtype: bool """ if replicas == None: logger.error('No replica count provided.') return False else: indices = ensure_list(indices) pruned = prune_closed(client, indices) if sorted(indices) != sorted(pruned): logger.warn('Skipping closed indices to prevent error which can leave indices unopenable.') logger.info('Updating index setting: number_of_replicas={0}'.format(replicas)) if len(pruned) > 0: try: client.indices.put_settings(index=to_csv(pruned), body='number_of_replicas={0}'.format(replicas)) return True except Exception: logger.error("Error changing replica count. Run with --debug flag and/or check Elasticsearch logs for more information.") return False else: logger.warn('No open indices found.') def replicas(client, indices, replicas=None): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg replicas: The number of replicas the indices should have :rtype: bool """ return change_replicas(client, indices, replicas=replicas) curator-3.4.1/curator/api/seal.py000066400000000000000000000047271265673226500167540ustar00rootroot00000000000000from .utils import * import elasticsearch import logging logger = logging.getLogger(__name__) def seal_indices(client, indices): """ Seal the indicated indices (perform a synced flush) if your Elasticsearch version supports it before closing. This method will ignore unavailable (including closed) indices. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: bool """ indices = prune_closed(client, indices) es_version = get_version(client) errcode = '' results = {} try: if es_version >= (1, 6, 0): if len(indices) > 0: results = client.indices.flush_synced(index=to_csv(indices)) else: logger.warn('No indices to seal.') return True else: logger.error('Your version of Elasticsearch ({0}) does not support index sealing (synced flush). Must be 1.6.0 or higher.'.format(es_version)) except Exception as e: logger.warn('Non-fatal error encountered.') try: logger.debug('Error: {0}. Message: {1}'.format(e.status_code, e.error)) results = e.info except AttributeError: logger.debug('Error: {0}'.format(e)) total = results.pop('_shards') if '_shards' in results else {"total":0,"successful":0,"failed":0} fails = [ i for i in sorted(results) if results[i]['failed'] > 0 ] if len(fails) > 0: logger.warn('{0} indices failed to seal (synced flush):'.format(len(fails))) for f in fails: reasons = set() for shard in results[f]['failures']: reasons.add(shard['reason']) logger.error('{0}: {1}'.format(f, reasons)) results.pop(f) # Remove fails from results if len(results) > 0: logger.info('All other indices provided have been successfully sealed. (Shown with --debug flag enabled.)') else: logger.info('Provided indices successfully sealed. (Shown with --debug flag enabled.)') if len(results) > 0: logger.debug('Successfully sealed indices: {0}'.format(list(results.keys()))) return True # Failures are sometimes expected. Log viewing is essential to ascertaining failures. def seal(client, indices): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: bool """ return seal_indices(client, indices) curator-3.4.1/curator/api/show.py000066400000000000000000000010141265673226500167720ustar00rootroot00000000000000from .utils import * def show(client, object_list, type=None): """ Helper method called by the CLI. :arg client: The Elasticsearch client connection :arg object_list: A list of indices or snapshots to show :arg object_type: `indices` or `snapshots` :rtype: bool """ for obj in ensure_list(object_list): if type == 'indices': print('{0}{1}'.format(obj, ' (CLOSED)' if index_closed(client, obj) else '')) else: print('{0}'.format(obj)) return True curator-3.4.1/curator/api/snapshot.py000066400000000000000000000134771265673226500176710ustar00rootroot00000000000000from .utils import * import logging logger = logging.getLogger(__name__) def create_snapshot(client, indices='_all', name=None, prefix='curator-', repository='', ignore_unavailable=False, include_global_state=True, partial=False, wait_for_completion=True, request_timeout=21600, skip_repo_validation=False): """ Create a snapshot of provided indices (or ``_all``) that are open. :arg client: The Elasticsearch client connection :arg indices: A list of indices to snapshot. Default is ``_all`` :arg name: What to name the snapshot. `prefix` + datestamp if omitted. :arg prefix: Override the default with this value. Defaults to ``curator-`` :arg repository: The Elasticsearch snapshot repository to use :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `True`) :type wait_for_completion: bool :arg ignore_unavailable: Ignore unavailable shards/indices. (default: `False`) :type ignore_unavailable: bool :arg include_global_state: Store cluster global state with snapshot. (default: `True`) :type include_global_state: bool :arg partial: Do not fail if primary shard is unavailable. (default: `False`) :type partial: bool :arg skip_repo_validation: Do not validate write access to repository on all cluster nodes before proceeding. (default: `False`). Useful for shared filesystems where intermittent timeouts can affect validation, but won't likely affect snapshot success. :type skip_repo_validation: bool :rtype bool: """ # Return True if it is skipped if not repository: logger.error('Missing required repository parameter') return False # This is to address older versions of Elasticsearch # Older versions do not have the _status endpoint # See https://github.com/elastic/curator/issues/379 no__status = (1, 1, 0) version_number = get_version(client) if version_number >= no__status: in_progress = client.snapshot.status()['snapshots'] if not len(in_progress) == 0: logger.error('Snapshot already in progress: {0}'.format(in_progress)) return False if not indices == '_all': indices = prune_closed(client, indices) if not indices: logger.error("No indices provided.") return False repo_access = (1, 4, 0) if version_number >= repo_access and not skip_repo_validation: try: nodes = client.snapshot.verify_repository(repository=repository)['nodes'] logger.debug('Nodes with verified repository access: {0}'.format(nodes)) except Exception as e: logger.error('Failed to verify all nodes have repository access:') try: if e.status_code == 404: logger.error('--- Repository "{0}" not found. Error: {1}, {2}'.format(repository, e.status_code, e.er)) else: logger.error('--- Got a {0} response from Elasticsearch. Error message: {1}'.format(e.status_code, e.error)) except AttributeError: logger.error('--- Error message: {0}'.format(e)) return False body=create_snapshot_body(indices, ignore_unavailable=ignore_unavailable, include_global_state=include_global_state, partial=partial) name = name if name else prefix + datetime.utcnow().strftime('%Y%m%d%H%M%S') logger.info("Snapshot name: {0}".format(name)) all_snaps = get_snapshots(client, repository=repository) if all_snaps is False: logger.error("Unable to find all snapshots in repository") return False if name in all_snaps: logger.error("A snapshot with name '{0}' already exists.".format(name)) return False try: client.snapshot.create(repository=repository, snapshot=name, body=body, wait_for_completion=wait_for_completion, request_timeout=request_timeout) if not wait_for_completion: logger.warn("Not waiting for completion. Remember to check for successful completion manually.") return True else: state = client.snapshot.get(repository=repository, snapshot=name)['snapshots'][0]['state'] if state == 'SUCCESS': logger.info("Snapshot {0} successfully completed.".format(name)) return True else: logger.error("Snapshot {0} completed with state: {0}".format(state)) return False except elasticsearch.TransportError: logger.error("Client raised a TransportError.") return False def delete_snapshot(client, snapshot=None, repository=None): """ Delete a single snapshot from a given repository by name :arg client: The Elasticsearch client connection :arg snapshot: The snapshot name :arg repository: The Elasticsearch snapshot repository to use """ if not repository: logger.error('Missing required repository parameter') return False if not snapshot: logger.error('Missing required snapshot parameter') return False if check_csv(snapshot): logger.error('Cannot delete multiple snapshots at once. CSV value or list detected: {0}'.format(snapshot)) return False try: logger.info("Deleting snapshot {0}".format(snapshot)) client.snapshot.delete(repository=repository, snapshot=snapshot) return True except elasticsearch.RequestError: logger.error("Unable to delete snapshot {0} from repository {1}. Run with --debug flag and/or check Elasticsearch logs for more information.".format(snapshot, repository)) return False curator-3.4.1/curator/api/utils.py000066400000000000000000000471141265673226500171650ustar00rootroot00000000000000from datetime import timedelta, datetime, date import elasticsearch import time import re import sys import logging import json import click logger = logging.getLogger(__name__) def get_alias(client, alias): """ Return information about the specified alias. :arg client: The Elasticsearch client connection :arg alias: Alias name to operate on. :rtype: list of strings """ if client.indices.exists_alias(name=alias): return client.indices.get_alias(name=alias).keys() else: logger.error('Unable to find alias {0}.'.format(alias)) return False def get_indices(client): try: indices = list(client.indices.get_settings( index='_all', params={'expand_wildcards': 'open,closed'})) logger.debug("All indices: {0}".format(indices)) return indices except Exception: logger.error("Failed to get indices.") return False def ensure_list(indices): """ Return a list, even if indices is a single value :arg indices: A list of indices to act upon :rtype: list """ if type(indices) is not type(list()): # in case of a single value passed indices = [indices] return indices def to_csv(indices): """ Return a csv string from a list of indices, or a single value if only one value is present :arg indices: A list of indices to act on, or a single value, which could be in the format of a csv string already. :rtype: str """ indices = ensure_list(indices) # in case of a single value passed if indices: return ','.join(sorted(indices)) else: return None def check_csv(value): """ Some of the curator methods should not operate against multiple indices at once. This method can be used to check if a list or csv has been sent. :arg value: The value to test, if list or csv string :rtype: bool """ if type(value) is type(list()): return True string = False # Python3 hack because it doesn't recognize unicode as a type anymore if sys.version_info < (3, 0): if type(value) is type(unicode()): value = str(value) if type(value) is type(str()): if len(value.split(',')) > 1: # It's a csv string. return True else: # There's only one value here, so it's not a csv string return False else: logger.error("Passed value: {0} is not a list or a string but is of type {1}".format(value, type(value))) sys.exit(1) def index_closed(client, index_name): """ Return `True` if the indicated index is closed. :arg client: The Elasticsearch client connection :arg index_name: The index name :rtype: bool """ try: index_metadata = client.cluster.state( index=index_name, metric='metadata', ) return index_metadata['metadata']['indices'][index_name]['state'] == 'close' except elasticsearch.TransportError as e: if e.status_code == 401: # This workaround using the _cat API is for AWS Elasticsearch, since it does # not allow users to poll the cluster state. It is slower than using the # cluster state, especially when iterating over large numbers of indices. try: if get_version(client) >= (1, 5, 0): indices = client.cat.indices(index=index_name, format='json', h='status') # This workaround is also for AWS, as the _cat API still returns text/plain # even with ``format='json'``. if not isinstance(indices, list): # content-type: text/plain indices = json.loads(indices) (index_info,) = indices return index_info['status'] == 'close' else: logger.error('Unable to determine whether index {0} is open or closed'.format(index_name)) raise except Exception: logger.error('Unable to determine whether index {0} is open or closed'.format(index_name)) raise else: logger.debug('Error {0}'.format(e)) else: logger.error('Unable to determine whether index {0} is open or closed'.format(index_name)) raise def get_segmentcount(client, index_name): """ Return a tuple of `(shardcount, segmentcount)` from the provided `index_name`. :arg client: The Elasticsearch client connection :arg index_name: The index name :rtype: tuple """ shards = client.indices.segments(index=index_name)['indices'][index_name]['shards'] segmentcount = 0 totalshards = 0 # We will increment this manually to capture all replicas... for shardnum in shards: for shard in range(0,len(shards[shardnum])): segmentcount += shards[shardnum][shard]['num_search_segments'] totalshards += 1 return totalshards, segmentcount def chunk_index_list(indices): """ This utility chunks very large index lists into 3KB chunks It measures the size as a csv string, then converts back into a list for the return value. :arg indices: A list of indices to act on. """ chunks = [] chunk = "" for index in indices: if len(chunk) < 3072: if not chunk: chunk = index else: chunk += "," + index else: chunks.append(chunk.split(',')) chunk = index chunks.append(chunk.split(',')) return chunks def optimized(client, index_name, max_num_segments=None): """ Check if an index is optimized. :arg client: The Elasticsearch client connection :arg index_name: The index name :arg max_num_segments: Merge to this number of segments per shard. :rtype: bool """ if not max_num_segments: logger.error("Mising value for max_num_segments.") raise ValueError if check_csv(index_name): logger.error("Must specify only a single index as an argument.") raise ValueError if index_closed(client, index_name): # Don't try to optimize a closed index logger.debug('Skipping index {0}: Already closed.'.format(index_name)) return True shards, segmentcount = get_segmentcount(client, index_name) logger.debug('Index {0} has {1} shards and {2} segments total.'.format(index_name, shards, segmentcount)) if segmentcount > (shards * max_num_segments): logger.debug('Flagging index {0} for optimization.'.format(index_name)) return False else: logger.debug('Skipping index {0}: Already optimized.'.format(index_name)) return True def get_version(client): """ Return the ES version number as a tuple. Omits trailing tags like -dev, or Beta :arg client: The Elasticsearch client connection :rtype: tuple """ version = client.info()['version']['number'] version = version.split('-')[0] if len(version.split('.')) > 3: version = version.split('.')[:-1] else: version = version.split('.') return tuple(map(int, version)) def is_master_node(client): """ Return `True` if the connected client node is the elected master node in the Elasticsearch cluster, otherwise return `False`. :arg client: The Elasticsearch client connection :rtype: bool """ my_node_id = list(client.nodes.info('_local')['nodes'])[0] master_node_id = client.cluster.state(metric='master_node')['master_node'] return my_node_id == master_node_id def get_repository(client, repository=''): """ Return configuration information for the indicated repository. :arg client: The Elasticsearch client connection :arg repository: The Elasticsearch snapshot repository to use :rtype: dict """ try: return client.snapshot.get_repository(repository=repository) except (elasticsearch.TransportError, elasticsearch.NotFoundError): logger.error("Repository {0} not found.".format(repository)) return False def get_snapshot(client, repository='', snapshot=''): """ Return information about a snapshot (or a comma-separated list of snapshots) If no snapshot specified, it will return all snapshots. If none exist, an empty dictionary will be returned. :arg client: The Elasticsearch client connection :arg repository: The Elasticsearch snapshot repository to use :arg snapshot: The snapshot name, or a comma-separated list of snapshots :rtype: dict """ if not repository: logger.error('Missing required repository parameter') return False try: return client.snapshot.get(repository=repository, snapshot=snapshot) except (elasticsearch.TransportError, elasticsearch.NotFoundError): logger.error("Snapshot: {0} or repository: {1} not found.".format(snapshot, repository)) return False def get_snapshots(client, repository=None): """ Get ``_all`` snapshots from repository and return a list. :arg client: The Elasticsearch client connection :arg repository: The Elasticsearch snapshot repository to use :rtype: list of strings """ if not repository: logger.error('Missing required repository parameter') return False try: allsnaps = client.snapshot.get(repository=repository, snapshot="_all")['snapshots'] return [snap['snapshot'] for snap in allsnaps if 'snapshot' in snap.keys()] except (elasticsearch.TransportError, elasticsearch.NotFoundError): logger.error("Unable to find all snapshots in repository: {0}".format(repository)) return False def create_snapshot_body(indices, ignore_unavailable=False, include_global_state=True, partial=False): """ Create the request body for creating a snapshot from the provided arguments. :arg indices: A single index, or list of indices to snapshot. :arg ignore_unavailable: Boolean. Ignore unavailable shards/indices. (default: `False`) :arg include_global_state: Boolean. Store cluster global state with snapshot. (default: `True`) :arg partial: Boolean. Do not fail if primary shard is unavailable. (default: `False`) :rtype: dict """ if not indices: logger.error('Missing required repository parameter') return False body = { "ignore_unavailable": ignore_unavailable, "include_global_state": include_global_state, "partial": partial, } if indices == '_all': body["indices"] = indices else: body["indices"] = to_csv(indices) return body def delete_callback(ctx, param, value): if not value: ctx.abort() def create_repo_body(repo_type=None, compress=True, chunk_size=None, max_restore_bytes_per_sec=None, max_snapshot_bytes_per_sec=None, location=None, bucket=None, region=None, base_path=None, access_key=None, secret_key=None, **kwargs): """ Build the 'body' portion for use in creating a repository. :arg repo_type: The type of repository (presently only `fs` and `s3`) :arg compress: Turn on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. (Default: `True`) :arg chunk_size: The chunk size can be specified in bytes or by using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size). :arg max_restore_bytes_per_sec: Throttles per node restore rate. Defaults to ``20mb`` per second. :arg max_snapshot_bytes_per_sec: Throttles per node snapshot rate. Defaults to ``20mb`` per second. :arg location: Location of the snapshots. Required. :arg bucket: `S3 only.` The name of the bucket to be used for snapshots. Required. :arg region: `S3 only.` The region where bucket is located. Defaults to `US Standard` :arg base_path: `S3 only.` Specifies the path within bucket to repository data. Defaults to value of ``repositories.s3.base_path`` or to root directory if not set. :arg access_key: `S3 only.` The access key to use for authentication. Defaults to value of ``cloud.aws.access_key``. :arg secret_key: `S3 only.` The secret key to use for authentication. Defaults to value of ``cloud.aws.secret_key``. :returns: A dictionary suitable for creating a repository from the provided arguments. :rtype: dict """ # This shouldn't happen, but just in case... if not repo_type: click.echo(click.style('Missing required parameter --repo_type', fg='red', bold=True)) sys.exit(1) argdict = locals() body = {} body['type'] = argdict['repo_type'] body['settings'] = {} settings = [] maybes = [ 'compress', 'chunk_size', 'max_restore_bytes_per_sec', 'max_snapshot_bytes_per_sec' ] s3 = ['bucket', 'region', 'base_path', 'access_key', 'secret_key'] settings += [i for i in maybes if argdict[i]] # Type 'fs' if argdict['repo_type'] == 'fs': settings.append('location') # Type 's3' if argdict['repo_type'] == 's3': settings += [i for i in s3 if argdict[i]] for k in settings: body['settings'][k] = argdict[k] return body def create_repository(client, **kwargs): """ Create repository with repository and body settings :arg client: The Elasticsearch client connection :arg repo_type: The type of repository (presently only `fs` and `s3`) :arg compress: Turn on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. (Default: `True`) :arg chunk_size: The chunk size can be specified in bytes or by using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size). :arg max_restore_bytes_per_sec: Throttles per node restore rate. Defaults to ``20mb`` per second. :arg max_snapshot_bytes_per_sec: Throttles per node snapshot rate. Defaults to ``20mb`` per second. :arg location: Location of the snapshots. Required. :arg bucket: `S3 only.` The name of the bucket to be used for snapshots. Required. :arg region: `S3 only.` The region where bucket is located. Defaults to `US Standard` :arg base_path: `S3 only.` Specifies the path within bucket to repository data. Defaults to value of ``repositories.s3.base_path`` or to root directory if not set. :arg access_key: `S3 only.` The access key to use for authentication. Defaults to value of ``cloud.aws.access_key``. :arg secret_key: `S3 only.` The secret key to use for authentication. Defaults to value of ``cloud.aws.secret_key``. :returns: A boolean value indicating success or failure. :rtype: boolean """ if not 'repository' in kwargs: click.echo(click.style('Missing required parameter --repository', fg='red', bold=True)) sys.exit(1) else: repository = kwargs['repository'] try: body = create_repo_body(**kwargs) logger.info("Checking if repository {0} already exists...".format(repository)) result = get_repository(client, repository=repository) logger.debug("Result = {0}".format(result)) if not result: logger.info("Repository {0} not in Elasticsearch. Continuing...".format(repository)) client.snapshot.create_repository(repository=repository, body=body) elif result is not None and repository not in result: logger.info("Repository {0} not in Elasticsearch. Continuing...".format(repository)) client.snapshot.create_repository(repository=repository, body=body) else: logger.error("Unable to create repository {0}. A repository with that name already exists.".format(repository)) sys.exit(1) except elasticsearch.TransportError as e: logger.error("Unable to create repository {0}. Response Code: {1}. Error: {2}. Check curator and elasticsearch logs for more information.".format(repository, e.status_code, e.error)) return False logger.info("Repository {0} creation initiated...".format(repository)) return True def verify_repository(client, repository=None): """ Verify the existence of a repository Args: client: The Elasticsearch client connection Kwargs: repository: The Elasticsearch snapshot repository to use Returns: A boolean value indicating success or failure. """ if not repository: click.echo(click.style('Missing required parameter --repository', fg='red', bold=True)) sys.exit(1) test_result = get_repository(client, repository) if repository in test_result: logger.info("Repository {0} creation validated.".format(repository)) return True else: logger.error("Repository {0} failed validation...".format(repository)) return False def prune_kibana(indices): """Remove any index named `.kibana`, `kibana-int`, or `.marvel-kibana` :arg indices: A list of indices to act upon. :rtype: list """ indices = ensure_list(indices) if '.marvel-kibana' in indices: indices.remove('.marvel-kibana') if 'kibana-int' in indices: indices.remove('kibana-int') if '.kibana' in indices: indices.remove('.kibana') return indices def prune_closed(client, indices): """ Return list of indices that are not closed. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: list """ return prune_open_or_closed(client, indices, append_closed=False) def prune_opened(client, indices): """ Return list of indices that are not open. :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :rtype: list """ return prune_open_or_closed(client, indices, append_closed=True) def prune_open_or_closed(client, indices, append_closed): indices = ensure_list(indices) retval = [] status = 'Closed' if append_closed else 'Opened' for idx in list(indices): if index_closed(client, idx) is append_closed: retval.append(idx) logger.debug('Including index {0}: {1}.'.format(idx, status)) else: logger.debug('Skipping index {0}: {1}.'.format(idx, status)) return sorted(retval) def prune_allocated(client, indices, key, value, allocation_type): """ Return list of indices that do not have the routing allocation rule of `key=value` :arg client: The Elasticsearch client connection :arg indices: A list of indices to act on :arg key: The allocation attribute to check for :arg value: The value to check for :arg allocation_type: Type of allocation to apply :rtype: list """ indices = ensure_list(indices) retval = [] for idx in indices: settings = client.indices.get_settings( index=idx, ) try: has_routing = settings[idx]['settings']['index']['routing']['allocation'][allocation_type][key] == value except KeyError: has_routing = False if has_routing: logger.debug('Skipping index {0}: Allocation rule {1} is already applied for type {2}.'.format(idx, key + "=" + value, allocation_type)) else: logger.info('Flagging index {0} to have allocation rule {1} applied for type {2}.'.format(idx, key + "=" + value, allocation_type)) retval.append(idx) return sorted(retval) curator-3.4.1/curator/cli/000077500000000000000000000000001265673226500154425ustar00rootroot00000000000000curator-3.4.1/curator/cli/__init__.py000066400000000000000000000005711265673226500175560ustar00rootroot00000000000000from .utils import * from .cli import * from .alias import * from .allocation import * from .bloom import * from .close import * from .delete import * from .opener import * from .optimize import * from .replicas import * from .seal import * from .show import * from .snapshot import * from .index_selection import * from .snapshot_selection import * from .es_repo_mgr import * curator-3.4.1/curator/cli/alias.py000066400000000000000000000012461265673226500171100ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('alias') @click.option('--name', help="Alias name", type=str) @click.option('--remove', is_flag=True, show_default=True, expose_value=True, help='Remove from alias rather than add.') @click.pass_context def alias(ctx, name, remove): """Index Aliasing""" if not name: msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet']) logger.error('Missing required parameter --name') msgout('Missing required parameter --name', error=True, quiet=ctx.parent.params['quiet']) sys.exit(1) alias.add_command(indices) curator-3.4.1/curator/cli/allocation.py000066400000000000000000000021201265673226500201340ustar00rootroot00000000000000import elasticsearch import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('allocation') @click.option('--rule', show_default=True, expose_value=True, type=str, help='Routing allocation rule to apply, e.g. tag=ssd') @click.option('--type', default='require', show_default=True, expose_value=True, type=str, help='Specify an allocation type, include, exclude or require') @click.pass_context def allocation(ctx, rule, type): """Index Allocation""" if not rule: msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet']) logger.error('Missing required parameter --rule') msgout('Missing required parameter --rule', error=True, quiet=ctx.parent.params['quiet']) sys.exit(1) if type not in ['require', 'include', 'exclude']: logger.error('--type can only be one of: require, include exclude') msgout('--type can only be one of: require, include exclude', error=True, quiet=ctx.parent.params['quiet']) sys.exit(1) allocation.add_command(indices) curator-3.4.1/curator/cli/bloom.py000066400000000000000000000006121265673226500171230ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('bloom') @click.option('--delay', type=int, default=0, show_default=True, expose_value=True, help='Pause *n* seconds after disabling bloom filter cache of an index') @click.pass_context def bloom(ctx, delay): """Disable bloom filter cache""" bloom.add_command(indices) curator-3.4.1/curator/cli/cli.py000066400000000000000000000102211265673226500165570ustar00rootroot00000000000000import elasticsearch import click import re from .utils import * from .. import __version__ import logging logger = logging.getLogger(__name__) try: from logging import NullHandler except ImportError: from logging import Handler class NullHandler(Handler): def emit(self, record): pass DEFAULT_ARGS = { 'host': 'localhost', 'url_prefix': '', 'port': 9200, 'http_auth': None, 'use_ssl': False, 'timeout': 30, 'dry_run': False, 'debug': False, 'log_level': 'INFO', 'logformat': 'default', } @click.group() @click.option('--host', help='Elasticsearch host.', default=DEFAULT_ARGS['host']) @click.option('--url_prefix', help='Elasticsearch http url prefix.', default=DEFAULT_ARGS['url_prefix']) @click.option('--port', help='Elasticsearch port.', default=DEFAULT_ARGS['port'], type=int) @click.option('--use_ssl', help='Connect to Elasticsearch through SSL.', is_flag=True, default=DEFAULT_ARGS['use_ssl']) @click.option('--certificate', help='Path to certificate to use for SSL validation. (OPTIONAL)', type=str, default=None) @click.option('--ssl-no-validate', help='Do not validate SSL certificate', is_flag=True) @click.option('--http_auth', help='Use Basic Authentication ex: user:pass', default=DEFAULT_ARGS['http_auth']) @click.option('--timeout', help='Connection timeout in seconds.', default=DEFAULT_ARGS['timeout'], type=int) @click.option('--master-only', is_flag=True, help='Only operate on elected master node.') @click.option('--dry-run', is_flag=True, help='Do not perform any changes.', default=DEFAULT_ARGS['dry_run']) @click.option('--debug', is_flag=True, help='Debug mode', default=DEFAULT_ARGS['debug']) @click.option('--loglevel', help='Log level', default=DEFAULT_ARGS['log_level']) @click.option('--logfile', help='log file') @click.option('--logformat', help='Log output format [default|logstash].', default=DEFAULT_ARGS['logformat']) @click.option('--quiet', help='Suppress command-line output.', is_flag=True) @click.version_option(version=__version__) @click.pass_context def cli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http_auth, timeout, master_only, dry_run, debug, loglevel, logfile, logformat, quiet): """ Curator for Elasticsearch indices. See http://elastic.co/guide/en/elasticsearch/client/curator/current """ # Setup logging if debug: numeric_log_level = logging.DEBUG or loglevel.upper() == 'DEBUG' format_string = '%(asctime)s %(levelname)-9s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s' else: numeric_log_level = getattr(logging, loglevel.upper(), None) format_string = '%(asctime)s %(levelname)-9s %(message)s' if not isinstance(numeric_log_level, int): raise ValueError('Invalid log level: {0}'.format(loglevel)) handler = logging.StreamHandler( open(logfile, 'a') if logfile else sys.stdout) if logformat == 'logstash': ctx.params['quiet'] = True handler.setFormatter(LogstashFormatter()) else: handler.setFormatter(logging.Formatter(format_string)) logging.root.addHandler(handler) logging.root.setLevel(numeric_log_level) logger = logging.getLogger('curator.cli') # Test whether certificate is a valid file path if use_ssl is True and certificate is not None: try: open(certificate, 'r') except IOError: logger.error('Could not open certificate at {0}'.format(certificate)) msgout('Error: Could not open certificate at {0}'.format(certificate), error=True, quiet=quiet) sys.exit(1) # Filter out logging from Elasticsearch and associated modules by default if not debug: for handler in logging.root.handlers: handler.addFilter( Whitelist( 'root', '__main__', 'curator', 'curator.curator', 'curator.api', 'curator.cli', 'curator.api', 'curator.cli' ) ) # Setting up NullHandler to handle nested elasticsearch.trace Logger # instance in elasticsearch python client logging.getLogger('elasticsearch.trace').addHandler(NullHandler()) curator-3.4.1/curator/cli/close.py000066400000000000000000000003151265673226500171200ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('close') @click.pass_context def close(ctx): """Close indices""" close.add_command(indices) curator-3.4.1/curator/cli/delete.py000066400000000000000000000013141265673226500172550ustar00rootroot00000000000000import click from .index_selection import * from .snapshot_selection import * import logging logger = logging.getLogger(__name__) @cli.group('delete') @click.option('--disk-space', type=float, expose_value=True, help='Delete indices beyond DISK_SPACE gigabytes.') @click.option('--reverse', type=bool, default=True, expose_value=True, show_default=True, is_eager=True, help='Only valid with --disk-space. Affects sort order of the indices. True means reverse-alphabetical (if dates are involved, older is deleted first).') @click.pass_context def delete(ctx, disk_space, reverse): """Delete indices or snapshots""" delete.add_command(indices) delete.add_command(snapshots) curator-3.4.1/curator/cli/es_repo_mgr.py000066400000000000000000000167131265673226500203250ustar00rootroot00000000000000import elasticsearch import click import re import sys from .utils import * from ..api import * from .. import __version__ import logging logger = logging.getLogger(__name__) try: from logging import NullHandler except ImportError: from logging import Handler class NullHandler(Handler): def emit(self, record): pass DEFAULT_ARGS = { 'host': 'localhost', 'url_prefix': '', 'port': 9200, 'http_auth': None, 'use_ssl': False, 'timeout': 300, 'dry_run': False, 'debug': False, 'log_level': 'INFO', 'logformat': 'default', } def show_repos(client): for repository in sorted(get_repository(client, '_all').keys()): print('{0}'.format(repository)) sys.exit(0) @click.command(short_help='Filesystem Repository') @click.option('--repository', required=True, type=str, help='Repository name') @click.option('--location', required=True, type=str, help='Shared file-system location. Must match remote path, & be accessible to all master & data nodes') @click.option('--compression', type=bool, default=True, show_default=True, help='Enable/Disable metadata compression.') @click.option('--chunk_size', type=str, help='Chunk size, e.g. 1g, 10m, 5k. [unbounded]') @click.option('--max_restore_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node restore rate (per second).') @click.option('--max_snapshot_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node snapshot rate (per second).') @click.pass_context def fs( ctx, repository, location, compression, chunk_size, max_restore_bytes_per_sec, max_snapshot_bytes_per_sec): """ Create a filesystem repository. """ client = get_client(**ctx.parent.parent.params) success = False if create_repository(client, repo_type='fs', **ctx.params): success = verify_repository(client, repository) if not success: sys.exit(1) @click.command(short_help='S3 Repository') @click.option('--repository', required=True, type=str, help='Repository name') @click.option('--bucket', required=True, type=str, help='S3 bucket name') @click.option('--region', type=str, help='S3 region. [US Standard]') @click.option('--base_path', type=str, help='S3 base path. [root]') @click.option('--access_key', type=str, help='S3 access key. [value of cloud.aws.access_key]') @click.option('--secret_key', type=str, help='S3 secret key. [value of cloud.aws.secret_key]') @click.option('--compression', type=bool, default=True, show_default=True, help='Enable/Disable metadata compression.') @click.option('--chunk_size', type=str, help='Chunk size, e.g. 1g, 10m, 5k. [unbounded]') @click.option('--max_restore_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node restore rate (per second).') @click.option('--max_snapshot_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node snapshot rate (per second).') @click.pass_context def s3( ctx, repository, bucket, region, base_path, access_key, secret_key, compression, chunk_size, max_restore_bytes_per_sec, max_snapshot_bytes_per_sec): """ Create an S3 repository. """ client = get_client(**ctx.parent.parent.params) success = False if create_repository(client, repo_type='s3', **ctx.params): success = verify_repository(client, repository) if not success: sys.exit(1) @click.group() @click.option('--host', help='Elasticsearch host.', default=DEFAULT_ARGS['host']) @click.option('--url_prefix', help='Elasticsearch http url prefix.', default=DEFAULT_ARGS['url_prefix']) @click.option('--port', help='Elasticsearch port.', default=DEFAULT_ARGS['port'], type=int) @click.option('--use_ssl', help='Connect to Elasticsearch through SSL.', is_flag=True, default=DEFAULT_ARGS['use_ssl']) @click.option('--certificate', help='Path to certificate to use for SSL validation. (OPTIONAL)', type=str, default=None) @click.option('--ssl-no-validate', help='Do not validate SSL certificate', is_flag=True) @click.option('--http_auth', help='Use Basic Authentication ex: user:pass', default=DEFAULT_ARGS['http_auth']) @click.option('--timeout', help='Connection timeout in seconds.', default=DEFAULT_ARGS['timeout'], type=int) @click.option('--master-only', is_flag=True, help='Only operate on elected master node.') @click.option('--debug', is_flag=True, help='Debug mode', default=DEFAULT_ARGS['debug']) @click.option('--loglevel', help='Log level', default=DEFAULT_ARGS['log_level']) @click.option('--logfile', help='log file') @click.option('--logformat', help='Log output format [default|logstash].', default=DEFAULT_ARGS['logformat']) @click.version_option(version=__version__) @click.pass_context def repomgrcli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http_auth, timeout, master_only, debug, loglevel, logfile, logformat): """Repository manager for Elasticsearch Curator. """ # Setup logging if debug: numeric_log_level = logging.DEBUG or loglevel.upper() == 'DEBUG' format_string = '%(asctime)s %(levelname)-9s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s' else: numeric_log_level = getattr(logging, loglevel.upper(), None) format_string = '%(asctime)s %(levelname)-9s %(message)s' if not isinstance(numeric_log_level, int): raise ValueError('Invalid log level: {0}'.format(loglevel)) handler = logging.StreamHandler( open(logfile, 'a') if logfile else sys.stdout) if logformat == 'logstash': handler.setFormatter(LogstashFormatter()) else: handler.setFormatter(logging.Formatter(format_string)) logging.root.addHandler(handler) logging.root.setLevel(numeric_log_level) # Filter out logging from Elasticsearch and associated modules by default if not debug: for handler in logging.root.handlers: handler.addFilter( Whitelist( 'root', '__main__', 'curator', 'curator.curator', 'curator.api', 'curator.cli', 'curator.api', 'curator.cli' ) ) # Setting up NullHandler to handle nested elasticsearch.trace Logger # instance in elasticsearch python client logging.getLogger('elasticsearch.trace').addHandler(NullHandler()) @repomgrcli.group('create') @click.pass_context def _create(ctx): """Create an Elasticsearch repository""" _create.add_command(fs) _create.add_command(s3) @repomgrcli.command('show') @click.pass_context def show(ctx): """ Show all repositories """ client = get_client(**ctx.parent.params) show_repos(client) @repomgrcli.command('delete') @click.option('--repository', required=True, help='Repository name', type=str) @click.option('--yes', is_flag=True, callback=delete_callback, expose_value=False, prompt='Are you sure you want to delete the repository?') @click.pass_context def _delete(ctx, repository): """Delete an Elasticsearch repository""" client = get_client(**ctx.parent.params) try: logger.info('Deleting repository {0}...'.format(repository)) client.snapshot.delete_repository(repository=repository) sys.exit(0) except elasticsearch.NotFoundError: logger.error("Unable to delete repository: {0} Not Found.".format(repository)) sys.exit(1) curator-3.4.1/curator/cli/index_selection.py000066400000000000000000000216331265673226500211750ustar00rootroot00000000000000from ..cli import * from ..api import * import elasticsearch import click import re import logging logger = logging.getLogger(__name__) ### INDICES @click.command(short_help="Index selection.") @click.option('--newer-than', type=int, callback=filter_callback, help='Include only indices newer than n time_units') @click.option('--older-than', type=int, callback=filter_callback, help='Include only indices older than n time_units') @click.option('--prefix', type=str, callback=filter_callback, help='Include only indices beginning with prefix.') @click.option('--suffix', type=str, callback=filter_callback, help='Include only indices ending with suffix.') @click.option('--time-unit', is_eager=True, type=click.Choice(['hours', 'days', 'weeks', 'months']), help='Unit of time to reckon by') @click.option('--timestring', type=str, is_eager=True, help="Python strftime string to match your index definition, e.g. 2014.07.15 would be %Y.%m.%d") @click.option('--regex', type=str, callback=filter_callback, help="Provide your own regex, e.g '^prefix-.*-suffix$'") @click.option('--exclude', multiple=True, callback=filter_callback, help='Exclude matching indices. Can be invoked multiple times.') @click.option('--index', multiple=True, help='Include the provided index in the list. Can be invoked multiple times.') @click.option('--all-indices', is_flag=True, help='Do not filter indices. Act on all indices.') @click.option('--closed-only', is_flag=True, help='Include only indices that are closed.') @click.pass_context def indices(ctx, newer_than, older_than, prefix, suffix, time_unit, timestring, regex, exclude, index, all_indices, closed_only): """ Get a list of indices to act on from the provided arguments, then perform the command [alias, allocation, bloom, close, delete, etc.] on the resulting list. """ # This top 'if' statement catches an edge-case I cannot depend upon click # to resolve. I cannot make options depend upon each other (yet), so I # have to test for this case here and act accordingly. if timestring and not ctx.obj['filters']: regex = r'^.*{0}.*$'.format(get_date_regex(timestring)) ctx.obj['filters'].append({ 'pattern': regex }) if not all_indices and not ctx.obj['filters'] and not index: logger.error('At least one filter must be supplied.') msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.parent.params['quiet']) msgout('ERROR. At least one filter must be supplied.', error=True, quiet=ctx.parent.parent.params['quiet']) sys.exit(1) if timestring and time_unit: if not validate_time_details(time_unit, timestring): msgout('ERROR. Timestring {0} does not contain time unit {1}'.format(timestring, time_unit)) sys.exit(1) logger.info("Job starting: {0} indices".format(ctx.parent.info_name)) # Base and client args are in the grandparent tier of the context logger.debug("Params: {0}".format(ctx.parent.parent.params)) # Set master_timeout to match 'timeout' if less than or equal to 300 seconds, # otherwise set to 300. Also, set this before overriding the timeout. master_timeout = ctx.parent.parent.params['timeout'] if ctx.parent.parent.params['timeout'] <= 300 else 300 master_timeout = master_timeout * 1000 # This value is in milliseconds, at least in Python. # If this is in error, it may be somewhere in the Python module or urllib3 # The Elasticsearch output was timing out deletes because they failed to # complete within 30ms (until I multiplied by 1000) override_timeout(ctx) # Get the client client = get_client(**ctx.parent.parent.params) # Get a master-list of indices indices = get_indices(client) logger.debug("Full list of indices: {0}".format(indices)) # Build index list if index and not ctx.obj['filters']: working_list = [] else: if indices: working_list = indices else: logger.error('No indices found in Elasticsearch') msgout('ERROR. No indices found in Elasticsearch.', error=True, quiet=ctx.parent.parent.params['quiet']) sys.exit(1) if closed_only and not all_indices: logger.info("Pruning open indices, leaving only closed indices.") working_list = prune_opened(client, working_list) # Override any other flags if --all_indices specified if all_indices: working_list = indices logger.info('Matching all indices. Ignoring flags other than --exclude.') logger.debug('All filters: {0}'.format(ctx.obj['filters'])) for f in ctx.obj['filters']: if all_indices and not 'exclude' in f: continue logger.debug('Filter: {0}'.format(f)) working_list = apply_filter(working_list, **f) if ctx.parent.info_name == "delete": # Protect against accidental delete logger.info("Pruning Kibana-related indices to prevent accidental deletion.") working_list = prune_kibana(working_list) # If there are manually added indices, we will add them here working_list.extend(in_list(index, indices)) if working_list and ctx.parent.info_name == 'delete': # If filter by disk space, filter the working_list by space: if ctx.parent.params['disk_space']: logger.info("Filtering to keep disk usage below {0} gigabytes".format(ctx.parent.params['disk_space'])) working_list = filter_by_space( client, working_list, disk_space=ctx.parent.params['disk_space'], reverse=ctx.parent.params['reverse'] ) if working_list: ### Issue #348 # I don't care about using only --timestring if it's a `show` or `dry_run` if timestring and not newer_than and not older_than \ and not (ctx.parent.info_name == 'show') \ and not ctx.parent.parent.params['dry_run']: if ctx.parent.parent.params['quiet']: # Don't output to stdout if 'quiet' (or logformat == logstash) logger.warn('You are using --timestring without --older-than or --newer-than.') logger.warn('This could result in actions being performed on all indices matching {0}'.format(timestring)) else: # Do this if not quiet mode. logger.warn('You are using --timestring without --older-than or --newer-than.') logger.warn('This could result in actions being performed on all indices matching {0}'.format(timestring)) msgout('You are using --timestring without --older-than or --newer-than.', warning=True, quiet=ctx.parent.parent.params['quiet']) msgout('This could result in actions being performed on all indices matching {0}'.format(timestring), warning=True, quiet=ctx.parent.parent.params['quiet']) msgout('Press CTRL-C to exit Curator before the timer expires:', error=True, quiet=ctx.parent.parent.params['quiet']) countdown(10) # Make a sorted, unique list of indices working_list = sorted(list(set(working_list))) logger.info('Action {0} will be performed on the following indices: {1}'.format(ctx.parent.info_name, working_list)) # Do action here!!! Don't forget to account for DRY_RUN!!! if ctx.parent.info_name == 'show': logger.info('Matching indices:') show(client, working_list, type='indices') else: if ctx.parent.parent.params['dry_run']: show_dry_run(client, working_list, ctx.parent.info_name, type='indices') else: # The snapshot command should get the full list, otherwise # the index list may need to be segmented. if len(to_csv(working_list)) > 3072 and not ctx.parent.info_name == 'snapshot': logger.warn('Very large list of indices. Breaking it up into smaller chunks.') index_lists = chunk_index_list(working_list) success = True for l in index_lists: retval = do_command(client, ctx.parent.info_name, l, ctx.parent.params, master_timeout) if not retval: success = False time.sleep(2) # Pause in between iterations exit_msg(success) else: retval = do_command(client, ctx.parent.info_name, working_list, ctx.parent.params, master_timeout) exit_msg(retval) else: logger.warn('No indices matched provided args: {0}'.format(ctx.params)) msgout('No indices matched provided args: {0}'.format(ctx.params), quiet=ctx.parent.parent.params['quiet']) sys.exit(0) curator-3.4.1/curator/cli/opener.py000066400000000000000000000003131265673226500173010ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('open') @click.pass_context def _open(ctx): """Open indices""" _open.add_command(indices) curator-3.4.1/curator/cli/optimize.py000066400000000000000000000013771265673226500176640ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('optimize') @click.option('--delay', type=int, default=0, show_default=True, expose_value=True, help='Pause *n* seconds after optimizing an index.') @click.option('--max_num_segments', type=int, default=2, show_default=True, expose_value=True, help='Merge to this number of segments per shard.') @click.option('--request_timeout', type=int, default=21600, show_default=True, expose_value=True, help='Allow this many seconds before the transaction times out.') @click.pass_context def optimize(ctx, delay, max_num_segments, request_timeout): """Optimize Indices""" optimize.add_command(indices) curator-3.4.1/curator/cli/replicas.py000066400000000000000000000012271265673226500176200ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('replicas') @click.option('--count', type=int, expose_value=True, help='Number of replicas the indices should have.') @click.pass_context def replicas(ctx, count): """Replica Count Per-shard""" if count == None: # Have to do this since 0 is valid msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet']) logger.error('Missing required parameter --count') msgout('Missing required parameter --count', error=True, quiet=ctx.parent.params['quiet']) sys.exit(1) replicas.add_command(indices) curator-3.4.1/curator/cli/seal.py000066400000000000000000000003661265673226500167450ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) @cli.group('seal') @click.pass_context def seal(ctx): """ Seal indices (Synced flush: ES 1.6.0+ only) """ seal.add_command(indices) curator-3.4.1/curator/cli/show.py000066400000000000000000000004241265673226500167740ustar00rootroot00000000000000import click from .index_selection import * from .snapshot_selection import * import logging logger = logging.getLogger(__name__) @cli.group('show') @click.pass_context def show(ctx): """Show indices or snapshots""" show.add_command(indices) show.add_command(snapshots) curator-3.4.1/curator/cli/snapshot.py000066400000000000000000000040761265673226500176620ustar00rootroot00000000000000import click from .index_selection import * import logging logger = logging.getLogger(__name__) DEFAULT_ARGS = { 'snapshot_prefix': 'curator-', 'wait_for_completion': True, 'include_global_state': True, 'skip_repo_validation': False, } @cli.group('snapshot') @click.option('--repository', help='Repository name.', expose_value=True) @click.option('--name', help='Override default name.', expose_value=True) @click.option('--prefix', help='Override default prefix.', expose_value=True, default=DEFAULT_ARGS['snapshot_prefix']) @click.option('--wait_for_completion', type=bool, show_default=True, default=DEFAULT_ARGS['wait_for_completion'], expose_value=True, help='Wait for snapshot to complete before returning.') @click.option('--ignore_unavailable', is_flag=True, expose_value=True, help='Ignore unavailable shards/indices.') @click.option('--include_global_state', type=bool, show_default=True, default=DEFAULT_ARGS['include_global_state'], expose_value=True, help='Store cluster global state with snapshot.') @click.option('--partial', is_flag=True, expose_value=True, help='Do not fail if primary shard is unavailable.') @click.option('--request_timeout', type=int, default=21600, show_default=True, expose_value=True, help='Allow this many seconds before the transaction times out.') @click.option('--skip-repo-validation', is_flag=True, expose_value=True, help='Skip repository access validation.') @click.pass_context def snapshot( ctx, repository, name, prefix, wait_for_completion, ignore_unavailable, include_global_state, partial, request_timeout, skip_repo_validation, ): """Take snapshots of indices (Backup)""" if not repository: msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet']) logger.error('Missing required parameter --repository') msgout('Missing required parameter --repository', error=True, quiet=ctx.parent.params['quiet']) sys.exit(1) snapshot.add_command(indices) curator-3.4.1/curator/cli/snapshot_selection.py000066400000000000000000000107651265673226500217310ustar00rootroot00000000000000from . import * from ..api import * import elasticsearch import click import re import logging logger = logging.getLogger(__name__) ### SNAPSHOTS @click.command(short_help="Snapshot selection.") @click.option('--newer-than', type=int, callback=filter_callback, help='Include only snapshots newer than n time_units') @click.option('--older-than', type=int, callback=filter_callback, help='Include only snapshots older than n time_units') @click.option('--prefix', type=str, callback=filter_callback, help='Include only snapshots beginning with prefix.') @click.option('--suffix', type=str, callback=filter_callback, help='Include only snapshots ending with suffix.') @click.option('--time-unit', is_eager=True, type=click.Choice(['hours', 'days', 'weeks', 'months']), help='Unit of time to reckon by') @click.option('--timestring', type=str, is_eager=True, default='%Y%m%d%H%M%S', help="Python strftime string to match your snapshot's definition, e.g. 20140715020304 would be %Y%m%d%H%M%S") @click.option('--regex', type=str, callback=filter_callback, help="Provide your own regex, e.g '^prefix-.*-suffix$'") @click.option('--exclude', multiple=True, callback=filter_callback, help='Exclude matching snapshots. Can be invoked multiple times.') @click.option('--snapshot', multiple=True, help='Include the provided snapshot in the list. Can be invoked multiple times.') @click.option('--all-snapshots', is_flag=True, help='Do not filter snapshots. Act on all snapshots.') @click.option('--repository', type=str, is_eager=True, expose_value=True, help='Repository name.') @click.pass_context def snapshots(ctx, newer_than, older_than, prefix, suffix, time_unit, timestring, regex, exclude, snapshot, all_snapshots, repository): """ Get a list of snapshots to act on from the provided arguments, then perform the command [delete, show] on the resulting list. """ if not repository: msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.parent.params['quiet']) logger.error('Missing required parameter --repository') msgout('Missing required parameter --repository', error=True, quiet=ctx.parent.parent.params['quiet']) sys.exit(1) logger.info("Job starting: {0} snapshots".format(ctx.parent.info_name)) # Base and client args are in the grandparent tier of the context if ctx.parent.parent.params['dry_run']: logging.info("DRY RUN MODE. No changes will be made.") client = get_client(**ctx.parent.parent.params) # Get a master-list of indices snapshots = get_snapshots(client, repository=repository) if snapshots: working_list = snapshots else: logger.error('No snapshots found in Elasticsearch.') msgout('No snapshots found in Elasticsearch.', error=True, quiet=ctx.parent.parent.params['quiet']) sys.exit(0) if all_snapshots: logger.info('Matching all snapshots. Ignoring flags other than --exclude.') else: logger.debug('All filters: {0}'.format(ctx.obj['filters'])) for f in ctx.obj['filters']: if all_snapshots and not 'exclude' in f: continue logger.debug('Filter: {0}'.format(f)) working_list = apply_filter(working_list, **f) # If there are manually added snapshots, we will add them here working_list.extend(in_list(snapshot, snapshots)) if working_list: # Make a sorted, unique list of indices working_list = sorted(list(set(working_list))) logger.debug('ACTION: {0} will be executed against the following snapshots: {1}'.format(ctx.parent.info_name, working_list)) if ctx.parent.info_name == 'show': logger.info('Matching snapshots:') show(client, working_list) elif ctx.parent.parent.params['dry_run']: show_dry_run(client, working_list, ctx.parent.info_name) elif ctx.parent.info_name == 'delete': success = True for snap in working_list: retval = delete_snapshot(client, snapshot=snap, repository=repository) # If we fail once, we fail completely if not retval: success = False exit_msg(success) else: logger.warn('No snapshots matched provided args.') msgout('No snapshots matched provided args.', quiet=ctx.parent.parent.params['quiet']) sys.exit(0) curator-3.4.1/curator/cli/utils.py000066400000000000000000000242641265673226500171640ustar00rootroot00000000000000import click import sys import re import time import logging import json from .utils import * import elasticsearch from ..api import * logger = logging.getLogger(__name__) # Elasticsearch versions supported version_max = (3, 0, 0) version_min = (1, 0, 0) REGEX_MAP = { 'timestring': r'^.*{0}.*$', 'newer_than': r'(?P{0})', 'older_than': r'(?P{0})', 'prefix': r'^{0}.*$', 'suffix': r'^.*{0}$', } def countdown(seconds): """Display an inline countdown to stdout.""" for i in range(seconds,0,-1): sys.stdout.write(str(i) + ' ') sys.stdout.flush() time.sleep(1) class LogstashFormatter(logging.Formatter): # The LogRecord attributes we want to carry over to the Logstash message, # mapped to the corresponding output key. WANTED_ATTRS = {'levelname': 'loglevel', 'funcName': 'function', 'lineno': 'linenum', 'message': 'message', 'name': 'name'} def converter(self, timevalue): return time.gmtime(timevalue) def format(self, record): timestamp = '%s.%03dZ' % ( self.formatTime(record, datefmt='%Y-%m-%dT%H:%M:%S'), record.msecs) result = {'message': record.getMessage(), '@timestamp': timestamp} for attribute in set(self.WANTED_ATTRS).intersection(record.__dict__): result[self.WANTED_ATTRS[attribute]] = getattr(record, attribute) return json.dumps(result, sort_keys=True) class Whitelist(logging.Filter): def __init__(self, *whitelist): self.whitelist = [logging.Filter(name) for name in whitelist] def filter(self, record): return any(f.filter(record) for f in self.whitelist) def exit_msg(success): """ Display a message corresponding to whether the job completed successfully or not, then exit. """ if success: logger.info("Job completed successfully.") else: logger.warn("Job did not complete successfully.") sys.exit(0) if success else sys.exit(1) def show_dry_run(client, items, command, type=None): """ Log dry run output with the command which would have been executed. """ logger.info("DRY RUN MODE. No changes will be made.") for item in items: if type == 'indices': logger.info("DRY RUN: {0}: {1}{2}".format(command, item, ' (CLOSED)' if index_closed(client, item) else '')) else: logger.info("DRY RUN: {0}: {1}".format(command, item)) def check_version(client): """ Verify version is within acceptable range. Exit with error if it is not. :arg client: The Elasticsearch client connection """ version_number = get_version(client) logger.debug('Detected Elasticsearch version {0}'.format(".".join(map(str,version_number)))) if version_number >= version_max or version_number < version_min: logger.error('Expected Elasticsearch version range > {0} < {1}'.format(".".join(map(str,version_min)),".".join(map(str,version_max)))) logger.error('Incompatible with version {0} of Elasticsearch.'.format(".".join(map(str,version_number)))) sys.exit(1) def check_master(client, master_only=False): """ Check if master node. If not, exit """ if master_only and not is_master_node(client): logger.info('Master-only flag detected. Connected to non-master node. Aborting.') sys.exit(0) def get_client(**kwargs): """Return an Elasticsearch client using the provided parameters """ kwargs['master_only'] = False if not 'master_only' in kwargs else kwargs['master_only'] kwargs['use_ssl'] = False if not 'use_ssl' in kwargs else kwargs['use_ssl'] kwargs['ssl_no_validate'] = False if not 'ssl_no_validate' in kwargs else kwargs['ssl_no_validate'] kwargs['certificate'] = False if not 'certificate' in kwargs else kwargs['certificate'] logger.debug("kwargs = {0}".format(kwargs)) master_only = kwargs.pop('master_only') if kwargs['use_ssl']: if kwargs['ssl_no_validate']: kwargs['verify_certs'] = False # Not needed, but explicitly defined else: logger.info('Attempting to verify SSL certificate.') # If user provides a certificate: if kwargs['certificate']: kwargs['verify_certs'] = True kwargs['ca_certs'] = kwargs['certificate'] else: # Try to use certifi certificates: try: import certifi kwargs['verify_certs'] = True kwargs['ca_certs'] = certifi.where() except ImportError: logger.warn('Unable to verify SSL certificate.') try: client = elasticsearch.Elasticsearch(**kwargs) # Verify the version is acceptable. check_version(client) # Verify "master_only" status, if applicable check_master(client, master_only=master_only) return client except Exception: logger.error('Connection failure.') sys.exit(1) def override_timeout(ctx): """ Override the default timeout for optimize and snapshot operations if the default value of 30 is provided at the command-line. """ timeout = 21600 if ctx.parent.info_name in ['optimize', 'snapshot']: if ctx.parent.parent.params['timeout'] == 30: logger.warn('Overriding default connection timeout. New timeout: {0}'.format(timeout)) ctx.parent.parent.params['timeout'] = timeout def filter_callback(ctx, param, value): """ Append a dict to ctx.obj['filters'] based on the arguments """ # Stop here if None or empty value, but zero values are okay if value == 0: argdict = {} elif not value: return value else: argdict = {} if param.name in ['older_than', 'newer_than']: if not ctx.params['time_unit'] : logger.error("Parameters --older-than and --newer-than require the --time-unit parameter") sys.exit(1) if not ctx.params['timestring']: logger.error("Parameters --older-than and --newer-than require the --timestring parameter") sys.exit(1) argdict = { "groupname":'date', "time_unit":ctx.params["time_unit"], "timestring": ctx.params['timestring'], "value": value, "method": param.name } date_regex = get_date_regex(ctx.params['timestring']) regex = REGEX_MAP[param.name].format(date_regex) elif param.name == 'regex': regex = r'{0}'.format(value) elif param.name in ['prefix', 'suffix']: regex = REGEX_MAP[param.name].format(value) if param.name == 'exclude': for e in value: argdict = {} argdict['pattern'] = '{0}'.format(e) argdict['exclude'] = True ctx.obj['filters'].append(argdict) logger.debug("Added filter: {0}".format(argdict)) else: logger.debug("REGEX = {0}".format(regex)) argdict['pattern'] = regex ctx.obj['filters'].append(argdict) logger.debug("Added filter: {0}".format(argdict)) logger.debug("New list of filters: {0}".format(ctx.obj['filters'])) return value def in_list(values, source_list): """ Return a list of values found inside source_list. While a list comprehension is faster, it doesn't log failures. :arg values: A list of items to compare to the ``source_list`` :arg source_list: A list of items """ retval = [] for v in values: if v in source_list: logger.info('Adding {0} from command-line argument'.format(v)) retval.append(v) else: logger.warn('{0} not found!'.format(v)) return retval def do_command(client, command, indices, params=None, master_timeout=30000): """ Do the command. """ if command == "alias": return alias( client, indices, alias=params['name'], remove=params['remove'] ) if command == "allocation": return allocation(client, indices, rule=params['rule'], allocation_type=params['type'] ) if command == "bloom": return bloom(client, indices, delay=params['delay']) if command == "close": return close(client, indices) if command == "delete": return delete(client, indices, master_timeout) if command == "open": return opener(client, indices) if command == "optimize": return optimize( client, indices, max_num_segments=params['max_num_segments'], delay=params['delay'], request_timeout=params['request_timeout'] ) if command == "replicas": return replicas(client, indices, replicas=params['count']) if command == "seal": return seal(client, indices) if command == "snapshot": return create_snapshot( client, indices=indices, name=params['name'], prefix=params['prefix'], repository=params['repository'], ignore_unavailable=params['ignore_unavailable'], include_global_state=params['include_global_state'], partial=params['partial'], wait_for_completion=params['wait_for_completion'], request_timeout=params['request_timeout'], skip_repo_validation=params['skip_repo_validation'], ) def msgout(msg, error=False, warning=False, quiet=False): """Output messages to stdout via click.echo if quiet=False""" if not quiet: if error: click.echo(click.style(click.style(msg, fg='red', bold=True))) elif warning: click.echo(click.style(click.style(msg, fg='yellow', bold=True))) else: click.echo(msg) def validate_time_details(time_unit, timestring): """ Validate that the appropriate element(s) for time_unit are in the timestring. """ retval = False if time_unit == 'hours': if '%H' in timestring: retval = True elif time_unit == 'days': if '%d' in timestring: retval = True elif time_unit == 'weeks': if '%W' in timestring: retval = True elif time_unit == 'months': if '%m' in timestring: retval = True return retval curator-3.4.1/curator/curator.py000077500000000000000000000001201265673226500167200ustar00rootroot00000000000000import click from .cli import cli def main(): cli( obj={ "filters": [] } ) curator-3.4.1/curator/es_repo_mgr.py000066400000000000000000000001241265673226500175430ustar00rootroot00000000000000import click from .cli import es_repo_mgr def main(): es_repo_mgr.repomgrcli() curator-3.4.1/docs/000077500000000000000000000000001265673226500141445ustar00rootroot00000000000000curator-3.4.1/docs/Changelog.rst000066400000000000000000000667261265673226500166060ustar00rootroot00000000000000.. _changelog: Changelog ========= 3.4.1 (10 February 2016) ------------------------ **General** * Update license copyright to 2016 * Use slim python version with Docker #527 (xaka) * Changed ``--master-only`` exit code to 0 when connected to non-master node #540 (wkruse) * Add ``cx_Freeze`` capability to ``setup.py``, plus a ``binary_release.py`` script to simplify binary package creation. #554 (untergeek) * Set Elastic as author. #555 (untergeek) * Put repository creation methods into API and document them. Requested in #550 (untergeek) **Bug fixes** * Fix sphinx documentation build error #506 (hydrapolic) * Ensure snapshots are found before iterating #507 (garyelephant) * Fix a doc inconsistency #509 (pmoust) * Fix a typo in `show` documentation #513 (pbamba) * Default to trying the cluster state for checking whether indices are closed, and then fall back to using the _cat API (for Amazon ES instances). #519 (untergeek) * Improve logging to show time delay between optimize runs, if selected. #525 (untergeek) * Allow elasticsearch-py module versions through 2.3.0 (a presumption at this point) #524 (untergeek) * Improve logging in snapshot api method to reveal when a repository appears to be missing. Reported in #551 (untergeek) * Test that ``--timestring`` has the correct variable for ``--time-unit``. Reported in #544 (untergeek) * Allocation will exit with exit_code 0 now when there are no indices to work on. Reported in #531 (untergeek) 3.4.0 (28 October 2015) ----------------------- **General** * API change in elasticsearch-py 1.7.0 prevented alias operations. Fixed in #486 (HonzaKral) * During index selection you can now select only closed indices with ``--closed-only``. Does not impact ``--all-indices`` Reported in #476. Fixed in #487 (Basster) * API Changes in Elasticsearch 2.0.0 required some refactoring. All tests pass for ES versions 1.0.3 through 2.0.0-rc1. Fixed in #488 (untergeek) * es_repo_mgr now has access to the same SSL options from #462. #489 (untergeek) * Logging improvements requested in #475. (untergeek) * Added ``--quiet`` flag. #494 (untergeek) * Fixed ``index_closed`` to work with AWS Elasticsearch. #499 (univerio) * Acceptable versions of Elasticsearch-py module are 1.8.0 up to 2.1.0 (untergeek) 3.3.0 (31 August 2015) ---------------------- **Announcement** * Curator is tested in Jenkins. Each commit to the master branch is tested with both Python versions 2.7.6 and 3.4.0 against each of the following Elasticsearch versions: * 1.7_nightly * 1.6_nightly * 1.7.0 * 1.6.1 * 1.5.1 * 1.4.4 * 1.3.9 * 1.2.4 * 1.1.2 * 1.0.3 * If you are using a version different from this, your results may vary. **General** * Allocation type can now also be ``include`` or ``exclude``, in addition to the the existing default ``require`` type. Add ``--type`` to the allocation command to specify the type. #443 (steffo) * Bump elasticsearch python module dependency to 1.6.0+ to enable synced_flush API call. Reported in #447 (untergeek) * Add SSL features, ``--ssl-no-validate`` and ``certificate`` to provide other ways to validate SSL connections to Elasticsearch. #436 (untergeek) **Bug fixes** * Delete by space was only reporting space used by primary shards. Fixed to show all space consumed. Reported in #455 (untergeek) * Update exit codes and messages for snapshot selection. Reported in #452 (untergeek) * Fix potential int/float casting issues. Reported in #465 (untergeek) 3.2.3 (16 July 2015) -------------------- **Bug fix** * In order to address customer and community issues with bulk deletes, the ``master_timeout`` is now invoked for delete operations. This should address 503s with 30s timeouts in the debug log, even when ``--timeout`` is set to a much higher value. The ``master_timeout`` is tied to the ``--timeout`` flag value, but will not exceed 300 seconds. #420 (untergeek) **General** * Mixing it up a bit here by putting `General` second! The only other changes are that logging has been improved for deletes so you won't need to have the ``--debug`` flag to see if you have error codes >= 400, and some code documentation improvements. 3.2.2 (13 July 2015) -------------------- **General** * This is a very minor change. The ``mock`` library recently removed support for Python 2.6. As many Curator users are using RHEL/CentOS 6, which is pinned to Python 2.6, this requires the mock version referenced by Curator to also be pinned to a supported version (``mock==1.0.1``). 3.2.1 (10 July 2015) -------------------- **General** * Added delete verification & retry (fixed at 3x) to potentially cover an edge case in #420 (untergeek) * Since GitHub allows rST (reStructuredText) README documents, and that's what PyPI wants also, the README has been rebuilt in rST. (untergeek) **Bug fixes** * If closing indices with ES 1.6+, and all indices are closed, ensure that the seal command does not try to seal all indices. Reported in #426 (untergeek) * Capture AttributeError when sealing indices if a non-TransportError occurs. Reported in #429 (untergeek) 3.2.0 (25 June 2015) -------------------- **New!** * Added support to manually seal, or perform a [synced flush](http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-synced-flush.html) on indices with the ``seal`` command. #394 (untergeek) * Added *experimental* support for SSL certificate validation. In order for this to work, you must install the ``certifi`` python module: ``pip install certifi`` This feature *should* automatically work if the ``certifi`` module is installed. Please report any issues. **General** * Changed logging to go to stdout rather than stderr. Reopened #121 and figured they were right. This is better. (untergeek) * Exit code 99 was unpopular. It has been removed. Reported in #371 and #391 (untergeek) * Add ``--skip-repo-validation`` flag for snapshots. Do not validate write access to repository on all cluster nodes before proceeding. Useful for shared filesystems where intermittent timeouts can affect validation, but won't likely affect snapshot success. Requested in #396 (untergeek) * An alias no longer needs to be pre-existent in order to use the alias command. #317 (untergeek) * es_repo_mgr now passes through upstream errors in the event a repository fails to be created. Requested in #405 (untergeek) **Bug fixes** * In rare cases, ``*`` wildcard would not expand. Replaced with _all. Reported in #399 (untergeek) * Beginning with Elasticsearch 1.6, closed indices cannot have their replica count altered. Attempting to do so results in this error: ``org.elasticsearch.ElasticsearchIllegalArgumentException: Can't update [index.number_of_replicas] on closed indices [[test_index]] - can leave index in an unopenable state`` As a result, the ``change_replicas`` method has been updated to prune closed indices. This change will apply to all versions of Elasticsearch. Reported in #400 (untergeek) * Fixed es_repo_mgr repository creation verification error. Reported in #389 (untergeek) 3.1.0 (21 May 2015) ------------------- **General** * If ``wait_for_completion`` is true, snapshot success is now tested and logged. Reported in #253 (untergeek) * Log & return false if a snapshot is already in progress (untergeek) * Logs individual deletes per index, even though they happen in batch mode. Also log individual snapshot deletions. Reported in #372 (untergeek) * Moved ``chunk_index_list`` from cli to api utils as it's now also used by ``filter.py`` * Added a warning and 10 second timer countdown if you use ``--timestring`` to filter indices, but do not use ``--older-than`` or ``--newer-than`` in conjunction with it. This is to address #348, which behavior isn't a bug, but prevents accidental action against all of your time-series indices. The warning and timer are not displayed for ``show`` and ``--dry-run`` operations. * Added tests for ``es_repo_mgr`` in #350 * Doc fixes **Bug fixes** * delete-by-space needed the same fix used for #245. Fixed in #353 (untergeek) * Increase default client timeout for ``es_repo_mgr`` as node discovery and availability checks for S3 repositories can take a bit. Fixed in #352 (untergeek) * If an index is closed, indicate in ``show`` and ``--dry-run`` output. Reported in #327. (untergeek) * Fix issue where CLI parameters were not being passed to the ``es_repo_mgr`` create sub-command. Reported in #337. (feltnerm) 3.0.3 (27 Mar 2015) ------------------- **Announcement** This is a bug fix release. #319 and #320 are affecting a few users, so this release is being expedited. Test count: 228 Code coverage: 99% **General** * Documentation for the CLI converted to Asciidoc and moved to http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html * Improved logging, and refactored a few methods to help with this. * Dry-run output is now more like v2, with the index or snapshot in the log line, along with the command. Several tests needed refactoring with this change, along with a bit of documentation. **Bug fixes** * Fix links to repository in setup.py. Reported in #318 (untergeek) * No more ``--delay`` with optimized indices. Reported in #319 (untergeek) * ``--request_timeout`` not working as expected. Reinstate the version 2 timeout override feature to prevent default timeouts for ``optimize`` and ``snapshot`` operations. Reported in #320 (untergeek) * Reduce index count to 200 for test.integration.test_cli_commands.TestCLISnapshot.test_cli_snapshot_huge_list in order to reduce or eliminate Jenkins CI test timeouts. Reported in #324 (untergeek) * ``--dry-run`` no longer calls ``show``, but will show output in the log, as in v2. This was a recurring complaint. See #328 (untergeek) 3.0.2 (23 Mar 2015) ------------------- **Announcement** This is a bug fix release. #307 and #309 were big enough to warrant an expedited release. **Bug fixes** * Purge unneeded constants, and clean up config options for snapshot. Reported in #303 (untergeek) * Don't split large index list if performing snapshots. Reported in #307 (untergeek) * Act correctly if a zero value for `--older-than` or `--newer-than` is provided. #309 (untergeek) 3.0.1 (16 Mar 2015) ------------------- **Announcement** The ``regex_iterate`` method was horribly named. It has been renamed to ``apply_filter``. Methods have been added to allow API users to build a filtered list of indices similarly to how the CLI does. This was an oversight. Props to @SegFaultAX for pointing this out. **General** * In conjunction with the rebrand to Elastic, URLs and documentation were updated. * Renamed horribly named `regex_iterate` method to `apply_filter` #298 (untergeek) * Added `build_filter` method to mimic CLI calls. #298 (untergeek) * Added Examples page in the API documentation. #298 (untergeek) **Bug fixes** * Refactored to show `--dry-run` info for `--disk-space` calls. Reported in #290 (untergeek) * Added list chunking so acting on huge lists of indices won't result in a URL bigger than 4096 bytes (Elasticsearch's default limit.) Reported in https://github.com/elastic/curator/issues/245#issuecomment-77916081 * Refactored `to_csv()` method to be simpler. * Added and removed tests according to changes. Code coverage still at 99% 3.0.0 (9 March 2015) -------------------- **Release Notes** The full release of Curator 3.0 is out! Check out all of the changes here! *Note:* This release is _not_ reverse compatible with any previous version. Because 3.0 is a major point release, there have been some major changes to both the API as well as the CLI arguments and structure. Be sure to read the updated command-line specific docs in the [wiki](https://github.com/elasticsearch/curator/wiki) and change your command-line arguments accordingly. The API docs are still at http://curator.readthedocs.org. Be sure to read the latest docs, or select the docs for 3.0.0. **General** * **Breaking changes to the API.** Because this is a major point revision, changes to the API have been made which are non-reverse compatible. Before upgrading, be sure to update your scripts and test them thoroughly. * **Python 3 support** Somewhere along the line, Curator would no longer work with curator. All tests now pass for both Python2 and Python3, with 99% code coverage in both environments. * **New CLI library.** Using Click now. http://click.pocoo.org/3/ This change is especially important as it allows very easy CLI integration testing. * **Pipelined filtering!** You can now use ``--older-than`` & ``--newer-than`` in the same command! You can also provide your own regex via the ``--regex`` parameter. You can use multiple instances of the ``--exclude`` flag. * **Manually include indices!** With the ``--index`` paramter, you can add an index to the working list. You can provide multiple instances of the ``--index`` parameter as well! * **Tests!** So many tests now. Test coverage of the API methods is at 100% now, and at 99% for the CLI methods. This doesn't mean that all of the tests are perfect, or that I haven't missed some scenarios. It does mean, however, that it will be much easier to write tests if something turns up missed. It also means that any new functionality will now need to have tests. * **Iteration changes** Methods now only iterate through each index when appropriate! In fact, the only commands that iterate are `alias` and `optimize`. The `bloom` command will iterate, but only if you have added the `--delay` flag with a value greater than zero. * **Improved packaging!** Methods have been moved into categories of ``api`` and ``cli``, and further broken out into individual modules to help them be easier to find and read. * Check for allocation before potentially re-applying an allocation rule. #273 (ferki) * Assigning replica count and routing allocation rules _can_ be done to closed indices. #283 (ferki) **Bug fixes** * Don't accidentally delete ``.kibana`` index. #261 (malagoli) * Fix segment count for empty indices. #265 (untergeek) * Change bloom filter cutoff Elasticsearch version to 1.4. Reported in #267 (untergeek) 3.0.0rc1 (5 March 2015) ----------------------- **Release Notes** RC1 is here! I'm re-releasing the Changes from all betas here, minus the intra-beta code fixes. Barring any show stoppers, the official release will be soon. **General** * **Breaking changes to the API.** Because this is a major point revision, changes to the API have been made which are non-reverse compatible. Before upgrading, be sure to update your scripts and test them thoroughly. * **Python 3 support** Somewhere along the line, Curator would no longer work with curator. All tests now pass for both Python2 and Python3, with 99% code coverage in both environments. * **New CLI library.** Using Click now. http://click.pocoo.org/3/ This change is especially important as it allows very easy CLI integration testing. * **Pipelined filtering!** You can now use ``--older-than`` & ``--newer-than`` in the same command! You can also provide your own regex via the ``--regex`` parameter. You can use multiple instances of the ``--exclude`` flag. * **Manually include indices!** With the ``--index`` paramter, you can add an index to the working list. You can provide multiple instances of the ``--index`` parameter as well! * **Tests!** So many tests now. Test coverage of the API methods is at 100% now, and at 99% for the CLI methods. This doesn't mean that all of the tests are perfect, or that I haven't missed some scenarios. It does mean, however, that it will be much easier to write tests if something turns up missed. It also means that any new functionality will now need to have tests. * Methods now only iterate through each index when appropriate! * Improved packaging! Hopefully the ``entry_point`` issues some users have had will be addressed by this. Methods have been moved into categories of ``api`` and ``cli``, and further broken out into individual modules to help them be easier to find and read. * Check for allocation before potentially re-applying an allocation rule. #273 (ferki) * Assigning replica count and routing allocation rules _can_ be done to closed indices. #283 (ferki) **Bug fixes** * Don't accidentally delete ``.kibana`` index. #261 (malagoli) * Fix segment count for empty indices. #265 (untergeek) * Change bloom filter cutoff Elasticsearch version to 1.4. Reported in #267 (untergeek) 3.0.0b4 (5 March 2015) ---------------------- **Notes** Integration testing! Because I finally figured out how to use the Click Testing API, I now have a good collection of command-line simulations, complete with a real back-end. This testing found a few bugs (this is why testing exists, right?), and fixed a few of them. **Bug fixes** * HUGE! `curator show snapshots` would _delete_ snapshots. This is fixed. * Return values are now being sent from the commands. * `scripttest` is no longer necessary (click.Test works!) * Calling `get_snapshot` without a snapshot name returns all snapshots 3.0.0b3 (4 March 2015) ---------------------- **Bug fixes** * setup.py was lacking the new packages "curator.api" and "curator.cli" The package works now. * Python3 suggested I had to normalize the beta tag to just b3, so that's also changed. * Cleaned out superfluous imports and logger references from the __init__.py files. 3.0.0-beta2 (3 March 2015) -------------------------- **Bug fixes** * Python3 issues resolved. Tests now pass on both Python2 and Python3 3.0.0-beta1 (3 March 2015) -------------------------- **General** * **Breaking changes to the API.** Because this is a major point revision, changes to the API have been made which are non-reverse compatible. Before upgrading, be sure to update your scripts and test them thoroughly. * **New CLI library.** Using Click now. http://click.pocoo.org/3/ * **Pipelined filtering!** You can now use ``--older-than`` & ``--newer-than`` in the same command! You can also provide your own regex via the ``--regex`` parameter. You can use multiple instances of the ``--exclude`` flag. * **Manually include indices!** With the ``--index`` paramter, you can add an index to the working list. You can provide multiple instances of the ``--index`` parameter as well! * **Tests!** So many tests now. Unit test coverage of the API methods is at 100% now. This doesn't mean that all of the tests are perfect, or that I haven't missed some scenarios. It does mean that any new functionality will need to also have tests, now. * Methods now only iterate through each index when appropriate! * Improved packaging! Hopefully the ``entry_point`` issues some users have had will be addressed by this. Methods have been moved into categories of ``api`` and ``cli``, and further broken out into individual modules to help them be easier to find and read. * Check for allocation before potentially re-applying an allocation rule. #273 (ferki) **Bug fixes** * Don't accidentally delete ``.kibana`` index. #261 (malagoli) * Fix segment count for empty indices. #265 (untergeek) * Change bloom filter cutoff Elasticsearch version to 1.4. Reported in #267 (untergeek) 2.1.2 (22 January 2015) ----------------------- **Bug fixes** * Do not try to set replica count if count matches provided argument. #247 (bobrik) * Fix JSON logging (Logstash format). #250 (magnusbaeck) * Fix bug in `filter_by_space()` which would match all indices if the provided patterns found no matches. Reported in #254 (untergeek) 2.1.1 (30 December 2014) ------------------------ **Bug fixes** * Renamed unnecessarily redundant ``--replicas`` to ``--count`` in args for ``curator_script.py`` 2.1.0 (30 December 2014) ------------------------ **General** * Snapshot name now appears in log output or STDOUT. #178 (untergeek) * Replicas! You can now change the replica count of indices. Requested in #175 (untergeek) * Delay option added to Bloom Filter functionality. #206 (untergeek) * Add 2-digit years as acceptable pattern (y vs. Y). Reported in #209 (untergeek) * Add Docker container definition #226 (christianvozar) * Allow the use of 0 with --older-than, --most-recent and --delete-older-than. See #208. #211 (bobrik) **Bug fixes** * Edge case where 1.4.0.Beta1-SNAPSHOT would break version check. Reported in #183 (untergeek) * Typo fixed. #193 (ferki) * Type fixed. #204 (gheppner) * Shows proper error in the event of concurrent snapshots. #177 (untergeek) * Fixes erroneous index display of ``_, a, l, l`` when --all-indices selected. Reported in #222 (untergeek) * Use json.dumps() to escape exceptions. Reported in #210 (untergeek) * Check if index is closed before adding to alias. Reported in #214 (bt5e) * No longer force-install argparse if pre-installed #216 (whyscream) * Bloom filters have been removed from Elasticsearch 1.5.0. Update methods and tests to act accordingly. #233 (untergeek) 2.0.2 (8 October 2014) ---------------------- **Bug fixes** * Snapshot name not displayed in log or STDOUT #185 (untergeek) * Variable name collision in delete_snapshot() #186 (untergeek) 2.0.1 (1 October 2014) ---------------------- **Bug fix** * Override default timeout when snapshotting --all-indices #179 (untergeek) 2.0.0 (25 September 2014) ------------------------- **General** * New! Separation of Elasticsearch Curator Python API and curator_script.py (untergeek) * New! ``--delay`` after optimize to allow cluster to quiesce #131 (untergeek) * New! ``--suffix`` option in addition to ``--prefix`` #136 (untergeek) * New! Support for wildcards in prefix & suffix #136 (untergeek) * Complete refactor of snapshots. Now supporting incrementals! (untergeek) **Bug fix** * Incorrect error msg if no indices sent to create_snapshot (untergeek) * Correct for API change coming in ES 1.4 #168 (untergeek) * Missing ``"`` in Logstash log format #143 (cassianoleal) * Change non-master node test to exit code 0, log as ``INFO``. #145 (untergeek) * `months` option missing from validate_timestring() (untergeek) 1.2.2 (29 July 2014) -------------------- **Bug fix** * Updated ``README.md`` to briefly explain what curator does #117 (untergeek) * Fixed ``es_repo_mgr`` logging whitelist #119 (untergeek) * Fixed absent ``months`` time-unit #120 (untergeek) * Filter out ``.marvel-kibana`` when prefix is ``.marvel-`` #120 (untergeek) * Clean up arg parsing code where redundancy exists #123 (untergeek) * Properly divide debug from non-debug logging #125 (untergeek) * Fixed ``show`` command bug caused by changes to command structure #126 (michaelweiser) 1.2.1 (24 July 2014) -------------------- **Bug fix** * Fixed the new logging when called by ``curator`` entrypoint. 1.2.0 (24 July 2014) -------------------- **General** * New! Allow user-specified date patterns: ``--timestring`` #111 (untergeek) * New! Curate weekly indices (must use week of year) #111 (untergeek) * New! Log output in logstash format ``--logformat logstash`` #111 (untergeek) * Updated! Cleaner default logs (debug still shows everything) (untergeek) * Improved! Dry runs are more visible in log output (untergeek) Errata * The ``--separator`` option was removed in lieu of user-specified date patterns. * Default ``--timestring`` for days: ``%Y.%m.%d`` (Same as before) * Default ``--timestring`` for hours: ``%Y.%m.%d.%H`` (Same as before) * Default ``--timestring`` for weeks: ``%Y.%W`` 1.1.3 (18 July 2014) -------------------- **Bug fix** * Prefix not passed in ``get_object_list()`` #106 (untergeek) * Use ``os.devnull`` instead of ``/dev/null`` for Windows #102 (untergeek) * The http auth feature was erroneously omitted #100 (bbuchacher) 1.1.2 (13 June 2014) -------------------- **Bug fix** * This was a showstopper bug for anyone using RHEL/CentOS with a Python 2.6 dependency for yum * Python 2.6 does not like format calls without an index. #96 via #95 (untergeek) * We won't talk about what happened to 1.1.1. No really. I hate git today :( 1.1.0 (12 June 2014) -------------------- **General** * Updated! New command structure * New! Snapshot to fs or s3 #82 (untergeek) * New! Add/Remove indices to alias #82 via #86 (cschellenger) * New! ``--exclude-pattern`` #80 (ekamil) * New! (sort of) Restored ``--log-level`` support #73 (xavier-calland) * New! show command-line options #82 via #68 (untergeek) * New! Shard Allocation Routing #82 via #62 (nickethier) **Bug fix** * Fix ``--max_num_segments`` not being passed correctly #74 (untergeek) * Change ``BUILD_NUMBER`` to ``CURATOR_BUILD_NUMBER`` in ``setup.py`` #60 (mohabusama) * Fix off-by-one error in time calculations #66 (untergeek) * Fix testing with python3 #92 (untergeek) Errata * Removed ``optparse`` compatibility. Now requires ``argparse``. 1.0.0 (25 Mar 2014) ------------------- **General** * compatible with ``elasticsearch-py`` 1.0 and Elasticsearch 1.0 (honzakral) * Lots of tests! (honzakral) * Streamline code for 1.0 ES versions (honzakral) **Bug fix** * Fix ``find_expired_indices()`` to not skip closed indices (honzakral) 0.6.2 (18 Feb 2014) ------------------- **General** * Documentation fixes #38 (dharrigan) * Add support for HTTPS URI scheme and ``optparse`` compatibility for Python 2.6 (gelim) * Add elasticsearch module version checking for future compatibility checks (untergeek) 0.6.1 (08 Feb 2014) ------------------- **General** * Added tarball versioning to ``setup.py`` (untergeek) **Bug fix** * Fix ``long_description`` by including ``README.md`` in ``MANIFEST.in`` (untergeek) * Incorrect version number in ``curator.py`` (untergeek) 0.6.0 (08 Feb 2014) ------------------- **General** * Restructured repository to a be a proper python package. (arieb) * Added ``setup.py`` file. (arieb) * Removed the deprecated file ``logstash_index_cleaner.py`` (arieb) * Updated ``README.md`` to fit the new package, most importantly the usage and installation. (arieb) * Fixes and package push to PyPI (untergeek) 0.5.2 (26 Jan 2014) ------------------- **General** * Fix boolean logic determining hours or days for time selection (untergeek) 0.5.1 (20 Jan 2014) ------------------- **General** * Fix ``can_bloom`` to compare numbers (HonzaKral) * Switched ``find_expired_indices()`` to use ``datetime`` and ``timedelta`` * Do not try and catch unrecoverable exceptions. (HonzaKral) * Future proofing the use of the elasticsearch client (i.e. work with version 1.0+ of Elasticsearch) (HonzaKral) Needs more testing, but should work. * Add tests for these scenarios (HonzaKral) 0.5.0 (17 Jan 2014) ------------------- **General** * Deprecated ``logstash_index_cleaner.py`` Use new ``curator.py`` instead (untergeek) * new script change: ``curator.py`` (untergeek) * new add index optimization (Lucene forceMerge) to reduce segments and therefore memory usage. (untergeek) * update refactor of args and several functions to streamline operation and make it more readable (untergeek) * update refactor further to clean up and allow immediate (and future) portability (HonzaKral) 0.4.0 ----- **General** * First version logged in ``CHANGELOG`` * new ``--disable-bloom-days`` feature requires 0.90.9+ http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings This can save a lot of heap space on cold indexes (i.e. not actively indexing documents) curator-3.4.1/docs/Makefile000066400000000000000000000152061265673226500156100ustar00rootroot00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Elasticsearch.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Elasticsearch.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/Elasticsearch" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Elasticsearch" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." curator-3.4.1/docs/asciidoc/000077500000000000000000000000001265673226500157225ustar00rootroot00000000000000curator-3.4.1/docs/asciidoc/commands/000077500000000000000000000000001265673226500175235ustar00rootroot00000000000000curator-3.4.1/docs/asciidoc/commands/alias.asciidoc000066400000000000000000000026731265673226500223240ustar00rootroot00000000000000[[alias]] == alias [float] Summary ~~~~~~~ Add indices to or remove them from aliases. NOTE: For Curator versions older than 3.2.0, the alias *must* already be in existence. [float] Flags ~~~~~ ------------------------------------------------------------------- $ curator alias --help Usage: curator alias [OPTIONS] COMMAND [ARGS]... Index Aliasing Options: --name TEXT Alias name --remove Remove from alias rather than add. [default: False] --help Show this message and exit. Commands: indices Index selection. -------------------------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Examples ~~~~~~~~ Add indices to `alias_name`: ----------------------------------------------------------------------- curator alias --name alias_name indices <> -----------------------------------------------------------------------   Remove indices from `alias_name`: -------------------------------------------------------------------------------- curator alias --name alias_name --remove indices <> --------------------------------------------------------------------------------   WARNING: There are currently no safety checks to preserve an alias. If you remove the last index from an alias, it will be deleted immediately. curator-3.4.1/docs/asciidoc/commands/allocation.asciidoc000066400000000000000000000104611265673226500233520ustar00rootroot00000000000000[[allocation]] == allocation [float] Summary ~~~~~~~ From the {ref}/shard-allocation-filtering.html[Elasticsearch Shard Allocation Filtering Documentation]: ________________________________________________________________________________ Allows to control the allocation of indices on nodes based on include/exclude filters. The filters can be set both on the index level and on the cluster level. Lets start with an example of setting it on the cluster level: Lets say we have 4 nodes, each has specific attribute called `tag` associated with it (the name of the attribute can be any name). Each node has a specific value associated with `tag`. Node 1 has a setting `node.tag: value1`, Node 2 a setting of `node.tag: value2`, and so on. We can create an index that will only deploy on nodes that have tag set to value1 and value2 by setting index.routing.allocation.include.tag to value1,value2. For example: ------------------------------------------------------------ curl -XPUT localhost:9200/test/_settings -d '{ "index.routing.allocation.include.tag" : "value1,value2" }' ------------------------------------------------------------ On the other hand, we can create an index that will be deployed on all nodes except for nodes with a `tag` of value `value3` by setting `index.routing.allocation.exclude.tag` to `value3`. For example: ------------------------------------------------------------ curl -XPUT localhost:9200/test/_settings -d '\{ + "index.routing.allocation.exclude.tag" : "value3" + }' ------------------------------------------------------------ `index.routing.allocation.require.*` can be used to specify a number of rules, all of which MUST match in order for a shard to be allocated to a node. ________________________________________________________________________________   In versions 3.2.3 and older, Curator used this last method, `index.routing.allocation.require.*`, to force routing. Beginning in version 3.3.0, Curator also allows the use of `include` and `exclude` tagging. [float] Use-case ^^^^^^^^ You may be wondering why this is useful. Imagine that you have a tiered Elasticsearch cluster, with indexing happening on nodes with SSDs, and longer-term storage on nodes with traditional spinning hard disks. Using routing allocation, you can do indexing on the SSD nodes, then after a few days use curator to shift them to change the routing allocation tags to re-allocate them onto the slower storage nodes. [float] Extra configuration needed ^^^^^^^^^^^^^^^^^^^^^^^^^^ You can assign tags to your indices with Curator, but no allocation will occur unless you have the tags set in each node's `elasticsearch.yml`: SSD nodes: node.tag: ssd Spinning disk nodes: node.tag: hdd These tags can be anything. You could use `fast` and `slow` if you wanted. `tag` is only the example provided here. You can use other descriptive markers: node.disk_type: ssd or node.disk_type: hdd In like fashion, you would need to guarantee that new indexes are tagged with the appropriate markers, e.g. `tag` and `ssd`. This should be handled with an Elasticsearch mapping template. Logstash comes with one by default, but it has no routing tags. You can use that template as a guide and add the appropriate tags. [float] Flags ~~~~~ -------------------------------------------------------------------------------- $ curator allocation --help Usage: curator allocation [OPTIONS] COMMAND [ARGS]... Index Allocation Options: --rule TEXT Routing allocation rule to apply, e.g. tag=ssd --type TEXT Specify an allocation type, include, exclude or require [default: require] --help Show this message and exit. Commands: indices Index selection. --------------------------------------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Example ~~~~~~~ Apply the `tag` value of `hdd`: ------------------------------------------------------------------------ curator allocation --rule tag=hdd indices <> ------------------------------------------------------------------------   This will append the setting `index.routing.allocation.require.tag=hdd` to the indices specified by <>. curator-3.4.1/docs/asciidoc/commands/bloom.asciidoc000066400000000000000000000037671265673226500223500ustar00rootroot00000000000000[[bloom]] == bloom IMPORTANT: Beginning with version 1.4, Elasticsearch no longer enables bloom filters for search. This command is unnecessary if your Elasticsearch version is 1.4 or higher. [float] Summary ~~~~~~~ From the http://www.elastic.co/guide/en/elasticsearch/reference/0.90/index-modules-codec.html#bloom-postings[Elasticsearch Bloom Filter Posting Format documentation] _______________________________________________________________________________ It can sometime make sense to disable bloom filters. For instance, if you are logging into an index per day, and you have thousands of indices, the bloom filters can take up a sizable amount of memory. For most queries you are only interested in recent indices, so you don’t mind CRUD operations on older indices taking slightly longer. In these cases you can disable loading of the bloom filter on a per-index basis by updating the index settings _______________________________________________________________________________   This setting is especially relevant to Logstash and Marvel use-cases. The `bloom` command allows you to disable the bloom filter cache for indices. [float] Flags ~~~~~ ----------------------------------------------------------------------------- $ curator bloom --help Usage: curator bloom [OPTIONS] COMMAND [ARGS]... Disable bloom filter cache Options: --delay INTEGER Pause *n* seconds after disabling bloom filter cache of an index [default: 0] --help Show this message and exit. Commands: indices Index selection. -----------------------------------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Example ~~~~~~~ Disable the bloom filters (for Elasticsearch versions < 1.4): ---------------------------------------------------- curator bloom indices <> ---------------------------------------------------- curator-3.4.1/docs/asciidoc/commands/close.asciidoc000066400000000000000000000032141265673226500223300ustar00rootroot00000000000000[[close]] == close [float] Summary ~~~~~~~ From the {ref}/indices-open-close.html[Elasticsearch Open/Close API documentation] ________________________________________________________________________________ The open and close index APIs allow to close an index, and later on re-open it. A closed index has almost no overhead on the cluster (except for maintaining its metadata & the disk space it consumes), and is blocked for read/write operations. A closed index can be opened which will then go through the normal recovery process. ________________________________________________________________________________ Sometimes it makes sense to close an index so as to reduce cluster overhead, but meet data retention requirements. The `close` command makes it possible to do in a scripted way. NOTE: Starting with Curator 3.2.0, if you have Elasticsearch 1.6.0 or later, the _close_ command will attempt to perform a <>, or _seal_ each index before closing, which should lead to faster recovery times. [float] Flags ~~~~~ ------------------------------------------------ $ curator close --help Usage: curator close [OPTIONS] COMMAND [ARGS]... Close indices Options: --help Show this message and exit. Commands: indices Index selection. ------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Example ~~~~~~~ Close matching indices: ---------------------------------------------------- curator close indices <> ---------------------------------------------------- curator-3.4.1/docs/asciidoc/commands/delete.asciidoc000066400000000000000000000062461265673226500224750ustar00rootroot00000000000000[[delete]] == delete [float] Summary ~~~~~~~ Delete indices or snapshots NOTE: For delete operations, all Kibana indices (.kibana, kibana-int, .marvel-kibana) will be filtered to prevent accidental deletion. If you wish to delete one of these indices, please use the <> flag to manually supply an index name. [float] Flags ~~~~~ ----------------------------------------------------------------------------- $ curator delete --help Usage: curator delete [OPTIONS] COMMAND [ARGS]... Delete indices or snapshots Options: --disk-space FLOAT Delete indices beyond DISK_SPACE gigabytes. --reverse BOOLEAN Only valid with --disk-space. Affects sort order of the indices. True means reverse-alphabetical (if dates are involved, older is deleted first). [default: True] --help Show this message and exit. Commands: indices Index selection. snapshots Snapshot selection. -----------------------------------------------------------------------------   IMPORTANT: This command requires either the <> <> for <>, or the <> <> for <>. [float] Examples ~~~~~~~~ Delete indices: ----------------------------------------------------- curator delete indices <> -----------------------------------------------------   Delete snapshots: ---------------------------------------------------------- curator delete snapshots <> ----------------------------------------------------------   Delete indices where disk space is in excess of 1024 gigabytes (1 terabyte): ----------------------------------------------------------------------- curator delete --disk-space 1024 indices <> -----------------------------------------------------------------------   NOTE: Deleting snapshots by space is not yet possible. [float] Deleting Indices By Space ^^^^^^^^^^^^^^^^^^^^^^^^^ This option is for those who want to retain indices based on disk consumption, rather than by a set number of days. There are some important caveats surrounding this choice. [float] Caveats +++++++ * *Elasticsearch cannot calculate the size of closed indices.* Elasticsearch does not keep tabs on how much disk-space closed indices consume. If you close indices, your space calculations will be inaccurate. * *Indices consume resources just by existing.* You could run into performance and/or operational snags in Elasticsearch as the count of indices climbs. * *You need to manually calculate how much space across all nodes.* The total you give will be the sum of all space consumed across all nodes in your cluster. If you use shard allocation to put more shards or indices on a single node, it will not affect the total space reported by the cluster, but you may still run out of space on that node. These are only a few of the caveats. This is still a valid use-case, especially for those running a single-node test box, however, so we include this option for your convenience. curator-3.4.1/docs/asciidoc/commands/index.asciidoc000066400000000000000000000010651265673226500223340ustar00rootroot00000000000000[[commands]] = Commands [partintro] -- * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> -- include::alias.asciidoc[] include::allocation.asciidoc[] include::bloom.asciidoc[] include::close.asciidoc[] include::delete.asciidoc[] include::open.asciidoc[] include::optimize.asciidoc[] include::replicas.asciidoc[] include::seal.asciidoc[] include::show.asciidoc[] include::snapshot.asciidoc[] curator-3.4.1/docs/asciidoc/commands/open.asciidoc000066400000000000000000000024751265673226500221740ustar00rootroot00000000000000[[open]] == open [float] Summary ~~~~~~~ From the {ref}/indices-open-close.html[Elasticsearch Open/Close API documentation] ________________________________________________________________________________ The open and close index APIs allow to close an index, and later on re-open it. A closed index has almost no overhead on the cluster (except for maintaining its metadata & the disk space it consumes), and is blocked for read/write operations. A closed index can be opened which will then go through the normal recovery process. ________________________________________________________________________________ The `open` command makes it possible to re-open closed indices in a scripted way. [float] Flags ~~~~~ ----------------------------------------------- $ curator open --help Usage: curator open [OPTIONS] COMMAND [ARGS]... Open indices Options: --help Show this message and exit. Commands: indices Index selection. -----------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Example ~~~~~~~ Open matching indices: --------------------------------------------------- curator open indices <> --------------------------------------------------- curator-3.4.1/docs/asciidoc/commands/optimize.asciidoc000066400000000000000000000063161265673226500230710ustar00rootroot00000000000000[[optimize]] == optimize [float] Summary ~~~~~~~ Optimize is a bit of a misnomer. It is in actuality a Lucene forceMerge operation. With time-series data in a per-day index, Lucene does a good job of keeping the number of segments low. However, if no new data is being ingested, no further segment merging will happen. There are some minor performance benefits from merging segments down to a smaller count, but a greater benefit when it comes to restarts [e.g. version upgrades, etc.] after a shutdown: with fewer segments to have to validate, the cluster comes back up sooner. [float] Segment count ^^^^^^^^^^^^^ Curator counts the number of segments per shard before performing an optimize call. If the count is at or below the threshold (default is 2 segments per shard) it will skip optimizing that index. In this manner, indices are not optimized repeatedly. [float] Extra disk space required ^^^^^^^^^^^^^^^^^^^^^^^^^ Performing this forceMerge requires a lot of extra disk space: https://issues.apache.org/jira/browse/LUCENE-6386?focusedCommentId=14392125&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14392125[3x what the original segments consume]. Fortunately, this extra space is only needed during the forceMerge operation. Be sure to keep enough disk space available so that this does not become an issue for you. [float] A word about timeouts ^^^^^^^^^^^^^^^^^^^^^ With some commands (e.g. `optimize`) the default behavior is to wait until the operation is complete before proceeding with the next step. Since these operations can take quite a long time it is advisable to set `--timeout` to a high value. If one is not set, a default of 6 hours (21600 seconds) will be applied. [float] Flags ~~~~~ ---------------------------------------------------------------------------- $ curator optimize --help Usage: curator optimize [OPTIONS] COMMAND [ARGS]... Optimize Indices Options: --delay INTEGER Pause *n* seconds after optimizing an index. [default: 0] --max_num_segments INTEGER Merge to this number of segments per shard. [default: 2] --request_timeout INTEGER Allow this many seconds before the transaction times out. [default: 21600] --help Show this message and exit. Commands: indices Index selection. ----------------------------------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Examples ~~~~~~~~ Optimize matching indices to 2 segments per shard (the default): ------------------------------------------------------- curator optimize indices <> -------------------------------------------------------   Optimize matching indices older than 2 days to 1 segment per shard, and delay 120 seconds between indices: ---------------------------------------------------------------------------------------- curator optimize --max_num_segments 1 --delay 120 indices <> ---------------------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/commands/replicas.asciidoc000066400000000000000000000023541265673226500230310ustar00rootroot00000000000000[[replicas]] == replicas [float] Summary ~~~~~~~ This command allows you to alter the number of replicas (per shard) for indices matching the specified criteria. This is useful if you would like to reduce your per-node shard count. WARNING: Setting this to zero will remove the default redundancy of 1 replica per shard. This is not recommended unless you have first created a backup/snapshot of the indices in question. [float] Flags ~~~~~ -------------------------------------------------------------- $ curator replicas --help Usage: curator replicas [OPTIONS] COMMAND [ARGS]... Replica Count Per-shard Options: --count INTEGER Number of replicas the indices should have. --help Show this message and exit. Commands: indices Index selection. --------------------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Example ~~~~~~~ Change the replica count to zero for matching indices: ----------------------------------------------------------------- curator replicas --count 0 indices <> ----------------------------------------------------------------- curator-3.4.1/docs/asciidoc/commands/seal.asciidoc000066400000000000000000000100171265673226500221460ustar00rootroot00000000000000[[seal]] == seal [float] Summary ~~~~~~~ IMPORTANT: This feature requires Elasticsearch version 1.6.0+ and Curator 3.2.0 From the {ref}/indices-synced-flush.html[Elasticsearch Synced Flush API documentation] ________________________________________________________________________________ Elasticsearch tracks the indexing activity of each shard. Shards that have not received any indexing operations for 5 minutes are automatically marked as inactive. This presents an opportunity for Elasticsearch to reduce shard resources and also perform a special kind of flush, called synced flush. A synced flush performs a normal flush, then adds a generated unique marker (sync_id) to all shards. Since the sync id marker was added when there were no ongoing indexing operations, it can be used as a quick way to check if the two shards' lucene indices are identical. This quick sync id comparison (if present) is used during recovery or restarts to skip the first and most costly phase of the process. In that case, no segment files need to be copied and the transaction log replay phase of the recovery can start immediately. Note that since the sync id marker was applied together with a flush, it is very likely that the transaction log will be empty, speeding up recoveries even more. This is particularly useful for use cases having lots of indices which are never or very rarely updated, such as time based data. This use case typically generates lots of indices whose recovery without the synced flush marker would take a long time. ... The Synced Flush API allows an administrator to initiate a synced flush manually. This can be particularly useful for a planned (rolling) cluster restart where you can stop indexing and don’t want to wait the default 5 minutes for idle indices to be sync-flushed automatically. ________________________________________________________________________________   As mentioned, Elasticsearch will _automatically_ seal indices after 5 minutes of inactivity. This command is here for times when you might need to manually seal (synced flush) an index. This command is also performed when closing indices. [NOTE] ================================================================================ From the {ref}/indices-synced-flush.html[Elasticsearch Synced Flush API documentation]: ____________________________________________________________________________ Synced flush is a best effort operation. Any ongoing indexing operations will cause the synced flush to fail on that shard. This means that some shards may be synced flushed while others aren’t. ____________________________________________________________________________   and also: ____________________________________________________________________________ It is harmless to request a synced flush while there is ongoing indexing. Shards that are idle will succeed and shards that are not will fail. Any shards that succeeded will have faster recovery times. ____________________________________________________________________________ ================================================================================ The seal command will always exit successfully, even if all shards fail the synced flush. As a result, you must pay attention to the logs to see which indices (and which shards) may have failed. All reasons for the failure which are returned will be logged with the correlating index name. [float] Flags ~~~~~ ------------------------------------------------ $ curator seal --help Usage: curator seal [OPTIONS] COMMAND [ARGS]... Seal indices (Synced flush: ES 1.6.0+ only) Options: --help Show this message and exit. Commands: indices Index selection. ------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Example ~~~~~~~ Seal matching indices: ---------------------------------------------------- curator seal indices <> ---------------------------------------------------- curator-3.4.1/docs/asciidoc/commands/show.asciidoc000066400000000000000000000027251265673226500222110ustar00rootroot00000000000000[[show]] == show [float] Summary ~~~~~~~ This command allows you to see any indices (or snapshots) matching <> or <>. Starting in v3.1.0, the term `(CLOSED)` will appear next to indices which are in a closed state. [float] Flags ~~~~~ ------------------------------------------ $ curator show [OPTIONS] COMMAND [ARGS]... Show indices or snapshots Options: --help Show this message and exit. Commands: indices Index selection. snapshots Snapshot selection. ------------------------------------------   IMPORTANT: This command requires either the <> <> for <>, or the <> <> for <>. [float] Examples ~~~~~~~~ Display a list of indices matching the selection parameters: ------------------------------------------------------------------ curator show indices <> ------------------------------------------------------------------   Display a list of snapshots in the named repository matching the selection parameters: ------------------------------------------------------------------------------------------------------ curator show snapshots <> ------------------------------------------------------------------------------------------------------ curator-3.4.1/docs/asciidoc/commands/snapshot.asciidoc000066400000000000000000000065621265673226500230730ustar00rootroot00000000000000[[snapshot]] == snapshot [float] Summary ~~~~~~~ This command allows you to capture snapshots of indices into a pre-existing repository. To create a repository you can {ref}/modules-snapshots.html#_repositories[use the API], or the `es_repo_mgr` script. There are other tools available. Snapshots are incremental. Snapshotting to a named snapshot is also possible, otherwise the default snapshot name will be the snapshot prefix + a timestamp: `curator-%Y%m%d%H%M%S` (or YearMonthDayHourMinuteSecond). The default snapshot prefix is `curator-`, but that can be overridden with the `--prefix` flag. With incremental snapshotting, only new data (segments) will be backed up. [float] Flags ~~~~~ ------------------------------------------------------------------------------ $ curator snapshot --help Usage: curator snapshot [OPTIONS] COMMAND [ARGS]... Take snapshots of indices (Backup) Options: --repository TEXT Repository name. --name TEXT Override default name. --prefix TEXT Override default prefix. --wait_for_completion BOOLEAN Wait for snapshot to complete before returning. [default: True] --ignore_unavailable Ignore unavailable shards/indices. --include_global_state BOOLEAN Store cluster global state with snapshot. [default: True] --partial Do not fail if primary shard is unavailable. --request_timeout INTEGER Allow this many seconds before the transaction times out. [default: 21600] --skip-repo-validation Skip repository access validation. --help Show this message and exit. Commands: indices Index selection. ------------------------------------------------------------------------------   IMPORTANT: This command requires the <> <> for <>. [float] Examples ~~~~~~~~ Snapshot indices matching the selection parameters to the named repository: ----------------------------------------------------------------------------------- curator snapshot --repository REPOSITORY_NAME indices <> -----------------------------------------------------------------------------------   [float] Regarding timeouts ^^^^^^^^^^^^^^^^^^ A default of 6 hours (21600 seconds) will be applied for the `--request_timeout`. Since a snapshot can take a long time, curator may disconnect and fail to continue with further operations if the timeout is not set high enough. This number may need to be higher, or could be reduced depending on your scenario. The log file timestamps will indicate how long it took to perform previous operations, which you could use as a guideline. [float] Regarding repositories ^^^^^^^^^^^^^^^^^^^^^^ Curator itself is completely repository agnostic. If you've created a repository in any way, curator can make use of it. However, it must be noted that the S3 repository type cannot be used without having the https://github.com/elasticsearch/elasticsearch-cloud-aws#s3-repository[AWS Cloud Plugin for Elasticsearch] installed on each master and data node in your cluster. The `es_repo_mgr` script can assist in creation of an S3 repository, but creation of the repository will fail without this plugin being installed first. curator-3.4.1/docs/asciidoc/flags/000077500000000000000000000000001265673226500170165ustar00rootroot00000000000000curator-3.4.1/docs/asciidoc/flags/all-indices.asciidoc000066400000000000000000000015371265673226500227100ustar00rootroot00000000000000[[all-indices]] == --all-indices [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> The `--all-indices` flag allows you to act on all indices in a cluster. Though this flag allows selection of all indices, the <> flag will allow you to exclude indices matching a pattern from the final list. NOTE: For <> operations, all Kibana indices (.kibana, kibana-int, .marvel-kibana) will be filtered to prevent accidental deletion. If you wish to delete one of these indices, please use the <> flag to manually supply an index name. [float] Flags ~~~~~ * `--all-indices` Use all indices in the cluster. [float] Example ~~~~~~~ Show all indices: ---------------------------------- curator show indices --all-indices ---------------------------------- curator-3.4.1/docs/asciidoc/flags/all-snapshots.asciidoc000066400000000000000000000013461265673226500233120ustar00rootroot00000000000000[[all-snapshots]] == --all-snapshots [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <>. The `--all-snapshots` flag allows you to act on all snapshots in a repository. Though this flag allows selection of all snapshots, the <> flag, `--exclude`, will allow you to exclude snapshots matching a pattern from the final list. [float] Flags ~~~~~ * `--all-snapshots` Use all snapshots in the repository. [float] Example ~~~~~~~ Show all snapshots in the named repository: ------------------------------------------------------------- curator show snapshots --repository REPO_NAME --all-snapshots ------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/certificate.asciidoc000066400000000000000000000013041265673226500227760ustar00rootroot00000000000000[[certificate]] == --certificate [float] Summary ~~~~~~~ Allows the use of a specified CA certificate file to validate the SSL certificate used by Elasticsearch. [float] Flags ~~~~~ * `--certificate` Path to certificate to use for SSL validation. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to a cluster at `https://example.com/` via SSL and verify the certificate with the provided CA certificate file: --------------------------------------------------------------------- curator --host example.com --port 443 --use_ssl --certificate /path/to/cacert.pem <> <> --------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/debug.asciidoc000066400000000000000000000006151265673226500216060ustar00rootroot00000000000000[[debug]] == --debug [float] Summary ~~~~~~~ Enable debug logging. This flag will will override <>. [float] Flags ~~~~~ * `--debug` Enable debug logging. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Enable debug output: -------------------------------------- curator --debug <> <> -------------------------------------- curator-3.4.1/docs/asciidoc/flags/dry-run.asciidoc000066400000000000000000000013061265673226500221160ustar00rootroot00000000000000[[dry-run]] == --dry-run [float] Summary ~~~~~~~ This flag allows you to test your command-line options. It can help you see which indices will be caught by operations without actually performing those operations. Starting in v3.1.0, the term `(CLOSED)` will appear next to indices which are in a closed state. [float] Flags ~~~~~ * `--dry-run` Do not actually perform the action. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Do a dry-run test of deleting indices: ---------------------------------------------------------------- curator --dry-run delete indices <> ---------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/exclude.asciidoc000066400000000000000000000027371265673226500221600ustar00rootroot00000000000000[[exclude]] == --exclude [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. Exclude indices or snapshots matching the provided pattern. Multiple items can be excluded by using the `--exclude` flag multiple times with different patterns. WARNING: `--exclude` patterns _will not_ exclude anything added with the `--index` flag. [float] Flags ~~~~~ * `--exclude` A regular expression pattern. This flag can be invoked multiple times. [float] Examples ~~~~~~~~ Show all indices, excluding indices beginning with log: --------------------------------------- curator show indices --exclude '^log.*' ---------------------------------------   Optimize all indices, excluding indices ending with `-prod` or `-qa`: ------------------------------------------------------------------ curator show indices --exclude '^.*-prod$' --exclude '^.*-qa$' ------------------------------------------------------------------   Delete all snapshots, excluding those with `2015` in the snapshot name. If the default naming pattern is used, with `curator-YearMonthDayHourMinuteSecond` as the pattern, and the year is 2015, this would effectively delete all `2014` snapshots: ---------------------------------------------------------------------- curator delete snapshots --repository REPO_NAME --exclude '^.*2015.*$' ---------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/help.asciidoc000066400000000000000000000030641265673226500214510ustar00rootroot00000000000000[[help]] == --help [float] Summary ~~~~~~~ Show available commands and arguments. [float] Flags ~~~~~ * `--help` [float] Example ~~~~~~~ Show the curator help output: -------------- curator --help --------------   The output of `curator --help`: ----------------------------------------------------------- $ curator --help Usage: curator [OPTIONS] COMMAND [ARGS]... Curator for Elasticsearch indices. See http://elastic.co/guide/en/elasticsearch/client/curator/current Options: --host TEXT Elasticsearch host. --url_prefix TEXT Elasticsearch http url prefix. --port INTEGER Elasticsearch port. --use_ssl Connect to Elasticsearch through SSL. --http_auth TEXT Use Basic Authentication ex: user:pass --timeout INTEGER Connection timeout in seconds. --master-only Only operate on elected master node. --dry-run Do not perform any changes. --debug Debug mode --loglevel TEXT Log level --logfile TEXT log file --logformat TEXT Log output format [default|logstash]. --version Show the version and exit. --help Show this message and exit. Commands: alias Index Aliasing allocation Index Allocation bloom Disable bloom filter cache close Close indices delete Delete indices or snapshots open Open indices optimize Optimize Indices replicas Replica Count Per-shard show Show indices or snapshots snapshot Take snapshots of indices (Backup) ----------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/host.asciidoc000066400000000000000000000011751265673226500214770ustar00rootroot00000000000000[[host]] == --host [float] Summary ~~~~~~~ By default, curator will connect to `localhost`. You can override this setting with the `--host` flag. [float] Flags ~~~~~ * `--host` IP address or resolvable host name or FQDN of Elasticsearch instance. Default: `localhost`. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to Elasticsearch instance on host `es.example.org` on port `12345`: ---------------------------------------------------------------- curator --host es.example.org --port 12345 <> <> ---------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/http_auth.asciidoc000066400000000000000000000012321265673226500225140ustar00rootroot00000000000000[[http_auth]] == --http_auth [float] Summary ~~~~~~~ Allow basic HTTP authentication (e.g. `user:pass`) to an Elasticsearch instance. [float] Flags ~~~~~ * `--http_auth` A username and password in the format of `username:password`. Default value is `None` IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to an elasticsearch instance protected by basic HTTP authentication: ------------------------------------------------------------------------------ curator --host es-host --port 9443 --http_auth user:pass <> <> ------------------------------------------------------------------------------ curator-3.4.1/docs/asciidoc/flags/index.asciidoc000066400000000000000000000035621265673226500216330ustar00rootroot00000000000000[[flags]] = Flags [partintro] -- * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> _Index and Snapshot Selection_ * <> * <> * <> * <> * <> * <> * <> * <> _Index selection only_ * <> * <> _Snapshot selection only_ * <> * <> * <> _Command flags_ * <> -- include::debug.asciidoc[] include::dry-run.asciidoc[] include::help.asciidoc[] include::host.asciidoc[] include::http_auth.asciidoc[] include::logfile.asciidoc[] include::logformat.asciidoc[] include::loglevel.asciidoc[] include::master-only.asciidoc[] include::quiet.asciidoc[] include::port.asciidoc[] include::timeout.asciidoc[] include::url_prefix.asciidoc[] include::use_ssl.asciidoc[] include::certificate.asciidoc[] include::ssl-no-validate.asciidoc[] include::version.asciidoc[] include::newer-than.asciidoc[] include::older-than.asciidoc[] include::prefix.asciidoc[] include::suffix.asciidoc[] include::time-unit.asciidoc[] include::timestring.asciidoc[] include::regex.asciidoc[] include::exclude.asciidoc[] include::index_flag.asciidoc[] include::all-indices.asciidoc[] include::snapshot_flag.asciidoc[] include::all-snapshots.asciidoc[] include::repository.asciidoc[] include::skip-repo-validation.asciidoc[] curator-3.4.1/docs/asciidoc/flags/index_flag.asciidoc000066400000000000000000000020341265673226500226150ustar00rootroot00000000000000[[index_flag]] == --index [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> Add an index (or indices) by name at the command-line. Indices added with the `--index` flag _will not_ be filtered by any of the other index selection flags. The index _must_ exist. This flag can be used in lieu of filtering from the list of all indices. This flag can be invoked multiple times. [float] Flags ~~~~~ * `--index` Include the named index in the list. Can be invoked multiple times. [float] Examples ~~~~~~~~ Include `index_name` as one of the indices to be shown in addition to the provided criteria: --------------------------------------------------------- curator show indices --prefix logstash --index index_name ---------------------------------------------------------   Delete only indices `index1` and `index2`: ---------------------------------------------------- curator delete indices --index index1 --index index2 ---------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/logfile.asciidoc000066400000000000000000000006201265673226500221350ustar00rootroot00000000000000[[logfile]] == --logfile [float] Summary ~~~~~~~ Directs log output to the specified file. [float] Flags ~~~~~ * `--logfile` Target file for log output. Default: `STDOUT` IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Set log output to `/tmp/curator.log`: ------------------ curator --logfile /tmp/curator.log <> <> ------------------ curator-3.4.1/docs/asciidoc/flags/logformat.asciidoc000066400000000000000000000012141265673226500225060ustar00rootroot00000000000000[[logformat]] == --logformat [float] Summary ~~~~~~~ Change the formatting of the log output. This can be either `default`, or `logstash` for JSON output. The default is `default` [float] Flags ~~~~~ * `--logformat` TEXT Log output format `[default|logstash]`. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Set log format to `logstash` and log to `/tmp/curator.log`: ----------------------------------------------------------------------------- curator --logfile /tmp/curator.log --logformat logstash <> <> ----------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/loglevel.asciidoc000066400000000000000000000006311265673226500223270ustar00rootroot00000000000000[[loglevel]] == --loglevel [float] Summary ~~~~~~~ Set the minimum acceptable log severity to display. Default: `INFO` [float] Flags ~~~~~ * `--loglevel` The minimum severity level to log. Default: `INFO` IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Set log severity to `WARN` level: ------------- curator --loglevel WARN <> <> ------------- curator-3.4.1/docs/asciidoc/flags/master-only.asciidoc000066400000000000000000000012521265673226500227700ustar00rootroot00000000000000[[master-only]] == --master-only [float] Summary ~~~~~~~ In some situations, primarily with automated deployments, it makes sense to install curator on every node. But you wouldn't want it to _run_ on each node. With the `--master-only` flag, this is possible. It tests for, and will only continue running on the node that is the elected master. [float] Flags ~~~~~ * `--master-only` Only operate on elected master node. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Operate only on the master node: ------------------------------------------- curator --master-only <> <> ------------------------------------------- curator-3.4.1/docs/asciidoc/flags/newer-than.asciidoc000066400000000000000000000023461265673226500225730ustar00rootroot00000000000000[[newer-than]] == --newer-than [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. Filter indices "newer than" the given number of <>. The indices must also have a <>. The value provided indicates a given number of <> ago to use as a reference point. All indices "newer than" that point will be included. [float] Flags ~~~~~ * `--newer-than` Include only indices newer than _n_ <> * <> Unit of time to reckon by. Can be one of `hours`, `days`, `weeks`, or `months` * <> http://strftime.org[Python strftime] string to match your index definition. 2014.07.15 would be `%Y.%m.%d` [float] Example ~~~~~~~ Snapshot daily Logstash indices newer than 2 days: -------------------------------------------------------------------------------------------------------------------- curator snapshot --repository REPO indices --time-unit days --newer-than 2 --timestring '%Y.%m.%d' --prefix logstash -------------------------------------------------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/older-than.asciidoc000066400000000000000000000022671265673226500225620ustar00rootroot00000000000000[[older-than]] == --older-than [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. Filter indices "older than" a given number of <>. The indices must also have a <>. The value provided indicates a given number of <> ago to use as a reference point. All indices "older than" that point will be included. [float] Flags ~~~~~ * `--older-than` Include only indices older than _n_ <> * <> Unit of time to reckon by. Can be one of `hours`, `days`, `weeks`, or `months` * <> http://strftime.org[Python strftime] string to match your index definition. 2014.07.15 would be `%Y.%m.%d` [float] Example ~~~~~~~ Delete hourly Logstash indices older than 48 hours: ----------------------------------------------------------------------------------------------------- curator delete indices --time-unit hours --older-than 48 --timestring '%Y.%m.%d.%H' --prefix logstash ----------------------------------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/port.asciidoc000066400000000000000000000011421265673226500215000ustar00rootroot00000000000000[[port]] == --port [float] Summary ~~~~~~~ By default, curator will connect to port `9200`. You can override this setting with the `--port` flag. [float] Flags ~~~~~ * `--port` Port remote Elasticsearch instance is running on. Default: `9200` IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to Elasticsearch instance on host `es.example.org` on port `12345`: ---------------------------------------------------------------- curator --host es.example.org --port 12345 <> <> ---------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/prefix.asciidoc000066400000000000000000000015211265673226500220120ustar00rootroot00000000000000[[prefix]] == --prefix [float] Incidence ~~~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. [float] Summary ~~~~~~~ Beginning with Curator v3, `--prefix` is interpreted as part of a regular expression pattern. This means that if you specify `--prefix l`, it will match indices or snapshots named, __logstash__, __local__, __l-2015.03.05__, or anything that begins with the letter __l__. The <> flag exhibits similar behavior, but with the pattern matching the end of the line instead. [float] Flags ~~~~~ * `--prefix` Prefix that indices or snapshots must match. [float] Example ~~~~~~~ Show indices beginning with the letter __l__ : -------------------------------- curator show indices --prefix l -------------------------------- curator-3.4.1/docs/asciidoc/flags/quiet.asciidoc000066400000000000000000000013151265673226500216450ustar00rootroot00000000000000[[quiet]] == --quiet [float] Summary ~~~~~~~ Disallows non-log STDOUT output, which is typically error messages. If no <> is specified, logging will still output to STDOUT by default. NOTE: If <> is specified, `--quiet` is automatically implied. [float] Flags ~~~~~ * `--quiet` Suppress command-line output. IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Silence non-log command-line output: ----------------------------------------------------------------------------- curator --logfile /tmp/curator.log --quiet <> <> ----------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/regex.asciidoc000066400000000000000000000010251265673226500216260ustar00rootroot00000000000000[[regex]] == --regex [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. Include only indices or snapshots matching the provided pattern. [float] Flags ~~~~~ * `--regex` A regular expression pattern. [float] Examples ~~~~~~~~ Show only indices beginning with log and ending with prod: ------------------------------------------ curator show indices --regex '^log.*prod$' ------------------------------------------ curator-3.4.1/docs/asciidoc/flags/repository.asciidoc000066400000000000000000000010521265673226500227330ustar00rootroot00000000000000[[repository]] == --repository [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> and the <> command. Provide the repository name for snapshot operations. [float] Flags ~~~~~ * `--repository` Repository name. [float] Examples ~~~~~~~~ Show all snapshots in the named repository: ------------------------------------------------------------- curator show snapshots --repository REPO_NAME --all-snapshots ------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/skip-repo-validation.asciidoc000066400000000000000000000025111265673226500245560ustar00rootroot00000000000000[[skip-repo-validation]] == --skip-repo-validation [float] Summary ~~~~~~~ New feature added in Curator 3.2.0 IMPORTANT: This flag only functions within the scope of the <> command. Skip {ref}/modules-snapshots.html#_repository_verification[repository access validation]. This is useful in cases where your shared filesystem (local or remote) is functional but slow enough to respond that Elasticsearch yields a false-positive test result. This condition is also usually intermittent, rather than constant. WARNING: Repository validation is performed in order to ensure your snapshots will all be able to write to the repository. It is recommended that you let Curator perform this valuable check each time a snapshot is created unless you are experiencing the intermittent timeouts described. [float] Flags ~~~~~ * `--skip-repo-validation` Skip repository access validation. [float] Examples ~~~~~~~~ Snapshot indices matching the selection parameters to the named repository, skipping repository validation: ---------------------------------------------------------------------------------------------------------- curator snapshot --skip-repo-validation --repository REPOSITORY_NAME indices <> ---------------------------------------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/snapshot_flag.asciidoc000066400000000000000000000021561265673226500233520ustar00rootroot00000000000000[[snapshot_flag]] == --snapshot [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <>. Add a snapshot (or snapshots) by name at the command-line. Snapshots added with the `--snapshot` flag _will not_ be filtered by any of the other snapshot selection flags. The snapshot _must_ exist. This flag can be used in lieu of filtering from the list of all snapshots. This flag can be invoked multiple times. [float] Flags ~~~~~ * `--snapshot` Include the named snapshot in the list. Can be invoked multiple times. [float] Examples ~~~~~~~~ Include `snapshot_name` as one of the snapshots to be shown in addition to the provided criteria: --------------------------------------------------------- curator show snapshots --repository REPO --prefix curator --snapshot snapshot_name --------------------------------------------------------- Delete only snapshots `snap1` and `snap2`: ---------------------------------------------------- curator delete snapshots --repository REPO --snapshot snap1 --snapshot snap2 ---------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/ssl-no-validate.asciidoc000066400000000000000000000017401265673226500235220ustar00rootroot00000000000000[[ssl-no-validate]] == --ssl-no-validate [float] Summary ~~~~~~~ If access to your Elasticsearch instance is protected by SSL encryption, you may use the `--ssl-no-validate` flag to disable SSL certificate verification. Valid use cases for this flag include the use of self-signed certificates that cannot be otherwise verified and would generate error messages. NOTE: The use of this flag will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior. [float] Flags ~~~~~ * `--ssl-no-validate` Do not validate SSL certificate IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to a cluster at `https://example.com/` via SSL but do not verify the certificate: --------------------------------------------------------------------- curator --host example.com --port 443 --use_ssl --ssl-no-validate <> <> --------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/suffix.asciidoc000066400000000000000000000015311265673226500220220ustar00rootroot00000000000000[[suffix]] == --suffix [float] Incidence ~~~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. [float] Summary ~~~~~~~ Beginning with Curator v3, `--suffix` is interpreted as part of a regular expression pattern. This means that if you specify `--suffix -prod`, it will match indices or snapshots named, __na-prod__, __local-prod__, __2015.03.05-prod__, or anything that ends with __-prod__. The <> flag exhibits similar behavior, but with the pattern matching the beginning of the line instead. [float] Flags ~~~~~ * `--suffix` Suffix that indices or snapshots must match. [float] Example ~~~~~~~ Show indices ending with __-prod__: ------------------------------------ curator show indices --suffix -prod ------------------------------------ curator-3.4.1/docs/asciidoc/flags/time-unit.asciidoc000066400000000000000000000022461265673226500224350ustar00rootroot00000000000000[[time-unit]] == --time-unit [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. As a part of index selection or snapshot selection you can filter indices or snapshots <> or <> a given point in time. The `--time-unit` flag is required when using <> or <> to help determine _how many_ `days`, `hours`, `weeks`, and `months`. The value you provide to `--older-than` or `--newer-than` will be multiplied by the <>: 5 `days`, 3 `hours`, etc. A <> is also required in order for Curator to recognize the date pattern in your index or snapshot names. [float] Flags ~~~~~ * `--time-unit` The time interval between indices, e.g. `hours`, `days`, `weeks`, or `months`(default: `days`) [float] Example ~~~~~~~ Show indices older than 5 days: ---------------------------------------------------------------------------- curator show indices --older-than 5 --time-unit days --timestring '%Y.%m.%d' ---------------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/timeout.asciidoc000066400000000000000000000014631265673226500222100ustar00rootroot00000000000000[[timeout]] == --timeout [float] Summary ~~~~~~~ By default, Curator has a 30 second timeout for most operations ( <> and <> being exceptions where the default is overridden if it is too low). You can override this setting with the `--timeout` flag. [float] Flags ~~~~~ * `--timeout` time in seconds to keep Elasticsearch client connection open. Default: `30` (seconds). IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to Elasticsearch instance on host `es.example.org` at port `12345` with a timeout of 60 seconds: ---------------------------------------------------------------- curator --host es.example.org --port 12345 --timeout 60 <> <> ---------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/timestring.asciidoc000066400000000000000000000027241265673226500227100ustar00rootroot00000000000000[[timestring]] == --timestring [float] Summary ~~~~~~~ IMPORTANT: This flag only functions within the scope of <> or <>. Timestring is the pattern used for matching the dates in indices and snapshots. You can construct date strings using https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior[python strftime formatting]. While it is often used in conjunction with the <> flag, a timestring pattern can be used to merely match indices with that pattern, rather than to filter by date, as with <> or <>. Starting in v3.1.0, if you specify `--timestring` without <> or <>, the process will continue, but with a message warning you that the operation will delete all indices with a time string, followed by a 10 second countdown timer, which provides the opportunity to cancel the operation. [float] Flags ~~~~~ * `--timestring` A python strftime string to match a date in one or more index or snapshot patterns. [float] Example ~~~~~~~ Show indices older than 30 days matching the `--timestring` `%Y%m%d`: ------------------------------------------------------------------------- curator show indices --older-than 30 --time-unit days --timestring %Y%m%d ------------------------------------------------------------------------- TIP: See the <> for information about escaping the `%` symbol in systemd setups. curator-3.4.1/docs/asciidoc/flags/url_prefix.asciidoc000066400000000000000000000015351265673226500227010ustar00rootroot00000000000000[[url_prefix]] == --url_prefix [float] Summary ~~~~~~~ In some cases you may be obliged to connect to your Elasticsearch cluster through a proxy of some kind. There may be a URL prefix before the API URI items, e.g. `http://example.com/elasticsearch/` as opposed to `http://localhost:9200`. In such a case, the `--url_prefix` flag is necessary. [float] Flags ~~~~~ * `--url_prefix` Elasticsearch http url prefix. Default: none IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Show indices of a cluster at `http://example.com/elasticsearch`: ------------------------------------------------------------------------------------------ curator --host example.com --port 80 --url_prefix elasticsearch show indices --all-indices ------------------------------------------------------------------------------------------ curator-3.4.1/docs/asciidoc/flags/use_ssl.asciidoc000066400000000000000000000036611265673226500222010ustar00rootroot00000000000000[[use_ssl]] == --use_ssl [float] Summary ~~~~~~~ If access to your Elasticsearch instance is protected by SSL encryption, you must use the `--use_ssl` flag. [float] SSL Certificate Validation ~~~~~~~~~~~~~~~~~~~~~~~~~~ Beginning with version 3.2.0, Curator added _experimental_ support for validating SSL certificates. These features were enhanced in version 3.3.0 to support additional SSL validation options. Depending on which validation option you choose, you may need to install an additional Python module. **If you have certificates signed by a commonly known Certificate Authority...** Chances are good that the CA certificates bundled with Mozilla Firefox will be able to validate your certificate. In this case, use of the `certifi` python module will work. This can be installed via `pip install certifi`. With the `certifi` module installed, if you are connecting to Elasticsearch via SSL, the verification should automatically take place. **If you have an unrecognized CA, use self-signed certificates, or do not wish to install the `certifi` module...** In this case you can use the <> flag to point to the file which holds the CA certificate to use for validation. This file should be in PEM format. **If you do not wish to validate your certificate at all...** This is not recommended, but if you so desire, you can use the <> flag. Use of this flag will likely still yield a warning about insecure connections, but still permit the use of SSL communications. [float] Flags ~~~~~ * `--use_ssl` (Set `use_ssl` to True) IMPORTANT: This flag must come before any <>. [float] Example ~~~~~~~ Connect to a cluster at `https://example.com/` via ssl: --------------------------------------------------------------------- curator --host example.com --port 443 --use_ssl <> <> --------------------------------------------------------------------- curator-3.4.1/docs/asciidoc/flags/version.asciidoc000066400000000000000000000004261265673226500222050ustar00rootroot00000000000000[[version]] == --version [float] Summary ~~~~~~~ Show the curator version release number, then exit. [float] Flags ~~~~~ * `--version` Show the version number. [float] Example ~~~~~~~ Show the curator version number: ----------------- curator --version ----------------- curator-3.4.1/docs/asciidoc/index.asciidoc000066400000000000000000000007501265673226500205330ustar00rootroot00000000000000:curator_version: 3.4.0 :curator_major: 3 :es_py_version: 1.8.0 :ref: http://www.elastic.co/guide/en/elasticsearch/reference/2.0 [[curator-reference]] = Curator Reference include::misc/about.asciidoc[] include::misc/getting_started.asciidoc[] include::flags/index.asciidoc[] include::commands/index.asciidoc[] include::subcommands/index.asciidoc[] include::selection/index.asciidoc[] include::misc/faq.asciidoc[] include::misc/contributing.asciidoc[] include::misc/glossary.asciidoc[] curator-3.4.1/docs/asciidoc/misc/000077500000000000000000000000001265673226500166555ustar00rootroot00000000000000curator-3.4.1/docs/asciidoc/misc/about.asciidoc000066400000000000000000000100151265673226500214640ustar00rootroot00000000000000[[about]] = About [partintro] -- Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your Elasticsearch indices. -- == Origin Curator was first called https://logstash.jira.com/browse/LOGSTASH-211[`clearESindices.py`]. Its sole function was to delete indices. It was almost immediately renamed to https://logstash.jira.com/browse/LOGSTASH-211[`logstash_index_cleaner.py`]. After a time it was briefly relocated under the https://github.com/elastic/logstash[logstash] repository as `expire_logs`, at which point it began to gain new functionality. Soon thereafter, Jordan Sissel was hired by Elastic (then still Elasticsearch), as was the original author of Curator. Not long after that it became Elasticsearch Curator and is now hosted at https://github.com/elastic/curator Curator now performs many operations on your Elasticsearch indices, from delete to snapshot to shard allocation routing. == Features Curator allows for many different operations to be performed to both indices and snapshots, including: * Add or remove indices from an <> * Shard routing <> * Disabling <> filter caches (for versions of Elasticsearch < 1.4) * <> indices * <> indices and snapshots * <> closed indices * <> indices * Change the number of <> per shard for indices * <> indices and snapshots * Take a <> (backup) of indices. [[about-cli]] == Command-Line Interface (CLI) Curator has always been a command-line tool. When Curator 2.0 was released, it was the first attempt to separate the <> from the CLI. This site provides the documentation for how to use Curator on the command-line. TIP: Learn more about <>. [[about-api]] == Application Program Interface (API) Since version 2.0, Curator ships with both an API and wrapper scripts (which are actually defined as entry points). The API, or Application Program Interface, allows you to write your own scripts to accomplish similar goals--or even new and different things--with the same code that Curator uses. The API documentation is not on this site, but is available at http://curator.readthedocs.org/. The Curator API is built using the http://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html[Elasticsearch Python API]. [[license]] == License Copyright 2011–2015 Elastic and contributors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. [[site-corrections]] == Site Corrections All documentation on this site allows for quick correction submission. To do this, use the "Edit" link to the right on any page. * You will need to log in with your GitHub account. * Make your corrections or additions to the documentation. * Please use the option to "Create a new branch for this commit and start a pull request." * Please make sure you have signed our http://www.elastic.co/contributor-agreement/[Contributor License Agreement]. We are not asking you to assign copyright to us, but to give us the right to distribute your code (even documentation corrections) without restriction. We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. You only need to sign the CLA once. If you are uncomfortable with this, feel free to submit a https://github.com/elastic/curator/issues[GitHub Issue] with your suggested correction instead. * Changes will be reviewed and merged, if acceptable. curator-3.4.1/docs/asciidoc/misc/command-line.asciidoc000066400000000000000000000057161265673226500227310ustar00rootroot00000000000000[[command-line]] == The Command-Line Interface === Structure Curator commands can be thought of as a series of nested <>, with each stage having its own options and <>. ------- curator [FLAGS] COMMAND [FLAGS] SUBCOMMAND [FLAGS] -------   The square braces indicate optional elements. Some commands have flags, some do not. Some of those flags are optional, some are mandatory per the command. See the list of <> and <> for more information. The first thing to know is that _help_ is never far away. You can use the <> flag at any stage to discover which <> are available: ------- curator --help curator COMMAND --help curator COMMAND SUBCOMMAND --help -------   Understand that using the `--help` flag in between two nested commands will result in the previous level `--help` output being shown. `curator --help COMMAND` will have the same output as `curator --help`, and likewise, `curator COMMAND --help SUBCOMMAND` will have the same output as `curator COMMAND --help`. The top-level help output looks like this: ------- $ curator --help Usage: curator [OPTIONS] COMMAND [ARGS]... Curator for Elasticsearch indices. See http://elastic.co/guide/en/elasticsearch/client/curator/current Options: --host TEXT Elasticsearch host. --url_prefix TEXT Elasticsearch http url prefix. --port INTEGER Elasticsearch port. --use_ssl Connect to Elasticsearch through SSL. --http_auth TEXT Use Basic Authentication ex: user:pass --timeout INTEGER Connection timeout in seconds. --master-only Only operate on elected master node. --dry-run Do not perform any changes. --debug Debug mode --loglevel TEXT Log level --logfile TEXT log file --logformat TEXT Log output format [default|logstash]. --version Show the version and exit. --help Show this message and exit. Commands: alias Index Aliasing allocation Index Allocation bloom Disable bloom filter cache close Close indices delete Delete indices or snapshots open Open indices optimize Optimize Indices replicas Replica Count Per-shard show Show indices or snapshots snapshot Take snapshots of indices (Backup) -------   Note that all available <> and <> for this level are shown. This pattern is repeated at each successive level of <> === Non-destructive Exploration The <> command will only ever show indices or snapshots matching the <> or <> parameters you provide. The <> flag will also show which indices or snapshots would have been acted on, but not perform the action. With these two options, you can explore your indices and work on building a set of <> criteria without fear of making a mistake. curator-3.4.1/docs/asciidoc/misc/contributing.asciidoc000066400000000000000000000014351265673226500230670ustar00rootroot00000000000000= Contributing to Curator [partintro] -- We welcome contributions and bug fixes to Curator's API and CLI. We are grateful for the many https://github.com/elastic/curator/blob/master/CONTRIBUTORS[contributors] who have helped Curator become what it is today. -- == How to contribute: Please read through our https://github.com/elastic/curator/blob/master/CONTRIBUTING.md[contribution] guide, and the Curator https://github.com/elastic/curator/blob/master/README.md[readme] document. A brief overview of the steps * fork the repo * make changes in your fork * add tests to cover your changes (if necessary) * run tests * sign the http://elastic.co/contributor-agreement/[CLA] * send a pull request! TIP: To submit documentation fixes for this site, see <> curator-3.4.1/docs/asciidoc/misc/examples.asciidoc000066400000000000000000000051241265673226500221750ustar00rootroot00000000000000[[examples]] == Examples === Delete _all_ indices with matching <> which are older than 30 days ----- curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \ --timestring '%Y.%m.%d' -----   In previous versions of Curator, you would have needed as many runs of the command-line as you had prefixes, no matter the matching pattern. Not so any more! With this example, _all_ indices with a pattern of `%Y.%m.%d` inside will be deleted. This is useful for managing both Logstash and Marvel indices with a single command, if you want the same retention period. NOTE: Kibana indices are omitted from deletion by default. === Alias a subset of indices ----- curator --host 10.0.0.2 alias --name lastmonth indices --newer-than 60 \ --older-than 30 --timestring '%Y.%m.%d' --time-unit days --prefix logstash -----   In this example, logstash indices older than 30 days--but newer than 60--are added to alias `lastmonth`. Please note that Curator does not create aliases. It can only add to them or remove from them. Let's take this example further, though. ----- curator --host 10.0.0.2 alias --name lastmonth --remove indices --older-than 60 \ --timestring '%Y.%m.%d' --time-unit days --prefix logstash curator --host 10.0.0.2 alias --name lastmonth indices --newer-than 60 \ --older-than 30 --timestring '%Y.%m.%d' --time-unit days --prefix logstash -----   The updated, 2 line example now _removes_ indices older than 60 days first, and then proceeds to make sure that everything between 30 and 60 days is in the alias. Note that Curator will not re-alias if the index is already in the alias. === Optimize only the specified indices ----- curator --host 10.0.0.2 optimize --max_num_segments 1 --delay 120 \ --index indexname1 --index indexname2 -----   This command will _only_ operate on the named indices, `indexname1` and `indexname2`. It will pause for `120` seconds after optimizing each index. === Show only indices with a timestring ----- curator --host 10.0.0.2 show indices --timestring '%Y.%m.%d' -----   While timestring may nearly always be seen alongside <>, <>, and <>, there may be cases when you just want to act on indices with a timestamp in the name. === Close all indices, excepting those matching patterns `do-not-touch` and `logstash-2015.03` ----- curator --host 10.0.0.2 close indices --exclude do-not-touch --exclude logstash-2015.03 -----   In this case, all indices saving those matching pattern `do-not-touch`, or all Logstash indices for the month of March 2015 will be closed. curator-3.4.1/docs/asciidoc/misc/exitcodes.asciidoc000066400000000000000000000006251265673226500223470ustar00rootroot00000000000000[[exit-codes]] == Exit Codes New in Curator version 3, exit codes will indicate success or failure. * `0` — Success * `1` — Failure NOTE: Up until Curator version 3.2.0, exit code `99` existed. This indicated that there were no matching indices to act on, but otherwise no failure existed. A warning message that no matching indices were found is still shown, but Curator now returns exit code 0. curator-3.4.1/docs/asciidoc/misc/faq.asciidoc000066400000000000000000000203321265673226500211240ustar00rootroot00000000000000[[faq]] = Frequently Asked Questions [partintro] -- This section will be updated as more frequently asked questions arise -- == Q: How can I report an error in the documentation? [float] A: Use the "Edit" link on any page ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ See <>. == Q: Can I delete only certain data from within indices? [float] A: It's complicated ~~~~~~~~~~~~~~~~~~~ [float] TL;DR: No. Curator can only delete entire indices. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [float] Full answer: ^^^^^^^^^^^^ As a thought exercise, think of Elasticsearch indices as being like databases, or tablespaces within a database. If you had hundreds of millions of rows to delete from your database, would you run a separate `DELETE from TABLE where date> flag. [float] The Problem: ^^^^^^^^^^^^ Illegal characters make it hard to delete indices. ------------------ % curl logs.example.com:9200/_cat/indices red }?ebc-2015.04.08.03 sip-request{ 5 1 0 0 632b 316b red }?ebc-2015.04.08.03 sip-response 5 1 0 0 474b 237b red ?ebc-2015.04.08.02 sip-request{ 5 1 0 0 474b 316b red eb 5 1 0 0 632b 316b red ?e 5 1 0 0 632b 316b ------------------   You can see it looks like there are some tab characters and maybe newline characters. This makes it hard to use the HTTP API to delete the indices. Dumping all the index settings out: [source,sh] ------- curl -XGET localhost:9200/*/_settings?pretty -------   ...reveals the index names as the first key in the resulting JSON. In this case, the names were very atypical: ------- }\b?\u0011ebc-2015.04.08.02\u000Bsip-request{ }\u0006?\u0011ebc-2015.04.08.03\u000Bsip-request{ }\u0003?\u0011ebc-2015.04.08.03\fsip-response ... -------   Curator lets you use a regular expression to select indices to perform actions on. To delete the first three from the above example, use `.*sip.*` as your regular expression: [source,sh] -------- curator delete indices --regex '.*sip.*' -------- WARNING: Before attempting a <>, see what will be affected by using the <> command first, or use the <> flag. TIP: You may find that your regular expression matches an index you do not want deleted. In that event, use the <> flag to exclude one or more indices, by name or by pattern. The next one is trickier. The real name of the index was `\n\u0011eb`. The regular expression `.*b$` did not work, but `\n.*` did: [source,sh] -------- curator delete indices --regex '\n.*' --------   The last index can be deleted with a regular expression of `.*e$`: [source,sh] -------- curator delete indices --regex '.*e$' -------- ''''' [[entrypoint-fix]] == Q: I'm getting `DistributionNotFound` and `entry_point` errors when I try to run Curator. What am I doing wrong? [float] A: You likely need to upgrade `setuptools` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are still unable to install, or get strange errors about dependencies you know you've installed, or messages mentioning `entry_point`, you may need to upgrade the `setuptools` package. This is especially common with RHEL and CentOS installs, and their variants, as they depend on Python 2.6. If you can run `pip install -U setuptools`, it should correct the problem. You may also be able to download and install manually: . `wget https://pypi.python.org/packages/source/s/setuptools/setuptools-15.1.tar.gz -O setuptools-15.1.tar.gz` . `pip install setuptools-15.1.tar.gz` Any dependencies this version of setuptools may require will have to be manually acquired and installed for your platform. For more information about setuptools, see https://pypi.python.org/pypi/setuptools This fix originally appeared https://github.com/elastic/curator/issues/56#issuecomment-77843587[here]. ''''' == Q: Snapshots seem to be working. Why am I getting `SnapshotMissingException` messages? [float] A: A load balancer, or other network fixture could be timing out your connection. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A gateway, proxy, or load balancer timeout will usually result in http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html[504 HTTP responses]: 504 Gateway Timeout:: The server, while acting as a gateway or proxy, did not receive a timely response from the upstream server specified by the URI (e.g. HTTP, FTP, LDAP) or some other auxiliary server (e.g. DNS) it needed to access in attempting to complete the request. An easy way to check is to turn on <> logging and look for 504 errors and long requests, like these: [source,txt] ------- 2015-04-02 13:18:37,702 DEBUG urllib3.connectionpool _make_request:368 "PUT /_snapshot/my_backup_s3_repository/curator-20150402121735?wait_for_completion=true HTTP/1.1" 504 0 2015-04-02 13:18:37,703 WARNING elasticsearch log_request_fail:81 PUT /_snapshot/my_backup_s3_repository/curator-20150402121735?wait_for_completion=true [status:504 request:59.777s] ------- Note the `status:504` and the `request:59.777s` at the right end of the line. In the case of these error messages, it turned out to be an AWS load balancer that had a 60 second timeout for connections. The fix was to increase the timeout. Another potential resolution would be to have Curator connect to a client directly, rather than through a load balancer or proxy. ''''' [[escape-percent]] == Q: How do I escape the `%` in the `--timestring` for use with systemd? [float] A: A double percent, `%%`, fixes the problem. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As seen here: https://discuss.elastic.co/t/curator-how-to-escape-characters-of-timestring-in-systemd/25222/2 --timestring '%%Y.%%m.%%d' ''''' curator-3.4.1/docs/asciidoc/misc/getting_started.asciidoc000066400000000000000000000007411265673226500235460ustar00rootroot00000000000000[[getting-started]] = Getting Started [partintro] -- Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your Elasticsearch indices. Curator performs many operations on your Elasticsearch indices, from delete to snapshot to shard allocation routing. -- include::installation.asciidoc[] include::upgrading.asciidoc[] include::command-line.asciidoc[] include::examples.asciidoc[] include::exitcodes.asciidoc[] curator-3.4.1/docs/asciidoc/misc/glossary.asciidoc000066400000000000000000000021501265673226500222160ustar00rootroot00000000000000[glossary] = Glossary of terms TIP: If you'd like to see a term defined here, use the "Edit" link to the right. See <>. [glossary] alias:: A pointer to one or more indices. See {ref}/indices-aliases.html[Index Aliases] in the official Elasticsearch documentation for more information. bloom:: This refers to a Bloom filter cache. A http://en.wikipedia.org/wiki/Bloom_filter[Bloom filter] can be used to speed up answers in a key-value storage system. Beginning with version 1.4, Elasticsearch no longer uses a Bloom filter cache for search operations. command:: An operation to perform on one or more indices or snapshots. See <>. subcommand:: A command which is nested, or comes after, a prior command. See <>. time-unit:: One of `hours`, `days`, `weeks`, or `months`. When acting on time-series indices, the <> is used for time calculation. timestring:: A http://strftime.org[Python strftime] string to match your index definition. 2014.07.15 would be `%Y.%m.%d`. See <>. curator-3.4.1/docs/asciidoc/misc/installation.asciidoc000066400000000000000000000176401265673226500230660ustar00rootroot00000000000000[[installation]] == Installation [float] Summary ~~~~~~~ There are a few ways to install curator. This document covers a few of them. If you are trying to upgrade curator, please see <>. [[pip]] === pip (recommended) The recommended installation procedure utilizes https://pip.pypa.io/en/latest/installing.html[python pip]: --------------------------------- pip install elasticsearch-curator ---------------------------------   [float] PIP installation of a specific version: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ---------------------------------------- pip install elasticsearch-curator==2.0.0 ----------------------------------------   Any other version-sepecific installation will follow this basic pattern. [float] Installation: PIP installation from source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download and install the https://github.com/shazow/urllib3[urllib3] dependency (1.8.3 or greater): . `wget https://pypi.python.org/packages/source/u/urllib3/urllib3-1.12.tar.gz` . `pip install urllib3-1.12.tar.gz`   Download and install the http://click.pocoo.org/[click] dependency (3.3 or greater): . `wget https://pypi.python.org/packages/source/c/click/click-5.1.tar.gz -O click-5.1.tar.gz` . `pip install click-5.1.tar.gz`   Download and install the https://github.com/elastic/elasticsearch-py[elasticsearch-py] dependency: . `wget https://github.com/elastic/elasticsearch-py/archive/`+pass:attributes[{es_py_version}].tar.gz -O elasticsearch-py.tar.gz+ . `pip install elasticsearch-py.tar.gz`   Download and install Curator: . `wget https://github.com/elastic/curator/archive/v`+pass:attributes[{curator_version}].tar.gz -O elasticsearch-curator.tar.gz+ . `pip install elasticsearch-curator.tar.gz` [[apt-repository]] === APT repository Starting with Elasticsearch Curator 3.2.0, repositories are available for APT and YUM based distributions. Note that we only provide binary packages, but no source packages, as the packages are created as part of the build process. We use the PGP key http://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4], Elastic's Signing Key, with fingerprint 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 to sign all our packages. It is available from http://pgp.mit.edu. [float] ==== Signing Key Download and install the Public Signing Key: [source,sh] -------------------------------------------------- wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - -------------------------------------------------- [float] ==== Repository Configuration Add the following in your `/etc/apt/sources.list.d/` directory in a file with a `.list` suffix, for example `curator.list` ["source","sh",subs="attributes,callouts"] -------------------------------------------------- deb http://packages.elastic.co/curator/{curator_major}/debian stable main -------------------------------------------------- [WARNING] ================================================== Use the file edit method described above to add the Curator repository. Do not use `add-apt-repository` as it will add a `deb-src` entry as well, but we do not provide a source package. If you have added the `deb-src` entry, you will see an error like the following: Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file) Just delete the `deb-src` entry from the `/etc/apt/sources.list.d/curator.list` file and the installation should work as expected. ================================================== [float] ==== Installation Run `sudo apt-get update` and the repository is ready for use. You can install it with: [source,sh] -------------------------------------------------- sudo apt-get update && sudo apt-get install python-elasticsearch-curator -------------------------------------------------- [[yum-repository]] === YUM repository Starting with Elasticsearch Curator 3.2.0, repositories are available for APT and YUM based distributions. Note that we only provide binary packages, but no source packages, as the packages are created as part of the build process. We use the PGP key http://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4], Elastic's Signing Key, with fingerprint 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 to sign all our packages. It is available from http://pgp.mit.edu. [float] ==== Signing Key Download and install the public signing key: [source,sh] -------------------------------------------------- rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch -------------------------------------------------- [float] ==== Repository Configuration Add the following in your `/etc/yum.repos.d/` directory in a file with a `.repo` suffix, for example `curator.repo` [WARNING] ======================================== The repositories are different for CentOS/RHEL 6 and 7 due to python library path differences. Be sure to use the correct version! RHEL/CentOS 6: ["source","sh",subs="attributes,callouts"] -------------------------------------------------- [curator-{curator_major}] name=CentOS/RHEL 6 repository for Elasticsearch Curator {curator_major}.x packages baseurl=http://packages.elastic.co/curator/{curator_major}/centos/6 gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 -------------------------------------------------- RHEL/CentOS 7: ["source","sh",subs="attributes,callouts"] -------------------------------------------------- [curator-{curator_major}] name=CentOS/RHEL 7 repository for Elasticsearch Curator {curator_major}.x packages baseurl=http://packages.elastic.co/curator/{curator_major}/centos/7 gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 -------------------------------------------------- ========================================= [float] ==== Installation And your repository is ready for use. You can install it with: [source,sh] ---------------------------------------- yum install python-elasticsearch-curator ---------------------------------------- [float] ==== Troubleshooting There are some pitfalls you may encounter: * `ImportError: No module named pkg_resources` + If you see this error: + [source,sh] ------------------------------ Traceback (most recent call last): File "/usr/bin/curator", line 5, in from pkg_resources import load_entry_point ImportError: No module named pkg_resources ------------------------------ + Then you will need to install python-setuptools (provided in the Curator repository): + [source,sh] ----------------------------- yum install python-setuptools ----------------------------- * `Requires: python-unittest2` (RHEL/CentOS 6 only) + If you see an error like this: + [source,sh] --------------------- Error: Package: python-elasticsearch-1.6.0-1.noarch (curator-3) Requires: python-unittest2 --------------------- + Then you will need to install python-unittest2, which is available in the `epel-release` repository: + [source,sh] ------------------ yum install epel-release ------------------ + After this is installed, run the `yum install python-elasticsearch-curator` command again. [[windows-binary]] === Windows Binary Package If you do not wish to install and maintain Python on Windows, there is a compiled binary version available (64bit only). It is in a directory with EXE files and all necessary libraries that Python requires. You can navigate to the directory and run the `curator` command just as you otherwise would. WARNING: If you do have Python installed, do not uncompress the zip file into your Python directory. It can cause library path collisions which will prevent Curator from properly functioning. * http://packages.elastic.co/curator/{curator_major}/windows/curator-{curator_version}-win64.zip[Download Curator] ** http://packages.elastic.co/curator/{curator_major}/windows/curator-{curator_version}-win64.zip.md5.txt[MD5] ** http://packages.elastic.co/curator/{curator_major}/windows/curator-{curator_version}-win64.zip.sha1.txt[SHA1] curator-3.4.1/docs/asciidoc/misc/upgrading.asciidoc000066400000000000000000000014461265673226500223420ustar00rootroot00000000000000[[upgrading]] == Upgrading If Curator has been installed via `pip`, the preferred way to upgrade is: ------------------------------------- pip install -U elasticsearch-curator ------------------------------------- If you need to manually upgrade, follow the <> for installing from source to obtain the most recent versions of both elasticsearch-py and Curator. You should then be able to run: ------------------------------------- pip install -U elasticsearch-py.tar.gz -------------------------------------   and then: ------------------------------------- pip install -U elasticsearch-curator.tar.gz ------------------------------------- If Curator has been installed via package repository, then use the standard upgrade method for your repository. curator-3.4.1/docs/asciidoc/selection/000077500000000000000000000000001265673226500177075ustar00rootroot00000000000000curator-3.4.1/docs/asciidoc/selection/index.asciidoc000066400000000000000000000077711265673226500225320ustar00rootroot00000000000000[[selection]] = Index & Snapshot Selection [partintro] -- The <> and <> <> allows <> to obtain a list of indices to act on. -- [[index-selection]] == Index Selection [float] Summary ~~~~~~~ The <> <> allows individual <> to obtain a list of indices to act on. ------------------------------------------------------------------------------ Options: --newer-than INTEGER Include only indices newer than n time_units --older-than INTEGER Include only indices older than n time_units --prefix TEXT Include only indices beginning with prefix. --suffix TEXT Include only indices ending with suffix. --time-unit [hours|days|weeks|months] Unit of time to reckon by --timestring TEXT Python strftime string to match your index definition, e.g. 2014.07.15 would be %Y.%m.%d --regex TEXT Provide your own regex, e.g '^prefix-.*-suffix$' --exclude TEXT Exclude matching indices. Can be invoked multiple times. --index TEXT Include the provided index in the list. Can be invoked multiple times. --all-indices Do not filter indices. Act on all indices. --help Show this message and exit. ------------------------------------------------------------------------------   [float] Options ~~~~~~~ * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> [[snapshot-selection]] == Snapshot Selection [float] Summary ~~~~~~~ The <> <> allows the <> and <> commands to obtain a list of snapshots to act on. ---------------------------------------------------------------------------- Options: --newer-than INTEGER Include only snapshots newer than n time_units --older-than INTEGER Include only snapshots older than n time_units --prefix TEXT Include only snapshots beginning with prefix. --suffix TEXT Include only snapshots ending with suffix. --time-unit [hours|days|weeks|months] Unit of time to reckon by --timestring TEXT Python strftime string to match your snapshot's definition, e.g. 20140715020304 would be %Y%m%d%H%M%S --regex TEXT Provide your own regex, e.g '^prefix-.*-suffix$' --exclude TEXT Exclude matching snapshots. Can be invoked multiple times. --snapshot TEXT Include the provided snapshot in the list. Can be invoked multiple times. --all-snapshots Do not filter snapshots. Act on all snapshots. --repository TEXT Repository name. --help Show this message and exit. ----------------------------------------------------------------------------   [float] Options ~~~~~~~ * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> curator-3.4.1/docs/asciidoc/subcommands/000077500000000000000000000000001265673226500202355ustar00rootroot00000000000000curator-3.4.1/docs/asciidoc/subcommands/index.asciidoc000066400000000000000000000112341265673226500230450ustar00rootroot00000000000000[[subcommand]] = Subcommands [partintro] -- The http://click.pocoo.org/3/[Click API] is used to form the command-line for Curator. Click allows Curator to have nested commands. In this way, a command-line structure can be built which allows multiple commands to all make use of the same subcommands. Available subcommands are <> for <>, and <> for <>. -- == Command "tree" This is tree view of all command and subcommands currently available: ---------------------------------- $ curator ├── alias │ └── indices ├── allocation │ └── indices ├── bloom │ └── indices ├── close │ └── indices ├── delete │ └── indices │ └── snapshots ├── open │ └── indices ├── optimize │ └── indices ├── replicas │ └── indices ├── seal │ └── indices ├── show │ └── indices │ └── snapshots └── snapshot └── indices ----------------------------------   [[indices-subcommand]] == indices Commands using the `indices` subcommand: * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> === Help Output This is the help output for <>. ---------- curator show indices --help Usage: curator show indices [OPTIONS] Get a list of indices to act on from the provided arguments, then perform the command [alias, allocation, bloom, close, delete, etc.] on the resulting list. Options: --newer-than INTEGER Include only indices newer than n time_units --older-than INTEGER Include only indices older than n time_units --prefix TEXT Include only indices beginning with prefix. --suffix TEXT Include only indices ending with suffix. --time-unit [hours|days|weeks|months] Unit of time to reckon by --timestring TEXT Python strftime string to match your index definition, e.g. 2014.07.15 would be %Y.%m.%d --regex TEXT Provide your own regex, e.g '^prefix-.*-suffix$' --exclude TEXT Exclude matching indices. Can be invoked multiple times. --index TEXT Include the provided index in the list. Can be invoked multiple times. --all-indices Do not filter indices. Act on all indices. --help Show this message and exit. ----------   [[snapshots-subcommand]] == snapshots Commands using the `snapshots` subcommand: * <> * <> === Help Output This is the help output for <>. ---------- curator show snapshots --help Usage: curator show snapshots [OPTIONS] Get a list of snapshots to act on from the provided arguments, then perform the command [delete, show] on the resulting list. Options: --newer-than INTEGER Include only snapshots newer than n time_units --older-than INTEGER Include only snapshots older than n time_units --prefix TEXT Include only snapshots beginning with prefix. --suffix TEXT Include only snapshots ending with suffix. --time-unit [hours|days|weeks|months] Unit of time to reckon by --timestring TEXT Python strftime string to match your snapshot's definition, e.g. 20140715020304 would be %Y%m%d%H%M%S --regex TEXT Provide your own regex, e.g '^prefix-.*-suffix$' --exclude TEXT Exclude matching snapshots. Can be invoked multiple times. --snapshot TEXT Include the provided snapshot in the list. Can be invoked multiple times. --all-snapshots Do not filter snapshots. Act on all snapshots. --repository TEXT Repository name. --help Show this message and exit. ---------- curator-3.4.1/docs/commands.rst000066400000000000000000000043201265673226500164760ustar00rootroot00000000000000.. _commands: Command Methods =============== Sections * `Aliasing Indices`_ * `Index Routing Allocation`_ * `Disabling Bloom Filters`_ * `Closing Indices`_ * `Deleting Indices`_ * `Opening Indices`_ * `Optimizing Indices`_ * `Changing Index Replica Count`_ * `Show Indices`_ * `Snapshot Indices`_ Aliasing Indices ---------------- alias +++++ .. automethod:: curator.api.alias add_to_alias ++++++++++++ .. automethod:: curator.api.add_to_alias remove_from_alias +++++++++++++++++ .. automethod:: curator.api.remove_from_alias Index Routing Allocation ------------------------ allocation ++++++++++ .. automethod:: curator.api.allocation apply_allocation_rule +++++++++++++++++++++ .. automethod:: curator.api.apply_allocation_rule Disabling Bloom Filters ----------------------- bloom +++++ .. automethod:: curator.api.bloom disable_bloom_filter ++++++++++++++++++++ .. automethod:: curator.api.disable_bloom_filter loop_bloom ++++++++++ .. automethod:: curator.api.loop_bloom Closing Indices --------------- close +++++ .. automethod:: curator.api.close close_indices +++++++++++++ .. automethod:: curator.api.close_indices Deleting Indices ---------------- delete ++++++ .. automethod:: curator.api.delete delete_indices ++++++++++++++ .. automethod:: curator.api.delete_indices Opening Indices --------------- opener ++++++ .. automethod:: curator.api.opener open_indices ++++++++++++ .. automethod:: curator.api.open_indices Optimizing Indices ------------------ optimize ++++++++ .. automethod:: curator.api.optimize optimize_index ++++++++++++++ .. automethod:: curator.api.optimize_index Changing Index Replica Count ---------------------------- replicas ++++++++ .. automethod:: curator.api.replicas change_replicas +++++++++++++++ .. automethod:: curator.api.change_replicas Sealing (Synced Flush) Indices ------------------------------ seal ++++ .. automethod:: curator.api.seal seal_indices ++++++++++++ .. automethod:: curator.api.seal_indices Show Indices ------------ show ++++ .. automethod:: curator.api.show Snapshot Indices ---------------- create_snapshot +++++++++++++++ .. automethod:: curator.api.create_snapshot delete_snapshot +++++++++++++++ .. automethod:: curator.api.delete_snapshot curator-3.4.1/docs/conf.py000066400000000000000000000211131265673226500154410ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Elasticsearch documentation build configuration file, created by # sphinx-quickstart on Mon May 6 15:38:41 2013. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os, re # Utility function to read from file. def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="../curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('../')) # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest'] autoclass_content = "both" # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Elasticsearch Curator' copyright = u'2011-2015, Elasticsearch' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The full version, including alpha/beta/rc tags. release = get_version() # The short X.Y version. version = '.'.join(release.split('.')[:2]) # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['_build'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. #keep_warnings = False # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'Curatordoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'ES_Curator.tex', u'Elasticsearch Curator Documentation', u'Aaron Mildenstein', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'curator', u'Elasticsearch Curator Documentation', [u'Aaron Mildenstein'], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'Curator', u'Elasticsearch Curator Documentation', u'Aaron Mildenstein', 'Curator', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. #texinfo_no_detailmenu = False curator-3.4.1/docs/examples.rst000066400000000000000000000151521265673226500165200ustar00rootroot00000000000000.. _examples: Examples ======== `build_filter` Examples ----------------------- Filter indices by prefix ++++++++++++++++++++++++ This example will generate a list of indices matching the prefix, 'logstash'. The effective regular expression would be: `^logstash.*$` `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. `_filter`'s contents: ``{'pattern': '^logstash.*$'}`` :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter(kindOf='prefix', value='logstash') working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then only be indices matching the `prefix`. Filter indices by suffix ++++++++++++++++++++++++ This example will generate a list of indices matching the suffix, '-prod'. The effective regular expression would be: `^.*-prod$` `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. `_filter`'s contents: ``{'pattern': '^.*-prod$'}`` :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter(kindOf='suffix', value='-prod') working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then only be indices matching the `suffix`. Filter indices by time (older_than) +++++++++++++++++++++++++++++++++++ This example will generate a list of indices matching the following criteria: * Have a date string of ``%Y.%m.%d`` * Use `days` as the unit of time measurement * Filter indices `older_than` 5 `days` `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. The resulting `_filter` dictionary will be: :: { 'pattern': '(?P\\d{4}\\.\\d{2}\\.\\d{2})', 'value': 5, 'groupname': 'date', 'time_unit': 'days', 'timestring': '%Y.%d.%m', 'method': 'older_than' } :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter(kindOf='suffix', value='-prod') working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then only be indices matching these criteria. Filter indices by time (newer_than) +++++++++++++++++++++++++++++++++++ This example will generate a list of indices matching the following criteria: * Have a date string of ``%Y.%m.%d`` * Use `days` as the unit of time measurement * Filter indices `newer_than` 5 `days` `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. The resulting `_filter` dictionary will be: :: { 'pattern': '(?P\\d{4}\\.\\d{2}\\.\\d{2})', 'value': 5, 'groupname': 'date', 'time_unit': 'days', 'timestring': '%Y.%d.%m', 'method': 'newer_than' } :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter( kindOf='newer_than', value=5, time_unit='days', timestring='%Y.%d.%m' ) working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then only be indices matching these criteria. Filter indices by custom regular expression +++++++++++++++++++++++++++++++++++++++++++ This example will generate a list of indices matching a custom regular expression ``(your expression)``. ``(your expression)`` needs to be a valid regular expression. `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. `_filter`'s contents: ``{'pattern': (your expression)}`` :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter(kindOf='regex', value=(your expression)) working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then only be indices matching (your expression) Filter indices by excluding matches +++++++++++++++++++++++++++++++++++ This example will generate a list of all indices `not` matching the pattern, 'dev-'. The effective regular expression would be: `^dev-.*$` `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. `_filter`'s contents: ``{'pattern': 'dev-', 'exclude': True}`` :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter(kindOf='exclude', value='dev-') working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then be all indices not matching the pattern, 'dev-'. .. note:: Any filter can become an `exclude` by adding ``'exclude':True`` to the `_filter` dictionary. Filter indices by time string as a pattern ++++++++++++++++++++++++++++++++++++++++++ This example will generate a list of indices having a matching time string, where `value` must be a valid python strftime string. `_filter` is a dictionary object. We send it as key word arguments (kwargs) to `apply_filter`. `_filter`'s contents: ``{'pattern': '(?P\\d{4}\\.\\d{2}\\.\\d{2})'}`` :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) _filter = curator.build_filter(kindOf='timestring', value='%Y.%m.%d') working_list = curator.apply_filter(indices, **_filter) The contents of `working_list` would then only be indices having a matching time string. More complex example mimicking the CLI ++++++++++++++++++++++++++++++++++++++ This example will show time-series indices matching `prefix`, `older_than` 30 `days` (the `time_unit`), and `newer_than` 60 `days`. :: import elasticsearch import curator client = elasticsearch.Elasticsearch() indices = curator.get_indices(client) filter_list = [] filter_list.append(curator.build_filter(kindOf='prefix', value='logstash')) filter_list.append( curator.build_filter( kindOf='older_than', value=30, time_unit='days', timestring='%Y.%d.%m' ) ) filter_list.append( curator.build_filter( kindOf='newer_than', value=60, time_unit='days', timestring='%Y.%d.%m' ) ) working_list = indices for filter in filter_list: working_list = curator.apply_filter(working_list, **filter) curator.show(working_list) curator-3.4.1/docs/filters.rst000066400000000000000000000013421265673226500163460ustar00rootroot00000000000000.. _filters: Filter Methods ============== Sections * `Regex`_ * `Date & Time`_ * `Disk Space`_ Regex ----- build_filter ++++++++++++ .. automethod:: curator.api.build_filter apply_filter +++++++++++++ .. automethod:: curator.api.apply_filter get_date_regex ++++++++++++++ .. automethod:: curator.api.get_date_regex Date & Time ----------- get_datetime +++++++++++++ .. automethod:: curator.api.get_datetime get_target_month ++++++++++++++++ .. automethod:: curator.api.get_target_month get_cutoff ++++++++++ .. automethod:: curator.api.get_cutoff timestamp_check +++++++++++++++ .. automethod:: curator.api.timestamp_check Disk Space ---------- filter_by_space +++++++++++++++ .. automethod:: curator.api.filter_by_space curator-3.4.1/docs/index.rst000066400000000000000000000057301265673226500160120ustar00rootroot00000000000000Elasticsearch Curator Python API ================================ The Elasticsearch Curator Python API helps you manage your indices and snapshots. .. note:: This documentation is for the Elasticsearch Curator Python API. Documentation for the Elasticsearch Curator *CLI* -- which uses this API and is installed as an entry_point as part of the package -- is available in the `Elastic guide`_. .. _Elastic guide: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html Compatibility ------------- The Elasticsearch Curator Python API is compatible with Elasticsearch versions 1.x through 2.0, and supports Python versions 2.6 and later. Example Usage ------------- :: import elasticsearch import curator client = elasticsearch.Elasticsearch() curator.close_indices(client, ['logstash-2014.08.16','logstash-2014.08.17']) curator.disable_bloom_filter(client, 'logstash-2014.08.31') curator.optimize_index(client, 'logstash-2014.08.31') curator.delete(client, ['logstash-2014.07.16', 'logstash-2014.07.17']) .. TIP:: See more examples in the :doc:`Examples ` page. Features -------- The API methods fall into the following categories: * :doc:`Commands ` take a single index, or in some cases a list of indices and perform an action on them. * :doc:`Filters ` are there to filter indices or snapshots based on provided criteria. * :doc:`Utilities ` are helper methods for commands and filters. Filtering indices is now handled by the :py:func:`curator.api.build_filter` method. Logging ~~~~~~~ The Elasticsearch Curator Python API uses the standard `logging library`_ from Python. It inherits two loggers from ``elasticsearch-py``: ``elasticsearch`` and ``elasticsearch.trace``. Clients use the ``elasticsearch`` logger to log standard activity, depending on the log level. The ``elasticsearch.trace`` logger logs requests to the server in JSON format as pretty-printed ``curl`` commands that you can execute from the command line. The ``elasticsearch.trace`` logger is not inherited from the base logger and must be activated separately. .. _logging library: http://docs.python.org/3.3/library/logging.html Contents -------- .. toctree:: :maxdepth: 2 commands filters utilities examples Changelog License ------- Copyright 2013–2015 Elastic and contributors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Indices and tables ------------------ * :ref:`genindex` * :ref:`search` curator-3.4.1/docs/utilities.rst000066400000000000000000000033141265673226500167120ustar00rootroot00000000000000.. _utilities: Utility & Helper Methods ======================== Sections * `Get Information`_ * `Verification`_ * `Index Pruning`_ * `Other`_ Get Information --------------- get_alias +++++++++ .. automethod:: curator.api.get_alias get_indices +++++++++++ .. automethod:: curator.api.get_indices get_repository ++++++++++++++ .. automethod:: curator.api.get_repository get_segmentcount ++++++++++++++++ .. automethod:: curator.api.get_segmentcount get_snapshot ++++++++++++ .. automethod:: curator.api.get_snapshot get_snapshots +++++++++++++ .. automethod:: curator.api.get_snapshots get_version +++++++++++ .. automethod:: curator.api.get_version Verification ------------ check_csv +++++++++ .. automethod:: curator.api.check_csv ensure_list +++++++++++ .. automethod:: curator.api.ensure_list index_closed ++++++++++++ .. automethod:: curator.api.index_closed optimized +++++++++ .. automethod:: curator.api.optimized is_master_node ++++++++++++++ .. automethod:: curator.api.is_master_node to_csv ++++++ .. automethod:: curator.api.to_csv verify_repository +++++++++++++++++ .. automethod:: curator.api.verify_repository Index Pruning ------------- prune_allocated +++++++++++++++ .. automethod:: curator.api.prune_allocated prune_closed ++++++++++++ .. automethod:: curator.api.prune_closed prune_kibana ++++++++++++ .. automethod:: curator.api.prune_kibana Other ----- create_snapshot_body ++++++++++++++++++++ .. automethod:: curator.api.create_snapshot_body create_repo_body ++++++++++++++++ .. automethod:: curator.api.create_repo_body create_repository +++++++++++++++++ .. automethod:: curator.api.create_repository chunk_index_list ++++++++++++++++ .. automethod:: curator.api.chunk_index_list curator-3.4.1/requirements.txt000066400000000000000000000000661265673226500165020ustar00rootroot00000000000000urllib3>=1.8.3 elasticsearch>=1.8.0,<2.1.0 click>=3.3 curator-3.4.1/run_curator.py000077500000000000000000000002171265673226500161340ustar00rootroot00000000000000#!/usr/bin/env python """Wrapper for running curator from source.""" from curator.curator import main if __name__ == '__main__': main() curator-3.4.1/run_es_repo_mgr.py000077500000000000000000000002271265673226500167570ustar00rootroot00000000000000#!/usr/bin/env python """Wrapper for running es_repo_mgr from source.""" from curator.es_repo_mgr import main if __name__ == '__main__': main() curator-3.4.1/setup.cfg000066400000000000000000000001031265673226500150270ustar00rootroot00000000000000[metadata] description-file = README.md [bdist_wheel] universal=1 curator-3.4.1/setup.py000066400000000000000000000104421265673226500147270ustar00rootroot00000000000000import os import re import sys from setuptools import setup # Utility function to read from file. def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION def get_install_requires(): res = ['elasticsearch>=1.8.0,<2.4.0' ] res.append('click>=3.3') return res try: ### cx_Freeze ### from cx_Freeze import setup, Executable # Dependencies are automatically detected, but it might need # fine tuning. buildOptions = dict(packages = [], excludes = []) base = 'Console' icon = None if os.path.exists('Elastic.ico'): icon = 'Elastic.ico' curator_exe = Executable( "run_curator.py", base=base, targetName = "curator", compress = True ) repo_mgr_exe = Executable( "run_es_repo_mgr.py", base=base, targetName = "es_repo_mgr", compress = True ) if sys.platform == "win32": curator_exe = Executable( "run_curator.py", base=base, targetName = "curator.exe", compress = True, icon = icon ) repo_mgr_exe = Executable( "run_es_repo_mgr.py", base=base, targetName = "es_repo_mgr.exe", compress = True, icon = icon ) setup( name = "elasticsearch-curator", version = get_version(), author = "Elastic", author_email = "info@elastic.co", description = "Tending your Elasticsearch indices", long_description=fread('README.rst'), url = "http://github.com/elastic/curator", download_url = "https://github.com/elastic/curator/tarball/v" + get_version(), license = "Apache License, Version 2.0", install_requires = get_install_requires(), keywords = "elasticsearch time-series indexed index-expiry", packages = ["curator", "curator.api", "curator.cli"], include_package_data=True, entry_points = { "console_scripts" : ["curator = curator.curator:main", "es_repo_mgr = curator.es_repo_mgr:main"] }, classifiers=[ "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", ], test_suite = "test.run_tests.run_all", tests_require = ["mock==1.0.1", "nose", "coverage", "nosexcover"], options = {"build_exe" : buildOptions}, executables = [curator_exe, repo_mgr_exe] ) ### end cx_Freeze ### except ImportError: print('Unable to load cx_Freeze. Cannot do \'build_exe\' or \'bdist_msi\'.\n') setup( name = "elasticsearch-curator", version = get_version(), author = "Elastic", author_email = "info@elastic.co", description = "Tending your Elasticsearch indices", long_description=fread('README.rst'), url = "http://github.com/elastic/curator", download_url = "https://github.com/elastic/curator/tarball/v" + get_version(), license = "Apache License, Version 2.0", install_requires = get_install_requires(), keywords = "elasticsearch time-series indexed index-expiry", packages = ["curator", "curator.api", "curator.cli"], include_package_data=True, entry_points = { "console_scripts" : ["curator = curator.curator:main", "es_repo_mgr = curator.es_repo_mgr:main"] }, classifiers=[ "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", ], test_suite = "test.run_tests.run_all", tests_require = ["mock==1.0.1", "nose", "coverage", "nosexcover"] ) curator-3.4.1/test/000077500000000000000000000000001265673226500141735ustar00rootroot00000000000000curator-3.4.1/test/__init__.py000066400000000000000000000000001265673226500162720ustar00rootroot00000000000000curator-3.4.1/test/integration/000077500000000000000000000000001265673226500165165ustar00rootroot00000000000000curator-3.4.1/test/integration/__init__.py000066400000000000000000000101611265673226500206260ustar00rootroot00000000000000import time import os import shutil import tempfile import random import string from datetime import timedelta, datetime, date from elasticsearch import Elasticsearch from elasticsearch.exceptions import ConnectionError from unittest import SkipTest, TestCase from mock import Mock client = None DATEMAP = { 'months': '%Y.%m', 'weeks': '%Y.%W', 'days': '%Y.%m.%d', 'hours': '%Y.%m.%d.%H', } host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 def get_client(): global client if client is not None: return client client = Elasticsearch([os.environ.get('TEST_ES_SERVER', {})], timeout=300) # wait for yellow status for _ in range(100): time.sleep(.1) try: client.cluster.health(wait_for_status='yellow') return client except ConnectionError: continue else: # timeout raise SkipTest("Elasticsearch failed to start.") def setup(): get_client() class Args(dict): def __getattr__(self, att_name): return self.get(att_name, None) class CuratorTestCase(TestCase): def setUp(self): super(CuratorTestCase, self).setUp() self.client = get_client() args = {} args['host'], args['port'] = host, port args['time_unit'] = 'days' args['prefix'] = 'logstash-' self.args = args dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) # This will create a psuedo-random temporary directory on the machine # which runs the unit tests, but NOT on the machine where elasticsearch # is running. This means tests may fail if run against remote instances # unless you explicitly set `self.args['location']` to a proper spot # on the target machine. self.args['location'] = tempfile.mkdtemp(suffix=dirname) self.args['repository'] = 'TEST_REPOSITORY' if not os.path.exists(self.args['location']): os.makedirs(self.args['location']) def tearDown(self): self.delete_repositories() self.client.indices.delete(index='*') self.client.indices.delete_template(name='*', ignore=404) if os.path.exists(self.args['location']): shutil.rmtree(self.args['location']) def parse_args(self): return Args(self.args) def create_indices(self, count, unit=None): now = datetime.utcnow() unit = unit if unit else self.args['time_unit'] format = DATEMAP[unit] if not unit == 'months': step = timedelta(**{unit: 1}) for x in range(count): self.create_index(self.args['prefix'] + now.strftime(format), wait_for_yellow=False) now -= step else: # months now = date.today() d = date(now.year, now.month, 1) self.create_index(self.args['prefix'] + now.strftime(format), wait_for_yellow=False) for i in range(1, count): if d.month == 1: d = date(d.year-1, 12, 1) else: d = date(d.year, d.month-1, 1) self.create_index(self.args['prefix'] + datetime(d.year, d.month, 1).strftime(format), wait_for_yellow=False) self.client.cluster.health(wait_for_status='yellow') def create_index(self, name, shards=1, wait_for_yellow=True): self.client.indices.create( index=name, body={'settings': {'number_of_shards': shards, 'number_of_replicas': 0}} ) if wait_for_yellow: self.client.cluster.health(wait_for_status='yellow') def create_repository(self): body = {'type':'fs', 'settings':{'location':self.args['location']}} self.client.snapshot.create_repository(repository=self.args['repository'], body=body) def delete_repositories(self): result = self.client.snapshot.get_repository(repository='_all') for repo in result: self.client.snapshot.delete_repository(repository=repo) def close_index(self, name): self.client.indices.close(index=name) curator-3.4.1/test/integration/test_api_commands.py000066400000000000000000000225501265673226500225650ustar00rootroot00000000000000from curator import api as curator from mock import patch, Mock from . import CuratorTestCase class TestAlias(CuratorTestCase): def test_add_to_pre_existent_alias(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') curator.add_to_alias(self.client, 'foo', alias=alias) self.assertEquals(2, len(self.client.indices.get_alias(name=alias))) def test_add_to_non_existent_alias(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') self.assertTrue(curator.add_to_alias(self.client, 'foo', alias="ooga")) def test_add_to_alias_with_closed(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') curator.close_indices(self.client, 'foo') self.assertTrue(curator.index_closed(self.client, 'foo')) curator.add_to_alias(self.client, 'foo', alias=alias) self.assertFalse(curator.add_to_alias(self.client, 'foo', alias=alias)) def test_add_to_alias_idx_already_in_alias(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') curator.add_to_alias(self.client, 'foo', alias=alias) self.assertTrue(curator.add_to_alias(self.client, 'foo', alias=alias)) def test_remove_from_alias_positive(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') curator.add_to_alias(self.client, 'foo', alias=alias) curator.remove_from_alias(self.client, 'dummy', alias=alias) self.assertEquals(1, len(self.client.indices.get_alias(name=alias))) def test_remove_from_alias_negative(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.assertFalse(curator.remove_from_alias(self.client, 'dummy', alias="ooga")) def test_full_alias_add_to_pre_existent_alias(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') self.assertTrue(curator.alias(self.client, 'foo', alias=alias)) self.assertEquals(2, len(self.client.indices.get_alias(name=alias))) def test_full_alias_add_non_existent_index(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.assertFalse(curator.alias(self.client, 'foo', alias='ooga')) def test_full_alias_remove_positive(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.create_index('foo') curator.add_to_alias(self.client, 'foo', alias=alias) curator.alias(self.client, 'dummy', alias=alias, remove=True) self.assertEquals(1, len(self.client.indices.get_alias(name=alias))) def test_full_alias_remove_negative(self): alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) self.assertFalse(curator.alias(self.client, 'dummy', alias="ooga", remove=True)) class TestChangeReplicas(CuratorTestCase): def test_index_replicas_can_be_modified(self): self.create_index('test_index') self.assertEquals('0', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) curator.change_replicas(self.client, 'test_index', replicas=1) self.assertEquals('1', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) def test_index_replicas_untouched(self): self.create_index('test_index') self.assertEquals('0', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) curator.change_replicas(self.client, 'test_index', replicas=0) self.assertEquals('0', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) def test_closed_index_replicas_cannot_be_modified(self): self.create_index('test_index') self.client.indices.close(index='test_index') index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('close', index_metadata['metadata']['indices']['test_index']['state']) self.assertEquals('0', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) curator.change_replicas(self.client, 'test_index', replicas=1) self.assertEquals('0', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) self.client.indices.open(index='test_index') index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('open', index_metadata['metadata']['indices']['test_index']['state']) self.assertEquals('0', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['number_of_replicas']) class TestChangeAllocation(CuratorTestCase): def test_index_allocation_can_be_modified(self): self.create_index('test_index') curator.apply_allocation_rule(self.client, 'test_index', rule="key=value") self.assertEquals('value', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['routing']['allocation']['require']['key']) def test_index_allocation_can_be_modified_for_include(self): self.create_index('test_index') curator.apply_allocation_rule(self.client, 'test_index', rule="key=value", allocation_type='include') self.assertEquals('value', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['routing']['allocation']['include']['key']) def test_index_allocation_can_be_modified_for_exclude(self): self.create_index('test_index') curator.apply_allocation_rule(self.client, 'test_index', rule="key=value", allocation_type='exclude') self.assertEquals('value', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['routing']['allocation']['exclude']['key']) def test_closed_index_allocation_can_be_modified(self): self.create_index('test_index') self.client.indices.close(index='test_index') index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('close', index_metadata['metadata']['indices']['test_index']['state']) curator.apply_allocation_rule(self.client, 'test_index', rule="key=value") self.assertEquals('value', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['routing']['allocation']['require']['key']) self.client.indices.open(index='test_index') index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('open', index_metadata['metadata']['indices']['test_index']['state']) self.assertEquals('value', self.client.indices.get_settings(index='test_index')['test_index']['settings']['index']['routing']['allocation']['require']['key']) class TestDeleteIndex(CuratorTestCase): def test_index_will_be_deleted(self): self.create_index('test_index') self.assertTrue(curator.delete_indices(self.client, 'test_index')) self.assertFalse(self.client.indices.exists('test_index')) class TestBloomIndex(CuratorTestCase): def test_bloom_filter_will_be_disabled(self): self.create_index('test_index') # Bloom filters have been removed from the 1.x branch after 1.4.0 no_more_bloom = (1, 4, 0) version_number = curator.get_version(self.client) if version_number < no_more_bloom: self.assertTrue(curator.disable_bloom_filter(self.client, 'test_index')) settings = self.client.indices.get_settings(index='test_index') self.assertEquals('false', settings['test_index']['settings']['index']['codec']['bloom']['load']) def test_closed_index_will_be_skipped(self): self.create_index('test_index') self.client.indices.close(index='test_index') self.assertTrue(curator.disable_bloom_filter(self.client, 'test_index')) index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('close', index_metadata['metadata']['indices']['test_index']['state']) class TestOptimizeIndex(CuratorTestCase): def test_optimized_index_will_be_skipped(self): self.create_index('test_index') self.client.create(index='test_index', doc_type='log', body={'message':'TEST DOCUMENT'}) # Will only have 1 segment self.assertTrue(curator.optimize_index(self.client, 'test_index', max_num_segments=4)) curator-3.4.1/test/integration/test_api_utils.py000066400000000000000000000102331265673226500221170ustar00rootroot00000000000000from curator import api as curator from mock import patch, Mock from . import CuratorTestCase class TestGetIndices(CuratorTestCase): def test_positive(self): self.create_index('test_index1') self.create_index('test_index2') l = sorted(curator.get_indices(self.client)) r = ["test_index1", "test_index2"] self.assertEqual(r, l) def test_negative(self): l = sorted(curator.get_indices(self.client)) r = ["test_index1", "test_index2"] self.assertNotEqual(r, l) def test_exception(self): client = "foo" l = curator.get_indices(client) self.assertFalse(l) class TestCloseIndex1(CuratorTestCase): def test_positive(self): self.create_index('test_index') self.client.indices.close('test_index') self.assertTrue(curator.index_closed(self.client, 'test_index')) def test_negative(self): self.create_index('test_index') self.assertFalse(curator.index_closed(self.client, 'test_index')) class TestCloseIndex2(CuratorTestCase): def test_index_will_be_closed(self): self.create_index('test_index') self.assertTrue(curator.close_indices(self.client, 'test_index')) index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('close', index_metadata['metadata']['indices']['test_index']['state']) def test_closed_index_will_be_skipped(self): self.create_index('test_index') self.client.indices.close(index='test_index') self.assertTrue(curator.close_indices(self.client, 'test_index')) index_metadata = self.client.cluster.state( index='test_index', metric='metadata', ) self.assertEquals('close', index_metadata['metadata']['indices']['test_index']['state']) class TestPruneClosed(CuratorTestCase): def test_positive(self): for i in range(1, 11): self.create_index('test_index{0}'.format(i)) self.client.indices.close(index='test_index1,test_index2,test_index3') l = curator.get_indices(self.client) l = sorted(curator.prune_closed(self.client, l)) self.assertEqual(7, len(l)) r = [ "test_index10", "test_index4", "test_index5", "test_index6", "test_index7", "test_index8", "test_index9", ] self.assertEqual(l, r) def test_negative(self): for i in range(1, 11): self.create_index('test_index{0}'.format(i)) self.client.indices.delete(index='test_index1,test_index2,test_index3') l = curator.get_indices(self.client) l = sorted(curator.prune_closed(self.client, l)) self.assertEqual(7, len(l)) r = [ "test_index10", "test_index4", "test_index5", "test_index6", "test_index7", "test_index8", "test_index9", ] self.assertEqual(l, r) class TestPruneOpened(CuratorTestCase): def test_positive(self): for i in range(1, 11): self.create_index('test_index{0}'.format(i)) self.client.indices.close(index='test_index1,test_index2,test_index3') l = curator.get_indices(self.client) l = sorted(curator.prune_opened(self.client, l)) self.assertEqual(3, len(l)) r = [ "test_index1", "test_index2", "test_index3" ] self.assertEqual(l, r) def test_negative(self): for i in range(1, 11): self.create_index('test_index{0}'.format(i)) self.client.indices.delete(index='test_index1,test_index2,test_index3') self.client.indices.close(index='test_index4') l = curator.get_indices(self.client) l = sorted(curator.prune_opened(self.client, l)) self.assertEqual(1, len(l)) r = ["test_index4"] self.assertEqual(l, r) class TestSegmentCount(CuratorTestCase): def test_simple(self): self.create_index('test_index', shards=2) self.client.index(index='test_index', doc_type='t', id=42, body={}) self.client.indices.refresh(index='test_index') self.assertEquals((2, 1), curator.get_segmentcount(self.client, 'test_index')) curator-3.4.1/test/integration/test_cli_commands.py000066400000000000000000001236611265673226500225700ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestGetClient(CuratorTestCase): def test_get_client_positive(self): client_args = {"host":host, "port":port} client = curator.get_client(**client_args) self.assertTrue(isinstance(client, elasticsearch.client.Elasticsearch)) def test_get_client_negative_connection_fail(self): client_args = {"host":host, "port":54321} with self.assertRaises(SystemExit) as cm: curator.get_client(**client_args) self.assertEqual(cm.exception.code, 1) class TestCLIIndexSelection(CuratorTestCase): def test_index_selection_only_timestamp_filter(self): self.create_indices(10) indices = curator.get_indices(self.client) # expected = sorted(indices, reverse=True)[:4] test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'close', 'indices', '--timestring', '%Y.%m.%d', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) def test_index_selection_no_filters(self): self.create_indices(1) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_index_selection_manual_selection_single(self): self.create_index('my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--index', 'my_index', ], obj={"filters":[]}) self.assertEqual(['my_index'], result.output.splitlines()[:1]) def test_index_selection_no_indices_test(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_index_selection_all_indices_single(self): self.create_index('my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(['my_index'], result.output.splitlines()[:1]) def test_index_selection_all_indices_exclude(self): self.create_index('my_index') self.create_index('your_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--all-indices', '--exclude', '^your.*$', ], obj={"filters":[]}) self.assertEqual(['my_index'], result.output.splitlines()[:1]) def test_index_selection_regex_match(self): self.create_index('my_index') self.create_index('your_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--regex', '^my.*$', ], obj={"filters":[]}) self.assertEqual(['my_index'], result.output.splitlines()[:1]) def test_index_selection_all_indices_skip_non_exclude_filters(self): self.create_index('my_index') self.create_index('your_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--all-indices', '--suffix', 'index', ], obj={"filters":[]}) self.assertEqual(['my_index', 'your_index'], result.output.splitlines()[:2]) def test_cli_show_indices_if_dry_run(self): self.create_indices(10) indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True)[:4] test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--dry-run', '--host', host, '--port', str(port), 'alias', '--name', 'dummy_alias', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True) # I tried doing a nested, double list comprehension here. # It works in the interpreter, but not here for some reason. output = [ x.split(' ')[-1:] for x in output ] output = [ x[0] for x in output if x[0].startswith('logstash') ] self.assertEqual(expected, output) def test_cli_no_indices_after_filtering(self): self.create_indices(10) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--exclude', 'log', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) def test_cli_closed_indices_only(self): self.create_index('open-one') self.create_index('closed-one') self.close_index('closed-one') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--closed-only', '--suffix', 'one', ], obj={"filters":[]}) self.assertEqual(['closed-one (CLOSED)'], result.output.splitlines()[:2]) class TestCLIAlias(CuratorTestCase): def test_alias_no_name_param(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'alias', 'indices', '--exclude', 'log', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_alias_all_indices_single(self): self.create_index('my_index') alias = 'testalias' self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'alias', '--name', alias, 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEquals(2, len(self.client.indices.get_alias(name=alias))) class TestCLIAllocation(CuratorTestCase): def test_allocation_no_rule_param(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'allocation', 'indices', '--exclude', 'log', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_allocation_all_indices_single(self): self.create_index('my_index') key = 'foo' value = 'bar' rule = key + '=' + value test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'allocation', '--rule', rule, 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEquals( value, self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']['routing']['allocation']['require'][key] ) def test_allocation_all_indices_include(self): self.create_index('my_index') key = 'foo' value = 'bar' rule = key + '=' + value allocation_type = 'include' test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'allocation', '--rule', rule, '--type', allocation_type, 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEquals( value, self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']['routing']['allocation'][allocation_type][key] ) def test_allocation_all_indices_exclude(self): self.create_index('my_index') key = 'foo' value = 'bar' rule = key + '=' + value allocation_type = 'exclude' test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'allocation', '--rule', rule, '--type', allocation_type, 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEquals( value, self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']['routing']['allocation'][allocation_type][key] ) def test_allocation_fail_on_bad_type(self): self.create_index('my_index') key = 'foo' value = 'bar' rule = key + '=' + value allocation_type = 'fail' test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'allocation', '--rule', rule, '--type', allocation_type, 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) class TestCLIBloom(CuratorTestCase): def test_bloom_cli(self): self.create_indices(5) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'bloom', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLIClose(CuratorTestCase): def test_close_cli(self): self.create_indices(5) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'close', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLIDelete(CuratorTestCase): def test_delete_indices_skip_kibana(self): self.create_index('my_index') self.create_index('.kibana') self.create_index('kibana-int') self.create_index('.marvel-kibana') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'delete', 'indices', '--all-indices', ], obj={"filters":[]}) l = curator.get_indices(self.client) self.assertEqual(sorted(['.kibana', '.marvel-kibana', 'kibana-int']), sorted(l)) def test_delete_indices_dry_run(self): self.create_indices(9) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--dry-run', '--host', host, '--port', str(port), 'delete', 'indices', '--all-indices', ], obj={"filters":[]}) l = curator.get_indices(self.client) self.assertEquals(9, len(l)) output = sorted(result.output.splitlines(), reverse=True) # I tried doing a nested, double list comprehension here. # It works in the interpreter, but not here for some reason. output = [ x.split(' ')[-1:] for x in output ] output = [ x[0] for x in output if x[0].startswith('logstash') ] self.assertEqual(sorted(l, reverse=True), output) def test_delete_indices_by_space_dry_run(self): for i in range(1,10): self.client.create( index="index" + str(i), doc_type='log', body={'message':'TEST DOCUMENT'}, ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--dry-run', '--host', host, '--port', str(port), 'delete', '--disk-space', '0.0000001', 'indices', '--all-indices', ], obj={"filters":[]}) l = curator.get_indices(self.client) self.assertEquals(9, len(l)) output = sorted(result.output.splitlines(), reverse=True) # I tried doing a nested, double list comprehension here. # It works in the interpreter, but not here for some reason. output = [ x.split(' ')[-1:] for x in output ] output = [ x[0] for x in output if x[0].startswith('index') ] self.assertEqual(sorted(l, reverse=True), output) def test_delete_indices_huge_list(self): self.create_indices(365) pre = curator.get_indices(self.client) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'delete', 'indices', '--all-indices', '--exclude', pre[0], ], obj={"filters":[]}) post = curator.get_indices(self.client) self.assertEquals(1, len(post)) def test_delete_indices_by_space_dry_run_huge_list(self): for i in range(100,150): self.client.create( index="superlongindexnamebyanystandardyouchoosethisissillyhowbigcanthisgetbeforeitbreaks" + str(i), doc_type='log', body={'message':'TEST DOCUMENT'}, ) l = curator.get_indices(self.client) self.assertEquals(50, len(l)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--dry-run', '--host', host, '--port', str(port), 'delete', '--disk-space', '0.0000001', 'indices', '--all-indices', ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True) # # I tried doing a nested, double list comprehension here. # # It works in the interpreter, but not here for some reason. output = [ x.split(' ')[-1:] for x in output ] output = [ x[0] for x in output if x[0].startswith('superlongindexname') ] self.assertEqual(sorted(l, reverse=True), output) class TestCLIOpen(CuratorTestCase): def test_open_cli(self): self.create_indices(5) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'open', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLIOptimize(CuratorTestCase): def test_optimize_cli(self): self.create_indices(5) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'optimize', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) def test_optimize_with_delay_cli(self): self.create_index("index_name") for i in ["1", "2", "3"]: self.client.create( index="index_name", doc_type='log', body={"doc" + i :'TEST DOCUMENT'}, ) # This should force each doc to be in its own segment. self.client.indices.flush(index="index_name", force=True) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'optimize', '--delay', 1, 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLIReplicas(CuratorTestCase): def test_replicas_no_count_param(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'replicas', 'indices', '--exclude', 'log', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_replicas_cli(self): self.create_indices(5) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'replicas', '--count', '2', 'indices', '--all-indices', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLISeal(CuratorTestCase): def test_cli_seal_indices(self): self.create_indices(10) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'seal', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLIShow(CuratorTestCase): def test_cli_show_indices(self): self.create_indices(10) indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True)[:4] test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True)[:4] self.assertEqual(expected, output) def test_cli_show_indices_hours(self): self.create_index('logstash-2016.02.10.00') indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--older-than', '5', '--timestring', '%Y.%m.%d.%H', '--time-unit', 'hours' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True) self.assertEqual(expected, output) def test_cli_show_indices_weeks(self): self.create_index('logstash-2016-03') indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--older-than', '1', '--timestring', '%Y-%W', '--time-unit', 'weeks' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True) self.assertEqual(expected, output) def test_cli_show_indices_months(self): self.create_index('logstash-2015.12') indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--older-than', '1', '--timestring', '%Y.%m', '--time-unit', 'months' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True) self.assertEqual(expected, output) def test_cli_show_indices_older_than_zero(self): self.create_indices(10) indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--older-than', '0', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True) self.assertEqual(expected, output) class TestCLISnapshot(CuratorTestCase): def test_snapshot_no_repository_param(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'snapshot', 'indices', '--exclude', 'log', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_cli_snapshot_indices(self): self.create_indices(5) self.create_repository() snap_name = 'snapshot1' indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True)[:4] test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'snapshot', '--repository', self.args['repository'], '--name', snap_name, 'indices', '--all-indices', ], obj={"filters":[]}) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(1, len(snapshot['snapshots'])) self.assertEqual(snap_name, snapshot['snapshots'][0]['snapshot']) def test_cli_snapshot_huge_list(self): self.create_indices(200) self.create_repository() snap_name = 'snapshot1' test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), '--timeout', 600, 'snapshot', '--repository', self.args['repository'], '--name', snap_name, 'indices', '--all-indices', ], obj={"filters":[]}) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(1, len(snapshot['snapshots'])) self.assertEqual(snap_name, snapshot['snapshots'][0]['snapshot']) self.assertEqual(200, len(snapshot['snapshots'][0]['indices'])) class TestCLISnapshotSelection(CuratorTestCase): def test_show_no_repository_param(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--exclude', 'log', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_snapshot_selection_show_all_snapshots(self): index_name = 'index1' snap_name = 'snapshot1' self.create_index(index_name) self.create_repository() self.client.create( index=index_name, doc_type='log', body={'message':'TEST DOCUMENT'} ) curator.create_snapshot( self.client, name=snap_name, indices=index_name, repository=self.args['repository'] ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--repository', self.args['repository'], '--all-snapshots', ], obj={"filters":[]}) self.assertEqual([snap_name], result.output.splitlines()[:1]) def test_snapshot_selection_show_all_snapshots_with_exclude(self): self.create_repository() for i in ["1", "2"]: self.create_index("index" + i) self.client.create( index="index" + i, doc_type='log', body={'message':'TEST DOCUMENT'}, ) curator.create_snapshot( self.client, name="snapshot" + i, indices="index" + i, repository=self.args['repository'] ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--repository', self.args['repository'], '--all-snapshots', '--exclude', '2', ], obj={"filters":[]}) self.assertEqual(['snapshot1'], result.output.splitlines()[:1]) def test_snapshot_selection_no_snapshots_found(self): self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--repository', self.args['repository'], '--all-snapshots', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) def test_snapshot_selection_show_filtered(self): self.create_repository() for i in ["1", "2", "3"]: self.create_index("index" + i) self.client.create( index="index" + i, doc_type='log', body={'message':'TEST DOCUMENT'}, ) curator.create_snapshot( self.client, name="snapshot" + i, indices="index" + i, repository=self.args['repository'] ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--repository', self.args['repository'], '--prefix', 'snap', '--suffix', '1', ], obj={"filters":[]}) self.assertEqual(['snapshot1'], result.output.splitlines()[:1]) def test_snapshot_selection_show_all_snapshots_with_exclude_and_other(self): self.create_repository() for i in ["1", "2"]: self.create_index("index" + i) self.client.create( index="index" + i, doc_type='log', body={'message':'TEST DOCUMENT'}, ) curator.create_snapshot( self.client, name="snapshot" + i, indices="index" + i, repository=self.args['repository'] ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--repository', self.args['repository'], '--all-snapshots', '--exclude', '2', '--prefix', 'missing', '--suffix', 'also_missing', ], obj={"filters":[]}) self.assertEqual(['snapshot1'], result.output.splitlines()[:1]) def test_snapshot_selection_delete_all_snapshots_with_dry_run(self): self.create_repository() for i in ["1", "2"]: self.create_index("index" + i) self.client.create( index="index" + i, doc_type='log', body={'message':'TEST DOCUMENT'}, ) curator.create_snapshot( self.client, name="snapshot" + i, indices="index" + i, repository=self.args['repository'] ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--dry-run', '--host', host, '--port', str(port), 'delete', 'snapshots', '--repository', self.args['repository'], '--all-snapshots', '--exclude', '2', ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True)[:4] # I tried doing a nested, double list comprehension here. # It works in the interpreter, but not here for some reason. output = [ x.split(' ')[-1:] for x in output ] output = [ x[0] for x in output if x[0].startswith('snapshot1') ] self.assertEqual(['snapshot1'], output) def test_snapshot_selection_delete_snapshot(self): self.create_repository() for i in ["1", "2"]: self.create_index("index" + i) self.client.create( index="index" + i, doc_type='log', body={'message':'TEST DOCUMENT'}, ) curator.create_snapshot( self.client, name="snapshot" + i, indices="index" + i, repository=self.args['repository'] ) result = curator.get_snapshot(self.client, self.args['repository'], '_all') self.assertEqual(2, len(result['snapshots'])) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'delete', 'snapshots', '--repository', self.args['repository'], '--exclude', '2', ], obj={"filters":[]}) result = curator.get_snapshot(self.client, self.args['repository'], '_all') self.assertEqual(1, len(result['snapshots'])) self.assertEqual('snapshot2', result['snapshots'][0]['snapshot']) def test_snapshot_selection_all_filtered_fail(self): self.create_repository() for i in ["1", "2", "3"]: self.create_index("index" + i) self.client.create( index="index" + i, doc_type='log', body={'message':'TEST DOCUMENT'}, ) curator.create_snapshot( self.client, name="snapshot" + i, indices="index" + i, repository=self.args['repository'] ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'snapshots', '--repository', self.args['repository'], '--prefix', 'no_match', ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) class TestCLILogging(CuratorTestCase): def test_logging_with_debug_flag(self): self.create_indices(10) indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True)[:4] test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--debug', '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True)[:4] self.assertEqual(expected, output) def test_logging_with_bad_loglevel_flag(self): self.create_indices(1) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--loglevel', 'FOO', '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) self.assertEqual('Invalid log level: FOO', str(result.exception)) def test_logging_with_logformat_logstash_flag(self): self.create_indices(10) indices = curator.get_indices(self.client) expected = sorted(indices, reverse=True)[:4] test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logformat', 'logstash', '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', '--time-unit', 'days' ], obj={"filters":[]}) output = sorted(result.output.splitlines(), reverse=True)[:4] self.assertEqual(expected, output) class TestCLIOptions(CuratorTestCase): def test_logstash_formatting(self): dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) logfile = tempfile.mkdtemp(suffix=dirname) + 'logfile' self.create_indices(1) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logformat', 'logstash', '--debug', '--host', host, '--port', str(port), 'show', 'indices', '--all-indices' ], obj={"filters":[]}) d = json.loads(result.output.splitlines()[:1][0]) keys = sorted(list(d.keys())) self.assertEqual(['@timestamp','function','linenum','loglevel','message','name'], keys) curator-3.4.1/test/integration/test_cli_utils.py000066400000000000000000000070771265673226500221310ustar00rootroot00000000000000import elasticsearch import curator import os import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestGetClient(CuratorTestCase): def test_get_client_positive(self): client_args = {"host":host, "port":port} client = curator.get_client(**client_args) self.assertTrue(isinstance(client, elasticsearch.client.Elasticsearch)) def test_get_client_negative_connection_fail(self): client_args = {"host":host, "port":54321} with self.assertRaises(SystemExit) as cm: curator.get_client(**client_args) self.assertEqual(cm.exception.code, 1) class TestCLIUtilsFilterCallback(CuratorTestCase): def test_filter_callback_without_timestring(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--older-than', '5', '--time-unit', 'days', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_filter_callback_without_timeunit(self): test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show', 'indices', '--newer-than', '5', '--timestring', '%Y.%m.%d', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) class TestSSLFlags(CuratorTestCase): def test_bad_certificate(self): self.create_indices(10) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), '--use_ssl', '--certificate', '/path/to/nowhere', 'show', 'indices', '--older-than', '5', '--time-unit', 'days', '--timestring', '%Y.%m.%d', ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) self.assertEqual('Error: Could not open certificate at /path/to/nowhere\n', result.output) def test_ssl_no_validate(self): self.create_indices(10) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), '--use_ssl', '--ssl-no-validate', 'show', 'indices', '--older-than', '5', '--time-unit', 'days', '--timestring', '%Y.%m.%d', ], obj={"filters":[]}) self.assertTrue(1, result.exit_code) curator-3.4.1/test/integration/test_es_repo_mgr.py000066400000000000000000000144631265673226500224400ustar00rootroot00000000000000import elasticsearch import curator import os import json import click import string, random, tempfile from click import testing as clicktest from mock import patch, Mock, MagicMock from . import CuratorTestCase import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestLoggingModules(CuratorTestCase): def test_logger_without_null_handler(self): mock = Mock() modules = {'logger': mock, 'logger.NullHandler': mock.module} with patch.dict('sys.modules', modules): self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show' ], obj={"filters":[]}) self.assertEqual(self.args['repository'], result.output.rstrip()) class TestCLIRepositoryCreate(CuratorTestCase): def test_create_fs_repository_success(self): test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'create', 'fs', '--repository', self.args['repository'], '--location', self.args['location'] ], obj={"filters":[]}) self.assertTrue(1, len(self.client.snapshot.get_repository(repository=self.args['repository']))) self.assertEqual(0, result.exit_code) def test_create_fs_repository_fail(self): test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'create', 'fs', '--repository', self.args['repository'], '--location', os.devnull ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) def test_create_s3_repository_fail(self): test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'create', 's3', '--bucket', 'mybucket', '--repository', self.args['repository'], ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) class TestCLIDeleteRepository(CuratorTestCase): def test_delete_repository_success(self): self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'delete', '--yes', # This ensures no prompting will happen '--repository', self.args['repository'] ], obj={"filters":[]}) self.assertFalse(curator.get_repository(self.client, self.args['repository'])) def test_delete_repository_notfound(self): test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'delete', '--yes', # This ensures no prompting will happen '--repository', self.args['repository'] ], obj={"filters":[]}) self.assertEqual(1, result.exit_code) class TestCLIShowRepositories(CuratorTestCase): def test_show_repository(self): self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', os.devnull, '--host', host, '--port', str(port), 'show' ], obj={"filters":[]}) self.assertEqual(self.args['repository'], result.output.rstrip()) class TestRepoMGR_CLIOptions(CuratorTestCase): def test_debug_logging(self): dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) logfile = tempfile.mkdtemp(suffix=dirname) + 'logfile' self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logfile', logfile, '--debug', '--host', host, '--port', str(port), 'show' ], obj={"filters":[]}) self.assertEqual(0, result.exit_code) def test_logstash_formatting(self): dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) logfile = tempfile.mkdtemp(suffix=dirname) + 'logfile' self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.repomgrcli, [ '--logformat', 'logstash', '--debug', '--host', host, '--port', str(port), 'show' ], obj={"filters":[]}) d = json.loads(result.output.splitlines()[:1][0]) keys = sorted(list(d.keys())) self.assertEqual(['@timestamp','function','linenum','loglevel','message','name'], keys) curator-3.4.1/test/run_tests.py000077500000000000000000000011461265673226500166000ustar00rootroot00000000000000#!/usr/bin/env python from __future__ import print_function import sys from os.path import dirname, abspath import nose def run_all(argv=None): sys.exitfunc = lambda: sys.stderr.write('Shutting down....\n') # always insert coverage when running tests through setup.py if argv is None: argv = [ 'nosetests', '--with-xunit', '--with-xcoverage', '--cover-package=curator', '--cover-erase', '--verbose', ] nose.run_exit( argv=argv, defaultTest=abspath(dirname(__file__)) ) if __name__ == '__main__': run_all(sys.argv) curator-3.4.1/test/unit/000077500000000000000000000000001265673226500151525ustar00rootroot00000000000000curator-3.4.1/test/unit/__init__.py000066400000000000000000000000001265673226500172510ustar00rootroot00000000000000curator-3.4.1/test/unit/test_api_commands.py000066400000000000000000001034131265673226500212170ustar00rootroot00000000000000from datetime import datetime, timedelta from unittest import TestCase from mock import Mock import sys try: from StringIO import StringIO except ImportError: from io import StringIO import elasticsearch from curator import api as curator named_index = 'index_name' named_indices = [ "index1", "index2" ] open_index = {'metadata': {'indices' : { named_index : {'state' : 'open'}}}} closed_index = {'metadata': {'indices' : { named_index : {'state' : 'close'}}}} cat_open_index = [{'status': 'open'}] cat_closed_index = [{'status': 'close'}] open_indices = { 'metadata': { 'indices' : { 'index1' : { 'state' : 'open' }, 'index2' : { 'state' : 'open' }}}} closed_indices = { 'metadata': { 'indices' : { 'index1' : { 'state' : 'close' }, 'index2' : { 'state' : 'close' }}}} fake_fail = Exception('Simulated Failure') four_oh_one = elasticsearch.TransportError(401, "simulated error") four_oh_four = elasticsearch.TransportError(404, "simulated error") named_alias = 'alias_name' allocation_in = {named_index: {'settings': {'index': {'routing': {'allocation': {'require': {'foo': 'bar'}}}}}}} allocation_out = {named_index: {'settings': {'index': {'routing': {'allocation': {'require': {'not': 'foo'}}}}}}} alias_retval = { "pre_aliased_index": { "aliases" : { named_alias : { }}}} aliases_retval = { "index1": { "aliases" : { named_alias : { } } }, "index2": { "aliases" : { named_alias : { } } }, } indices_space = { 'indices' : { 'index1' : { 'index' : { 'primary_size_in_bytes': 1083741824 }}, 'index2' : { 'index' : { 'primary_size_in_bytes': 1083741824 }}}} shards = { 'indices': { named_index: { 'shards': { '0': [ { 'num_search_segments' : 15 }, { 'num_search_segments' : 21 } ], '1': [ { 'num_search_segments' : 19 }, { 'num_search_segments' : 16 } ] }}}} optimize_tuple = (4, 71) snap_name = 'snap_name' repo_name = 'repo_name' snap_running = { 'snapshots': ['running'] } nosnap_running = { 'snapshots': [] } snapshot = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} partial = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'PARTIAL', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} snapshots = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }, { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:02.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': 'snapshot2', 'end_time': '2015-01-01T00:00:03.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} snap_body_all = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "_all" } snap_body = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "index1,index2" } verified_nodes = {'nodes': {'nodeid1': {'name': 'node1'}, 'nodeid2': {'name': 'node2'}}} synced_pass = { "_shards":{"total":1,"successful":1,"failed":0}, "index_name":{ "total":1,"successful":1,"failed":0, "failures":[], } } synced_fail = { "_shards":{"total":1,"successful":0,"failed":1}, "index_name":{ "total":1,"successful":0,"failed":1, "failures":[ {"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":True,"node":"nodeid1","relocating_node":None,"shard":0,"index":"index_name"}}, ] } } sync_conflict = elasticsearch.ConflictError(409, u'{"_shards":{"total":1,"successful":0,"failed":1},"index_name":{"total":1,"successful":0,"failed":1,"failures":[{"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":true,"node":"nodeid1","relocating_node":null,"shard":0,"index":"index_name"}}]}})', synced_fail) synced_fails = { "_shards":{"total":2,"successful":1,"failed":1}, "index1":{ "total":1,"successful":0,"failed":1, "failures":[ {"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":True,"node":"nodeid1","relocating_node":None,"shard":0,"index":"index_name"}}, ] }, "index2":{ "total":1,"successful":1,"failed":0, "failures":[] }, } class TestAlias(TestCase): def test_add_to_alias_bad_csv(self): client = Mock() c = "a,b,c,d" self.assertFalse(curator.add_to_alias(client, c)) def test_add_to_alias_no_alias_arg(self): client = Mock() self.assertFalse(curator.add_to_alias(client, named_index)) def test_add_to_alias_alias_not_found(self): client = Mock() client.indices.exists_alias.return_value = False client.cluster.state.return_value = open_index client.info.return_value = {'version': {'number': '1.4.4'} } self.assertTrue(curator.add_to_alias(client, named_index, alias=named_alias)) def test_add_to_alias_exception_test(self): client = Mock() client.indices.get_alias.return_value = alias_retval client.indices.exists_alias.return_value = True client.cluster.state.return_value = open_index client.indices.update_aliases.side_effect = fake_fail client.info.return_value = {'version': {'number': '1.4.4'} } self.assertFalse(curator.add_to_alias(client, named_index, alias=named_alias)) def test_remove_from_alias_bad_csv(self): client = Mock() self.assertFalse(curator.remove_from_alias(client, "a,b,c,d")) def test_remove_from_alias_no_alias_arg(self): client = Mock() self.assertFalse(curator.remove_from_alias(client, named_index)) def test_remove_from_alias_alias_not_found(self): client = Mock() client.indices.exists_alias.return_value = False self.assertFalse(curator.remove_from_alias(client, named_index, alias=named_alias)) def test_remove_from_alias_exception_raised(self): client = Mock() client.indices.exists_alias.return_value = True client.indices.get_alias.return_value = aliases_retval client.indices.update_aliases.side_effect = fake_fail self.assertRaises(Exception, curator.remove_from_alias(client, "index1", alias=named_alias)) def test_remove_from_alias_exception_return_false(self): client = Mock() client.indices.exists_alias.return_value = True client.indices.get_alias.return_value = aliases_retval client.indices.update_aliases.side_effect = fake_fail self.assertFalse(curator.remove_from_alias(client, "index1", alias=named_alias)) def test_remove_from_alias_index_not_found_in_alias(self): client = Mock() client.indices.exists_alias.return_value = True client.indices.get_alias.return_value = aliases_retval self.assertFalse(curator.remove_from_alias(client, "foo", alias=named_alias)) class TestAllocate(TestCase): def test_apply_allocation_rule_param_check(self): client = Mock() # Testing for the omission of the rule param self.assertFalse(curator.apply_allocation_rule(client, named_indices)) def test_allocation_rule_positive(self): client = Mock() client.indices.get_settings.return_value = allocation_out client.cluster.state.return_value = open_index client.indices.put_settings.return_value = None self.assertTrue(curator.apply_allocation_rule(client, named_index, rule="foo=bar")) def test_apply_allocation_rule_negative(self): client = Mock() client.cluster.state.return_value = open_index client.indices.get_settings.return_value = allocation_in client.indices.put_settings.return_value = None # It should return true now because after skipping one, it results in an empty list. #531 self.assertTrue(curator.apply_allocation_rule(client, named_index, rule="foo=bar")) def test_apply_allocation_rule_empty_list(self): client = Mock() # It should return true now, even with an empty list. #531 self.assertTrue(curator.apply_allocation_rule(client, [], rule="foo=bar")) def test_allocation_positive(self): client = Mock() client.cluster.state.return_value = open_index client.indices.get_settings.return_value = allocation_out client.indices.put_settings.return_value = None self.assertTrue(curator.allocation(client, named_index, rule="foo=bar")) def test_allocation_negative_exception(self): client = Mock() client.cluster.state.return_value = open_index client.indices.get_settings.return_value = allocation_out client.indices.put_settings.side_effect = fake_fail self.assertFalse(curator.allocation(client, named_index, rule="foo=bar")) def test_allocation_wrong_type_param(self): client = Mock() client.cluster.state.return_value = open_index client.indices.get_settings.return_value = allocation_out client.indices.put_settings.return_value = None self.assertFalse(curator.allocation(client, named_index, rule="foo=bar", allocation_type="wrong_type")) def test_allocation_good_type_param(self): client = Mock() client.cluster.state.return_value = open_index client.indices.get_settings.return_value = allocation_out client.indices.put_settings.return_value = None self.assertTrue(curator.allocation(client, named_index, rule="foo=bar", allocation_type="require")) self.assertTrue(curator.allocation(client, named_index, rule="foo=bar", allocation_type="include")) self.assertTrue(curator.allocation(client, named_index, rule="foo=bar", allocation_type="exclude")) class TestBloom(TestCase): def test_disable_bloom_no_more_bloom_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } self.assertTrue(curator.disable_bloom_filter(client, named_index)) def test_disable_bloom_no_more_bloom_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.cluster.state.return_value = open_index client.indices.put_settings.return_value = None self.assertTrue(curator.disable_bloom_filter(client, named_index)) def test_disable_bloom_exception_test(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.cluster.state.return_value = open_index client.indices.put_settings.side_effect = fake_fail self.assertRaises(Exception, curator.disable_bloom_filter(client, named_index)) def test_disable_bloom_with_delay_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.cluster.state.return_value = open_indices self.assertTrue(curator.disable_bloom_filter( client, named_indices, delay=1 )) def test_disable_bloom_with_delay_negative(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.cluster.state.return_value = open_indices client.indices.put_settings.side_effect = fake_fail self.assertFalse(curator.disable_bloom_filter( client, named_indices, delay=1 )) def test_bloom_full_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.cluster.state.return_value = open_index client.indices.put_settings.return_value = None self.assertTrue(curator.bloom(client, named_index)) def test_bloom_full_negative(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.cluster.state.return_value = open_index client.indices.put_settings.side_effect = fake_fail self.assertFalse(curator.bloom(client, named_index)) class TestClose(TestCase): def test_close_indices_positive_presyncflush(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.indices.flush.return_value = None client.indices.close.return_value = None self.assertTrue(curator.close_indices(client, named_index)) def test_close_indices_negative_presyncflush(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.indices.flush.return_value = None client.indices.close.side_effect = fake_fail client.indices.close.return_value = None self.assertFalse(curator.close_indices(client, named_index)) def test_full_close_positive_presyncflush(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.indices.flush.return_value = None client.indices.close.return_value = None self.assertTrue(curator.close(client, named_index)) def test_full_close_negative_presyncflush(self): client = Mock() client.info.return_value = {'version': {'number': '1.3.4'} } client.indices.flush.return_value = None client.indices.close.side_effect = fake_fail client.indices.close.return_value = None self.assertFalse(curator.close(client, named_index)) def test_close_indices_positive_cat(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_pass client.info.return_value = {'version': {'number': '1.6.0'} } client.indices.close.return_value = None self.assertTrue(curator.close_indices(client, named_index)) def test_close_indices_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.6.0'} } client.cluster.state.return_value = open_index client.indices.flush_synced.return_value = synced_pass client.indices.close.return_value = None self.assertTrue(curator.close_indices(client, named_index)) def test_close_indices_negative_cat(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_fail client.info.return_value = {'version': {'number': '1.6.0'} } client.indices.close.side_effect = fake_fail client.indices.close.return_value = None self.assertFalse(curator.close_indices(client, named_index)) def test_close_indices_negative(self): client = Mock() client.info.return_value = {'version': {'number': '1.6.0'} } client.indices.flush_synced.return_value = synced_fail client.cluster.state.return_value = open_index client.indices.close.side_effect = fake_fail client.indices.close.return_value = None self.assertFalse(curator.close_indices(client, named_index)) def test_full_close_positive_cat(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_pass client.info.return_value = {'version': {'number': '1.6.0'} } client.indices.close.return_value = None self.assertTrue(curator.close(client, named_index)) def test_full_close_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.6.0'} } client.cluster.state.return_value = open_index client.indices.flush_synced.return_value = synced_pass client.indices.close.return_value = None self.assertTrue(curator.close(client, named_index)) def test_full_close_negative_cat(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_fail client.info.return_value = {'version': {'number': '1.6.0'} } client.indices.close.side_effect = fake_fail client.indices.close.return_value = None self.assertFalse(curator.close(client, named_index)) def test_full_close_negative(self): client = Mock() client.info.return_value = {'version': {'number': '1.6.0'} } client.cluster.state.return_value = open_index client.indices.flush_synced.return_value = synced_fail client.indices.close.side_effect = fake_fail client.indices.close.return_value = None self.assertFalse(curator.close(client, named_index)) class TestDelete(TestCase): def test_delete_indices_positive(self): client = Mock() client.info.return_value = {'version': {'number': '2.0.0'} } client.indices.delete.return_value = None self.assertTrue(curator.delete_indices(client, named_indices)) def test_delete_indices_negative(self): client = Mock() client.info.return_value = {'version': {'number': '2.0.0'} } client.indices.delete.side_effect = fake_fail self.assertFalse(curator.delete_indices(client, named_indices)) # This test needs to be able to have get_settings return two different # values on subsequent calls. I don't know how to do that, if it can # be done. Integration testing can cover testing this method, though. # # def test_full_delete_positive(self): # client = Mock() # client.indices.delete.return_value = None # client.indices.get_settings.return_value = named_indices # self.assertTrue(curator.delete(client, named_indices)) def test_full_delete_negative(self): client = Mock() client.info.return_value = {'version': {'number': '1.7.2'} } client.indices.delete.return_value = None client.indices.get_settings.return_value = named_indices self.assertFalse(curator.delete(client, named_indices)) def test_full_delete_exception(self): client = Mock() client.info.return_value = {'version': {'number': '1.7.2'} } client.indices.delete.side_effect = fake_fail client.indices.get_settings.return_value = named_indices self.assertFalse(curator.delete(client, named_indices)) class TestOpen(TestCase): def test_opener_positive(self): client = Mock() client.indices.open.return_value = None self.assertTrue(curator.open_indices(client, named_indices)) def test_opener_negative(self): client = Mock() client.indices.open.side_effect = fake_fail self.assertFalse(curator.open_indices(client, named_indices)) def test_full_opener_positive(self): client = Mock() client.indices.open.return_value = None self.assertTrue(curator.opener(client, named_indices)) def test_full_opener_negative(self): client = Mock() client.indices.open.side_effect = fake_fail self.assertFalse(curator.opener(client, named_indices)) class TestOptimize(TestCase): def test_optimize_index_positive(self): client = Mock() client.indices.segments.return_value = shards client.cluster.state.return_value = open_index client.indices.optimize.return_value = None client.info.return_value = {'version': {'number': '1.4.4'} } self.assertTrue(curator.optimize_index(client, named_index, max_num_segments=2)) def test_optimize_index_negative(self): client = Mock() client.indices.segments.return_value = shards client.cluster.state.return_value = open_index client.indices.optimize.side_effect = fake_fail client.info.return_value = {'version': {'number': '1.4.4'} } self.assertFalse(curator.optimize_index(client, named_index, max_num_segments=2)) def test_optimize_positive(self): client = Mock() client.indices.segments.return_value = shards client.cluster.state.return_value = open_index client.indices.optimize.return_value = None client.info.return_value = {'version': {'number': '1.4.4'} } self.assertTrue(curator.optimize(client, named_index, max_num_segments=2)) def test_optimize_negative(self): client = Mock() client.indices.segments.return_value = shards client.cluster.state.return_value = open_index client.indices.optimize.side_effect = fake_fail client.info.return_value = {'version': {'number': '1.4.4'} } self.assertFalse(curator.optimize(client, named_index, max_num_segments=2)) class TestReplicas(TestCase): def test_change_replicas_param_check(self): client = Mock() # Testing for the omission of the replicas param self.assertFalse(curator.change_replicas(client, named_indices)) def test_change_replicas_positive(self): client = Mock() client.cluster.state.return_value = open_indices client.indices.put_settings.return_value = None client.info.return_value = {'version': {'number': '1.4.4'} } self.assertTrue(curator.change_replicas(client, named_indices, replicas=0)) def test_change_replicas_negative(self): client = Mock() client.cluster.state.return_value = open_indices client.indices.put_settings.side_effect = fake_fail client.info.return_value = {'version': {'number': '1.4.4'} } self.assertFalse(curator.change_replicas(client, named_indices, replicas=0)) def test_replicas_positive(self): client = Mock() client.cluster.state.return_value = open_indices client.indices.put_settings.return_value = None client.info.return_value = {'version': {'number': '1.4.4'} } self.assertTrue(curator.replicas(client, named_indices, replicas=0)) def test_replicas_negative(self): client = Mock() client.cluster.state.return_value = open_indices client.indices.put_settings.side_effect = fake_fail client.info.return_value = {'version': {'number': '1.4.4'} } self.assertFalse(curator.replicas(client, named_indices, replicas=0)) class TestSeal(TestCase): # The seal_indices method pretty much always returns True, requiring log # viewing to ascertain if one or more indices failed to seal. def test_seal_indices_good_version(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_pass client.info.return_value = {'version': {'number': '1.6.0'} } self.assertTrue(curator.seal_indices(client, named_index)) def test_seal_indices_bad_version(self): client = Mock() client.cluster.state.return_value = open_index client.indices.flush_synced.return_value = synced_pass client.info.return_value = {'version': {'number': '1.3.4'} } self.assertTrue(curator.seal_indices(client, named_index)) def test_seal_indices_conflicterror(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_fail client.indices.flush_synced.side_effect = sync_conflict client.info.return_value = {'version': {'number': '1.6.0'} } self.assertTrue(curator.seal_indices(client, named_index)) def test_seal_indices_onepass_onefail(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_fails client.info.return_value = {'version': {'number': '1.6.0'} } self.assertTrue(curator.seal_indices(client, named_index)) def test_seal_indices_attribute_exception(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.cat.indices.return_value = cat_open_index client.indices.flush_synced.return_value = synced_fail client.indices.flush_synced.side_effect = fake_fail client.info.return_value = {'version': {'number': '1.6.0'} } self.assertTrue(curator.seal_indices(client, named_index)) class TestShow(TestCase): def setUp(self): self.held, sys.stdout = sys.stdout, StringIO() def test_show_positive(self): client = Mock() client.cluster.state.return_value = open_index curator.show(client, named_index) self.assertEqual(sys.stdout.getvalue(),'index_name\n') def test_show_positive_list(self): client = Mock() client.cluster.state.return_value = open_indices curator.show(client, named_indices) self.assertEqual(sys.stdout.getvalue(),'index1\nindex2\n') class TestSnapshot(TestCase): def test_create_snapshot_missing_arg_repository(self): client = Mock() self.assertFalse(curator.create_snapshot(client, name=snap_name)) def test_create_snapshot_in_progress(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.snapshot.status.return_value = snap_running self.assertFalse(curator.create_snapshot(client, indices=[], repository=repo_name, name=snap_name)) def test_create_snapshot_in_progress_old_version(self): client = Mock() client.info.return_value = {'version': {'number': '1.0.3'} } client.snapshot.status.return_value = snap_running self.assertFalse(curator.create_snapshot(client, indices=[], repository=repo_name, name=snap_name)) def test_create_snapshot_empty_arg_indices(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.snapshot.status.return_value = nosnap_running self.assertFalse(curator.create_snapshot(client, indices=[], repository=repo_name, name=snap_name)) def test_create_snapshot_verify_nodes_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshots client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.status.return_value = nosnap_running self.assertTrue( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name='not_snap_name' ) ) def test_create_snapshot_verify_nodes_negative(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshots client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.verify_repository.side_effect = fake_fail client.snapshot.status.return_value = nosnap_running self.assertFalse( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name='not_snap_name' ) ) def test_create_snapshot_verify_nodes_four_oh_one(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshots client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.verify_repository.side_effect = four_oh_one client.snapshot.status.return_value = nosnap_running self.assertFalse( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name='not_snap_name' ) ) def test_create_snapshot_verify_nodes_four_oh_four(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshots client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.verify_repository.side_effect = four_oh_four client.snapshot.status.return_value = nosnap_running self.assertFalse( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name='not_snap_name' ) ) def test_create_snapshot_name_collision(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshots client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.status.return_value = nosnap_running self.assertFalse( self.assertFalse( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name=snap_name ) ) ) def test_create_snapshot_exception(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshots client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.create.side_effect = elasticsearch.TransportError client.snapshot.status.return_value = nosnap_running self.assertFalse( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name='not_snap_name' ) ) def test_create_snapshot_no_wait_for_completion(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = snapshot client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.status.return_value = nosnap_running self.assertTrue( curator.create_snapshot( client, indices=named_indices, repository=repo_name, wait_for_completion=False, name='not_snap_name' ) ) def test_create_snapshot_incomplete(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = open_indices client.snapshot.get.return_value = partial client.snapshot.verify_repository.return_value = verified_nodes client.snapshot.status.return_value = nosnap_running self.assertFalse( curator.create_snapshot( client, indices=named_indices, repository=repo_name, name='not_snap_name' ) ) class TestDeleteSnapshot(TestCase): def test_delete_snapshot_missing_arg_repository(self): client = Mock() self.assertFalse(curator.delete_snapshot(client, snapshot=snap_name)) def test_delete_snapshot_missing_arg_name(self): client = Mock() self.assertFalse(curator.delete_snapshot(client, repository=repo_name)) def test_delete_snapshot_snapshot_is_list(self): client = Mock() self.assertFalse(curator.delete_snapshot(client, repository=repo_name, snapshot=['snap1', 'snap2'])) def test_delete_snapshot_positive(self): client = Mock() client.snapshot.delete.return_value = None self.assertTrue(curator.delete_snapshot(client, repository=repo_name, snapshot=snap_name)) def test_delete_snapshot_negative(self): client = Mock() client.snapshot.delete.side_effect = elasticsearch.RequestError self.assertFalse(curator.delete_snapshot(client, repository=repo_name, snapshot=snap_name)) curator-3.4.1/test/unit/test_api_filter.py000066400000000000000000000354611265673226500207120ustar00rootroot00000000000000from datetime import datetime, timedelta from unittest import TestCase from mock import Mock from curator import api as curator named_index = 'index_name' named_indices = [ "index1", "index2" ] open_index = {'metadata': {'indices' : { named_index : {'state' : 'open'}}}} closed_index = {'metadata': {'indices' : { named_index : {'state' : 'close'}}}} open_indices = { 'metadata': { 'indices' : { 'index1' : { 'state' : 'open' }, 'index2' : { 'state' : 'open' }}}} closed_indices = { 'metadata': { 'indices' : { 'index1' : { 'state' : 'close' }, 'index2' : { 'state' : 'close' }}}} fake_fail = Exception('Simulated Failure') indices_stats = { 'indices' : { 'index1' : { 'total' : { 'store' : { 'size_in_bytes' : 1083741824 }}}, 'index2' : { 'total' : { 'store' : { 'size_in_bytes' : 1083741824 }}}}} re_test_indices = [ "logstash-2014.12.31", "logstash-2014.12.30", "logstash-2014.12.29", ".marvel-2015.12.31", ".marvel-2015.12.30", ".marvel-2015.12.29", "2014.11.30", "2014.11.29", "2014.12.28", "2014.10.31-buh", "2014.10.30-buh", "2014.10.29-buh", "2015-01-01", "2015-01-02", "2015-01-03", "20150101", "20150102", "20150103", "foo", "bar", "baz", "neeble", "logstash-foo", "logstash-bar", "foo-logstash", "bar-logstash", "foologstash", "barlogstash", "logstashfoo", "logstashbar", ] class FilterBySpace(TestCase): def test_filter_by_space_param_check(self): client = Mock() # Testing for the omission of the disk_space param self.assertFalse(curator.filter_by_space(client, named_indices)) def test_filter_by_space_all_indices_closed(self): client = Mock() ds = 100.0 client.cluster.state.return_value = closed_indices client.info.return_value = {'version': {'number': '1.4.4'} } self.assertEqual([], curator.filter_by_space(client, named_indices, disk_space=ds)) def test_filter_by_space_no_deletions_positive(self): client = Mock() ds = 10.0 client.cluster.state.return_value = open_indices # Build return value of over 1G in size for each index client.indices.stats.return_value = indices_stats client.info.return_value = {'version': {'number': '1.4.4'} } self.assertEqual([], curator.filter_by_space(client, named_indices, disk_space=ds)) def test_filter_by_space_one_deletion(self): client = Mock() ds = 2.0 client.cluster.state.return_value = open_indices # Build return value of over 1G in size for each index client.indices.stats.return_value = indices_stats client.info.return_value = {'version': {'number': '1.4.4'} } self.assertEqual(["index1"], curator.filter_by_space(client, named_indices, disk_space=ds)) def test_filter_by_space_one_deletion_no_reverse(self): client = Mock() ds = 2.0 client.cluster.state.return_value = open_indices # Build return value of over 1G in size for each index client.indices.stats.return_value = indices_stats client.info.return_value = {'version': {'number': '1.4.4'} } self.assertEqual(["index2"], curator.filter_by_space(client, named_indices, disk_space=ds, reverse=False)) class TestBuildFilter(TestCase): def test_build_filter_missing_param_kindOf(self): self.assertEqual({}, curator.build_filter()) def test_build_filter_missing_param_value(self): self.assertEqual({}, curator.build_filter(kindOf='regex')) def test_build_filter_missing_param_time_unit(self): self.assertEqual({}, curator.build_filter(kindOf='older_than', value=5)) def test_build_filter_missing_param_timestring(self): self.assertEqual( {}, curator.build_filter( kindOf='older_than', value=5, time_unit='days' ), ) def test_build_filter_newer_than_positive(self): self.assertEqual( { 'pattern': '(?P\\d{4}\\.\\d{2}\\.\\d{2})', 'value': 5, 'groupname': 'date', 'time_unit': 'days', 'timestring': '%Y.%d.%m', 'method': 'newer_than' }, curator.build_filter( kindOf='newer_than', value=5, time_unit='days', timestring='%Y.%d.%m', ), ) def test_build_filter_newer_than_zero(self): self.assertEqual( { 'pattern': '(?P\\d{4}\\.\\d{2}\\.\\d{2})', 'value': 0, 'groupname': 'date', 'time_unit': 'days', 'timestring': '%Y.%d.%m', 'method': 'newer_than' }, curator.build_filter( kindOf='newer_than', value=0, time_unit='days', timestring='%Y.%d.%m', ), ) def test_build_filter_regex_positive(self): self.assertEqual( {'pattern': '^logs'}, curator.build_filter(kindOf='regex', value='^logs'), ) def test_build_filter_exclude_positive(self): self.assertEqual( {'pattern': '^logs', 'exclude': True}, curator.build_filter(kindOf='exclude', value='^logs'), ) def test_build_filter_prefix_positive(self): self.assertEqual( {'pattern': '^logs.*$'}, curator.build_filter(kindOf='prefix', value='logs'), ) def test_build_filter_suffix_positive(self): self.assertEqual( {'pattern': '^.*logs$'}, curator.build_filter(kindOf='suffix', value='logs'), ) def test_build_filter_timestring_positive(self): self.assertEqual( {'pattern': '^.*\\d{4}\\.\\d{2}\\.\\d{2}.*$'}, curator.build_filter(kindOf='timestring', value='%Y.%m.%d'), ) class TestRegexIterate(TestCase): def test_apply_filter_missing_param_pattern(self): self.assertFalse(curator.apply_filter(re_test_indices)) # The apply_filter method filters a list of indices based on regex patterns def test_apply_filter_filter_none(self): pattern = r'.*' self.assertEqual( re_test_indices, curator.apply_filter(re_test_indices, pattern=pattern) ) def test_apply_filter_prefix(self): pattern = r'^foo.*$' expected = [ 'foo', 'foo-logstash', 'foologstash' ] self.assertEqual( expected, curator.apply_filter(re_test_indices, pattern=pattern) ) def test_apply_filter_suffix(self): pattern = r'^.*foo$' expected = [ 'foo', 'logstash-foo', 'logstashfoo' ] self.assertEqual( expected, curator.apply_filter(re_test_indices, pattern=pattern) ) def test_apply_filter_timestring(self): pattern = r'^.*\d{4}.\d{2}.\d{2}.*$' expected = [ 'logstash-2014.12.31', 'logstash-2014.12.30', 'logstash-2014.12.29', '.marvel-2015.12.31', '.marvel-2015.12.30', '.marvel-2015.12.29', '2014.11.30', '2014.11.29', '2014.12.28', '2014.10.31-buh', '2014.10.30-buh', '2014.10.29-buh', '2015-01-01', '2015-01-02', '2015-01-03', ] self.assertEqual( expected, curator.apply_filter(re_test_indices, pattern=pattern) ) def test_apply_filter_newer_than(self): pattern = r'(?P\d{4}.\d{2}.\d{2})' expected = [ 'logstash-2014.12.31', 'logstash-2014.12.30', 'logstash-2014.12.29', '.marvel-2015.12.31', '.marvel-2015.12.30', '.marvel-2015.12.29', '2014.12.28', ] t = datetime(2014, 12, 1, 2, 34, 56) self.assertEqual(expected, curator.apply_filter(re_test_indices, pattern=pattern, groupname='date', timestring='%Y.%m.%d', time_unit='days', method='newer_than', value=2, utc_now=t ) ) def test_apply_filter_newer_than_negative_value(self): pattern = r'(?P\d{4}.\d{2}.\d{2})' expected = [ '.marvel-2015.12.31', '.marvel-2015.12.30', '.marvel-2015.12.29' ] t = datetime(2014, 12, 1, 2, 34, 56) self.assertEqual(expected, curator.apply_filter( re_test_indices, pattern=pattern, groupname='date', timestring='%Y.%m.%d', time_unit='days', method='newer_than', value=-30, utc_now=t ) ) def test_apply_filter_older_than_date_only(self): pattern = r'(?P\d{4}-\d{2}-\d{2})' expected = [ '2015-01-01', '2015-01-02', '2015-01-03', ] t = datetime(2015, 2, 1, 2, 34, 56) self.assertEqual(expected, curator.apply_filter( re_test_indices, pattern=pattern, groupname='date', timestring='%Y-%m-%d', time_unit='days', method='older_than', value=2, utc_now=t ) ) def test_apply_filter_older_than_with_prefix_and_suffix(self): pattern = r'(?P\d{4}.\d{2}.\d{2})' expected = [ 'logstash-2014.12.31', 'logstash-2014.12.30', 'logstash-2014.12.29', '2014.11.30', '2014.11.29', '2014.12.28', '2014.10.31-buh', '2014.10.30-buh', '2014.10.29-buh' ] t = datetime(2015, 2, 1, 2, 34, 56) self.assertEqual(expected, curator.apply_filter( re_test_indices, pattern=pattern, groupname='date', timestring='%Y.%m.%d', time_unit='days', method='older_than', value=2, utc_now=t ) ) def test_apply_filter_exclude(self): pattern = r'201|logstash' expected = ['foo', 'bar', 'baz', 'neeble'] self.assertEqual( expected, curator.apply_filter(re_test_indices, pattern=pattern, exclude=True) ) class TestGetDateRegex(TestCase): def test_get_date_regex_arbitrary(self): self.assertEqual('\\a\\a\\a\\a', curator.get_date_regex('aaaa')) def test_get_date_regex_arbitrary_with_percent(self): self.assertEqual('\\a\\a\\a\\a', curator.get_date_regex('%a%a%a%a')) def test_get_date_regex_date_map(self): self.assertEqual('\\d{4}\\.\\d{2}\\.\\d{2}\\-\\d{2}\\-\\d{2}\\d{2}\\d{2}\\d{2}\\d{2}', curator.get_date_regex('%Y.%y.%m-%W-%U%d%H%M%S')) def test_get_date_regex_date_map2(self): self.assertEqual('\\d{4}\\-\\d{3}', curator.get_date_regex('%Y-%j')) class TestGetIndexTime(TestCase): def test_get_datetime_week_fix_W(self): utc_now = datetime(2015, 2, 1, 2, 34, 56) expected = datetime(2015, 1, 26, 0, 00, 00) weeknow = utc_now.strftime('%Y-%W') self.assertEqual(expected, curator.get_datetime(weeknow, '%Y-%W')) def test_get_datetime_week_fix_U(self): utc_now = datetime(2015, 2, 21, 2, 34, 56) expected = datetime(2015, 2, 16, 0, 00, 00) weeknow = utc_now.strftime('%Y-%U') self.assertEqual(expected, curator.get_datetime(weeknow, '%Y-%U')) def test_get_datetime_month_fix_positive(self): utc_now = datetime(2015, 2, 22, 2, 34, 56) expected = datetime(2015, 2, 1, 0, 00, 00) weeknow = utc_now.strftime('%Y-%m') self.assertEqual(expected, curator.get_datetime(weeknow, '%Y-%m')) def test_get_datetime_month_fix_negative(self): utc_now = datetime(2015, 2, 22, 2, 34, 56) expected = datetime(2015, 2, 22, 0, 00, 00) weeknow = utc_now.strftime('%Y-%m-%d') self.assertEqual(expected, curator.get_datetime(weeknow, '%Y-%m-%d')) class TestGetTargetMonth(TestCase): def test_get_target_month_same_year(self): before = datetime(2015, 2, 1, 2, 34, 56) after = datetime(2015, 1, 1, 0, 0, 0) self.assertEqual(after, curator.get_target_month(1, utc_now=before)) def test_get_target_negative_value_month_same_year(self): before = datetime(2015, 2, 1, 2, 34, 56) after = datetime(2015, 3, 1, 0, 0, 0) self.assertEqual(after, curator.get_target_month(-1, utc_now=before)) def test_get_target_month_previous_year(self): before = datetime(2015, 2, 1, 2, 34, 56) after = datetime(2014, 12, 1, 0, 0, 0) self.assertEqual(after, curator.get_target_month(2, utc_now=before)) def test_get_target_negative_value_month_next_year(self): before = datetime(2015, 2, 1, 2, 34, 56) after = datetime(2016, 1, 1, 0, 0, 0) self.assertEqual(after, curator.get_target_month(-11, utc_now=before)) def test_month_bump_exception(self): datestamp = datetime(2015, 2, 1, 2, 34, 56) self.assertRaises(ValueError, curator.month_bump,datestamp, sign='foo') class TestGetCutoff(TestCase): def test_get_cutoff_param_check(self): # Testing for the omission of the unit_count param self.assertFalse(curator.get_cutoff()) def test_get_cutoff_non_integer(self): self.assertFalse(curator.get_cutoff(unit_count='foo')) def test_get_cutoff_weeks(self): fakenow = datetime(2015, 2, 3, 4, 5, 6) cutoff = datetime(2015, 2, 2, 0, 0, 0) self.assertEqual(cutoff, curator.get_cutoff(1, time_unit='weeks', utc_now=fakenow)) def test_get_cutoff_months(self): fakenow = datetime(2015, 2, 3, 4, 5, 6) cutoff = datetime(2015, 1, 1, 0, 0, 0) self.assertEqual(cutoff, curator.get_cutoff(1, time_unit='months', utc_now=fakenow)) def test_get_cutoff_negtive_value_weeks(self): fakenow = datetime(2015, 2, 3, 4, 5, 6) cutoff = datetime(2015, 3, 9, 0, 0, 0) self.assertEqual(cutoff, curator.get_cutoff(-5, time_unit='weeks', utc_now=fakenow)) def test_get_cutoff_negtive_value_months(self): fakenow = datetime(2015, 2, 3, 4, 5, 6) cutoff = datetime(2015, 7, 1, 0, 0, 0) self.assertEqual(cutoff, curator.get_cutoff(-5, time_unit='months', utc_now=fakenow)) class TestTimestampCheck(TestCase): # Only the final else statement was not listed as "covered" by nose def test_timestamp_check_else(self): ts = '%Y.%m.%d' tu = 'days' v = 30 timestamp = '2015.01.01' utc_now = datetime(2015, 1, 5) self.assertFalse(curator.timestamp_check(timestamp, timestring=ts, time_unit=tu, value=v, utc_now=utc_now)) def test_timestamp_check_get_cutoff_with_none(self): ts = '%Y.%m.%d' tu = 'days' v = None timestamp = '2015.01.01' utc_now = datetime(2015, 1, 5) self.assertFalse(curator.timestamp_check(timestamp, timestring=ts, time_unit=tu, value=v, utc_now=utc_now)) def test_timestamp_check_get_cutoff_with_non_int(self): ts = '%Y.%m.%d' tu = 'days' v = 'foo' timestamp = '2015.01.01' utc_now = datetime(2015, 1, 5) self.assertFalse(curator.timestamp_check(timestamp, timestring=ts, time_unit=tu, value=v, utc_now=utc_now)) curator-3.4.1/test/unit/test_api_utils.py000066400000000000000000000342611265673226500205620ustar00rootroot00000000000000from datetime import datetime, timedelta from unittest import TestCase from mock import Mock import elasticsearch from curator import api as curator named_index = 'index_name' open_index = {'metadata': {'indices' : { named_index : {'state' : 'open'}}}} closed_index = {'metadata': {'indices' : { named_index : {'state' : 'close'}}}} named_indices = [ "index1", "index2" ] named_alias = 'alias_name' alias_retval = { "pre_aliased_index": { "aliases" : { named_alias : { }}}} aliases_retval = { "index1": { "aliases" : { named_alias : { } } }, "index2": { "aliases" : { named_alias : { } } }, } fake_fail = Exception('Simulated Failure') four_oh_one = elasticsearch.TransportError(401, "simulated error") repo_name = 'repo_name' test_repo = {repo_name: {'type': 'fs', 'settings': {'compress': 'true', 'location': '/tmp/repos/repo_name'}}} test_repos = {'TESTING': {'type': 'fs', 'settings': {'compress': 'true', 'location': '/tmp/repos/TESTING'}}, repo_name: {'type': 'fs', 'settings': {'compress': 'true', 'location': '/rmp/repos/repo_name'}}} snap_name = 'snap_name' snapshot = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} snapshots = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }, { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:02.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': 'snapshot2', 'end_time': '2015-01-01T00:00:03.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} snap_body_all = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "_all" } snap_body = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "index1,index2" } class TestGetAlias(TestCase): def test_get_alias_positive(self): client = Mock() client.indices.exists_alias.return_value = True client.indices.get_alias.return_value = aliases_retval retval = sorted(curator.get_alias(client, named_alias)) self.assertEqual(named_indices, retval) def test_get_alias_negative(self): client = Mock() client.indices.exists_alias.return_value = False self.assertFalse(curator.get_alias(client, named_alias)) class TestEnsureList(TestCase): def test_ensure_list_returns_lists(self): l = ["a", "b", "c", "d"] e = ["a", "b", "c", "d"] self.assertEqual(e, curator.ensure_list(l)) l = "abcd" e = ["abcd"] self.assertEqual(e, curator.ensure_list(l)) l = [["abcd","defg"], 1, 2, 3] e = [["abcd","defg"], 1, 2, 3] self.assertEqual(e, curator.ensure_list(l)) l = {"a":"b", "c":"d"} e = [{"a":"b", "c":"d"}] self.assertEqual(e, curator.ensure_list(l)) class TestTo_CSV(TestCase): def test_to_csv_will_return_csv(self): l = ["a", "b", "c", "d"] c = "a,b,c,d" self.assertEqual(c, curator.to_csv(l)) def test_to_csv_will_return_single(self): l = ["a"] c = "a" self.assertEqual(c, curator.to_csv(l)) def test_to_csv_will_return_None(self): l = [] self.assertIsNone(curator.to_csv(l)) class TestCheckCSV(TestCase): def test_check_csv_positive(self): c = "1,2,3" self.assertTrue(curator.check_csv(c)) def test_check_csv_negative(self): c = "12345" self.assertFalse(curator.check_csv(c)) def test_check_csv_list(self): l = ["1", "2", "3"] self.assertTrue(curator.check_csv(l)) def test_check_csv_unicode(self): u = u'test' self.assertFalse(curator.check_csv(u)) def test_check_csv_wrong_value(self): v = 123 with self.assertRaises(SystemExit) as cm: curator.check_csv(v) self.assertEqual(cm.exception.code, 1) class TestPruneKibana(TestCase): def test_prune_kibana_positive(self): l = [ "logstash-2015.02.25", "logstash-2015.02.24", ".kibana", ".marvel-kibana", "kibana-int", ".marvel-2015.02.25", ".marvel-2015.02.24", ] r = [ "logstash-2015.02.25", "logstash-2015.02.24", ".marvel-2015.02.25", ".marvel-2015.02.24", ] self.assertEqual(r, curator.prune_kibana(l)) def test_prune_kibana_negative(self): l = [ "logstash-2015.02.25", "logstash-2015.02.24", ".marvel-2015.02.25", ".marvel-2015.02.24", ] r = [ "logstash-2015.02.25", "logstash-2015.02.24", ".marvel-2015.02.25", ".marvel-2015.02.24", ] self.assertEqual(r, curator.prune_kibana(l)) def test_prune_kibana_empty(self): l = [ ".kibana", ".marvel-kibana", "kibana-int", ] r = [] self.assertEqual(r, curator.prune_kibana(l)) class TestIndexClosed(TestCase): def test_cat_indices_json(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.info.return_value = {'version': {'number': '1.7.2'} } client.cat.indices.return_value = [{'status': 'close'}] closed = curator.index_closed(client, named_index) self.assertEqual(closed, True) def test_cat_indices_text_plain(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.info.return_value = {'version': {'number': '1.5.0'} } client.cat.indices.return_value = u'[{"status":"close"}]' closed = curator.index_closed(client, named_index) self.assertEqual(closed, True) def test_closed(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.4'} } client.cluster.state.return_value = closed_index closed = curator.index_closed(client, named_index) self.assertEqual(closed, True) def test_open_cat(self): client = Mock() client.cluster.state.side_effect = four_oh_one client.info.return_value = {'version': {'number': '1.7.2'} } client.cat.indices.return_value = [{'status': 'open'}] closed = curator.index_closed(client, named_index) self.assertEqual(closed, False) def test_open(self): client = Mock() client.cluster.state.return_value = open_index closed = curator.index_closed(client, named_index) self.assertEqual(closed, False) class TestGetVersion(TestCase): def test_positive(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9'} } version = curator.get_version(client) self.assertEqual(version, (9,9,9)) def test_negative(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9'} } version = curator.get_version(client) self.assertNotEqual(version, (8,8,8)) def test_dev_version_4_dots(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9.dev'} } version = curator.get_version(client) self.assertEqual(version, (9,9,9)) def test_dev_version_with_dash(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9-dev'} } version = curator.get_version(client) self.assertEqual(version, (9,9,9)) class TestOptimized(TestCase): def test_optimized_index_bad_csv(self): client = Mock() self.assertRaises(ValueError, curator.optimized, client, "a,b,c,d", max_num_segments=2) def test_optimized_index_missing_arg(self): client = Mock() self.assertRaises(ValueError, curator.optimized, client, named_index) def test_optimized_index_closed(self): client = Mock() client.info.return_value = {'version': {'number': '1.4.0'} } client.cluster.state.return_value = closed_index self.assertTrue(curator.optimized(client, named_index, max_num_segments=2)) class TestIsMasterNode(TestCase): def test_positive(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "foo" : "bar"} } client.cluster.state.return_value = { "master_node" : "foo" } self.assertTrue(curator.is_master_node(client)) def test_negative(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "bad" : "mojo"} } client.cluster.state.return_value = { "master_node" : "foo" } self.assertFalse(curator.is_master_node(client)) class TestGetIndexTime(TestCase): def test_get_datetime(self): for text, datestring, dt in [ ('2014.01.19', '%Y.%m.%d', datetime(2014, 1, 19)), ('14.01.19', '%y.%m.%d', datetime(2014, 1, 19)), ('2014-01-19', '%Y-%m-%d', datetime(2014, 1, 19)), ('2010-12-29', '%Y-%m-%d', datetime(2010, 12, 29)), ('2012-12', '%Y-%m', datetime(2012, 12, 1)), ('2011.01', '%Y.%m', datetime(2011, 1, 1)), ('2014-28', '%Y-%W', datetime(2014, 7, 14)), ('2010.12.29.12', '%Y.%m.%d.%H', datetime(2010, 12, 29, 12)), ('2009101112136', '%Y%m%d%H%M%S', datetime(2009, 10, 11, 12, 13, 6)), ]: self.assertEqual(dt, curator.get_datetime(text, datestring)) class TestGetRepository(TestCase): def test_get_repository_missing_arg(self): client = Mock() client.snapshot.get_repository.return_value = {} self.assertEqual({}, curator.get_repository(client)) def test_get_repository_positive(self): client = Mock() client.snapshot.get_repository.return_value = test_repo self.assertEqual(test_repo, curator.get_repository(client, repository=repo_name)) def test_get_repository_transporterror_negative(self): client = Mock() client.snapshot.get_repository.side_effect = elasticsearch.TransportError self.assertFalse(curator.get_repository(client, repository=repo_name)) def test_get_repository_notfounderror_negative(self): client = Mock() client.snapshot.get_repository.side_effect = elasticsearch.NotFoundError self.assertFalse(curator.get_repository(client, repository=repo_name)) def test_get_repository__all_positive(self): client = Mock() client.snapshot.get_repository.return_value = test_repos self.assertEqual(test_repos, curator.get_repository(client)) class TestGetSnapshot(TestCase): def test_get_snapshot_missing_repository_arg(self): client = Mock() self.assertFalse(curator.get_snapshot(client, snapshot=snap_name)) def test_get_snapshot_positive(self): client = Mock() client.snapshot.get.return_value = snapshot self.assertEqual(snapshot, curator.get_snapshot(client, repository=repo_name, snapshot=snap_name)) def test_get_snapshot_transporterror_negative(self): client = Mock() client.snapshot.get_repository.return_value = test_repo client.snapshot.get.side_effect = elasticsearch.TransportError self.assertFalse(curator.get_snapshot(client, repository=repo_name, snapshot=snap_name)) def test_get_snapshot_notfounderror_negative(self): client = Mock() client.snapshot.get_repository.return_value = test_repo client.snapshot.get.side_effect = elasticsearch.NotFoundError self.assertFalse(curator.get_snapshot(client, repository=repo_name, snapshot=snap_name)) class TestGetSnapshots(TestCase): def test_get_snapshots_missing_repository_arg(self): client = Mock() self.assertFalse(curator.get_snapshots(client)) def test_get_snapshots_positive(self): client = Mock() client.snapshot.get.return_value = snapshot self.assertEqual(['snap_name'], curator.get_snapshots(client, repository=repo_name)) def test_get_snapshots_multiple_positive(self): client = Mock() client.snapshot.get.return_value = snapshots self.assertEqual(['snap_name', 'snapshot2'], curator.get_snapshots(client, repository=repo_name)) def test_get_snapshots_transporterror_negative(self): client = Mock() client.snapshot.get_repository.return_value = test_repo client.snapshot.get.side_effect = elasticsearch.TransportError self.assertFalse(curator.get_snapshots(client, repository=repo_name)) def test_get_snapshots_notfounderror_negative(self): client = Mock() client.snapshot.get_repository.return_value = test_repo client.snapshot.get.side_effect = elasticsearch.NotFoundError self.assertFalse(curator.get_snapshots(client, repository=repo_name)) class TestCreateSnapshotBody(TestCase): def test_create_snapshot_body_empty_arg(self): self.assertFalse(curator.create_snapshot_body([])) def test_create_snapshot_body__all_positive(self): self.assertEqual(snap_body_all, curator.create_snapshot_body('_all')) def test_create_snapshot_body_positive(self): self.assertEqual(snap_body, curator.create_snapshot_body(named_indices)) curator-3.4.1/test/unit/test_cli_utils.py000066400000000000000000000126751265673226500205650ustar00rootroot00000000000000from datetime import datetime, timedelta from unittest import TestCase from mock import Mock import sys import click from click import testing as clicktest import logging logger = logging.getLogger(__name__) import curator named_indices = [ "index1", "index2" ] named_alias = 'alias_name' alias_retval = { "pre_aliased_index": { "aliases" : { named_alias : { }}}} aliases_retval = { "index1": { "aliases" : { named_alias : { } } }, "index2": { "aliases" : { named_alias : { } } }, } fake_fail = Exception('Simulated Failure') repo_name = 'repo_name' test_repo = {repo_name: {'type': 'fs', 'settings': {'compress': 'true', 'location': '/tmp/repos/repo_name'}}} test_repos = {'TESTING': {'type': 'fs', 'settings': {'compress': 'true', 'location': '/tmp/repos/TESTING'}}, repo_name: {'type': 'fs', 'settings': {'compress': 'true', 'location': '/rmp/repos/repo_name'}}} snap_name = 'snap_name' snapshot = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} snapshots = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-01-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }, { 'duration_in_millis': 60000, 'start_time': '2015-01-01T00:00:02.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': 'snapshot2', 'end_time': '2015-01-01T00:00:03.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 0 }]} snap_body_all = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "_all" } snap_body = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "index1,index2" } class TestExitMsg(TestCase): def test_exit_msg_positive(self): with self.assertRaises(SystemExit) as cm: curator.exit_msg(True) self.assertEqual(cm.exception.code, 0) def test_exit_msg_negative(self): with self.assertRaises(SystemExit) as cm: curator.exit_msg(False) self.assertEqual(cm.exception.code, 1) class TestCheckVersion(TestCase): def test_check_version_positive(self): client = Mock() client.info.return_value = {'version': {'number': '1.1.1'} } self.assertIsNone(curator.check_version(client)) def test_check_version_less_than(self): client = Mock() client.info.return_value = {'version': {'number': '0.90.3'} } with self.assertRaises(SystemExit) as cm: curator.check_version(client) self.assertEqual(cm.exception.code, 1) def test_check_version_greater_than(self): client = Mock() client.info.return_value = {'version': {'number': '3.0.1'} } with self.assertRaises(SystemExit) as cm: curator.check_version(client) self.assertEqual(cm.exception.code, 1) class TestCheckMaster(TestCase): def test_check_master_positive(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "foo" : "bar"} } client.cluster.state.return_value = { "master_node" : "foo" } self.assertIsNone(curator.check_master(client, master_only=True)) def test_check_master_negative(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "bad" : "mojo"} } client.cluster.state.return_value = { "master_node" : "foo" } with self.assertRaises(SystemExit) as cm: curator.check_master(client, master_only=True) self.assertEqual(cm.exception.code, 0) class TestInList(TestCase): def test_in_list_positive(self): v = ['a', 'b'] s = ['a', 'b', 'c', 'd'] self.assertEqual(v, curator.in_list(v, s)) def test_in_list_negative(self): v = ['a', 'b', 'q'] s = ['a', 'b', 'c', 'd'] self.assertEqual(['a', 'b'], curator.in_list(v, s)) class TestGetClient(TestCase): def test_certificate_logic(self): client = Mock() kwargs = { 'use_ssl' : True, 'certificate' : 'mycert.pem' } with self.assertRaises(SystemExit) as cm: curator.get_client(**kwargs) self.assertEqual(sys.stdout.getvalue(),'ERROR: Connection failure.\n') self.assertEqual(cm.exception.code, 1) curator-3.4.1/test/unit/test_es_repo_mgr.py000066400000000000000000000064661265673226500211000ustar00rootroot00000000000000from datetime import datetime, timedelta from unittest import TestCase from mock import patch, Mock import click import sys try: from StringIO import StringIO except ImportError: from io import StringIO import elasticsearch from curator import api as curator fake_fail = Exception('Simulated Failure') repo_name = 'repo_name' class TestCreateRepoBody(TestCase): def test_create_repo_body_missing_repo_type(self): with self.assertRaises(SystemExit) as cm: curator.create_repo_body() self.assertEqual(cm.exception.code, 1) def test_create_repo_body_s3(self): body = curator.create_repo_body(repo_type='s3') self.assertEqual(body['type'], 's3') class TestCreateRepository(TestCase): def test_create_repository_missing_arg(self): client = Mock() with self.assertRaises(SystemExit) as cm: curator.create_repository(client) self.assertEqual(cm.exception.code, 1) def test_create_repository_empty_result_call(self): client = Mock() client.snapshot.get_repository.return_value = None self.assertTrue(curator.create_repository(client, repository="repo", repo_type="fs")) def test_create_repository_repo_not_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'not_your_repo':{'foo':'bar'}} self.assertTrue(curator.create_repository(client, repository="repo", repo_type="fs")) def test_create_repository_repo_already_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'repo':{'foo':'bar'}} with self.assertRaises(SystemExit) as cm: curator.create_repository(client, repository="repo", repo_type="fs") self.assertEqual(cm.exception.code, 1) def test_create_repository_exception(self): client = Mock() client.snapshot.get_repository.return_value = {'not_your_repo':{'foo':'bar'}} client.snapshot.create_repository.side_effect = elasticsearch.TransportError(500, "Error message", {"message":"Error"}) self.assertRaises(Exception, curator.create_repository(client, repository="repo", repo_type="fs")) class TestVerifyRepository(TestCase): def test_verify_repository_missing_arg(self): client = Mock() with self.assertRaises(SystemExit) as cm: curator.verify_repository(client) self.assertEqual(cm.exception.code, 1) def test_verify_repository_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'repo':{'foo':'bar'}} self.assertTrue(curator.verify_repository(client, repository="repo")) def test_verify_repository_repo_not_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'not_your_repo':{'foo':'bar'}} self.assertFalse(curator.verify_repository(client, repository="repo")) class TestDeleteCallback(TestCase): def test_delete_callback_no_value(self): ctx = Mock() param = None value = None self.assertEqual(curator.delete_callback(ctx, param, value), None) def test_delete_callback_with_value(self): ctx = Mock() param = None value = True ctx.abort.return_value = None self.assertEqual(curator.delete_callback(ctx, param, value), None)