pax_global_header00006660000000000000000000000064147115524150014517gustar00rootroot0000000000000052 comment=ddbe0c6efc0f47b5cd120344ae26fa102ad6156c powa-collector-1.3.0/000077500000000000000000000000001471155241500144525ustar00rootroot00000000000000powa-collector-1.3.0/.github/000077500000000000000000000000001471155241500160125ustar00rootroot00000000000000powa-collector-1.3.0/.github/workflows/000077500000000000000000000000001471155241500200475ustar00rootroot00000000000000powa-collector-1.3.0/.github/workflows/powa_collector.yml000066400000000000000000000017451471155241500236150ustar00rootroot00000000000000name: Trigger build and push of powa-collector image on: push: # https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet tags-ignore: - 'debian/*' env: TARGET_REPO: "powa-podman" EVENT_TYPE: "powa-collector" jobs: trigger_build: name: Trigger build and push of powa-collector in powa-podman repo runs-on: ubuntu-latest steps: - name: Trigger the powa-collector repository dispatch run: | # Set variables org="${{ github.repository_owner }}" repo="${{ env.TARGET_REPO }}" event_type="${{ env.EVENT_TYPE }}" curl -L \ -X POST \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer ${{ secrets.DISPATCH_TOKEN }}" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/repos/${org}/${repo}/dispatches \ -d "{\"event_type\": \"${event_type}\"}" powa-collector-1.3.0/.github/workflows/powa_collector_git.yml000066400000000000000000000015611471155241500244540ustar00rootroot00000000000000name: Trigger build and push of powa-collector-git image on: push: branches: [master] env: TARGET_REPO: "powa-podman" EVENT_TYPE: "powa-collector-git" jobs: trigger_build: name: Trigger build and push of powa-collector-git in powa-podman repo runs-on: ubuntu-latest steps: - name: Trigger the powa-collector-git repository dispatch run: | # Set variables org="${{ github.repository_owner }}" repo="${{ env.TARGET_REPO }}" event_type="${{ env.EVENT_TYPE }}" curl -L \ -X POST \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer ${{ secrets.DISPATCH_TOKEN }}" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/repos/${org}/${repo}/dispatches \ -d "{\"event_type\": \"${event_type}\"}" powa-collector-1.3.0/.gitignore000066400000000000000000000001221471155241500164350ustar00rootroot00000000000000.*.sw* *.pyc *.conf **/__pycache__/ .pypirc build/ dist/ powa_collector.egg-info/ powa-collector-1.3.0/CHANGELOG000066400000000000000000000062131471155241500156660ustar00rootroot000000000000001.3.0: New Features: - Support for upcoming PoWA v5 new extension model (Julien Rouhaud) - Support snapshot of per-database datasources (Julien Rouhaud) - Support snapshot of per-database catalogs (Julien Rouhaud) - Support on-demand snapshot of any remote server (Julien Rouhaud) - Support on-demand catalog snapshot of any remote server (Julien Rouhaud) - Add permission checking based on new PoWA predefine roles for on-demande request (Julien Rouhaud) - Support any extension installed in any schema (Julien Rouhaud) - Optionally emit logs using the server alias rather than the server hostname and port (Julien Rouhaud, thanks to Marc Cousin for the request) Bugfixes: - Redact the password (if provided) in the worker representation logs (Julien Rouhaud per report from Marc Cousin) - Fix the logic to spread the activity on server start (Julien Rouhaud) - Fix configuration file error handling (Julien Rouhaud) Misc: - Check for PoWA version incompatibility between remote servers and the repository server (Julien Rouhaud) 1.2.0: New Features: - Automatically detect hypopg on remote servers (Julien Rouhaud, thanks to github user MikhailSaynukov for the request) Bugfixes: - Fix sleep time calculation (Marc Cousin) - Properly detect -Infinity as an unknown last snapshot (Julien Rouhaud) - Properly handle error happening when retrieving the list of remote servers (Julien Rouhaud) - Properly detect stop condition after checking if PoWA must be loaded (Julien Rouhaud) - Close all thread's connections in case of uncatched error during snapshot (Marc Cousin) Misc: - Immediately exit the worker thread if PoWA isn't present or can't be loaded (Julien Rouhaud) - Improve server list stdout logging when no server is found (Julien Rouhaud) - Do PoWA extension sanity checks for the dedicated repo connection too (Julien Rouhaud) - Fix compatibility with python 3.9 (Julien Rouhaud, per report from Christoph Berg) 1.1.1: Bugfix: - Make sure that repo connection is available when getting repo powa version (Julien Rouhaud, thanks to Adrien Nayrat for the report and testing the patch) 1.1.0: New features: - Avoid explicit "LOAD 'powa'" with poWA 4.1.0, so a superuser isn't required anymore when PoWA isn't in shared_preload_libraries (Julien Rouhaud) - Store postgres and handled extensions versions on repository server (Julien Rouhaud) Bug fixes: - Handle errors that might happen during snapshot (Julien Rouhaud) 1.0.0: New features: - Handle the new query_cleanup query that may be run after getting remote data. Bugfix: - Let workers quit immediately if they're asked to stop. 0.0.3 Bugfix: - Support standard_conforming_strings = off - Better error message for remote servers lacking powa extension (Thomas Reiss and Julien Rouhaud) 0.0.2 Bugfix: - Ignore deactivated servers Miscellaneous: - Set lock_timeout to 2s for every pg connection - Fully qualify all objects in SQL queries 0.0.1 Initial release powa-collector-1.3.0/LICENSE000066400000000000000000000016521471155241500154630ustar00rootroot00000000000000Copyright (c) 2018-2024, The PoWA-team Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies. IN NO EVENT SHALL The PoWA-team BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF The PoWA-team HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The PoWA-team SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND The PoWA-team HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. powa-collector-1.3.0/README.md000066400000000000000000000047151471155241500157400ustar00rootroot00000000000000Overview ======== This repository contains the `powa-collector` tool, a simple multi-threaded python program that performs the snapshots for all the remote servers configured in a powa repository database (in the **powa_servers** table). Requirements ============ This program requires python 2.7 or python 3. The required dependencies are listed in the **requirements.txt** file. Configuration ============= Copy the provided `powa-collector.conf-dist` file to a new `powa-collector.conf` file, and adapt the **dsn** specification to be able to connect to the wanted main PoWA repository. Usage ===== To start the program, simply run the powa-collector.py program. A `SIGTERM` or a `Keyboard Interrupt` on the program will cleanly stop all the thread and exit the program. A `SIGHUP` will reload the configuration. Protocol ======== A minimal communication protocol is implented, using the LISTEN/NOTIFY facility provided by postgres, which is used by the powa-web project. You can send queries to collector by sending messages on the "powa_collector" channel. The collector will send answers on the channel you specified, so make sure to listen on it before sending any query to not miss answers. The requests are of the following form: COMMAND RESPONSE_CHANNEL OPTIONAL_ARGUMENTS - COMMAND: mandatory argument describing the query. The following commands are supported: - RELOAD: reload the configuration and report that the main thread successfully received the command. The reload will be attempted even if no response channel was provided. - WORKERS_STATUS: return a JSON (srvid is the key, status is the content) describing the status of each remote server thread. Command is ignored if no response channel was provided. This command accept an optional argument to get the status of a single remote server, identified by its srvid. If no worker exists for this server, an empty JSON will be returned. - RESPONSE_CHANNEL: mandatory argument to describe the NOTIFY channel the client listens a response on. '-' can be used if no answer should be sent. - OPTIONAL_ARGUMENTS: space separated list of arguments, specific to the underlying command. The answers are of the form: COMMAND STATUS DATA - COMMAND: same as the command in the query - STATUS: OK or KO. - DATA: reason for the failure if status is KO, otherwise the data for the answer. powa-collector-1.3.0/powa-collector.conf-dist000066400000000000000000000001561471155241500212160ustar00rootroot00000000000000{ "repository": { "dsn": "postgresql://powa_user@localhost:5432/powa" }, "debug": false } powa-collector-1.3.0/powa-collector.py000077500000000000000000000020651471155241500177640ustar00rootroot00000000000000#!/usr/bin/env python import getopt import os import sys from powa_collector import PowaCollector, getVersion def usage(rc): """Show tool usage """ print("""Usage: %s [ -? | --help ] [ -V | --version ] -? | --help Show this message and exits -v | --version Report powa-collector version and exits See https://powa.readthedocs.io/en/latest/powa-collector/ for more information about this tool. """ % os.path.basename(__file__)) sys.exit(rc) def main(): """Instantiates and starts a PowaCollector object """ try: opts, args = getopt.getopt(sys.argv[1:], "?V", ["help", "version"]) except getopt.GetoptError as e: print(str(e)) usage(1) for o, a in opts: if o in ("-?", "--help"): usage(0) elif o in ("-V", "--version"): print("%s version %s" % (os.path.basename(__file__), getVersion())) sys.exit(0) else: assert False, "unhandled option" app = PowaCollector() app.main() if (__name__ == "__main__"): main() powa-collector-1.3.0/powa_collector/000077500000000000000000000000001471155241500174665ustar00rootroot00000000000000powa-collector-1.3.0/powa_collector/__init__.py000066400000000000000000000400431471155241500216000ustar00rootroot00000000000000""" PowaCollector: powa-collector main application. It takes a simple configuration file in json format, where repository.dsn should point to an URI to connect to the repository server. The list of remote servers and their configuration will be retrieved from this repository server. It maintains a persistent dedicated connection to the repository server, for monitoring and communication purpose. It also starts one thread per remote server. These threads are kept in the "workers" dict attribute, with the key being the textual identifier (host:port). See powa_worker.py for more details about those threads. The main thread will intercept the following signals: - SIGHUP: reload configuration and log and changes done - SIGTERM: cleanly terminate all threads and exit A minimal communication protocol is implemented, using the LISTEN/NOTIFY facility provided by postgres. The dedicated main thread repository connection listens on the "powa_collector" channel. A client, such as powa-web, can send requests on this channel and the main thread will act and respond accordingly. The requests are of the following form: COMMAND RESPONSE_CHANNEL OPTIONAL_ARGUMENTS See the README.md file for the full protocol documentation. """ from powa_collector.options import (parse_options, get_full_config, add_servers_config) from powa_collector.powa_worker import PowaThread from powa_collector.customconn import get_connection from powa_collector.notify import (notify_parse_force_snapshot, notify_parse_refresh_db_cat, notify_allowed) from powa_collector.utils import conf_are_equal import psycopg2 import select import logging import json import signal import time __VERSION__ = '1.3.0' __VERSION_NUM__ = [int(part) for part in __VERSION__.split('.')] def getVersion(): """Return powa_collector's version as a string""" return __VERSION__ class PowaCollector(): """Main powa collector's class. This manages all collection tasks Declare all attributes here, we don't want dynamic attributes """ def __init__(self): """Instance creator. Sets logging, signal handlers, and basic structure""" self.workers = {} self.logger = logging.getLogger("powa-collector") self.stopping = False raw_options = parse_options() loglevel = logging.INFO if (raw_options["debug"]): loglevel = logging.DEBUG extra = {'threadname': '-'} logging.basicConfig( format='%(asctime)s %(threadname)s %(levelname)-6s: %(message)s ', level=loglevel) self.logger = logging.LoggerAdapter(self.logger, extra) signal.signal(signal.SIGHUP, self.sighandler) signal.signal(signal.SIGTERM, self.sighandler) def __get_worker_by_srvid(self, srvid): """Get the worker thread for the given srvid, if any""" for k, worker in self.workers.items(): worker_srvid = self.config["servers"][k]["srvid"] if (srvid != worker_srvid): continue return worker return None def connect(self, options): """Connect to the repository Used for communication with powa-web and users of the communication repository Persistent Threads will use distinct connections """ try: self.logger.debug("Connecting on repository...") self.__repo_conn = get_connection(self.logger, options["debug"], options["repository"]['dsn']) self.__repo_conn.autocommit = True self.logger.debug("Connected.") cur = self.__repo_conn.cursor() # Setup a 2s lock_timeout if there's no inherited lock_timeout cur.execute("""SELECT pg_catalog.set_config(name, '2000', false) FROM pg_catalog.pg_settings WHERE name = 'lock_timeout' AND setting = '0'""") cur.execute("SET application_name = %s", ('PoWA collector - main thread' + ' (' + __VERSION__ + ')', )) # Listen on our dedicated powa_collector notify channel cur.execute("LISTEN powa_collector") # Check if powa-archivist is installed on the repository server cur.execute("""SELECT regexp_split_to_array(extversion, '\\.'), extversion FROM pg_catalog.pg_extension WHERE extname = 'powa'""") ver = cur.fetchone() cur.close() if ver is None: self.__repo_conn.close() self.__repo_conn = None self.logger.error("PoWA extension not found on repository " "server") return False elif (int(ver[0][0]) < 4): self.__repo_conn.close() self.__repo_conn = None self.logger.error("Incompatible PoWA version, found %s," " requires at least 4.0.0" % ver[1]) return False except psycopg2.Error as e: self.__repo_conn = None self.logger.error("Error connecting:\n%s", e) return False return True def process_notifications(self): """Process PostgreSQL NOTIFY messages. These come mainly from the UI, to ask us to reload our configuration, or to display the workers status. """ if (not self.__repo_conn): return self.__repo_conn.poll() cur = self.__repo_conn.cursor() while (self.__repo_conn.notifies): notify = self.__repo_conn.notifies.pop(0) pid = notify.pid payload = notify.payload.split(' ') status = '' cmd = payload.pop(0) channel = "-" status = "OK" data = None # the channel is mandatory, but if the message can be processed # without answering, we'll try to if (len(payload) > 0): channel = payload.pop(0) self.logger.debug("Received async command: %s %s %r" % (cmd, channel, payload)) if (not notify_allowed(pid, self.__repo_conn)): status = 'ERROR' data = 'Permission denied' else: try: (status, data) = self.__process_one_notification(cmd, channel, payload) except Exception as e: status = 'ERROR' data = str(e) # if there was a response channel, reply back if (channel != '-'): # We need an extra {} around the data as the custom connection # will call format() on the overall string, which would fail # with a json dump as-is. We therefore also need to make sure # that any non-empty data looks like a json. payload = ("%(cmd)s %(status)s %(data)s" % {'cmd': cmd, 'status': status, 'data': data}) # with default configuration, postgres only accept up to 8k # bytes payload. If the payload is longer, just warn the # caller that it didn't fit. # XXX we could implement multi-part answer, but if we ever # reach that point, we should consider moving to a table if (len(payload.encode('utf-8')) >= 8000): payload = ("%(cmd)s %(status)s %(data)s" % {'cmd': cmd, 'status': "KO", 'data': "{ANSWER TOO LONG}"}) cur.execute('NOTIFY "' + channel + '", %(payload)s', {'payload': payload}) cur.close() def __process_one_notification(self, cmd, channel, notif): """ Process a single notification, called by process_notifications. """ status = "OK" data = '' if (cmd == "RELOAD"): self.reload_conf() data = 'OK' elif (cmd == "WORKERS_STATUS"): # ignore the message if no channel was received if (channel != '-'): # did the caller request a single server only? We ignore # anything but the first parameter passed if (len(notif) > 0 and notif[0].isdigit()): w_id = int(notif[0]) data = json.dumps(self.list_workers(w_id, False)) else: data = json.dumps(self.list_workers(None, False)) elif (cmd == "FORCE_SNAPSHOT"): r_srvid = notify_parse_force_snapshot(notif) worker = self.__get_worker_by_srvid(r_srvid) if (worker is None): raise Exception("Server id %d not found" % r_srvid) request_time = time.time() (status, data) = worker.request_forced_snapshot() # If a snapshot could be scheduled, wait just a bit and see if we # can report that the snapshot has already begun or will be started # shortly. It's possible, although unlikely, that the snapshot was # started and finished during that time. We don't try to properly # detect that case, as it should only happen in toy setups. We # might want to change that later if we want to provide a way to # inform callers of the completion of the snapshot they requested. if (status == 'OK'): time.sleep(0.1) if worker.is_snapshot_in_progress(): data = 'Snapshot in progress' else: data = 'Snapshot wil begin shortly' elif (cmd == "REFRESH_DB_CAT"): (r_srvid, r_dbnames) = notify_parse_refresh_db_cat(notif) worker = self.__get_worker_by_srvid(r_srvid) if (worker is None): raise Exception("Server id %d not found" % r_srvid) (status, data) = worker.register_cat_refresh(r_dbnames) # everything else is unhandled else: status = 'UNKNOWN' data = 'Message "%s" invalid, command ignored' % cmd return (status, data) def main(self): """Start the active loop. Connect or reconnect to the repository and starts threads to manage the monitored servers """ raw_options = parse_options() self.logger.info("Starting powa-collector...") if (not self.connect(raw_options)): exit(1) try: self.config = add_servers_config(self.__repo_conn, raw_options) except psycopg2.Error as e: self.__repo_conn.close() self.__repo_conn = None self.logger.error("Error retrieving the list of remote servers:" "\n%s", e) exit(1) for k, conf in self.config["servers"].items(): self.register_worker(k, self.config["repository"], conf, raw_options['debug']) self.list_workers() try: while (not self.stopping): if (self.__repo_conn is not None): try: cur = self.__repo_conn.cursor() cur.execute("SELECT 1") cur.close() except Exception: self.__repo_conn = None self.logger.warning("Connection was dropped!") if (not self.__repo_conn): self.connect(raw_options) select.select([self.__repo_conn], [], [], 10) self.process_notifications() except KeyboardInterrupt: self.logger.debug("KeyboardInterrupt caught") self.logger.info("Stopping all workers and exiting...") self.stop_all_workers() def register_worker(self, name, repository, config, debug): """Add a worker thread to a server""" self.workers[name] = PowaThread(name, repository, config, debug) self.workers[name].start() def stop_all_workers(self): """Ask all worker threads to stop This is asynchronous, no guarantee """ for k, worker in self.workers.items(): worker.ask_to_stop() def sighandler(self, signum, frame): """Manage signal handlers: reload conf on SIGHUB, shutdown on SIGTERM""" if (signum == signal.SIGHUP): self.logger.debug("SIGHUP caught") self.reload_conf() elif (signum == signal.SIGTERM): self.logger.debug("SIGTERM caught") self.stop_all_workers() self.stopping = True else: self.logger.error("Unhandled signal %d" % signum) def list_workers(self, wanted_srvid=None, tostdout=True): """List all workers and their status""" res = {} if (tostdout): self.logger.info('List of workers:') if (tostdout and len(self.workers.items()) == 0): self.logger.info('No worker') for k, worker in self.workers.items(): # self.logger.info(" %s%s" % (k, "" if (worker.is_alive()) else # " (stopped)")) worker_srvid = self.config["servers"][k]["srvid"] # ignore this entry if caller want information for only one server if (wanted_srvid is not None and wanted_srvid != worker_srvid): continue status = "Unknown" if (worker.is_stopping()): status = "stopping" elif (worker.is_alive()): status = worker.get_status() else: status = "stopped" if (tostdout): self.logger.info("%r (%s)" % (worker, status)) else: res[worker_srvid] = status return res def reload_conf(self): """Reload configuration: - reparse the configuration - stop and start workers if necessary - for those whose configuration has changed, ask them to reload - update dep versions: recompute powa version's and its dependencies """ self.list_workers() self.logger.info('Reloading...') config_new = get_full_config(self.__repo_conn) # check for removed servers for k, worker in self.workers.items(): if (worker.is_alive()): continue if (worker.is_stopping()): self.logger.warning("The worker %s is stoping" % k) if (k not in config_new["servers"]): self.logger.info("%s has been removed, stopping it..." % k) worker.ask_to_stop() # check for added servers for k in config_new["servers"]: if (k not in self.workers or not self.workers[k].is_alive()): self.logger.info("%s has been added, registering it..." % k) self.register_worker(k, config_new["repository"], config_new["servers"][k], config_new['debug']) # check for updated configuration for k in config_new["servers"]: cur = config_new["servers"][k] if (not conf_are_equal(cur, self.workers[k].get_config())): self.workers[k].ask_reload(cur) # also try to reconnect if the worker experienced connection issue elif(self.workers[k].get_status() != "running"): self.workers[k].ask_reload(cur) # update stored versions for k in config_new["servers"]: self.workers[k].ask_update_dep_versions() self.config = config_new self.logger.info('Reload done') powa-collector-1.3.0/powa_collector/customconn.py000066400000000000000000000117351471155241500222370ustar00rootroot00000000000000import psycopg2 from psycopg2.extensions import connection as _connection, cursor as _cursor from psycopg2.extras import DictCursor import time from powa_collector.utils import get_powa_version class CustomConnection(_connection): """ Custom psycopg2 connection class that takes care of expanding extension schema and optionally logs various information at debug level, both on successful execution and in case of error. Supports either plain cursor (through CustomCursor) or DictCursor (through CustomDictCursor). Before execution, and if a _nsps object is found cached in the connection, the query will be formatted using its _nsps dict, which contains a list of extension_name -> escaped_schema_name mapping. All you need to do is pass query strings of the form SELECT ... FROM {extension_name}.some_relation ... """ def initialize(self, logger, debug): self._logger = logger self._debug = debug def cursor(self, *args, **kwargs): factory = kwargs.get('cursor_factory') if factory is None: kwargs['cursor_factory'] = CustomCursor elif factory == DictCursor: kwargs['cursor_factory'] = CustomDictCursor else: msg = "Unsupported cursor_factory: %s" % factory.__name__ self._logger.error(msg) raise Exception(msg) return _connection.cursor(self, *args, **kwargs) class CustomDictCursor(DictCursor): def execute(self, query, params=None): query = resolve_nsps(query, self.connection) self.timestamp = time.time() try: return super(CustomDictCursor, self).execute(query, params) except Exception as e: log_query(self, query, params, e) raise e finally: log_query(self, query, params) class CustomCursor(_cursor): def execute(self, query, params=None): query = resolve_nsps(query, self.connection) self.timestamp = time.time() try: return super(CustomCursor, self).execute(query, params) except Exception as e: log_query(self, query, params, e) raise e finally: log_query(self, query, params) def resolve_nsps(query, connection): if hasattr(connection, '_nsps'): return query.format(**connection._nsps) return query def log_query(cls, query, params=None, exception=None): t = round((time.time() - cls.timestamp) * 1000, 2) fmt = '' if exception is not None: fmt = "Error during query execution:\n{}\n".format(exception) fmt += "query on {dsn}: {ms} ms\n{query}" if params is not None: fmt += "\n{params}" cls.connection._logger.debug(fmt.format(ms=t, dsn=cls.connection.dsn, query=query, params=params)) def get_connection(logger, debug, *args, **kwargs): """ Create a new connection using the CustomConnection factory. For handling connections on a specific database rather than the dedicated powa database (for the per-database snapshot feature) caller can specify an "override_dbname" named argument that will replace the "dbname" named argument. Note that in that case the extension schema expansion feature is not available. """ new_dbname = kwargs.pop("override_dbname", None) if (new_dbname is not None): # If caller provided an override_dbname it better also provided a # dbname argument (which should be the case for remote server standard # connection kwargs, but not the repository server). if ('dbname' not in kwargs): msg = "Tried to specify an override_dbname without providing " \ "a dbname. kwargs:\n%r" % kwargs self._logger.error(msg) raise Exception(msg) kwargs['dbname'] = new_dbname kwargs['connection_factory'] = CustomConnection conn = psycopg2.connect(*args, **kwargs) conn.initialize(logger, debug) cur = conn.cursor() # The automatic extension schema qualification is only available for # standard global and module connections, not per-database module # connections. if (new_dbname is None): # retrieve and cache the quoted schema for all installed extensions logger.debug("Retrieving extension schemas...") cur.execute("""SELECT extname, quote_ident(nspname) AS nsp FROM pg_catalog.pg_extension e JOIN pg_catalog.pg_namespace n ON n.oid = e.extnamespace""") ext_nsps = {row[0]: row[1] for row in cur.fetchall()} logger.debug("extension schemas: %r" % ext_nsps) conn._nsps = ext_nsps # If using powa 5+ (or connecting to a non-powa database) make sure # everything has to be fully qualified. powa_ver = get_powa_version(conn) if (powa_ver is None or int(powa_ver[0][0]) >= 5): logger.debug("Setting secure search_path") cur.execute("SET search_path TO 'pg_catalog'"); cur.close() conn.commit() return conn powa-collector-1.3.0/powa_collector/notify.py000066400000000000000000000066131471155241500213560ustar00rootroot00000000000000""" Helper functions for the NOTIFY-based communication processing. """ import psycopg2 from powa_collector.utils import get_powa_version def notify_parse_force_snapshot(notif): """Parse the payload of a received FORCE_SNAPSHOT notification""" if (len(notif) != 1): raise Exception('Command "%r" malformed' % notif) try: r_srvid = int(notif[0]) notif.pop(0) except: raise Exception(('invalid remote server id %s' % notif[0])) return r_srvid def notify_parse_refresh_db_cat(notif): """Parse the payload of a received REFRESH_DB_CAT notification""" if (len(notif) < 2): raise Exception('Command "%r" malformed' % notif) try: r_srvid = int(notif[0]) notif.pop(0) except: raise Exception(('invalid remote server id %s' % notif[0])) try: r_nb_db = int(notif[0]) notif.pop(0) except: raise Exception(('invalid number of databases %s' % notif[0])) r_dbnames = notif if (r_nb_db != len(r_dbnames)): raise Exception('Caller asked for %d db, but found %d: %r' % (r_nb_db, len(r_dbnames), r_dbnames)) return (r_srvid, r_dbnames) def notify_allowed(pid, conn): """ Check that the role used in the given backend, identified by its pid, is allowed to send notifications. """ powa_roles = ['powa_signal_backend', 'powa_write_all_data', 'powa_admin'] roles_to_test = [] cur = conn.cursor() powa_ver = get_powa_version(conn) # Should not happen if not powa_ver: return False # pg 14+ introduced predefined roles if conn.server_version >= 140000: roles_to_test.append('pg_signal_backend') try: # powa 5+ introduced various powa_* predefined roles so try them, but # also possibly pg14+ predefined roles if (int(powa_ver[0][0]) >= 5): cur.execute("""WITH s(v) AS ( SELECT unnest(%s) UNION ALL SELECT rolname FROM {powa}.powa_roles WHERE powa_role = ANY (%s) ) SELECT bool_or(pg_has_role(usesysid, v, 'USAGE')) FROM pg_stat_activity a CROSS JOIN s WHERE pid = %s""", (roles_to_test, powa_roles, pid)) # if powa 4- but we have a list of predefined roles, check those elif (len(roles_to_test) > 0): cur.execute("""WITH s(v) AS (SELECT unnest(%s)) SELECT bool_or(pg_has_role(usesysid, v, 'USAGE')) FROM pg_stat_activity a CROSS JOIN s WHERE pid = %s""", (roles_to_test, pid)) # if not (both old powa and postgres version), fallback to testing # whether the given role has enough privileges to INSERT data in # powa_statements to decide whether they should be allowed to use the # nofication system. else: cur.execute("""SELECT has_table_privilege(usesysid, '{powa}.powa_statements', 'INSERT') FROM pg_stat_activity WHERE pid = %s""", (pid, )) except psycopg2.Error as e: conn.rollback() return False row = cur.fetchone() # Connection already closed, deny if (not row): return False return (row[0] is True) powa-collector-1.3.0/powa_collector/options.py000066400000000000000000000064521471155241500215420ustar00rootroot00000000000000""" Simple configuration file handling, as a JSON. """ import json import os import sys SAMPLE_CONFIG_FILE = """ { "repository": { "dsn": "postgresql://powa_user@localhost:5432/powa", }, "use_server_alias": false "debug": false } """ CONF_LOCATIONS = [ '/etc/powa-collector.conf', os.path.expanduser('~/.config/powa-collector.conf'), os.path.expanduser('~/.powa-collector.conf'), './powa-collector.conf' ] def get_full_config(conn): """Return the full configuration, consisting of the information from the local configuration file and the remote servers stored on the repository database. """ return add_servers_config(conn, parse_options()) def add_servers_config(conn, config): """Add the activated remote servers stored on the repository database to a given configuration JSON. """ if ("servers" not in config): config["servers"] = {} cur = conn.cursor() cur.execute(""" SELECT id, hostname, port, username, password, dbname, frequency, coalesce(alias, '') AS alias FROM {powa}.powa_servers s WHERE s.id > 0 AND s.frequency > 0 ORDER BY id """) for row in cur: parms = {} parms["host"] = row[1] parms["port"] = row[2] parms["user"] = row[3] if (row[4] is not None): parms["password"] = row[4] parms["dbname"] = row[5] key = row[1] + ':' + str(row[2]) config["servers"][key] = {} config["servers"][key]["dsn"] = parms config["servers"][key]["frequency"] = row[6] if (config["use_server_alias"] and row[7] != ''): config["servers"][key]["alias"] = row[7] else: config["servers"][key]["alias"] = key config["servers"][key]["srvid"] = row[0] conn.commit() return config def parse_options(): """Look for the configuration file in all supported location, parse it and return the resulting JSON, also adding the implicit values if needed. """ options = None for possible_config in CONF_LOCATIONS: options = parse_file(possible_config) if (options is not None): break if (options is None): print("Could not find the configuration file in any of the expected" + " locations:") for possible_config in CONF_LOCATIONS: print("\t- %s" % possible_config) sys.exit(1) if ('repository' not in options or 'dsn' not in options["repository"]): print("The configuration file is invalid, it should contains" + " a repository.dsn entry") print("Place and adapt the following content in one of those " "locations:""") print("\n\t".join([""] + CONF_LOCATIONS)) print(SAMPLE_CONFIG_FILE) sys.exit(1) for opt in ['use_server_alias', 'debug']: if (opt not in options): options[opt] = False return options def parse_file(filepath): """Read a configuration file and return the JSON """ try: return json.load(open(filepath)) except IOError: return None except Exception as e: print("Error parsing config file %s:" % filepath) print("\t%s" % e) sys.exit(1) powa-collector-1.3.0/powa_collector/powa_worker.py000066400000000000000000001202611471155241500224010ustar00rootroot00000000000000""" PowaThread: powa-collector dedicated remote server thread. One of such thread is started per remote server by the main thred. Each threads will use 2 connections: - a persistent dedicated connection to the remote server, where it'll get the source data - a connection to the repository server, to write the source data and perform the snapshot. This connection is created and dropped at each checkpoint """ from collections import defaultdict from decimal import Decimal import threading import time import psycopg2 from psycopg2.extras import DictCursor import logging import random from powa_collector.customconn import get_connection from powa_collector.snapshot import (get_global_snapfuncs_sql, get_src_query, get_db_mod_snapfuncs_sql, get_db_cat_snapfuncs_sql, get_global_tmp_name, get_nsp, copy_remote_data_to_repo) from powa_collector.utils import get_powa_version class PowaThread (threading.Thread): """A Powa collector thread. Derives from threading.Thread Manages a monitored remote server. """ def __init__(self, name, repository, config, debug): """Instance creator. Starts threading and logger""" threading.Thread.__init__(self) # we use this event to sleep on the worker main loop. It'll be set by # the main thread through one of the public functions, when a SIGHUP # was received to notify us to reload our config, or if we should # terminate. All public functions will first set the required event # before setting this one, to avoid missing an event in case of the # sleep ends at exactly the same time self.__stop_sleep = threading.Event() # this event is set when we should terminate the thread self.__stopping = threading.Event() # this event is set when we should reload the configuration self.__got_sighup = threading.Event() # this event is set when we should force an immediate snapshot self.__force_snapshot = threading.Event() # this event is set internally while a snapshot is being performed self.__snapshot_in_progress = threading.Event() # protects __register_cat_refresh_dbnames only self.__register_cat_refresh_lock = threading.Lock() # Info for registering catalog refresh. None means no refresh asked, # empty array means all db, otherwise a list of dbnames. There's no # associated Event as this won't change the normal snapshot scheduling. self.__register_cat_refresh_dbnames = None self.__connected = threading.Event() # the alias can be the dsn is use_server_alias is false or there's no # alias defined for that remote server. self.name = config["alias"] self.__repository = repository self.__config = config self.__pending_config = None self.__update_dep_versions = False self.__remote_conn = None self.__repo_conn = None self.__last_repo_conn_errored = False self.logger = logging.getLogger("powa-collector") # last snapshot time, None if unknown self.__last_snap_time = None self.__debug = debug extra = {'threadname': self.name} self.logger = logging.LoggerAdapter(self.logger, extra) self.logger.debug("Creating worker %s: %r" % (name, config)) def __repr__(self): dsn = self.__config["dsn"].copy() if ("password" in dsn): dsn["password"] = "" return ("%s: %s" % (self.name, dsn)) def __maybe_load_powa(self, conn): """Loads Powa if it's not already and it's needed. Only supports 4.0+ extension, and this version can be loaded on the fly """ ver = get_powa_version(conn) if (not ver): self.logger.error("PoWA extension not found") self.__disconnect_all() self.__stopping.set() return elif (int(ver[0][0]) < 4): self.logger.error("Incompatible PoWA version, found %s," " requires at least 4.0.0" % ver[1]) self.__disconnect_all() self.__stopping.set() return # make sure the GUC are present in case powa isn't in # shared_preload_librairies. This is only required for powa # 4.0.x. if (int(ver[0][0]) == 4 and int(ver[0][1]) == 0): try: cur = conn.cursor() cur.execute("LOAD 'powa'") cur.close() conn.commit() except psycopg2.Error as e: self.logger.error("Could not load extension powa:\n%s" % e) self.__disconnect_all() self.__stopping.set() def __save_versions(self): """Save the versions we collect on the remote server in the repository""" srvid = self.__config["srvid"] if (self.__repo_conn is None): self.__connect() ver = get_powa_version(self.__repo_conn) # Check and update PG and dependencies versions, for powa 4.1+ if (not ver or (int(ver[0][0]) == 4 and int(ver[0][1]) == 0)): self.__disconnect_repo() return self.logger.debug("Checking postgres and dependencies versions") if (self.__remote_conn is None or self.__repo_conn is None): self.logger.error("Could not check PoWA") return cur = self.__remote_conn.cursor() repo_cur = self.__repo_conn.cursor() cur.execute(""" SELECT setting FROM pg_settings WHERE name = 'server_version' --WHERE name = 'server_version_num' """) server_num = cur.fetchone() repo_cur.execute(""" SELECT version FROM {powa}.powa_servers WHERE id = %(srvid)s """, {'srvid': srvid}) repo_num = cur.fetchone() if (repo_num is None or repo_num[0] != server_num[0]): try: repo_cur.execute(""" UPDATE {powa}.powa_servers SET version = %(version)s WHERE id = %(srvid)s """, {'srvid': srvid, 'version': server_num[0]}) self.__repo_conn.commit() except Exception as e: self.logger.warning("Could not save server version" + ": %s" % (e)) self.__repo_conn.rollback() tbl_config = "powa_extension_config" if ((int(ver[0][0]) <= 4)): tbl_config = "powa_extensions" hypo_ver = None repo_cur.execute(""" SELECT extname, version FROM {powa}.""" + tbl_config + """ WHERE srvid = %(srvid)s """ % {'srvid': srvid}) exts = repo_cur.fetchall() for ext in exts: if (ext[0] == 'hypopg'): hypo_ver = ext[1] cur.execute(""" SELECT extversion FROM pg_extension WHERE extname = %(extname)s """, {'extname': ext[0]}) remote_ver = cur.fetchone() if (not remote_ver): self.logger.debug("No version found for extension " + "%s on server %d" % (ext[0], srvid)) continue if (ext[1] is None or ext[1] != remote_ver[0]): try: repo_cur.execute(""" UPDATE {powa}.""" + tbl_config + """ SET version = %(version)s WHERE srvid = %(srvid)s AND extname = %(extname)s """, {'version': remote_ver, 'srvid': srvid, 'extname': ext[0]}) self.__repo_conn.commit() except Exception as e: self.logger.warning("Could not save version for extension " + "%s: %s" % (ext[0], e)) self.__repo_conn.rollback() # Special handling of hypopg, which isn't required to be installed in # the powa dedicated database. cur.execute(""" SELECT default_version FROM pg_available_extensions WHERE name = 'hypopg' """) remote_ver = cur.fetchone() if (remote_ver is None): try: repo_cur.execute(""" DELETE FROM {powa}.""" + tbl_config + """ WHERE srvid = %(srvid)s AND extname = 'hypopg' """, {'srvid': srvid, 'hypo_ver': remote_ver}) self.__repo_conn.commit() except Exception as e: self.logger.warning("Could not save version for extension " + "hypopg: %s" % (e)) self.__repo_conn.rollback() elif (remote_ver != hypo_ver): try: if (hypo_ver is None): repo_cur.execute(""" INSERT INTO {powa}.""" + tbl_config + """ (srvid, extname, version) VALUES (%(srvid)s, 'hypopg', %(hypo_ver)s) """, {'srvid': srvid, 'hypo_ver': remote_ver}) else: repo_cur.execute(""" UPDATE {powa}.""" + tbl_config + """ SET version = %(hypo_ver)s WHERE srvid = %(srvid)s AND extname = 'hypopg' """, {'srvid': srvid, 'hypo_ver': remote_ver}) self.__repo_conn.commit() except Exception as e: self.logger.warning("Could not save version for extension " + "hypopg: %s" % (e)) self.__repo_conn.rollback() self.__disconnect_repo() def __check_powa(self): """Check that Powa is ready on the remote server.""" if (self.__remote_conn is None): self.__connect() if (self.is_stopping()): return # make sure the GUC are present in case powa isn't in # shared_preload_librairies. This is only required for powa # 4.0.x. if (self.__remote_conn is not None): self.__maybe_load_powa(self.__remote_conn) if (self.is_stopping()): return # Check and update PG and dependencies versions if possible self.__save_versions() def __reload(self): """Reload configuration Disconnect from everything, read new configuration, reconnect, update dependencies, check Powa is still available The new session could be totally different """ self.logger.info("Reloading configuration") if (self.__pending_config is not None): self.__config = self.__pending_config self.__pending_config = None self.__disconnect_all() self.__connect() if (self.__update_dep_versions): self.__update_dep_versions = False self.__check_powa() self.__got_sighup.clear() def __report_error(self, msg, replace=True): """Store errors in the repository database. replace means we overwrite current stored errors in the database for this server. Else we append""" if (self.__repo_conn is not None): if (type(msg).__name__ == 'list'): error = msg else: error = [msg] srvid = self.__config["srvid"] cur = self.__repo_conn.cursor() cur.execute("SAVEPOINT metas") try: if (replace): cur.execute("""UPDATE {powa}.powa_snapshot_metas SET errors = %s WHERE srvid = %s """, (error, srvid)) else: cur.execute("""UPDATE {powa}.powa_snapshot_metas SET errors = pg_catalog.array_cat(errors, %s) WHERE srvid = %s """, (error, srvid)) cur.execute("RELEASE metas") except psycopg2.Error as e: err = "Could not report error for server %d:\n%s" % (srvid, e) self.logger.warning(err) cur.execute("ROLLBACK TO metas") self.__repo_conn.commit() def __connect(self): """Connect to a remote server Override lock_timeout, application name""" if ('dsn' not in self.__repository or 'dsn' not in self.__config): self.logger.error("Missing connection info") self.__stopping.set() return try: if (self.__repo_conn is None): self.logger.debug("Connecting on repository...") self.__repo_conn = get_connection(self.logger, self.__debug, self.__repository['dsn']) self.logger.debug("Connected.") # make sure the GUC are present in case powa isn't in # shared_preload_librairies. This is only required for powa # 4.0.x. self.__maybe_load_powa(self.__repo_conn) # Return now if __maybe_load_powa asked to stop if (self.is_stopping()): return cur = self.__repo_conn.cursor() cur.execute("""SELECT pg_catalog.set_config(name, '2000', false) FROM pg_catalog.pg_settings WHERE name = 'lock_timeout' AND setting = '0'""") cur.execute("SET application_name = %s", ('PoWA collector - repo_conn for worker ' + self.name,)) cur.close() self.__repo_conn.commit() self.__last_repo_conn_errored = False if (self.__remote_conn is None): self.logger.debug("Connecting on remote database...") self.__remote_conn = get_connection(self.logger, self.__debug, **self.__config['dsn']) self.logger.debug("Connected.") # make sure the GUC are present in case powa isn't in # shared_preload_librairies. This is only required for powa # 4.0.x. if (self.__remote_conn is not None): self.__maybe_load_powa(self.__remote_conn) # Return now if __maybe_load_powa asked to stop if (self.is_stopping()): return cur = self.__remote_conn.cursor() cur.execute("""SELECT pg_catalog.set_config(name, '2000', false) FROM pg_catalog.pg_settings WHERE name = 'lock_timeout' AND setting = '0'""") cur.execute("SET application_name = %s", ('PoWA collector - worker ' + self.name,)) cur.close() self.__remote_conn.commit() self.__connected.set() except psycopg2.Error as e: self.logger.error("Error connecting on %s:\n%s" % (self.__config["dsn"], e)) if (self.__repo_conn is not None): self.__report_error("%s" % (e)) else: self.__last_repo_conn_errored = True def __disconnect_all(self): """Disconnect from remote server and repository server""" if (self.__remote_conn is not None): self.logger.info("Disconnecting from remote server") self.__remote_conn.close() self.__remote_conn = None if (self.__repo_conn is not None): self.logger.info("Disconnecting from repository") self.__disconnect_repo() self.__connected.clear() def __disconnect_repo(self): """Disconnect from repo""" if (self.__repo_conn is not None): self.__repo_conn.close() self.__repo_conn = None def __disconnect_all_and_exit(self): """Disconnect all and stop the thread""" # this is the exit point self.__disconnect_all() self.logger.info("stopped") self.__stopping.clear() def __worker_main(self): """The thread's main loop Get latest snapshot timestamp for the remote server and determine how long to sleep before performing the next snapshot. Add a random seed to avoid doing all remote servers simultaneously""" self.__last_snap_time = None self.__check_powa() # __check_powa() is only responsible for making sure that the remote # connection is opened. if (self.__repo_conn is None): self.__connect() # if this worker has been restarted, restore the previous snapshot # time to try to keep up on the same frequency if (not self.is_stopping() and self.__repo_conn is not None): cur = None try: cur = self.__repo_conn.cursor() cur.execute("""SELECT EXTRACT(EPOCH FROM snapts) FROM {powa}.powa_snapshot_metas WHERE srvid = %d """ % self.__config["srvid"]) row = cur.fetchone() if not row: self.logger.error("Server %d was not correctly registered" " (no powa_snapshot_metas record)" % self.__config["srvid"]) self.logger.debug("Server configuration details:\n%r" % self.__config) self.logger.error("Stopping worker for server %d" % self.__config["srvid"]) self.__stopping.set() if row: self.__last_snap_time = float(row[0]) self.logger.debug("Retrieved last snapshot time:" + " %r" % self.__last_snap_time) cur.close() self.__repo_conn.commit() except Exception as e: self.logger.warning("Could not retrieve last snapshot" + " time: %s" % (e)) if (cur is not None): cur.close() self.__repo_conn.rollback() # Normalize unknkown last snapshot time if (self.__last_snap_time == Decimal('-Infinity')): self.__last_snap_time = None # if this worker was stopped longer than the configured frequency, # assign last snapshot time to a random time between now and now minus # duration. This will help to spread the snapshots and avoid activity # spikes if the collector itself was stopped for a long time, or if a # lot of new servers were added freq = self.__config["frequency"] if (not self.is_stopping() and ( self.__last_snap_time is None or ((time.time() - self.__last_snap_time) > freq) )): random.seed() r = random.randint(0, self.__config["frequency"] - 1) self.logger.debug("Spreading snapshot: setting last snapshot to" + " %d seconds ago (frequency: %d)" % (r, freq)) self.__last_snap_time = time.time() - r while (not self.is_stopping()): if (self.__got_sighup.isSet()): self.__reload() if ((self.__last_snap_time is None) or ((time.time() - self.__last_snap_time) >= freq) or (self.__force_snapshot.isSet())): try: self.__snapshot_in_progress.set() if (self.__force_snapshot.isSet()): self.__force_snapshot.clear() self.__last_snap_time = time.time() self.__take_snapshot() self.__snapshot_in_progress.clear() except psycopg2.Error as e: self.logger.error("Error during snapshot: %s" % e) # It will reconnect automatically at next snapshot self.__disconnect_all() time_to_sleep = max(self.__config["frequency"] - \ (time.time() - self.__last_snap_time), 0) # sleep until the scheduled processing time, or if the main thread # asked us to perform an action or if we were asked to stop. if (time_to_sleep > 0 and not self.is_stopping()): self.__stop_sleep.wait(time_to_sleep) # clear the event if it has been set. We'll process all possible # event triggered by it within the next iteration if (self.__stop_sleep.isSet()): self.__stop_sleep.clear() # main loop is over, disconnect and quit self.__disconnect_all_and_exit() def __get_global_snapfuncs(self, powa_ver): """ Get the list of global snapshot functions (in the dedicated powa database), and their associated query_src """ srvid = self.__config["srvid"] cur = self.__repo_conn.cursor(cursor_factory=DictCursor) cur.execute("SAVEPOINT snapshots") try: cur.execute(get_global_snapfuncs_sql(powa_ver), (srvid,)) snapfuncs = cur.fetchall() cur.execute("RELEASE snapshots") except psycopg2.Error as e: cur.execute("ROLLBACK TO snapshots") err = "Error while getting snapshot functions:\n%s" % (e) self.logger.error(err) self.logger.error("Exiting worker for server %s..." % srvid) self.__stopping.set() return None cur.close() if (not snapfuncs): self.logger.info("No datasource configured for server %d" % srvid) self.logger.debug("Committing transaction") self.__repo_conn.commit() self.__disconnect_repo() return None return snapfuncs def __get_global_src_data(self, powa_ver, ins): """ Retrieve the source global data (in the powa database) from the foreign server, and insert them in the *_src_tmp tables on the repository server. """ srvid = self.__config["srvid"] errors = [] snapfuncs = self.__get_global_snapfuncs(powa_ver) if not snapfuncs: # __get_global_snapfuncs already took care of reporting errors return errors data_src = self.__remote_conn.cursor() for snapfunc in snapfuncs: if (self.is_stopping()): return errors kind_name = snapfunc["name"] query_source = snapfunc["query_source"] cleanup_sql = snapfunc["query_cleanup"] function_name = snapfunc["function_name"] external = snapfunc["external"] self.logger.debug("Working on %s", kind_name) # get the SQL needed to insert the query_src data on the remote # server into the transient unlogged table on the repository server if (query_source is None): self.logger.warning("No query_source for %s" % function_name) continue # execute the query_src functions on the remote server to get its # local data (srvid 0) r_nsp = get_nsp(self.__remote_conn, external, kind_name) self.logger.debug("Calling %s.%s(0)..." % (r_nsp, query_source)) data_src_sql = get_src_query(r_nsp, query_source, srvid) tbl_nsp = get_nsp(self.__repo_conn, external, kind_name) target_tbl_name = get_global_tmp_name(tbl_nsp, query_source) errors.extend(copy_remote_data_to_repo(self, kind_name, data_src, data_src_sql, ins, target_tbl_name, cleanup_sql)) data_src.close() return errors def __get_db_mod_snapfuncs(self, srvid): """ Get the list of per-db module, with their associated query_source and dbnames. """ server_version_num = self.__remote_conn.server_version db_mod_queries = defaultdict(list) cur = self.__repo_conn.cursor(cursor_factory=DictCursor) cur.execute("SAVEPOINT db_snapshots") try: cur.execute(get_db_mod_snapfuncs_sql(srvid, server_version_num)) mod_snapfuncs = cur.fetchall() cur.execute("RELEASE db_snapshots") except psycopg2.Error as e: cur.execute("ROLLBACK TO db_snapshots") err = "Error while getting db module snapshot functions:\n%s" % (e) self.logger.error(err) self.logger.error("Exiting worker for server %s..." % srvid) self.__stopping.set() return None cur.close() for func in mod_snapfuncs: row = (func['db_module'], func['query_source'], func['tmp_table']) if (func['dbnames'] is None): db_mod_queries[None].append(row) else: for dbname in func['dbnames']: db_mod_queries[dbname].append(row) return db_mod_queries def __get_db_cat_snapfuncs(self, srvid): """ Get the list of per-db catalogs, with their associated query_source and dbnames. """ server_version_num = self.__remote_conn.server_version cat_queries = [] cur = self.__repo_conn.cursor(cursor_factory=DictCursor) cur.execute("SAVEPOINT db_catalog") forced_dbnames = None try: # Make a copy of the catalog refresh registration if any while # holding a lock, and also clear the registration list at the same # time. # It means that any cat refresh registered after that point won't # be lost and will be treated once during the next snapshot. with self.__register_cat_refresh_lock: if (self.__register_cat_refresh_dbnames is not None): forced_dbnames = self.__register_cat_refresh_dbnames.copy() self.__register_cat_refresh_dbnames = None force = True if forced_dbnames is not None else False cur.execute(get_db_cat_snapfuncs_sql(srvid, server_version_num, force)) cat_snapfuncs = cur.fetchall() cur.execute("RELEASE db_catalog") except psycopg2.Error as e: cur.execute("ROLLBACK TO db_catalog") err = "Error while getting db catalog snapshot functions:\n%s" % (e) self.logger.error(err) self.logger.error("Exiting worker for server %s..." % srvid) self.__stopping.set() return None for func in cat_snapfuncs: row = (func['catname'], func['query_source'], func['tmp_table'], func['excluded_dbnames']) cat_queries.append(row) return (cat_queries, forced_dbnames) def __get_db_src_data(self, powa_ver, now, ins): """ Retrieve the source per-database data from the foreign server, and insert them in the *_src_tmp tables on the repository server. This handles both db_modules and catalog datasources. """ srvid = self.__config["srvid"] errors = [] # This is a powa 5+ feature if (int(powa_ver[0][0]) < 5): return errors dbnames = self.__get_remote_dbnames() db_mod_queries = self.__get_db_mod_snapfuncs(srvid) (db_cat_queries, forced_cat_dbnames) = self.__get_db_cat_snapfuncs(srvid) for dbname in dbnames: # Skip that database if no module configured for it do_db_module = (None in db_mod_queries or dbname in db_mod_queries) do_db_cat = False if (forced_cat_dbnames is not None): if (len(forced_cat_dbnames) == 0): do_db_cat = True else: do_db_cat = (dbname in forced_cat_dbnames) if not do_db_cat: for (_, _, _, excluded_dbnames) in db_cat_queries: if (dbname not in excluded_dbnames): do_db_cat = True break # Skip this database if there's no db_module or catalog to import if (not do_db_module and not do_db_cat): continue self.logger.debug("Working on remote database %s", dbname) errors.extend(self.__get_db_src_data_onedb(now, dbname, ins, db_mod_queries, db_cat_queries, forced_cat_dbnames)) if (self.is_stopping()): if (len(errors) > 0): self.__report_error(errors) return [] return errors def __get_db_src_data_onedb(self, now, dbname, ins, db_mod_queries, db_cat_queries, forced_cat_dbnames): """ Per-database worker function for __get_db_src_data(), taking care of db modules and catalog import. """ srvid = self.__config["srvid"] errors = [] self.logger.debug("Working on remote database %s", dbname) try: dbconn = get_connection(self.logger, self.__debug, override_dbname=dbname, **self.__config['dsn']) except psycopg2.Error as e: err = "Could not connect to remote database %s:\n%s" % (dbname, e) self.logger.warning(err) errors.append(err) return errors data_src = dbconn.cursor() # first, process the enabled db_modules on that database for row in (db_mod_queries.get(None, []) + \ db_mod_queries.get(dbname, [])): if (self.is_stopping()): dbconn.close() return errors (db_module, query_source, tmp_table) = row data_src_sql = """SELECT %d AS srvid, '%s' AS ts, d.dbid, src.* FROM (%s) src CROSS JOIN ( SELECT oid AS dbid FROM pg_catalog.pg_database WHERE datname = current_database() ) d""" % (srvid, now, query_source) self.logger.debug("Db module %s, calling SQL:\n%s" % (db_module, data_src_sql)) errors.extend(copy_remote_data_to_repo(self, db_module, data_src, data_src_sql, ins, tmp_table)) # then process the outdated catalogs on that databasr for row in db_cat_queries: (catname, query_source, tmp_table, excluded_dbnames) = row # Ignore this catalog if excluded for this database if (forced_cat_dbnames is not None): if (len(forced_cat_dbnames) == 0): # empty means to process all databases pass elif (dbname not in forced_cat_dbnames): continue if (dbname in excluded_dbnames): continue data_src_sql = """SELECT %d AS srvid, d.dbid, src.* FROM (%s) src CROSS JOIN ( SELECT oid AS dbid FROM pg_catalog.pg_database WHERE datname = current_database() ) d""" % (srvid, query_source) self.logger.debug("Catalog %s, calling SQL:\n:%s" % (catname, data_src_sql)) errors.extend(copy_remote_data_to_repo(self, catname, data_src, data_src_sql, ins, tmp_table)) data_src.close() dbconn.close() return errors def __get_remote_dbnames(self): """ Get the list of databases on the remote servers """ res = [] cur = self.__remote_conn.cursor() cur.execute("""SELECT datname FROM pg_catalog.pg_database WHERE datallowconn""") for row in cur.fetchall(): res.append(row[0]) cur.close() return res def __take_snapshot(self): """Main part of the worker thread. This function will call all the query_src functions enabled for the target server, and insert all the retrieved rows on the repository server, in unlogged tables, and finally call powa_take_snapshot() on the repository server to finish the distant snapshot. All is done in one transaction, so that there won't be concurrency issues if a snapshot takes longer than the specified interval. This also ensure that all rows will see the same snapshot timestamp. """ srvid = self.__config["srvid"] if (self.is_stopping()): return self.__connect() if (self.__remote_conn is None): self.logger.error("No connection to remote server, snapshot skipped") return if (self.__repo_conn is None): self.logger.error("No connection to repository server, snapshot skipped") return powa_ver = get_powa_version(self.__repo_conn) powa_remote_ver = get_powa_version(self.__remote_conn) # We require the same powa major version (X.Y) on the repository and # each remote server. if (powa_ver[0][0] != powa_remote_ver[0][0] or powa_ver[0][1] != powa_remote_ver[0][1]): error = ("Incompatible PoWA version between the repository server" + " (%s.X) and the remote host (%s.X)" % ( powa_ver[0][0] + "." + powa_ver[0][1], powa_remote_ver[0][0] + "." + powa_remote_ver[0][1])) self.__report_error(error) self.__disconnect_repo() return ins = self.__repo_conn.cursor() # Retrieve the global data from the remote server errors = self.__get_global_src_data(powa_ver, ins) if (self.is_stopping()): if (len(errors) > 0): self.__report_error(errors) return with self.__remote_conn.cursor() as cur: cur.execute("SELECT now()") now = cur.fetchone()[0] # Retrieve the per-db data from the remote server errors.extend(self.__get_db_src_data(powa_ver, now, ins)) if (self.is_stopping()): if (len(errors) > 0): self.__report_error(errors) return # call powa_take_snapshot() for the given server self.logger.debug("Calling powa_take_snapshot(%d)..." % (srvid)) sql = ("SELECT {powa}.powa_take_snapshot(%(srvid)d)" % {'srvid': srvid}) try: ins.execute("SAVEPOINT powa_take_snapshot") ins.execute(sql) val = ins.fetchone()[0] if (val != 0): self.logger.warning("Number of errors during snapshot: %d", val) self.logger.warning(" Check the logs on the repository server") ins.execute("RELEASE powa_take_snapshot") except psycopg2.Error as e: err = "Error while taking snapshot for server %d:\n%s" % (srvid, e) self.logger.warning(err) errors.append(err) ins.execute("ROLLBACK TO powa_take_snapshot") # Manually reset existing errors as powa_take_snapshot, which # should be responsible for it, failed ins.execute("""UPDATE {powa}.powa_snapshot_metas SET errors = NULL WHERE srvid = %(srvid)s""", {'srvid': srvid}) ins.execute("SET application_name = %s", ('PoWA collector - repo_conn for worker ' + self.name,)) ins.close() # we need to report and append errors after calling powa_take_snapshot, # since this function will first reset errors if (len(errors) > 0): self.__report_error(errors, False) # and finally commit the transaction self.logger.debug("Committing transaction") self.__repo_conn.commit() self.__remote_conn.commit() self.__disconnect_repo() def is_stopping(self): """Is the thread currently stopping""" return self.__stopping.isSet() def get_config(self): """Returns the thread's config""" return self.__config def ask_to_stop(self): """Ask the thread to stop""" self.__stopping.set() self.logger.info("Asked to stop...") self.__stop_sleep.set() def run(self): """Start the main loop of the thread""" if (not self.is_stopping()): self.logger.info("Starting worker") self.__worker_main() def ask_reload(self, new_config): """Ask the thread to reload""" self.logger.debug("Reload asked") self.__pending_config = new_config self.__got_sighup.set() self.__stop_sleep.set() def ask_update_dep_versions(self): """Ask the thread to recompute its dependencies""" self.logger.debug("Version dependencies reload asked") self.__update_dep_versions = True self.__got_sighup.set() self.__stop_sleep.set() def get_status(self): """Get the status: ok, not connected to repo, or not connected to remote""" if (self.__repo_conn is None and self.__last_repo_conn_errored): return "no connection to repository server" if (self.__remote_conn is None): return "no connection to remote server" else: return "running" def is_snapshot_in_progress(self): """Returns whether the worker is currently performing a snapshot""" return self.__snapshot_in_progress.isSet() def request_forced_snapshot(self): """Ask for an immediate snapshot""" self.logger.debug('Forced snapshot required') if (self.__snapshot_in_progress.isSet()): return ('ERROR', 'A snapshot is already in progress') if (self.__force_snapshot.isSet()): return ('ERROR', 'A forced snapshot is already scheduled') if ((time.time() - self.__last_snap_time ) < 10): return ('ERROR', 'Last snapshot was less than 10s. ago') self.__force_snapshot.set() self.__stop_sleep.set() return ('OK', '') def register_cat_refresh(self, dbnames=[]): """ Register a catalog refresh on the wanted databases (or all) during the next snapshot. """ self.logger.debug('Cat refresh required on databases %r', dbnames) with self.__register_cat_refresh_lock: if (self.__register_cat_refresh_dbnames is None): self.__register_cat_refresh_dbnames = dbnames.copy() elif (len(dbnames) == 0): # passed empty array means all databases self.__register_cat_refresh_dbnames = [] # If the stored array is empty, it means someone asked to refresh # the catalogs on all databases, so nothing to do. Otherwise, just # add the givend databases elif (len(self.__register_cat_refresh_dbnames) != 0): # it's ok to have duplicates, we don't expect too many request, # and we want to minimize the lock time self.__register_snapshot.extend(dbnames) return ('OK', 'Catalogs will be refreshed during the next snapshot') powa-collector-1.3.0/powa_collector/snapshot.py000066400000000000000000000130161471155241500217000ustar00rootroot00000000000000from os import SEEK_SET import psycopg2 import sys if (sys.version_info < (3, 0)): from StringIO import StringIO else: from io import StringIO def get_global_snapfuncs_sql(ver): """Get the list of enabled global functions for snapshotting""" # XXX should we ignore entries without query_src? if (int(ver[0][0]) >= 5): return """SELECT name, external, query_source, query_cleanup, function_name FROM {powa}.powa_functions pf -- FIXME -- JOIN pg_extension ext ON ext.extname = pf.module -- JOIN pg_namespace nsp ON nsp.oid = ext.extnamespace WHERE operation = 'snapshot' AND enabled AND srvid = %s ORDER BY priority""" else: return """SELECT module AS name, false AS external, query_source, query_cleanup, function_name FROM {powa}.powa_functions pf -- FIXME -- JOIN pg_extension ext ON ext.extname = pf.module -- JOIN pg_namespace nsp ON nsp.oid = ext.extnamespace WHERE operation = 'snapshot' AND enabled AND srvid = %s ORDER BY priority""" def get_src_query(schema, src_fct, srvid): """ Get the SQL query for global dataousource we'll use to get results from a snapshot function """ return ("SELECT %(srvid)d, * FROM %(schema)s.%(fname)s(0)" % {'fname': src_fct, 'schema': schema, 'srvid': srvid}) def get_db_mod_snapfuncs_sql(srvid, server_version_num): """ Get the SQL query for a db_module we'll use to get results from a snapshot function """ return ("""SELECT db_module, query_source, tmp_table, dbnames FROM {powa}.powa_db_functions(%(srvid)d, %(server_version_num)d) WHERE operation = 'snapshot' ORDER BY priority """ % {'srvid': srvid, 'server_version_num': server_version_num}) def get_db_cat_snapfuncs_sql(srvid, server_version_num, force=False): """ Get the SQL query for a db catalog we'll use to get results from a snapshot function """ if force: interval = 'NULL' else: interval = "'1 year'" return ("""SELECT catname, query_source, tmp_table, excluded_dbnames FROM {powa}.powa_catalog_functions( %(srvid)d, %(server_version_num)d, %(interval)s )""" % {'srvid': srvid, 'server_version_num': server_version_num, 'interval': interval}) def get_global_tmp_name(schema, src_fct): """Get the temp table name we'll use to spool changes""" return "%s.%s_tmp" % (schema, src_fct) def get_nsp(conn, external, module): if external: return conn._nsps[module] else: return conn._nsps['powa'] def copy_remote_data_to_repo(cls, data_name, data_src, data_src_sql, data_ins, target_tbl_name, cleanup_sql=None): """ Retrieve the wanted datasource from the given connection and insert it on the repository server in the given table. data_src: the cursor to use to get the data on the remote server data_src_sql: the SQL query to execute to get the datasource data data_name: a string describing the datasource data_ins: the cursor to use to write the data on the repository server target_tbl_name: a string containing the (fully qualified and properly quoted) target table name to COPY data to on the repository server cleanup_sql: an optional SQL query to execute after executing data_src_sql. Note that this query is executed even if there's an error during data_src_sql execution """ errors = [] src_ok = True # use savepoint, there could be an error while retrieving data on the # remote server. data_src.execute("SAVEPOINT src") # XXX should we use os.pipe() or a temp file instead, to avoid too # much memory consumption? buf = StringIO() try: data_src.copy_expert("COPY (%s) TO stdout" % data_src_sql, buf) except psycopg2.Error as e: src_ok = False err = "Error retrieving datasource data %s:\n%s" % (data_name, e) errors.append(err) data_src.execute("ROLLBACK TO src") # execute the cleanup query if provided if (cleanup_sql is not None): data_src.execute("SAVEPOINT src") try: cls.logger.debug("Calling %s..." % cleanup_sql) data_src.execute(cleanup_sql) except psycopg2.Error as e: err = "Error while calling %s:\n%s" % (cleanup_sql, e) errors.append(err) data_src.execute("ROLLBACK TO src") # If user want to stop the collector or if any error happened during # datasource retrieval, there's nothing more to do so simply inform caller # of the errors. if (cls.is_stopping() or not src_ok): return errors # insert the data to the transient unlogged table data_ins.execute("SAVEPOINT data") buf.seek(0, SEEK_SET) try: # For data import the schema is now on the repository server data_ins.copy_expert("COPY %s FROM stdin" % target_tbl_name, buf) except psycopg2.Error as e: err = "Error while inserting data:\n%s" % e cls.logger.warning(err) errors.append(err) cls.logger.warning("Giving up for datasource %s", data_name) data_ins.execute("ROLLBACK TO data") buf.close() return errors powa-collector-1.3.0/powa_collector/utils.py000066400000000000000000000014371471155241500212050ustar00rootroot00000000000000""" General functions shared between the main thread and the remote server threads. """ def get_powa_version(conn): """Get powa's extension version""" cur = conn.cursor() cur.execute("""SELECT regexp_split_to_array(extversion, '\\.'), extversion FROM pg_catalog.pg_extension WHERE extname = 'powa'""") res = cur.fetchone() cur.close() return res def conf_are_equal(conf1, conf2): """Compare two configurations, returns True if equal""" for k in conf1.keys(): if (k not in conf2): return False if (conf1[k] != conf2[k]): return False for k in conf2.keys(): if (k not in conf1): return False if (conf1[k] != conf2[k]): return False return True powa-collector-1.3.0/requirements.txt000066400000000000000000000000111471155241500177260ustar00rootroot00000000000000psycopg2 powa-collector-1.3.0/setup.py000066400000000000000000000021701471155241500161640ustar00rootroot00000000000000import setuptools __VERSION__ = None with open("powa_collector/__init__.py", "r") as fh: for line in fh: if line.startswith('__VERSION__'): __VERSION__ = line.split('=')[1].replace("'", '').strip() break requires = ['psycopg2'] setuptools.setup( name="powa-collector", version=__VERSION__, author="powa-team", license='Postgresql', author_email="rjuju123@gmail.com", description="PoWA collector, a collector for performing remote snapshot with PoWA", long_description="See https://powa.readthedocs.io/", long_description_content_type="text/markdown", url="https://powa.readthedocs.io/", packages=setuptools.find_packages(), install_requires=requires, scripts=["powa-collector.py"], classifiers=[ "Development Status :: 5 - Production/Stable", "Programming Language :: Python :: 2", "Programming Language :: Python :: 3", "License :: OSI Approved :: PostgreSQL License", "Operating System :: OS Independent", "Intended Audience :: System Administrators", "Topic :: Database :: Front-Ends" ], ) powa-collector-1.3.0/test.py000077500000000000000000000005011471155241500160020ustar00rootroot00000000000000import psycopg2 def main(): try: print("test") conn = psycopg2.connect("host=localhost") cur = conn.cursor() cur.execute("LOAD pouet") cur.close() conn.close() except psycopg2.Error as e: print("error: %s" % (e)) if (__name__ == "__main__"): main()