skytools-2.1.13/0000755000175000017500000000000011727601174012477 5ustar markomarkoskytools-2.1.13/NEWS0000644000175000017500000004400311727600317013175 0ustar markomarko 2012-03-13 - SkyTools 2.1.13 - "Pailapsiin Overdose" * Convert SkyTools to use symbolic isolation level constants. Pscyopg 2.4.2/2.4.3 have renumbered isolation levels, which made "londiste copy" lose data. Pscyopg 2.4.4 fixes it. But make SkyTools conform to new Pscyopg conventions so Londiste can survive such changes in the future. This also makes Londiste work with those 2 Pscyopg versions. * Use REPEATABLE READ isolation level on copy. Pre-9.1 READ_COMMITTED and SERIALIZABLE were the same. 9.1+ introduce new SERIALIZABLE which does more than we want. * londiste add-table: make trigger check sql 9.1-compatible. (Sébastien Lardière) * Make C modules compile on 9.2. * v2.1.12 broke psycopg <= 2.0.9. Fix. (Dimitri Fontaine) * Fix rare crash in pgq.insert_event() & pgq triggers. (Backport from 3.0) * walmgr: - Sync changes from 3.0 branch. (Martin Pihlak) - Add support for detecting stale locks and releasing them instead of aborting (Steve Singer) - Move the pg_stop_backup() into a finally: block. Some instances were reported where the base backup failed with some issue but pg_stop_backup() hadn't been called and had to be called manually. This should make that less likely (Steve Singer) 2010-11-10 - SkyTools 2.1.12 - "Portable Minefield" To apply Londiste database-side fixes, run 'londiste.upgrade.sql' where needed. = Features = * Support Postgres 9.0. (Devrim Gündüz, Jason Buberel, Sébastien Lardière) * walmgr: Introduce a 'slave_pg_xlog' configuration variable. This allows master and slave pg_xlog files to be in different locations. During restore this directory is symlinked to slave pg_xlog. (Steve Singer) * walmgr: Introduce a 'backup_datadir' configuration variable to control whether the slave data directory is kept or overwritten during restore. (Steve Singer) * walmgr: consider wal_level during setup/stop (Martin Pihlak) * skytools.dbstruct: support version-dependant sql. * psycopgwrapper: always set db.server_version = Fixes = * londiste copy: restore index clustering info. (André Malo) * londiste repair: use subprocess module instead os.popen4, where available. (Shoaib Mir) * pgqadm: Remove unnecessary JOIN in refresh_queues(). (Artyom Nosov) * londiste add: dont send new table/seq list into queue, not useful, only can cause surprise removal in case of several 'provider add'+ 'subscriber add' pairs. * londiste launcher: don't interpret file 'londiste' as module. Needed for 3.0 compatibility. * walmgr: "sync" now omits unneeded WAL segments if the database has been cleanly shut down. This greatly reduces sync time during planned switchovers as usually there is only a single WAL file to be synched to slave. * londiste.link: Add missing quoting. The functions are unused, thus missed the last round of fixes. * Building from CVS/GIT assumes --with-asciidoc automatically. * Newer asciidoc/docbook do not need fixman.py hack. Remove it. 2010-02-03 - SkyTools 2.1.11 - "Replicates Like Randy Rabbit" = Fixes = * londiste, pgq smart triggers: crash in case the table info cache invalidation callback was called from signal handler. Fix it by moving cache operations out of callback. * walmgr: - symlink handling for pg_log and pg_xlog - Set archive_command to "/bin/true" for restored postgresql.conf (Martin Pihlak, Mark Kirkwood) * londiste copy: - Make constraint dropping work for inherited tables (Hannu Krosing) - Do not restore fkeys to tables with unknown replication status or coming from different queue. (Hannu Krosing) - Use TRUNCATE ONLY on 8.4+. (Sergey Konoplev) - Replace last copy pidfile check with signal_pidfile, thus avoiding potential infinite stall with dead copy. * londiste repair: set LC_ALL=C when calling 'sort' to have byte based sorting. * skytools.DBScript: - Add --version switch to show skytools version. (Hannu Krosing) - Safer pidfile handling - avoid creating zero-length files on disk full situation by deleting them on write error. * pgq.Event: ev.retry, ev.ev_retry fields for retry count. (Nico Mandery) * pgq.maint_retry_events(): lock table to allow only single mover. * pgq.logutriga() did not put custom pkey= value into events. * pgq.logutriga() and pgq.sqltriga() did allow UPDATE and DELETE on tables without pkey, running into SQL errors downstream. They should throw error in such case. * Fix DeprecationWarning on Python 2.6 vs. 'import sets'. * make deb: Work around Debian's --install-layout related braindamage. 2009-08-31 - SkyTools 2.1.10 - "As Stable As A Falling Anvil" = Fixes = * pgqadm: rename 'as' function argument as it's keyword in Python 2.6 * londiste provider add-seq: - Detect existing sequences. - Make --all ignore pgq, londiste and temp schemas. - Proper quoting was missing in few places. * walmgr: Create pg_xlog/archive_status directory for slave restore. (Mark Kirkwood) * londiste provider add: detect and warn about user AFTER triggers that run before Londiste one. * docs: - Documentation for log_size, log_count and use_skylog options. - Refresh dependency list, add rsync - Mention --with-asciidoc in INSTALL. * --with-asciidoc: make it tolerate missing asciidoc and/or xmlto. * deb84 Makefile target. 2009-03-13 - SkyTools 2.1.9 - "Six Pack of Luck" = WalMgr improvements = * walmgr.py: WAL files purge procedure now pays attention to recovery restart points. (%r patch by Omar Kilani, Mark Kirkwood) * walmgr.py: archive_mode GUC is now set during setup. Followed by an optional restart if master_restart_cmd is specified (Mark Kirkwood) * walmgr.py: PostgreSQL configuration files are backed up to "config_backup" and restored to "slave_config_dir". (Mark Kirkwood) * walmgr.py: Backups taken from slave now generate backup_label and history files. * walmgr.py Configuration files now default to PostgreSQL 8.3 = Fixes = * londiste copy: Add missing identifier quoting for TRUNCATE and ADD CONSTRAINT. (Luc Van Hoeylandt) * Quote identifiers starting with numeric. (Andrew Dunstan) * Fix crash with pgq.sqltrigq/pgq.logutriga, which automatically detect table structure. Problem was handling table invalidation event during internal query. (Götz Lange, André Malo) * Fix 'uninitialized variable' warning and potentially bad code in pgq.sqltriga(). (Götz Lange, André Malo) * skytools._cquoting: Fix crash with Decimal type. (May affects users who have compiled psycopg2 to use Decimal() instead of float for NUMERIC) * skytools.DBStruct: The DEFAULT keyword was missing when creating table columns with default value. Table creation functionality is unused in 2.1.x thus no users should be affected. * skytools.magic_insert: be more flexible about whats dict. In particular, now accept DictRow. * configure.ac: don't detect xmlto and asciidoc version if --with-asciidoc is not used. Otherwise unnecessary error message was printed. * Add few headers for Postgres 8.4 that are not included automatically anymore. (Devrim Gündüz) 2008-10-12 - SkyTools 2.1.8 - "Perestroika For Your Data" = Fixes = * deb: Make debian package accept skytools-8.3 too. * add skylog.ini as installalble file (David Fetter) * Londiste: - Document the fkeys, triggers, restore-triggers commands. - Run ANALYZE after table is copied over. - Fix "provider add-seq --all" * pgq.SerialConsumer (used by Londiste) - fix position check, which got confused when sequence numbers did large jump. * PgQ database functions: - pgq.maint_rotate_step1() function removed tick #1 even if consumer was registered on it. - pgq triggers: import cache invalidation from HEAD. Now pgq.sqltriga() and pgq.logutriga() should be preferable to pgq.logtriga(). - uninstall_pgq.sql: Correct syntax for DROP SCHEMA CASCADE. * skytools.DBScript, used by all Python scripts: - Don't try to stay running if MemoryError was thrown. - Detect stale pidfile and ignore it. - Stop hooking SIGTERM anymore. Python signal handling and libpq polling did not interact well. * scriptmgr: - Don't crash on missing pidfile. - Fix few typos on scriptmgr (Martin Pihlak) * walmgr (Martin Pihlak): - improved error messages on startup if no config file specified. - walmgr.py stop now also stops syncdaemon - pg_auth file is copied to slave as part of archiving. restored during boot. - cleanup in remote_walmgr() to always pass config file to slave walmgr. - master_backup() now reports exceptions. = Features = * londiste -s/-k now kill "copy" process with SIGTERM. Previously the process was left running. On startup "replay" will check if "copy" process had died before finishing and launches one if needed. (Dimitri Fontaine) * New skytools.signal_pidfile() function. * Londiste: New lock_timeout parameter to limit time locks are held on provider. Applies to operations: provider add/remove, compare, repair. * Londiste: on psycopg2 v2.0.6+ use new .copy_expert() API, which does not crash on tables with large number of columns. 2008-06-03 - SkyTools 2.1.7 - "Attack of the Werebugs" = Fixes = * Fix pgq trigger compilation with Postgres 8.2 (Marcin Stępnicki) * Replace `make` calls with $(MAKE) in Makefiles (Pierre-Emmanuel André) * londiste: Fix incompatibility with Python 2.3 (Dimitri Fontaine) * walmgr: Fix typo in config symlinking code. (pychecker) * bulk_loader: Fix typo in temp table check. (pychecker) * Install upgrade .sql files. Otherwise skytools_upgrade.py could be used only from source directory. * pgq.Consumer: Fix bug in retry/failed event handling. (pychecker) * pgq: Fix pgq.maint_retry_events() - it could create double events when amount of event to be moved back into main queue was more than 10. = Features = * Quoting of table and column names in Londiste and dispatcher scripts. = Upgrade procedure = * Database code under pgq and londiste schemas need to be upgraded. That can be done on running databases with following command: $ skytools_upgrade.py "connstr" Or by applying 'londiste.upgrade.sql' and 'pgq.upgrade.sql' by hand. The changes were only in functions, no table sctructure changed. 2008-04-05 - SkyTools 2.1.6 - "Quick Bugfix Release" Now we have upgrade script, see 'man skytools_upgrade' for info how to upgrade database code. = Fixes = * Include upgrade sql scripts in .tgz * Fix 'londiste provider seqs' * Fix 'londiste provider fkeys' in no tables added. * Fix "londiste copy" pidfile timing race * Fix Solaris build - avoid grep -q / define HAVE_UNSETENV * Fix "make debXX" when several Postgres version are installed. * New-style AC_OUTPUT usage. * Disable manpage creation by default, --with-asciidoc to enable. They are still included in .tgz so users should have less problems now. * Restore iter-on-values behaviour for rows from curs.fetch*. The attempt to make them iter-on-keys seems now misguided, as iter-on-values is already used in existing code, and iter-on-keys returns keys in random order. * londiste subscriber add: Dont drop triggers on table if --expect-sync is used. * londiste copy: drop triggers and fkeys in case "replay" or "subscriber add" was skipped * walmgr restore: better detection if old postmaster is running (Charles Duffy) * walmgr xrestore: detect the death of parent process * walmgr restore: create pg_tblspc - its required for 8.3 (Zoltán Böszörményi) * walmgr restore: copy old config files over if exist (Zoltán Böszörményi) = Features = * Table name globbing for Londiste commands (Erik Jones) * New manpages for scripts (Asko Oja & me) * Upgrade script with manpage: scripts/skytools_upgrade.py * Add .version() function to pgq_ext & londiste schemas. * pgqadm: allow parameters without queue_ prefix in 'config' command. * skytools Python module: - intern() keys in db_urldecode() to decrease memory usage - udp-logger: more fields: hostaddr, service_name - udp-logger: dont cache udp socket, seem to hang in some cases - DBScript.get_database() allows explicit connect string - DBScript allows disabling config file loading - magic_insert on dicts allows missing columns, uses NULL - new parsing functions: - parse_pgarray(), parse_logtriga_sql(), parse_tabbed_table(), - parse_statements() - this one is used to split SQL install files to separate statements. - exists_function() checks both 'public' and 'pg_catalog' for unqualified functions names. - skytools.quoting: add C implementation for quote_literal, quote_copy, quote_bytea_raw, unescape, db_urlencode, db_urldecode. This gives 20x speedup for db_urlencode and 50x for db_urldecode on some real-life data. - unquote_ident(), unquote_literal() 2007-11-19 - SkyTools 2.1.5 - "Enterprise-Grade Duct Tape" = Big changes = * Lot of new docs [Dimitri Fontaine, Asko Oja, Marko Kreen] * Support for fkey and trigger handling in Londiste. [Erik Jones] * Rewrite pgq.insert_event() and log triggers in C, thus SkyTools does not depend on PL/Python anymore. = Small changes = * pgq+txid: convert to new API appearing in 8.3 /contrib/txid/ * Support psycopg2, preferring it to psycopg1. * Improved bulk_loader, using temp tables exclusively. * skytools.config: API change to allow usage without config file. * skytools module: quote_ident(), quote_fqident() * install .sql files under share/skytools in addition to contrib/ * pgqadm: also vacuums londiste and pgq_ext tables, if they exist * londiste: provider add/remove --all [Hans-Juergen Schoenig] * backend modules support 8.3 * pgq: switch pgq_lazy_fetch=NROWS for pgq.Consumer, which makes it use cursor for event fetching, thus allowing larger batches * txid: use bsearch() for larger snapshots = Fixes = * londiste fkeys: look also at dependers not only dependencies. * pgq.consumer: make queue seeking in case of failover more strict. * scriptmgr: dont die on user error. * pgq: there was still fallout from reorg - 2 missing indexes. * Due to historical reasons SerialConsumer (and thus Londiste) accessed completed tick table directly, not via functions. Make it use functions again. * londiste: set client_encoding on subscriber same as on provider * londiste: remove tbl should work also if table is already dropped [Dimitri Fontaine] * couple walmgr fixed [Martin Pihlak] = Upgrade procedure for database code = * PgQ (used on Londiste provider side), table structure, plpgsql functions: $ psql dbname -f upgrade/final/v2.1.5.pgq_core.sql * PgQ new insert_event(), written in C: $ psql dbname -f sql/pgq/lowlevel/pgq_lowlevel.sql * PgQ new triggers (sqltriga, logtriga, logutriga), written in C: $ psql dbname -f sql/pgq/triggers/pgq_triggers.sql * Londiste (both provider and subscriber side) $ psql dbname -f upgrade/final/v2.1.5.londiste.sql * pgq_ext: $ psql dbname -f upgrade/final/v2.1.5.pgq_ext.sql 2007-04-16 - SkyTools 2.1.4 - "Sweets from last Christmas" = Fixes = * logtriga.c was supposed to survive mismatched column string, but the logic were buggy. Thanks go to Dmitriy V'jukov for good analysis. * Couple of scripts were not converted to new API. Fix it. * Quiet a warning in textbuf.c * Make configure and Makefiles survive on BSD's where 'make' is not GNU make. Thanks to Hans-Juergen Schoening. = Features = * Published WalMgr was an old version. Sync with internal code, where Martin has done lot of enhancements. * Small incompat change in PGQ: add fields to pgq.get_consumer_info() return type. Example upgrade procedure: DROP TYPE pgq.ret_consumer_info cascade; \i structure/types.sql \i functions/pgq.get_consumer_info.sql It will show some errors but thats ok. Its annoying but needed for the tick_id cleanup in SerialConsumer/Londiste. 2007-04-10 - SkyTools 2.1.3 - "Brown Paper Bag" Still managed to sneak in a last-minute typo. * Fix copy-paste error in table_copy.py * Remember to bump version in pgq.version() 2007-04-09 - SkyTools 2.1.2 - "Help screen works" Most fallout from reorg is hopefully cleaned now. * Dumb bug in ticker wich made it almost non-working, except it managed to complete the regression test... * Move --skip-truncate switch from 'copy' to 'londiste add'. * 'londiste install' also installs plpgsql+plpythonu. * SQL installer logs full path to file it found. * Change pgq.get_queue_info() FOR loop variable to record instead text that was reported to fail, although it did work here. * Remember where the SQL files were installed. 2007-04-06 - SkyTools 2.1.1 - "Needs more thrust to fly" SkyTools got big reorg before release, but with the hoopla with the 3 projects at once, it did not get much testing... There are some untested areas still, but at least pgq/londiste are in better shape now. * pgqadm: finish conversion... * londiste.Syncer: - Convert to new API - Avoid ticking, parallel ticks are dangerous - Bad arg in repair * pgq: - too aggressive check in register_consumer - Algo desc for batch_event_sql * Add some knobs to make regtests for londiste pass more predictibly. 2007-03-13 - SkyTools 2.1 - "Radioactive Candy" * Final public release. skytools-2.1.13/doc/0000755000175000017500000000000011727601174013244 5ustar markomarkoskytools-2.1.13/doc/pgq-nodupes.txt0000644000175000017500000000266311670174255016260 0ustar markomarko= Avoiding duplicate events = It is pretty burdensome to check if event is already processed, especially on bulk data moving. Here's a way how this can be avoided. First, consumer must guarantee that it processes all events in one tx. Consumer itself can tag events for retry, but then it must be able to handle them later. == Only one db == If the PgQ queue and event data handling happen in same database, the consumer must simply call pgq.finish_batch() inside the event-processing transaction. == Several databases == If the event processing happens in different database, the consumer must store the batch_id into destination database, inside the same transaction as the event processing happens. - Only after committing it, consumer can call pgq.finish_batch() in queue database and commit that. - As the batches come in sequence, there's no need to remember full log of batch_id's, it's enough to keep the latest batch_id. - Then at the start of every batch, consumer can check if the batch_id already exists in destination database, and if it does, then just tag batch done, without processing. With this, there's no need for consumer to check for already processed events. == Note == This assumes the event processing is transaction-able - failures will be rollbacked. If event processing includes communication with world outside database, eg. sending email, such handling won't work. skytools-2.1.13/doc/queue_splitter.10000644000175000017500000001473611727600402016404 0ustar markomarko'\" t .\" Title: queue_splitter .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "QUEUE_SPLITTER" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" queue_splitter \- PgQ consumer that transports events from one queue into several target queues .SH "SYNOPSIS" .sp .nf queue_splitter\&.py [switches] config\&.ini .fi .SH "DESCRIPTION" .sp queue_spliter is PgQ consumer that transports events from source queue into several target queues\&. ev_extra1 field in each event shows into which target queue it must go\&. (pgq\&.logutriga() puts there the table name\&.) .sp One use case is to move events from OLTP database to batch processing server\&. By using queue spliter it is possible to move all kinds of events for batch processing with one consumer thus keeping OLTP database less crowded\&. .SH "QUICK-START" .sp Basic queue_splitter setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} pgq must be installed both in source and target databases\&. See pgqadm man page for details\&. Target database must also have pgq_ext schema installed\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} edit a queue_splitter configuration file, say queue_splitter_sourcedb_sourceq_targetdb\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} create source and target queues .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini create .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} launch queue splitter in daemon mode .sp .if n \{\ .RS 4 .\} .nf $ queue_splitter\&.py queue_splitter_sourcedb_sourceq_targetdb\&.ini \-d .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} start producing and consuming events .RE .SH "CONFIG" .SS "Common configuration parameters" .PP job_name .RS 4 Name for particulat job the script does\&. Script will log under this name to logdb/logserver\&. The name is also used as default for PgQ consumer name\&. It should be unique\&. .RE .PP pidfile .RS 4 Location for pid file\&. If not given, script is disallowed to daemonize\&. .RE .PP logfile .RS 4 Location for log file\&. .RE .PP loop_delay .RS 4 If continuisly running process, how long to sleep after each work loop, in seconds\&. Default: 1\&. .RE .PP connection_lifetime .RS 4 Close and reconnect older database connections\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. .RE .SS "Common PgQ consumer parameters" .PP pgq_queue_name .RS 4 Queue name to attach to\&. No default\&. .RE .PP pgq_consumer_id .RS 4 Consumers ID to use when registering\&. Default: %(job_name)s .RE .SS "queue_splitter parameters" .PP src_db .RS 4 Source database\&. .RE .PP dst_db .RS 4 Target database\&. .RE .SS "Example config file" .sp .if n \{\ .RS 4 .\} .nf [queue_splitter] job_name = queue_spliter_sourcedb_sourceq_targetdb .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf src_db = dbname=sourcedb dst_db = dbname=targetdb .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf pgq_queue_name = sourceq .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf logfile = ~/log/%(job_name)s\&.log pidfile = ~/pid/%(job_name)s\&.pid .fi .if n \{\ .RE .\} .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE .SH "USECASE" .sp How to to process events created in secondary database with several queues but have only one queue in primary database\&. This also shows how to insert events into queues with regular SQL easily\&. .sp .if n \{\ .RS 4 .\} .nf CREATE SCHEMA queue; CREATE TABLE queue\&.event1 ( \-\- this should correspond to event internal structure \-\- here you can put checks that correct data is put into queue id int4, name text, \-\- not needed, but good to have: primary key (id) ); \-\- put data into queue in urlencoded format, skip actual insert CREATE TRIGGER redirect_queue1_trg BEFORE INSERT ON queue\&.event1 FOR EACH ROW EXECUTE PROCEDURE pgq\&.logutriga(\*(Aqsinglequeue\*(Aq, \*(AqSKIP\*(Aq); \-\- repeat the above for event2 .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf \-\- now the data can be inserted: INSERT INTO queue\&.event1 (id, name) VALUES (1, \*(Aquser\*(Aq); .fi .if n \{\ .RE .\} .sp If the queue_splitter is put on "singlequeue", it spreads the event on target to queues named "queue\&.event1", "queue\&.event2", etc\&. This keeps PgQ load on primary database minimal both CPU\-wise and maintenance\-wise\&. skytools-2.1.13/doc/queue_splitter.txt0000644000175000017500000000534111670174255017064 0ustar markomarko= queue_splitter(1) = == NAME == queue_splitter - PgQ consumer that transports events from one queue into several target queues == SYNOPSIS == queue_splitter.py [switches] config.ini == DESCRIPTION == queue_spliter is PgQ consumer that transports events from source queue into several target queues. `ev_extra1` field in each event shows into which target queue it must go. (`pgq.logutriga()` puts there the table name.) One use case is to move events from OLTP database to batch processing server. By using queue spliter it is possible to move all kinds of events for batch processing with one consumer thus keeping OLTP database less crowded. == QUICK-START == Basic queue_splitter setup and usage can be summarized by the following steps: 1. pgq must be installed both in source and target databases. See pgqadm man page for details. Target database must also have pgq_ext schema installed. 2. edit a queue_splitter configuration file, say queue_splitter_sourcedb_sourceq_targetdb.ini 3. create source and target queues $ pgqadm.py ticker.ini create 4. launch queue splitter in daemon mode $ queue_splitter.py queue_splitter_sourcedb_sourceq_targetdb.ini -d 5. start producing and consuming events == CONFIG == include::common.config.txt[] === queue_splitter parameters === src_db:: Source database. dst_db:: Target database. === Example config file === [queue_splitter] job_name = queue_spliter_sourcedb_sourceq_targetdb src_db = dbname=sourcedb dst_db = dbname=targetdb pgq_queue_name = sourceq logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid == COMMAND LINE SWITCHES == include::common.switches.txt[] == USECASE == How to to process events created in secondary database with several queues but have only one queue in primary database. This also shows how to insert events into queues with regular SQL easily. CREATE SCHEMA queue; CREATE TABLE queue.event1 ( -- this should correspond to event internal structure -- here you can put checks that correct data is put into queue id int4, name text, -- not needed, but good to have: primary key (id) ); -- put data into queue in urlencoded format, skip actual insert CREATE TRIGGER redirect_queue1_trg BEFORE INSERT ON queue.event1 FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('singlequeue', 'SKIP'); -- repeat the above for event2 -- now the data can be inserted: INSERT INTO queue.event1 (id, name) VALUES (1, 'user'); If the queue_splitter is put on "singlequeue", it spreads the event on target to queues named "queue.event1", "queue.event2", etc. This keeps PgQ load on primary database minimal both CPU-wise and maintenance-wise. skytools-2.1.13/doc/pgqadm.10000644000175000017500000002172111727600371014600 0ustar markomarko'\" t .\" Title: pgqadm .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "PGQADM" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" pgqadm \- PgQ ticker and administration interface .SH "SYNOPSIS" .sp .nf pgqadm\&.py [option] config\&.ini command [arguments] .fi .SH "DESCRIPTION" .sp PgQ is Postgres based event processing system\&. It is part of SkyTools package that contains several useful implementations on this engine\&. Main function of PgQadm is to maintain and keep healthy both pgq internal tables and tables that store events\&. .sp SkyTools is scripting framework for Postgres databases written in Python that provides several utilities and implements common database handling logic\&. .sp Event \- atomic piece of data created by Producers\&. In PgQ event is one record in one of tables that services that queue\&. Event record contains some system fields for PgQ and several data fileds filled by Producers\&. PgQ is neither checking nor enforcing event type\&. Event type is someting that consumer and produser must agree on\&. PgQ guarantees that each event is seen at least once but it is up to consumer to make sure that event is processed no more than once if that is needed\&. .sp Batch \- PgQ is designed for efficiency and high throughput so events are grouped into batches for bulk processing\&. Creating these batches is one of main tasks of PgQadm and there are several parameters for each queue that can be use to tune size and frequency of batches\&. Consumerss receive events in these batches and depending on business requirements process events separately or also in batches\&. .sp Queue \- Event are stored in queue tables i\&.e queues\&. Several producers can write into same queeu and several consumers can read from the queue\&. Events are kept in queue until all the consumers have seen them\&. We use table rotation to decrease hard disk io\&. Queue can contain any number of event types it is up to Producer and Consumer to agree on what types of events are passed and how they are encoded For example Londiste producer side can produce events for more tables tan consumer side needs so consumer subscribes only to those tables it needs and events for other tables are ignores\&. .sp Producer \- applicatione that pushes event into queue\&. Prodecer can be written in any langaage that is able to run stored procedures in Postgres\&. .sp Consumer \- application that reads events from queue\&. Consumers can be written in any language that can interact with Postgres\&. SkyTools package contains several useful consumers written in Python that can be used as they are or as good starting points to write more complex consumers\&. .SH "QUICK-START" .sp Basic PgQ setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} create the database .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} edit a PgQ ticker configuration file, say ticker\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} install PgQ internal tables .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini install .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} launch the PgQ ticker on databse machine as daemon .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py \-d ticker\&.ini ticker .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} create queue .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini create .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 6.\h'+01'\c .\} .el \{\ .sp -1 .IP " 6." 4.2 .\} register or run consumer to register it automatically .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini register .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 7.\h'+01'\c .\} .el \{\ .sp -1 .IP " 7." 4.2 .\} start producing events .RE .SH "CONFIG" .sp .if n \{\ .RS 4 .\} .nf [pgqadm] job_name = pgqadm_somedb .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf db = dbname=somedb .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # how often to run maintenance [seconds] maint_delay = 600 .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # how often to check for activity [seconds] loop_delay = 0\&.1 .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf logfile = ~/log/%(job_name)s\&.log pidfile = ~/pid/%(job_name)s\&.pid .fi .if n \{\ .RE .\} .SH "COMMANDS" .SS "ticker" .sp Start ticking & maintenance process\&. Usually run as daemon with \-d option\&. Must be running for PgQ to be functional and for consumers to see any events\&. .SS "status" .sp Show overview of registered queues and consumers and queue health\&. This command is used when you want to know what is happening inside PgQ\&. .SS "install" .sp Installs PgQ schema into database from config file\&. .SS "create " .sp Create queue tables into pgq schema\&. As soon as queue is created producers can start inserting events into it\&. But you must be aware that if there are no consumers on the queue the events are lost until consumer is registered\&. .SS "drop " .sp Drop queue and all it\(cqs consumers from PgQ\&. Queue tables are dropped and all the contents are lost forever so use with care as with most drop commands\&. .SS "register " .sp Register given consumer to listen to given queue\&. First batch seen by this consumer is the one completed after registration\&. Registration happens automatically when consumer is run first time so using this command is optional but may be needed when producers start producing events before consumer can be run\&. .SS "unregister " .sp Removes consumer from given queue\&. Note consumer must be stopped before issuing this command otherwise it automatically registers again\&. .SS "config [ [= \&... ]]" .sp Show or change queue config\&. There are several parameters that can be set for each queue shown here with default values: .PP queue_ticker_max_lag (2) .RS 4 If no tick has happend during given number of seconds then one is generated just to keep queue lag in control\&. It may be increased if there is no need to deliver events fast\&. Not much room to decrease it :) .RE .PP queue_ticker_max_count (200) .RS 4 Threshold number of events in filling batch that triggers tick\&. Can be increased to encourage PgQ to create larger batches or decreased to encourage faster ticking with smaller batches\&. .RE .PP queue_ticker_idle_period (60) .RS 4 Number of seconds that can pass without ticking if no events are coming to queue\&. These empty ticks are used as keep alive signals for batch jobs and monitoring\&. .RE .PP queue_rotation_period (2 hours) .RS 4 Interval of time that may pass before PgQ tries to rotate tables to free up space\&. Not PgQ can not rotate tables if there are long transactions in database like VACUUM or pg_dump\&. May be decreased if low on disk space or increased to keep longer history of old events\&. To small values might affect performance badly because postgres tends to do seq scans on small tables\&. Too big values may waste disk space\&. .RE .sp Looking at queue config\&. .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py mydb\&.ini config testqueue queue_ticker_max_lag = 3 queue_ticker_max_count = 500 queue_ticker_idle_period = 60 queue_rotation_period = 7200 $ pgqadm\&.py conf/pgqadm_myprovider\&.ini config testqueue queue_ticker_max_lag=10 queue_ticker_max_count=300 Change queue bazqueue config to: queue_ticker_max_lag=\*(Aq10\*(Aq, queue_ticker_max_count=\*(Aq300\*(Aq $ .fi .if n \{\ .RE .\} .SH "COMMON OPTIONS" .PP \-h, \-\-help .RS 4 show help message .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program verbose .RE .PP \-d, \-\-daemon .RS 4 go background .RE .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE skytools-2.1.13/doc/cube_dispatcher.10000644000175000017500000002027311727600375016460 0ustar markomarko'\" t .\" Title: cube_dispatcher .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "CUBE_DISPATCHER" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" cube_dispatcher \- PgQ consumer that is used to write source records into partitoned tables .SH "SYNOPSIS" .sp .nf cube_dispatcher\&.py [switches] config\&.ini .fi .SH "DESCRIPTION" .sp cube_dispatcher is PgQ consumer that reads url encoded records from source queue and writes them into partitioned tables according to configuration file\&. Used to prepare data for business intelligence\&. Name of the table is read from producer field in event\&. Batch creation time is used for partitioning\&. All records created in same day will go into same table partion\&. If partiton does not exist cube dispatcer will create it according to template\&. .sp Events are usually procuded by pgq\&.logutriga()\&. Logutriga adds all the data of the record into the event (also in case of updates and deletes)\&. .sp cube_dispatcher can be used in to modes: .PP keep_all .RS 4 keeps all the data that comes in\&. If record is updated several times during one day then table partiton for that day will contain several instances of that record\&. .RE .PP keep_latest .RS 4 only last instance of each record is kept for each day\&. That also means that all tables must have primary keys so cube dispatcher can delete previous versions of records before inserting new data\&. .RE .SH "QUICK-START" .sp Basic cube_dispatcher setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} pgq and logutriga must be installed in source databases\&. See pgqadm man page for details\&. target database must also have pgq_ext schema\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} edit a cube_dispatcher configuration file, say cube_dispatcher_sample\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} create source queue .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini create .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} create target database and parent tables in it\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} launch cube dispatcher in daemon mode .sp .if n \{\ .RS 4 .\} .nf $ cube_dispatcher\&.py cube_dispatcher_sample\&.ini \-d .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 6.\h'+01'\c .\} .el \{\ .sp -1 .IP " 6." 4.2 .\} start producing events (create logutriga trggers on tables) CREATE OR REPLACE TRIGGER trig_cube_replica AFTER INSERT OR UPDATE ON some_table FOR EACH ROW EXECUTE PROCEDURE pgq\&.logutriga(\fI\fR) .RE .SH "CONFIG" .SS "Common configuration parameters" .PP job_name .RS 4 Name for particulat job the script does\&. Script will log under this name to logdb/logserver\&. The name is also used as default for PgQ consumer name\&. It should be unique\&. .RE .PP pidfile .RS 4 Location for pid file\&. If not given, script is disallowed to daemonize\&. .RE .PP logfile .RS 4 Location for log file\&. .RE .PP loop_delay .RS 4 If continuisly running process, how long to sleep after each work loop, in seconds\&. Default: 1\&. .RE .PP connection_lifetime .RS 4 Close and reconnect older database connections\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. .RE .SS "Common PgQ consumer parameters" .PP pgq_queue_name .RS 4 Queue name to attach to\&. No default\&. .RE .PP pgq_consumer_id .RS 4 Consumers ID to use when registering\&. Default: %(job_name)s .RE .SS "Config options specific to cube_dispatcher" .PP src_db .RS 4 Connect string for source database where the queue resides\&. .RE .PP dst_db .RS 4 Connect string for target database where the tables should be created\&. .RE .PP mode .RS 4 Operation mode for cube_dispatcher\&. Either keep_all or keep_latest\&. .RE .PP dateformat .RS 4 Optional parameter to specify how to suffix data tables\&. Default is YYYY_MM_DD which creates per\-day tables\&. With YYYY_MM per\-month tables can be created\&. If explicitly set empty, partitioning is disabled\&. .RE .PP part_template .RS 4 SQL fragment for table creation\&. Various magic replacements are done there: .RE .PP _PKEY .RS 4 comma separated list of primery key columns\&. .RE .PP _PARENT .RS 4 schema\-qualified parent table name\&. .RE .PP _DEST_TABLE .RS 4 schema\-qualified partition table\&. .RE .PP _SCHEMA_TABLE .RS 4 same as \fIDEST_TABLE but dots replaced with "_\fR", to allow use as index names\&. .RE .SS "Example config file" .sp .if n \{\ .RS 4 .\} .nf [cube_dispatcher] job_name = some_queue_to_cube .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf src_db = dbname=sourcedb_test dst_db = dbname=dataminedb_test .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf pgq_queue_name = udata\&.some_queue .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf logfile = ~/log/%(job_name)s\&.log pidfile = ~/pid/%(job_name)s\&.pid .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # how many rows are kept: keep_latest, keep_all mode = keep_latest .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # to_char() fmt for table suffix #dateformat = YYYY_MM_DD # following disables table suffixes: #dateformat = .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf part_template = create table _DEST_TABLE (like _PARENT); alter table only _DEST_TABLE add primary key (_PKEY); .fi .if n \{\ .RE .\} .SH "LOGUTRIGA EVENT FORMAT" .sp PgQ trigger function pgq\&.logutriga() sends table change event into queue in following format: .PP ev_type .RS 4 (op || ":" || pkey_fields)\&. Where op is either "I", "U" or "D", corresponging to insert, update or delete\&. And pkey_fields is comma\-separated list of primary key fields for table\&. Operation type is always present but pkey_fields list can be empty, if table has no primary keys\&. Example: I:col1,col2 .RE .PP ev_data .RS 4 Urlencoded record of data\&. It uses db\-specific urlecoding where existence of \fI=\fR is meaningful \- missing \fI=\fR means NULL, present \fI=\fR means literal value\&. Example: id=3&name=str&nullvalue&emptyvalue= .RE .PP ev_extra1 .RS 4 Fully qualified table name\&. .RE .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE skytools-2.1.13/doc/pgq-sql.txt0000644000175000017500000001021111670174255015366 0ustar markomarko= PgQ - queue for PostgreSQL = == Queue creation == pgq.create_queue(queue_name text) Initialize event queue. Returns 0 if event queue already exists, 1 otherwise. == Producer == pgq.insert_event(queue_name text, ev_type, ev_data) pgq.insert_event(queue_name text, ev_type, ev_data, extra1, extra2, extra3, extra4) Generate new event. This should be called inside main tx - thus rollbacked with it if needed. == Consumer == pgq.register_consumer(queue_name text, consumer_id text) Attaches this consumer to particular event queue. Returns 0 if the consumer was already attached, 1 otherwise. pgq.unregister_consumer(queue_name text, consumer_id text) Unregister and drop resources allocated to customer. pgq.next_batch(queue_name text, consumer_id text) Allocates next batch of events to consumer. Returns batch id (int8), to be used in processing functions. If no batches are available, returns NULL. That means that the ticker has not cut them yet. This is the appropriate moment for consumer to sleep. pgq.get_batch_events(batch_id int8) `pgq.get_batch_events()` returns a set of events in this batch. There may be no events in the batch. This is normal. The batch must still be closed with pgq.finish_batch(). Event fields: (ev_id int8, ev_time timestamptz, ev_txid int8, ev_retry int4, ev_type text, ev_data text, ev_extra1, ev_extra2, ev_extra3, ev_extra4) pgq.event_failed(batch_id int8, event_id int8, reason text) Tag event as 'failed' - it will be stored, but not further processing is done. pgq.event_retry(batch_id int8, event_id int8, retry_seconds int4) Tag event for 'retry' - after x seconds the event will be re-inserted into main queue. pgq.finish_batch(batch_id int8) Tag batch as finished. Until this is not done, the consumer will get same batch again. After calling finish_batch consumer cannot do any operations with events of that batch. All operations must be done before. == Failed queue operation == Events tagged as failed just stay on their queue. Following functions can be used to manage them. pgq.failed_event_list(queue_name, consumer) pgq.failed_event_list(queue_name, consumer, cnt, offset) pgq.failed_event_count(queue_name, consumer) Get info about the queue. Event fields are same as for pgq.get_batch_events() pgq.failed_event_delete(queue_name, consumer, event_id) pgq.failed_event_retry(queue_name, consumer, event_id) Remove an event from queue, or retry it. == Info operations == pgq.get_queue_info() Get list of queues. Result: (queue_name, queue_ntables, queue_cur_table, queue_rotation_period, queue_switch_time, queue_external_ticker, queue_ticker_max_count, queue_ticker_max_lag, queue_ticker_idle_period, ticker_lag) pgq.get_consumer_info() pgq.get_consumer_info(queue_name) pgq.get_consumer_info(queue_name, consumer) Get list of active consumers. Result: (queue_name, consumer_name, lag, last_seen, last_tick, current_batch, next_tick) pgq.get_batch_info(batch_id) Get info about batch. Result fields: (queue_name, consumer_name, batch_start, batch_end, prev_tick_id, tick_id, lag) == Notes == Consumer *must* be able to process same event several times. == Example == First, create event queue: select pgq.create_queue('LogEvent'); Then, producer side can do whenever it wishes: select pgq.insert_event('LogEvent', 'data', 'DataFor123'); First step for consumer is to register: select pgq.register_consumer('LogEvent', 'TestConsumer'); Then it can enter into consuming loop: begin; select pgq.next_batch('LogEvent', 'TestConsumer'); [into batch_id] commit; That will reserve a batch of events for this consumer. To see the events in batch: select * from pgq.get_batch_events(batch_id); That will give all events in batch. The processing does not need to be happen all in one transaction, framework can split the work how it wants. If a events failed or needs to be tried again, framework can call: select pgq.event_retry(batch_id, event_id, 60); select pgq.event_failed(batch_id, event_id, 'Record deleted'); When all done, notify database about it: select pgq.finish_batch(batch_id) skytools-2.1.13/doc/common.logutriga.txt0000644000175000017500000000131211670174255017270 0ustar markomarko PgQ trigger function `pgq.logutriga()` sends table change event into queue in following format: ev_type:: `(op || ":" || pkey_fields)`. Where op is either "I", "U" or "D", corresponging to insert, update or delete. And `pkey_fields` is comma-separated list of primary key fields for table. Operation type is always present but pkey_fields list can be empty, if table has no primary keys. Example: `I:col1,col2` ev_data:: Urlencoded record of data. It uses db-specific urlecoding where existence of '=' is meaningful - missing '=' means NULL, present '=' means literal value. Example: `id=3&name=str&nullvalue&emptyvalue=` ev_extra1:: Fully qualified table name. skytools-2.1.13/doc/londiste.10000644000175000017500000003005611727600366015155 0ustar markomarko'\" t .\" Title: londiste .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "LONDISTE" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" londiste \- PostgreSQL replication engine written in python .SH "SYNOPSIS" .sp .nf londiste\&.py [option] config\&.ini command [arguments] .fi .SH "DESCRIPTION" .sp Londiste is the PostgreSQL replication engine portion of the SkyTools suite, by Skype\&. This suite includes packages implementing specific replication tasks and/or solutions in layers, building upon each other\&. .sp PgQ is a generic queue implementation based on ideas from Slony\-I\(cqs snapshot based event batching\&. Londiste uses PgQ as its transport mechanism to implement a robust and easy to use replication solution\&. .sp Londiste is an asynchronous master\-slave(s) replication system\&. Asynchronous means that a transaction commited on the master is not guaranteed to have made it to any slave at the master\(cqs commit time; and master\-slave means that data changes on slaves are not reported back to the master, it\(cqs the other way around only\&. .sp The replication is trigger based, and you choose a set of tables to replicate from the provider to the subscriber(s)\&. Any data changes occuring on the provider (in a replicated table) will fire the londiste trigger, which fills a queue of events for any subscriber(s) to care about\&. .sp A replay process consumes the queue in batches, and applies all given changes to any subscriber(s)\&. The initial replication step involves using the PostgreSQL\(cqs COPY command for efficient data loading\&. .SH "QUICK-START" .sp Basic londiste setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} create the subscriber database, with tables to replicate .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} edit a londiste configuration file, say conf\&.ini, and a PgQ ticker configuration file, say ticker\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} install londiste on the provider and subscriber nodes\&. This step requires admin privileges on both provider and subscriber sides, and both install commands can be run remotely: .sp .if n \{\ .RS 4 .\} .nf $ londiste\&.py conf\&.ini provider install $ londiste\&.py conf\&.ini subscriber install .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} launch the PgQ ticker on the provider machine: .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py \-d ticker\&.ini ticker .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} launch the londiste replay process: .sp .if n \{\ .RS 4 .\} .nf $ londiste\&.py \-d conf\&.ini replay .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 6.\h'+01'\c .\} .el \{\ .sp -1 .IP " 6." 4.2 .\} add tables to replicate from the provider database: .sp .if n \{\ .RS 4 .\} .nf $ londiste\&.py conf\&.ini provider add table1 table2 \&.\&.\&. .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 7.\h'+01'\c .\} .el \{\ .sp -1 .IP " 7." 4.2 .\} add tables to replicate to the subscriber database: .sp .if n \{\ .RS 4 .\} .nf $ londiste\&.py conf\&.ini subscriber add table1 table2 \&.\&.\&. .fi .if n \{\ .RE .\} .RE .sp To replicate to more than one subscriber database just repeat each of the described subscriber steps for each subscriber\&. .SH "COMMANDS" .sp The londiste command is parsed globally, and has both options and subcommands\&. Some options are reserved to a subset of the commands, and others should be used without any command at all\&. .SH "GENERAL OPTIONS" .sp This section presents options available to all and any londiste command\&. .PP \-h, \-\-help .RS 4 show this help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .SH "PROVIDER COMMANDS" .sp .if n \{\ .RS 4 .\} .nf $ londiste\&.py config\&.ini provider .fi .if n \{\ .RE .\} .sp Where command is one of: .SS "provider install" .sp Installs code into provider and subscriber database and creates queue\&. Equivalent to doing following by hand: .sp .if n \{\ .RS 4 .\} .nf CREATE LANGUAGE plpgsql; CREATE LANGUAGE plpython; \ei \&.\&.\&./contrib/txid\&.sql \ei \&.\&.\&./contrib/pgq\&.sql \ei \&.\&.\&./contrib/londiste\&.sql select pgq\&.create_queue(queue name); .fi .if n \{\ .RE .\} .SS "provider add \&..." .sp Registers table(s) on the provider database and adds the londiste trigger to the table(s) which will send events to the queue\&. Table names can be schema qualified with the schema name defaulting to public if not supplied\&. .PP \-\-all .RS 4 Register all tables in provider database, except those that are under schemas \fIpgq\fR, \fIlondiste\fR, \fIinformation_schema\fR or \fIpg_*\fR\&. .RE .SS "provider remove
\&..." .sp Unregisters table(s) on the provider side and removes the londiste triggers from the table(s)\&. The table removal event is also sent to the queue, so all subscribers unregister the table(s) on their end as well\&. Table names can be schema qualified with the schema name defaulting to public if not supplied\&. .SS "provider add\-seq \&..." .sp Registers a sequence on provider\&. .SS "provider remove\-seq \&..." .sp Unregisters a sequence on provider\&. .SS "provider tables" .sp Shows registered tables on provider side\&. .SS "provider seqs" .sp Shows registered sequences on provider side\&. .SH "SUBSCRIBER COMMANDS" .sp .if n \{\ .RS 4 .\} .nf londiste\&.py config\&.ini subscriber .fi .if n \{\ .RE .\} .sp Where command is one of: .SS "subscriber install" .sp Installs code into subscriber database\&. Equivalent to doing following by hand: .sp .if n \{\ .RS 4 .\} .nf CREATE LANGUAGE plpgsql; \ei \&.\&.\&./contrib/londiste\&.sql .fi .if n \{\ .RE .\} .sp This will be done under the Postgres Londiste user, if the tables should be owned by someone else, it needs to be done by hand\&. .SS "subscriber add
\&..." .sp Registers table(s) on subscriber side\&. Table names can be schema qualified with the schema name defaulting to public if not supplied\&. .sp Switches (optional): .PP \-\-all .RS 4 Add all tables that are registered on provider to subscriber database .RE .PP \-\-force .RS 4 Ignore table structure differences\&. .RE .PP \-\-expect\-sync .RS 4 Table is already synced by external means so initial COPY is unnecessary\&. .RE .PP \-\-skip\-truncate .RS 4 When doing initial COPY, don\(cqt remove old data\&. .RE .SS "subscriber remove
\&..." .sp Unregisters table(s) from subscriber\&. No events will be applied to the table anymore\&. Actual table will not be touched\&. Table names can be schema qualified with the schema name defaulting to public if not supplied\&. .SS "subscriber add\-seq \&..." .sp Registers a sequence on subscriber\&. .SS "subscriber remove\-seq \&..." .sp Unregisters a sequence on subscriber\&. .SS "subscriber resync
\&..." .sp Tags table(s) as "not synced"\&. Later the replay process will notice this and launch copy process(es) to sync the table(s) again\&. .SS "subscriber tables" .sp Shows registered tables on the subscriber side, and the current state of each table\&. Possible state values are: .PP NEW .RS 4 the table has not yet been considered by londiste\&. .RE .PP in\-copy .RS 4 Full\-table copy is in progress\&. .RE .PP catching\-up .RS 4 Table is copied, missing events are replayed on to it\&. .RE .PP wanna\-sync: .RS 4 The "copy" process catched up, wants to hand the table over to "replay"\&. .RE .PP do\-sync: .RS 4 "replay" process is ready to accept it\&. .RE .PP ok .RS 4 table is in sync\&. .RE .SS "subscriber fkeys" .sp Show pending and active foreign keys on tables\&. Takes optional type argument \- pending or active\&. If no argument is given, both types are shown\&. .sp Pending foreign keys are those that were removed during COPY time but have not restored yet, The restore happens autmatically if both tables are synced\&. .SS "subscriber triggers" .sp Show pending and active triggers on tables\&. Takes optional type argument \- pending or active\&. If no argument is given, both types are shown\&. .sp Pending triggers keys are those that were removed during COPY time but have not restored yet, The restore of triggers does not happen autmatically, it needs to be done manually with restore\-triggers command\&. .SS "subscriber restore\-triggers
" .sp Restores all pending triggers for single table\&. Optionally trigger name can be given as extra argument, then only that trigger is restored\&. .SS "subscriber register" .sp Register consumer on queue\&. This usually happens automatically when replay is launched, but .SS "subscriber unregister" .sp Unregister consumer from provider\(cqs queue\&. This should be done if you want to shut replication down\&. .SH "REPLICATION COMMANDS" .SS "replay" .sp The actual replication process\&. Should be run as daemon with \-d switch, because it needs to be always running\&. .sp It\(cqs main task is to get batches of events from PgQ and apply them to subscriber database\&. .sp Switches: .PP \-d, \-\-daemon .RS 4 go background .RE .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE .SH "UTILITY COMMAND" .SS "repair
\&..." .sp Attempts to achieve a state where the table(s) is/are in sync, compares them, and writes out SQL statements that would fix differences\&. .sp Syncing happens by locking provider tables against updates and then waiting until the replay process has applied all pending changes to subscriber database\&. As this is dangerous operation, it has a hardwired limit of 10 seconds for locking\&. If the replay process does not catch up in that time, the locks are released and the repair operation is cancelled\&. .sp Comparing happens by dumping out the table contents of both sides, sorting them and then comparing line\-by\-line\&. As this is a CPU and memory\-hungry operation, good practice is to run the repair command on a third machine to avoid consuming resources on either the provider or the subscriber\&. .SS "compare
\&..." .sp Syncs tables like repair, but just runs SELECT count(*) on both sides to get a little bit cheaper, but also less precise, way of checking if the tables are in sync\&. .SH "CONFIGURATION" .sp Londiste and PgQ both use INI configuration files, your distribution of skytools include examples\&. You often just have to edit the database connection strings, namely db in PgQ ticker\&.ini and provider_db and subscriber_db in londiste conf\&.ini as well as logfile and pidfile to adapt to you system paths\&. .sp See londiste(5)\&. .SH "SEE ALSO" .sp londiste(5) .sp \m[blue]\fBhttps://developer\&.skype\&.com/SkypeGarage/DbProjects/SkyTools/\fR\m[] .sp \m[blue]\fBReference guide\fR\m[]\&\s-2\u[1]\d\s+2 .SH "NOTES" .IP " 1." 4 Reference guide .RS 4 \%http://skytools.projects.postgresql.org/doc/londiste.ref.html .RE skytools-2.1.13/doc/skytools_upgrade.txt0000644000175000017500000000137711670174255017415 0ustar markomarko= skytools_upgrade(1) = == NAME == skytools_upgrade - utility for upgrading Skytools code in databases. == SYNOPSIS == skytools_upgrade.py connstr [connstr ..] == DESCRIPTION == It connects to given database, then looks for following schemas: pgq:: Main PgQ code. pgq_ext:: PgQ batch/event tracking in remote database. londiste:: Londiste replication. If schema exists, its version is detected by querying .version() function under schema. If the function does not exists, there is some heiristics built in to differentiate between 2.1.4 and 2.1.5 version of ther schemas. If detected that version is older that current, it is upgraded by applying upgrade scripts in order. == COMMAND LINE SWITCHES == include::common.switches.txt[] skytools-2.1.13/doc/bulk_loader.txt0000644000175000017500000000535411670174255016301 0ustar markomarko = bulk_loader(1) = == NAME == bulk_loader - PgQ consumer that loads urlencoded records to slow databases == SYNOPSIS == bulk_loader.py [switches] config.ini == DESCRIPTION == bulk_loader is PgQ consumer that reads url encoded records from source queue and writes them into tables according to configuration file. It is targeted to slow databases that cannot handle applying each row as separate statement. Originally written for BizgresMPP/greenplumDB which have very high per-statement overhead, but can also be used to load regular PostgreSQL database that cannot manage regular replication. Behaviour properties: - reads urlencoded "logutriga" records. - does not do partitioning, but allows optionally redirect table events. - does not keep event order. - always loads data with COPY, either directly to main table (INSERTs) or to temp tables (UPDATE/COPY) then applies from there. Events are usually procuded by `pgq.logutriga()`. Logutriga adds all the data of the record into the event (also in case of updates and deletes). == QUICK-START == Basic bulk_loader setup and usage can be summarized by the following steps: 1. pgq and logutriga must be installed in source databases. See pgqadm man page for details. target database must also have pgq_ext schema. 2. edit a bulk_loader configuration file, say bulk_loader_sample.ini 3. create source queue $ pgqadm.py ticker.ini create 4. Tune source queue to have big batches: $ pgqadm.py ticker.ini config ticker_max_count="10000" ticker_max_lag="10 minutes" ticker_idle_period="10 minutes" 5. create target database and tables in it. 6. launch bulk_loader in daemon mode $ bulk_loader.py -d bulk_loader_sample.ini 7. start producing events (create logutriga trggers on tables) CREATE OR REPLACE TRIGGER trig_bulk_replica AFTER INSERT OR UPDATE ON some_table FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('') == CONFIG == include::common.config.txt[] === Config options specific to `bulk_loader` === src_db:: Connect string for source database where the queue resides. dst_db:: Connect string for target database where the tables should be created. remap_tables:: Optional parameter for table redirection. Contains comma-separated list of : pairs. Eg: `oldtable1:newtable1, oldtable2:newtable2`. load_method:: Optional parameter for load method selection. Available options: 0:: UPDATE as UPDATE from temp table. This is default. 1:: UPDATE as DELETE+COPY from temp table. 2:: merge INSERTs with UPDATEs, then do DELETE+COPY from temp table. == LOGUTRIGA EVENT FORMAT == include::common.logutriga.txt[] == COMMAND LINE SWITCHES == include::common.switches.txt[] skytools-2.1.13/doc/cube_dispatcher.txt0000644000175000017500000000722511670174255017141 0ustar markomarko = cube_dispatcher(1) = == NAME == cube_dispatcher - PgQ consumer that is used to write source records into partitoned tables == SYNOPSIS == cube_dispatcher.py [switches] config.ini == DESCRIPTION == cube_dispatcher is PgQ consumer that reads url encoded records from source queue and writes them into partitioned tables according to configuration file. Used to prepare data for business intelligence. Name of the table is read from producer field in event. Batch creation time is used for partitioning. All records created in same day will go into same table partion. If partiton does not exist cube dispatcer will create it according to template. Events are usually procuded by `pgq.logutriga()`. Logutriga adds all the data of the record into the event (also in case of updates and deletes). `cube_dispatcher` can be used in to modes: keep_all:: keeps all the data that comes in. If record is updated several times during one day then table partiton for that day will contain several instances of that record. keep_latest:: only last instance of each record is kept for each day. That also means that all tables must have primary keys so cube dispatcher can delete previous versions of records before inserting new data. == QUICK-START == Basic cube_dispatcher setup and usage can be summarized by the following steps: 1. pgq and logutriga must be installed in source databases. See pgqadm man page for details. target database must also have pgq_ext schema. 2. edit a cube_dispatcher configuration file, say cube_dispatcher_sample.ini 3. create source queue $ pgqadm.py ticker.ini create 4. create target database and parent tables in it. 5. launch cube dispatcher in daemon mode $ cube_dispatcher.py cube_dispatcher_sample.ini -d 6. start producing events (create logutriga trggers on tables) CREATE OR REPLACE TRIGGER trig_cube_replica AFTER INSERT OR UPDATE ON some_table FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('') == CONFIG == include::common.config.txt[] === Config options specific to `cube_dispatcher` === src_db:: Connect string for source database where the queue resides. dst_db:: Connect string for target database where the tables should be created. mode:: Operation mode for cube_dispatcher. Either `keep_all` or `keep_latest`. dateformat:: Optional parameter to specify how to suffix data tables. Default is `YYYY_MM_DD` which creates per-day tables. With `YYYY_MM` per-month tables can be created. If explicitly set empty, partitioning is disabled. part_template:: SQL fragment for table creation. Various magic replacements are done there: _PKEY:: comma separated list of primery key columns. _PARENT:: schema-qualified parent table name. _DEST_TABLE:: schema-qualified partition table. _SCHEMA_TABLE:: same as _DEST_TABLE but dots replaced with "__", to allow use as index names. === Example config file === [cube_dispatcher] job_name = some_queue_to_cube src_db = dbname=sourcedb_test dst_db = dbname=dataminedb_test pgq_queue_name = udata.some_queue logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid # how many rows are kept: keep_latest, keep_all mode = keep_latest # to_char() fmt for table suffix #dateformat = YYYY_MM_DD # following disables table suffixes: #dateformat = part_template = create table _DEST_TABLE (like _PARENT); alter table only _DEST_TABLE add primary key (_PKEY); == LOGUTRIGA EVENT FORMAT == include::common.logutriga.txt[] == COMMAND LINE SWITCHES == include::common.switches.txt[] skytools-2.1.13/doc/walmgr.txt0000644000175000017500000002175411670174255015311 0ustar markomarko = walmgr(1) = == NAME == walmgr - tools for managing WAL-based replication for PostgreSQL. == SYNOPSIS == walmgr.py command == DESCRIPTION == It is both admin and worker script for PostgreSQL PITR replication. == QUICK START == 1. Set up passwordless ssh authentication from master to slave master$ test -f ~/.ssh/id_dsa.pub || ssh-keygen -t dsa master$ cat ~/.ssh/id_dsa.pub | ssh slave cat \>\> .ssh/authorized_keys 2. Configure paths master$ edit master.ini slave$ edit slave.ini Make sure that walmgr.py executable has same pathname on slave and master. 3. Start archival process and create a base backup master$ ./walmgr.py master.ini setup master$ ./walmgr.py master.ini backup Note: starting from PostgreSQL 8.3 the archiving is enabled by setting archive_mode GUC to on. However changing this parameter requires the server to be restarted. 4. Prepare postgresql.conf and pg_hba.conf on slave and start replay master$ scp $PGDATA/*.conf slave: slave$ ./walmgr.py slave.ini restore For debian based distributions the standard configuration files are located in /etc/postgresql/x.x/main directory. If another scheme is used the postgresql.conf and pg_hba.conf should be copied to slave full_backup directory. Make sure to disable archive_command in slave config. 'walmgr.py restore' moves data in place, creates recovery.conf and starts postmaster in recovery mode. 5. In-progress WAL segments can be backup by command: master$ ./walmgr.py master.ini sync 6. If need to stop replay on slave and boot into normal mode, do: slave$ ./walmgr.py slave.ini boot == GENERAL OPTIONS == Common options to all walmgr.py commands. -h, --help:: show this help message and exit -q, --quiet:: make program silent -v, --verbose:: make program more verbose -n, --not-really:: Show what would be done without actually doing anything. == MASTER COMMANDS == === setup === Sets up postgres archiving, creates necessary directory structures on slave. === sync === Synchronizes in-progress WAL files to slave. === syncdaemon === Start WAL synchronization in daemon mode. This will start periodically synching the in-progress WAL files to slave. The following parameters are used to drive the syncdaemon: loop_delay - how long to sleep between the synchs. use_xlog_functions - use record based shipping to synchronize in-progress WAL segments. === stop === Deconfigures postgres archiving. === periodic === Runs periodic command, if configured. This enables to execute arbitrary commands on interval, useful for synchronizing scripts, config files, crontabs etc. === listbackups === List backup sets available on slave node. === backup === Creates a new base backup from master database. Will purge expired backups and WAL files on slave if `keep_backups` is specified. During a backup a lock file is created in slave `completed_wals` directory. This is to prevent simultaneous backups and resulting corruption. If running backup is terminated, the BACKUPLOCK file may have to be removed manually. === restore === EXPERIMENTAL. Attempts to restore the backup from slave to master. == SLAVE COMMANDS == === boot === Stop log playback and bring the database up. === pause === Pauses WAL playback. === continue === Continues previously paused WAL playback. === listbackups === Lists available backups. === backup === EXPERIMENTAL. Creates a new backup from slave data. Log replay is paused, slave data directory is backed up to `full_backup` directory and log replay resumed. Backups are rotated as needed. The idea is to move the backup load away from production node. Usable from postgres 8.2 and up. === restore [src][dst] === Restores the specified backup set to target directory. If specified without arguments the latest backup is *moved* to slave data directory (doesn't obey retention rules). If src backup is specified the backup is copied (instead of moving). Alternative destination directory can be specified with `dst`. == CONFIGURATION == === Common settings === ==== job_name ==== Optional. Indentifies this script, used in logging. Keep unique if using central logging. ==== logfile ==== Where to log. ==== use_skylog ==== Optional. If nonzero, skylog.ini is used for log configuration. === Master settings === ==== pidfile ==== Pid file location for syncdaemon mode (if running with -d). Otherwise not required. ==== master_db ==== Database to connect to for pg_start_backup() etc. It is not a good idea to use `dbname=template` if running syncdaemon in record shipping mode. ==== master_data ==== Master data directory location. ==== master_config ==== Master postgresql.conf file location. This is where `archive_command` gets updated. ==== master_restart_cmd ==== The command to restart master database, this used after changing `archive_mode` parameter. Leave unset, if you cannot afford to restart the database at setup/stop. ==== slave ==== Slave host and base directory. ==== slave_config ==== Configuration file location for the slave walmgr. ==== completed_wals ==== Slave directory where archived WAL files are copied. ==== partial_wals ==== Slave directory where incomplete WAL files are stored. ==== full_backup ==== Slave directory where full backups are stored. ==== config_backup ==== Slave directory where configuration file backups are stored. Optional. ==== loop_delay ==== The frequency of syncdaemon updates. In record shipping mode only incremental updates are sent, so smaller interval can be used. ==== use_xlog_functions ==== Use pg_xlog functions for record based shipping (available in 8.2 and up). ==== compression ==== If nonzero, a -z flag is added to rsync cmdline. Will reduce network traffic at the cost of extra CPU time. ==== periodic_command ==== Shell script to be executed at specified time interval. Can be used for synchronizing scripts, config files etc. ==== command_interval ==== How ofter to run periodic command script. In seconds, and only evaluated at log switch times. ==== hot_standby === Boolean. If set to true, walmgr setup will set wal_level to hot_standby (9.0 and newer). === Sample master.ini === [wal-master] logfile = master.log pidfile = master.pid master_db = dbname=template1 master_data = /var/lib/postgresql/8.0/main master_config = /etc/postgresql/8.0/main/postgresql.conf slave = slave:/var/lib/postgresql/walshipping completed_wals = %(slave)s/logs.complete partial_wals = %(slave)s/logs.partial full_backup = %(slave)s/data.master loop_delay = 10.0 use_xlog_functions = 1 compression = 1 === Slave settings === ==== slave_data ==== Postgres data directory for the slave. This is where the restored backup is copied/moved. ==== slave_config_dir ==== Directory for postgres configuration files. If specified, "walmgr restore" attempts to restore configuration files from config_backup directory. ==== slave_stop_cmd ==== Script to stop postmaster on slave. ==== slave_start_cmd ==== Script to start postmaster on slave. ==== slave ==== Base directory for slave files (logs.complete, data.master etc) ==== slave_bin ==== Specifies the location of postgres binaries (pg_controldata, etc). Needed if they are not already in the PATH. ==== completed_wals ==== Directory where complete WAL files are stored. Also miscellaneous control files are created in this directory (BACKUPLOCK, STOP, PAUSE, etc.). ==== partial_wals ==== Directory where partial WAL files are stored. ==== full_backup ==== Directory where full backups are stored. ==== keep_backups ==== Number of backups to keep. Also all WAL files needed to bring earliest backup up to date are kept. The backups are rotated before new backup is started, so at one point there is actually one less backup available. It probably doesn't make sense to specify `keep_backups` if periodic backups are not performed - the WAL files will pile up quickly. Backups will be named data.master, data.master.0, data.master.1 etc. ==== archive_command ==== Script to execute before rotating away the oldest backup. If it fails backups will not be rotated. ==== slave_pg_xlog ==== Set slave_pg_xlog to the directory on the slave where pg_xlog files get written to. On a restore to the slave walmgr.py will create a symbolic link from data/pg_xlog to this location. ==== backup_datadir ==== Set backup_datadir to 'no' to prevent walmgr.py from making a backup of the data directory when restoring to the slave. This defaults to 'yes' === Sample slave.ini === [wal-slave] logfile = slave.log slave_data = /var/lib/postgresql/8.0/main slave_stop_cmd = /etc/init.d/postgresql-8.0 stop slave_start_cmd = /etc/init.d/postgresql-8.0 start slave = /var/lib/postgresql/walshipping completed_wals = %(slave)s/logs.complete partial_wals = %(slave)s/logs.partial full_backup = %(slave)s/data.master keep_backups = 5 backup_datadir = yes skytools-2.1.13/doc/overview.txt0000644000175000017500000001361611670174255015664 0ustar markomarko#pragma section-numbers 2 = SkyTools = [[TableOfContents]] == Intro == This is package of tools we use at Skype to manage our cluster of [http://www.postgresql.org PostgreSQL] servers. They are put together for our own convinience and also because they build on each other, so managing them separately is pain. The code is hosted at [http://pgfoundry.org PgFoundry] site: http://pgfoundry.org/projects/skytools/ There are our [http://pgfoundry.org/frs/?group_id=1000206 downloads] and [http://lists.pgfoundry.org/mailman/listinfo/skytools-users mailing list]. Also [http://pgfoundry.org/scm/?group_id=1000206 CVS] and [http://pgfoundry.org/tracker/?group_id=1000206 bugtracker]. Combined todo list for all the modules: [http://skytools.projects.postgresql.org/doc/TODO.html TODO.html] == High-level tools == Those are script that are meant for end-user. In our case that means database administrators. === Londiste === Replication engine written in Python. It uses PgQ as transport mechanism. Its main goals are robustness and easy usage. Thus its not as complete and featureful as Slony-I. [http://pgsql.tapoueh.org/londiste.html Tutorial] written by Dimitri Fontaine. Documentation: * [http://skytools.projects.postgresql.org/doc/londiste.cmdline.html Usage guide] * [http://skytools.projects.postgresql.org/doc/londiste.config.html Config file] * [http://skytools.projects.postgresql.org/doc/londiste.ref.html Low-level reference] ''' Features ''' * Tables can be added one-by-one into set. * Initial COPY for one table does not block event replay for other tables. * Can compare tables on both sides. * Supports sequences. * Easy installation. ''' Missing features ''' * Does not understand cascaded replication, when one subscriber acts as provider to another one and it dies, the last one loses sync with the first one. In other words - it understands only pair of servers. ''' Sample usage ''' {{{ ## install pgq on provider: $ pgqadm.py provider_ticker.ini install ## run ticker on provider: $ pgqadm.py provider_ticker.ini ticker -d ## install Londiste in provider $ londiste.py replic.ini provider install ## install Londiste in subscriber $ londiste.py replic.ini subscriber install ## start replication daemon $ londiste.py replic.ini replay -d ## activate tables on provider $ londiste.py replic.ini provider add users orders ## add tables to subscriber $ londiste.py replic.ini subscriber add users }}} === PgQ === Generic queue implementation. Based on ideas from [http://www.slony1.info/ Slony-I] - snapshot based event batching. ''' Features ''' * Generic multi-consumer, multi-producer queue. * There can be several consumers on one queue. * It is guaranteed that each of them sees a event at least once. But it's not guaranteed that it sees it only once. * The goal is to provide a clean API as SQL functions. The frameworks on top of that don't need to understand internal details. ''' Technical design ''' * Events are batched using snapshots (like Slony-I). * Consumers are poll-only, they don't need to do any administrative work. * Queue administration is separate process from consumers. * Tolerant of long transactions. * Easy to monitor. ''' Docs ''' * [http://skytools.projects.postgresql.org/doc/pgq-sql.html SQL API overview] * [http://skytools.projects.postgresql.org/pgq/ SQL API detailed docs] * [http://skytools.projects.postgresql.org/doc/pgq-admin.html Administrative tool usage] === WalMgr === Python script for hot failover. Tries to make setup initial copy and later switch easy for admins. * Docs: [http://skytools.projects.postgresql.org/doc/walmgr.html walmgr.html] Sample: {{{ [ .. prepare config .. ] master$ walmgr master.ini setup master$ walmgr master.ini backup slave$ walmgr slave.ini restore [ .. main server down, switch failover server to normal mode: ] slave$ walmgr slave.ini boot }}} == Low-level tools == Those are building blocks for the PgQ and Londiste. Useful for database developers. === txid === Provides 8-byte transaction id-s for external usage. === logtriga === Trigger function for table event logging in "partial SQL" format. Based on Slony-I logtrigger. Used in londiste for replication. === logutriga === Trigger function for table event logging in urlencoded format. Written in PL/Python. For cases where data manipulation is necessary. == Developement frameworks == === skytools - Python framework for database scripting === This collect various utilities for Python scripts for databases. ''' Topics ''' * Daemonization * Logging * Configuration. * Skeleton class for scripts. * Quoting (SQL/COPY) * COPY helpers. * Database object lookup. * Table structure detection. Documentation: http://skytools.projects.postgresql.org/api/ === pgq - Python framework for PgQ consumers === This builds on scripting framework above. Docs: * [http://skytools.projects.postgresql.org/api/ Python API docs] == Sample scripts == Those are specialized script that are based on skytools/pgq framework. Can be considered examples, although they are used in production in Skype. === Special data moving scripts === There are couple of scripts for situations where regular replication does not fit. They all operate on `logutriga()` urlencoded queues. * `cube_dispatcher`: Multi-table partitioning on change date, with optional keep-all-row-versions mode. * `table_dispatcher`: configurable partitioning for one table. * `bulk_loader`: aggregates changes for slow databases. Instead of each change in separate statement, does minimal amount of DELETE-s and then big COPY. || Script || Supported operations || Number of tables || Partitioning || || table_dispatcher || INSERT || 1 || any || || cube_dispatcher || INSERT/UPDATE || any || change time || || bulk_loader || INSERT/UPDATE/DELETE || any || none || === queue_mover === Simply copies all events from one queue to another. === scriptmgr === Allows to start and stop several scripts together. skytools-2.1.13/doc/Makefile0000644000175000017500000000574011670174255014714 0ustar markomarko include ../config.mak wiki = https://developer.skype.com/SkypeGarage/DbProjects/SkyTools web = mkz@shell.pgfoundry.org:/home/pgfoundry.org/groups/skytools/htdocs/ EPYDOC = epydoc EPYARGS = --no-private --url="http://pgfoundry.org/projects/skytools/" \ --name="Skytools" --html --no-private -v HTMLS = londiste.cmdline.html londiste.config.html README.html INSTALL.html \ londiste.ref.html TODO.html pgq-sql.html pgq-admin.html pgq-nodupes.html \ $(SCRIPT_HTMLS) SCRIPT_TXTS = walmgr.txt cube_dispatcher.txt table_dispatcher.txt \ queue_mover.txt queue_splitter.txt bulk_loader.txt \ scriptmgr.txt skytools_upgrade.txt SCRIPT_MANS = $(SCRIPT_TXTS:.txt=.1) SCRIPT_HTMLS = $(SCRIPT_TXTS:.txt=.html) COMMON = common.switches.txt common.config.txt common.logutriga.txt GETATTRS = python ./getattrs.py all: man man: londiste.1 londiste.5 pgqadm.1 $(SCRIPT_MANS) html: $(HTMLS) install: man mkdir -p $(DESTDIR)/$(mandir)/man1 mkdir -p $(DESTDIR)/$(mandir)/man5 install -m 644 londiste.1 $(DESTDIR)/$(mandir)/man1 install -m 644 londiste.5 $(DESTDIR)/$(mandir)/man5 install -m 644 pgqadm.1 $(DESTDIR)/$(mandir)/man1 for m in $(SCRIPT_MANS); do \ install -m 644 $$m $(DESTDIR)/$(mandir)/man1 ; \ done old.wiki.upload: devupload.sh overview.txt $(wiki) #devupload.sh TODO.txt $(wiki)/ToDo #devupload.sh londiste.txt $(wiki)/LondisteUsage #devupload.sh londiste.ref.txt $(wiki)/LondisteReference #devupload.sh pgq-sql.txt $(wiki)/PgQdocs #devupload.sh pgq-nodupes.txt $(wiki)/PgqNoDupes #devupload.sh walmgr.txt $(wiki)/WalMgr #devupload.sh pgq-admin.txt $(wiki)/PgqAdm PY_PKGS = skytools pgq londiste # skytools.config skytools.dbstruct skytools.gzlog \ # skytools.quoting skytools.scripting skytools.sqltools \ # pgq pgq.consumer pgq.event pgq.maint pgq.producer pgq.status pgq.ticker \ # londiste londiste.compare londiste.file_read londiste.file_write \ # londiste.installer londiste.playback londiste.repair londiste.setup \ # londiste.syncer londiste.table_copy apidoc: rm -rf api mkdir -p api cd ../python && $(EPYDOC) $(EPYARGS) -o ../doc/api $(PY_PKGS) apiupload: apidoc rsync -rtlz api/* $(web)/api cd ../sql/pgq && rm -rf docs/html && $(MAKE) dox rsync -rtlz ../sql/pgq/docs/html/* $(web)/pgq/ clean: rm -rf api *.html distclean: clean rm -rf ../sql/pgq/docs/pgq realclean: distclean rm -f *.[15] *.xml ifneq ($(ASCIIDOC),no) ifneq ($(XMLTO),no) londiste.1: londiste.cmdline.xml $(XMLTO) man $< londiste.5: londiste.config.xml $(XMLTO) man $< pgqadm.1: pgq-admin.xml $(XMLTO) man $< walmgr.1: walmgr.xml $(XMLTO) man $< %.xml: %.txt $(COMMON) $(ASCIIDOC) -b docbook -d manpage `$(GETATTRS) $<` -o - $< \ | cat > $@ %.1: %.xml $(XMLTO) man $< endif %.html: %.txt $(COMMON) $(ASCIIDOC) -a toc `$(GETATTRS) $<` $< README.html: ../README cat $< \ | sed -e 's,doc/\([!-~]*\)[.]txt,link:\1.html[],g' \ -e 's,http:[!-~]*,&[],g' \ | $(ASCIIDOC) -o $@ - INSTALL.html: ../INSTALL $(ASCIIDOC) -o $@ $< endif web: $(HTMLS) rsync -avz $(HTMLS) $(web)/doc/ skytools-2.1.13/doc/walmgr.10000644000175000017500000003407311727600373014626 0ustar markomarko'\" t .\" Title: walmgr .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "WALMGR" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" walmgr \- tools for managing WAL\-based replication for PostgreSQL\&. .SH "SYNOPSIS" .sp .nf walmgr\&.py command .fi .SH "DESCRIPTION" .sp It is both admin and worker script for PostgreSQL PITR replication\&. .SH "QUICK START" .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} Set up passwordless ssh authentication from master to slave .sp .if n \{\ .RS 4 .\} .nf master$ test \-f ~/\&.ssh/id_dsa\&.pub || ssh\-keygen \-t dsa master$ cat ~/\&.ssh/id_dsa\&.pub | ssh slave cat \e>\e> \&.ssh/authorized_keys .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} Configure paths .sp .if n \{\ .RS 4 .\} .nf master$ edit master\&.ini slave$ edit slave\&.ini .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf Make sure that walmgr\&.py executable has same pathname on slave and master\&. .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} Start archival process and create a base backup .sp .if n \{\ .RS 4 .\} .nf master$ \&./walmgr\&.py master\&.ini setup master$ \&./walmgr\&.py master\&.ini backup .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf Note: starting from PostgreSQL 8\&.3 the archiving is enabled by setting archive_mode GUC to on\&. However changing this parameter requires the server to be restarted\&. .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} Prepare postgresql\&.conf and pg_hba\&.conf on slave and start replay .sp .if n \{\ .RS 4 .\} .nf master$ scp $PGDATA/*\&.conf slave: slave$ \&./walmgr\&.py slave\&.ini restore .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf For debian based distributions the standard configuration files are located in /etc/postgresql/x\&.x/main directory\&. If another scheme is used the postgresql\&.conf and pg_hba\&.conf should be copied to slave full_backup directory\&. Make sure to disable archive_command in slave config\&. .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf \*(Aqwalmgr\&.py restore\*(Aq moves data in place, creates recovery\&.conf and starts postmaster in recovery mode\&. .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} In\-progress WAL segments can be backup by command: .sp .if n \{\ .RS 4 .\} .nf master$ \&./walmgr\&.py master\&.ini sync .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 6.\h'+01'\c .\} .el \{\ .sp -1 .IP " 6." 4.2 .\} If need to stop replay on slave and boot into normal mode, do: .sp .if n \{\ .RS 4 .\} .nf slave$ \&./walmgr\&.py slave\&.ini boot .fi .if n \{\ .RE .\} .RE .SH "GENERAL OPTIONS" .sp Common options to all walmgr\&.py commands\&. .PP \-h, \-\-help .RS 4 show this help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-n, \-\-not\-really .RS 4 Show what would be done without actually doing anything\&. .RE .SH "MASTER COMMANDS" .SS "setup" .sp Sets up postgres archiving, creates necessary directory structures on slave\&. .SS "sync" .sp Synchronizes in\-progress WAL files to slave\&. .SS "syncdaemon" .sp Start WAL synchronization in daemon mode\&. This will start periodically synching the in\-progress WAL files to slave\&. .sp The following parameters are used to drive the syncdaemon: loop_delay \- how long to sleep between the synchs\&. use_xlog_functions \- use record based shipping to synchronize in\-progress WAL segments\&. .SS "stop" .sp Deconfigures postgres archiving\&. .SS "periodic" .sp Runs periodic command, if configured\&. This enables to execute arbitrary commands on interval, useful for synchronizing scripts, config files, crontabs etc\&. .SS "listbackups" .sp List backup sets available on slave node\&. .SS "backup" .sp Creates a new base backup from master database\&. Will purge expired backups and WAL files on slave if keep_backups is specified\&. During a backup a lock file is created in slave completed_wals directory\&. This is to prevent simultaneous backups and resulting corruption\&. If running backup is terminated, the BACKUPLOCK file may have to be removed manually\&. .SS "restore " .sp EXPERIMENTAL\&. Attempts to restore the backup from slave to master\&. .SH "SLAVE COMMANDS" .SS "boot" .sp Stop log playback and bring the database up\&. .SS "pause" .sp Pauses WAL playback\&. .SS "continue" .sp Continues previously paused WAL playback\&. .SS "listbackups" .sp Lists available backups\&. .SS "backup" .sp EXPERIMENTAL\&. Creates a new backup from slave data\&. Log replay is paused, slave data directory is backed up to full_backup directory and log replay resumed\&. Backups are rotated as needed\&. The idea is to move the backup load away from production node\&. Usable from postgres 8\&.2 and up\&. .SS "restore [src][dst]" .sp Restores the specified backup set to target directory\&. If specified without arguments the latest backup is \fBmoved\fR to slave data directory (doesn\(cqt obey retention rules)\&. If src backup is specified the backup is copied (instead of moving)\&. Alternative destination directory can be specified with dst\&. .SH "CONFIGURATION" .SS "Common settings" .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBjob_name\fR .RS 4 .sp Optional\&. Indentifies this script, used in logging\&. Keep unique if using central logging\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBlogfile\fR .RS 4 .sp Where to log\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBuse_skylog\fR .RS 4 .sp Optional\&. If nonzero, skylog\&.ini is used for log configuration\&. .RE .SS "Master settings" .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBpidfile\fR .RS 4 .sp Pid file location for syncdaemon mode (if running with \-d)\&. Otherwise not required\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBmaster_db\fR .RS 4 .sp Database to connect to for pg_start_backup() etc\&. It is not a good idea to use dbname=template if running syncdaemon in record shipping mode\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBmaster_data\fR .RS 4 .sp Master data directory location\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBmaster_config\fR .RS 4 .sp Master postgresql\&.conf file location\&. This is where archive_command gets updated\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBmaster_restart_cmd\fR .RS 4 .sp The command to restart master database, this used after changing archive_mode parameter\&. Leave unset, if you cannot afford to restart the database at setup/stop\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave\fR .RS 4 .sp Slave host and base directory\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_config\fR .RS 4 .sp Configuration file location for the slave walmgr\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBcompleted_wals\fR .RS 4 .sp Slave directory where archived WAL files are copied\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBpartial_wals\fR .RS 4 .sp Slave directory where incomplete WAL files are stored\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBfull_backup\fR .RS 4 .sp Slave directory where full backups are stored\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBconfig_backup\fR .RS 4 .sp Slave directory where configuration file backups are stored\&. Optional\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBloop_delay\fR .RS 4 .sp The frequency of syncdaemon updates\&. In record shipping mode only incremental updates are sent, so smaller interval can be used\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBuse_xlog_functions\fR .RS 4 .sp Use pg_xlog functions for record based shipping (available in 8\&.2 and up)\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBcompression\fR .RS 4 .sp If nonzero, a \-z flag is added to rsync cmdline\&. Will reduce network traffic at the cost of extra CPU time\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBperiodic_command\fR .RS 4 .sp Shell script to be executed at specified time interval\&. Can be used for synchronizing scripts, config files etc\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBcommand_interval\fR .RS 4 .sp How ofter to run periodic command script\&. In seconds, and only evaluated at log switch times\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBhot_standby ===\fR .RS 4 .sp Boolean\&. If set to true, walmgr setup will set wal_level to hot_standby (9\&.0 and newer)\&. .RE .SS "Sample master\&.ini" .sp .if n \{\ .RS 4 .\} .nf [wal\-master] logfile = master\&.log pidfile = master\&.pid master_db = dbname=template1 master_data = /var/lib/postgresql/8\&.0/main master_config = /etc/postgresql/8\&.0/main/postgresql\&.conf slave = slave:/var/lib/postgresql/walshipping completed_wals = %(slave)s/logs\&.complete partial_wals = %(slave)s/logs\&.partial full_backup = %(slave)s/data\&.master loop_delay = 10\&.0 use_xlog_functions = 1 compression = 1 .fi .if n \{\ .RE .\} .SS "Slave settings" .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_data\fR .RS 4 .sp Postgres data directory for the slave\&. This is where the restored backup is copied/moved\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_config_dir\fR .RS 4 .sp Directory for postgres configuration files\&. If specified, "walmgr restore" attempts to restore configuration files from config_backup directory\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_stop_cmd\fR .RS 4 .sp Script to stop postmaster on slave\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_start_cmd\fR .RS 4 .sp Script to start postmaster on slave\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave\fR .RS 4 .sp Base directory for slave files (logs\&.complete, data\&.master etc) .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_bin\fR .RS 4 .sp Specifies the location of postgres binaries (pg_controldata, etc)\&. Needed if they are not already in the PATH\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBcompleted_wals\fR .RS 4 .sp Directory where complete WAL files are stored\&. Also miscellaneous control files are created in this directory (BACKUPLOCK, STOP, PAUSE, etc\&.)\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBpartial_wals\fR .RS 4 .sp Directory where partial WAL files are stored\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBfull_backup\fR .RS 4 .sp Directory where full backups are stored\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBkeep_backups\fR .RS 4 .sp Number of backups to keep\&. Also all WAL files needed to bring earliest .sp backup up to date are kept\&. The backups are rotated before new backup is started, so at one point there is actually one less backup available\&. .sp It probably doesn\(cqt make sense to specify keep_backups if periodic backups are not performed \- the WAL files will pile up quickly\&. .sp Backups will be named data\&.master, data\&.master\&.0, data\&.master\&.1 etc\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBarchive_command\fR .RS 4 .sp Script to execute before rotating away the oldest backup\&. If it fails backups will not be rotated\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBslave_pg_xlog\fR .RS 4 .sp Set slave_pg_xlog to the directory on the slave where pg_xlog files get written to\&. On a restore to the slave walmgr\&.py will create a symbolic link from data/pg_xlog to this location\&. .RE .sp .it 1 an-trap .nr an-no-space-flag 1 .nr an-break-flag 1 .br .ps +1 \fBbackup_datadir\fR .RS 4 .sp Set backup_datadir to \fIno\fR to prevent walmgr\&.py from making a backup of the data directory when restoring to the slave\&. This defaults to \fIyes\fR .RE .SS "Sample slave\&.ini" .sp .if n \{\ .RS 4 .\} .nf [wal\-slave] logfile = slave\&.log slave_data = /var/lib/postgresql/8\&.0/main slave_stop_cmd = /etc/init\&.d/postgresql\-8\&.0 stop slave_start_cmd = /etc/init\&.d/postgresql\-8\&.0 start slave = /var/lib/postgresql/walshipping completed_wals = %(slave)s/logs\&.complete partial_wals = %(slave)s/logs\&.partial full_backup = %(slave)s/data\&.master keep_backups = 5 backup_datadir = yes .fi .if n \{\ .RE .\} skytools-2.1.13/doc/getattrs.py0000755000175000017500000000020711670174255015457 0ustar markomarko#! /usr/bin/env python import sys buf = open(sys.argv[1], "r").read().lower() if buf.find("pgq consumer") >= 0: print "-a pgq" skytools-2.1.13/doc/londiste.cmdline.txt0000644000175000017500000002266011670174255017250 0ustar markomarko= londiste(1) = == NAME == londiste - PostgreSQL replication engine written in python == SYNOPSIS == londiste.py [option] config.ini command [arguments] == DESCRIPTION == Londiste is the PostgreSQL replication engine portion of the SkyTools suite, by Skype. This suite includes packages implementing specific replication tasks and/or solutions in layers, building upon each other. PgQ is a generic queue implementation based on ideas from Slony-I's snapshot based event batching. Londiste uses PgQ as its transport mechanism to implement a robust and easy to use replication solution. Londiste is an asynchronous master-slave(s) replication system. Asynchronous means that a transaction commited on the master is not guaranteed to have made it to any slave at the master's commit time; and master-slave means that data changes on slaves are not reported back to the master, it's the other way around only. The replication is trigger based, and you choose a set of tables to replicate from the provider to the subscriber(s). Any data changes occuring on the provider (in a replicated table) will fire the londiste trigger, which fills a queue of events for any subscriber(s) to care about. A replay process consumes the queue in batches, and applies all given changes to any subscriber(s). The initial replication step involves using the PostgreSQL's COPY command for efficient data loading. == QUICK-START == Basic londiste setup and usage can be summarized by the following steps: 1. create the subscriber database, with tables to replicate 2. edit a londiste configuration file, say conf.ini, and a PgQ ticker configuration file, say ticker.ini 3. install londiste on the provider and subscriber nodes. This step requires admin privileges on both provider and subscriber sides, and both install commands can be run remotely: $ londiste.py conf.ini provider install $ londiste.py conf.ini subscriber install 4. launch the PgQ ticker on the provider machine: $ pgqadm.py -d ticker.ini ticker 5. launch the londiste replay process: $ londiste.py -d conf.ini replay 6. add tables to replicate from the provider database: $ londiste.py conf.ini provider add table1 table2 ... 7. add tables to replicate to the subscriber database: $ londiste.py conf.ini subscriber add table1 table2 ... To replicate to more than one subscriber database just repeat each of the described subscriber steps for each subscriber. == COMMANDS == The londiste command is parsed globally, and has both options and subcommands. Some options are reserved to a subset of the commands, and others should be used without any command at all. == GENERAL OPTIONS == This section presents options available to all and any londiste command. -h, --help:: show this help message and exit -q, --quiet:: make program silent -v, --verbose:: make program more verbose == PROVIDER COMMANDS == $ londiste.py config.ini provider Where command is one of: === provider install === Installs code into provider and subscriber database and creates queue. Equivalent to doing following by hand: CREATE LANGUAGE plpgsql; CREATE LANGUAGE plpython; \i .../contrib/txid.sql \i .../contrib/pgq.sql \i .../contrib/londiste.sql select pgq.create_queue(queue name); === provider add
... === Registers table(s) on the provider database and adds the londiste trigger to the table(s) which will send events to the queue. Table names can be schema qualified with the schema name defaulting to public if not supplied. --all:: Register all tables in provider database, except those that are under schemas 'pgq', 'londiste', 'information_schema' or 'pg_*'. === provider remove
... === Unregisters table(s) on the provider side and removes the londiste triggers from the table(s). The table removal event is also sent to the queue, so all subscribers unregister the table(s) on their end as well. Table names can be schema qualified with the schema name defaulting to public if not supplied. === provider add-seq ... === Registers a sequence on provider. === provider remove-seq ... === Unregisters a sequence on provider. === provider tables === Shows registered tables on provider side. === provider seqs === Shows registered sequences on provider side. == SUBSCRIBER COMMANDS == londiste.py config.ini subscriber Where command is one of: === subscriber install === Installs code into subscriber database. Equivalent to doing following by hand: CREATE LANGUAGE plpgsql; \i .../contrib/londiste.sql This will be done under the Postgres Londiste user, if the tables should be owned by someone else, it needs to be done by hand. === subscriber add
... === Registers table(s) on subscriber side. Table names can be schema qualified with the schema name defaulting to `public` if not supplied. Switches (optional): --all:: Add all tables that are registered on provider to subscriber database --force:: Ignore table structure differences. --expect-sync:: Table is already synced by external means so initial COPY is unnecessary. --skip-truncate:: When doing initial COPY, don't remove old data. === subscriber remove
... === Unregisters table(s) from subscriber. No events will be applied to the table anymore. Actual table will not be touched. Table names can be schema qualified with the schema name defaulting to public if not supplied. === subscriber add-seq ... === Registers a sequence on subscriber. === subscriber remove-seq ... === Unregisters a sequence on subscriber. === subscriber resync
... === Tags table(s) as "not synced". Later the replay process will notice this and launch copy process(es) to sync the table(s) again. === subscriber tables === Shows registered tables on the subscriber side, and the current state of each table. Possible state values are: NEW:: the table has not yet been considered by londiste. in-copy:: Full-table copy is in progress. catching-up:: Table is copied, missing events are replayed on to it. wanna-sync::: The "copy" process catched up, wants to hand the table over to "replay". do-sync::: "replay" process is ready to accept it. ok:: table is in sync. === subscriber fkeys === Show pending and active foreign keys on tables. Takes optional type argument - `pending` or `active`. If no argument is given, both types are shown. Pending foreign keys are those that were removed during COPY time but have not restored yet, The restore happens autmatically if both tables are synced. === subscriber triggers === Show pending and active triggers on tables. Takes optional type argument - `pending` or `active`. If no argument is given, both types are shown. Pending triggers keys are those that were removed during COPY time but have not restored yet, The restore of triggers does not happen autmatically, it needs to be done manually with `restore-triggers` command. === subscriber restore-triggers
=== Restores all pending triggers for single table. Optionally trigger name can be given as extra argument, then only that trigger is restored. === subscriber register === Register consumer on queue. This usually happens automatically when `replay` is launched, but === subscriber unregister === Unregister consumer from provider's queue. This should be done if you want to shut replication down. == REPLICATION COMMANDS == === replay === The actual replication process. Should be run as daemon with -d switch, because it needs to be always running. It's main task is to get batches of events from PgQ and apply them to subscriber database. Switches: -d, --daemon:: go background -r, --reload:: reload config (send SIGHUP) -s, --stop:: stop program safely (send SIGINT) -k, --kill:: kill program immidiately (send SIGTERM) == UTILITY COMMAND == === repair
... === Attempts to achieve a state where the table(s) is/are in sync, compares them, and writes out SQL statements that would fix differences. Syncing happens by locking provider tables against updates and then waiting until the replay process has applied all pending changes to subscriber database. As this is dangerous operation, it has a hardwired limit of 10 seconds for locking. If the replay process does not catch up in that time, the locks are released and the repair operation is cancelled. Comparing happens by dumping out the table contents of both sides, sorting them and then comparing line-by-line. As this is a CPU and memory-hungry operation, good practice is to run the repair command on a third machine to avoid consuming resources on either the provider or the subscriber. === compare
... === Syncs tables like repair, but just runs SELECT count(*) on both sides to get a little bit cheaper, but also less precise, way of checking if the tables are in sync. == CONFIGURATION == Londiste and PgQ both use INI configuration files, your distribution of skytools include examples. You often just have to edit the database connection strings, namely db in PgQ ticker.ini and provider_db and subscriber_db in londiste conf.ini as well as logfile and pidfile to adapt to you system paths. See `londiste(5)`. == SEE ALSO == `londiste(5)` https://developer.skype.com/SkypeGarage/DbProjects/SkyTools/[] http://skytools.projects.postgresql.org/doc/londiste.ref.html[Reference guide] skytools-2.1.13/doc/common.config.txt0000644000175000017500000000225511670174255016547 0ustar markomarko === Common configuration parameters === job_name:: Name for particulat job the script does. Script will log under this name to logdb/logserver. The name is also used as default for PgQ consumer name. It should be unique. pidfile:: Location for pid file. If not given, script is disallowed to daemonize. logfile:: Location for log file. loop_delay:: If continuisly running process, how long to sleep after each work loop, in seconds. Default: 1. connection_lifetime:: Close and reconnect older database connections. log_count:: Number of log files to keep. Default: 3 log_size:: Max size for one log file. File is rotated if max size is reached. Default: 10485760 (10M) use_skylog:: If set, search for `[./skylog.ini, ~/.skylog.ini, /etc/skylog.ini]`. If found then the file is used as config file for Pythons `logging` module. It allows setting up fully customizable logging setup. ifdef::pgq[] === Common PgQ consumer parameters === pgq_queue_name:: Queue name to attach to. No default. pgq_consumer_id:: Consumers ID to use when registering. Default: %(job_name)s endif::pgq[] skytools-2.1.13/doc/TODO.txt0000644000175000017500000000056511670174255014562 0ustar markomarko = Skytools ToDo list = == Ideas for 2.1 == * Frozen code, no new features. * Fix things in existing code. == Ideas for 2.2 == * Use pgq.sqltriga() in Londiste by default * Backport triggers code from 3.0 * Backport truncate trigger 3.0 * londiste: support creating slave from master by pg_dump / PITR. * Backport EXECUTE 3.0? * Use session_replication_role? skytools-2.1.13/doc/scriptmgr.10000644000175000017500000001317511727600405015343 0ustar markomarko'\" t .\" Title: scriptmgr .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "SCRIPTMGR" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" scriptmgr \- utility for controlling other skytools scripts\&. .SH "SYNOPSIS" .sp .nf scriptmgr\&.py [switches] config\&.ini [\-a | job_name \&.\&.\&. ] .fi .SH "DESCRIPTION" .sp scriptmgr is used to manage several scripts together\&. It discovers potential jobs based on config file glob expression\&. From config file it gets both job_name and service type (that is the main section name eg [cube_dispatcher])\&. For each service type there is subsection in the config how to handle it\&. Unknown services are ignored\&. .SH "COMMANDS" .SS "status" .sp .if n \{\ .RS 4 .\} .nf scriptmgr config\&.ini status .fi .if n \{\ .RE .\} .sp Show status for all known jobs\&. .SS "start" .sp .if n \{\ .RS 4 .\} .nf scriptmgr config\&.ini start \-a scriptmgr config\&.ini start job_name1 job_name2 \&.\&.\&. .fi .if n \{\ .RE .\} .sp launch script(s) that are not running\&. .SS "stop" .sp .if n \{\ .RS 4 .\} .nf scriptmgr config\&.ini stop \-a scriptmgr config\&.ini stop job_name1 job_name2 \&.\&.\&. .fi .if n \{\ .RE .\} .sp stop script(s) that are running\&. .SS "restart" .sp .if n \{\ .RS 4 .\} .nf scriptmgr config\&.ini restart \-a scriptmgr config\&.ini restart job_name1 job_name2 \&.\&.\&. .fi .if n \{\ .RE .\} .sp restart scripts\&. .SS "reload" .sp .if n \{\ .RS 4 .\} .nf scriptmgr config\&.ini reload \-a scriptmgr config\&.ini reload job_name1 job_name2 \&.\&.\&. .fi .if n \{\ .RE .\} .sp Send SIGHUP to scripts that are running\&. .SH "CONFIG" .SS "Common configuration parameters" .PP job_name .RS 4 Name for particulat job the script does\&. Script will log under this name to logdb/logserver\&. The name is also used as default for PgQ consumer name\&. It should be unique\&. .RE .PP pidfile .RS 4 Location for pid file\&. If not given, script is disallowed to daemonize\&. .RE .PP logfile .RS 4 Location for log file\&. .RE .PP loop_delay .RS 4 If continuisly running process, how long to sleep after each work loop, in seconds\&. Default: 1\&. .RE .PP connection_lifetime .RS 4 Close and reconnect older database connections\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. .RE .SS "scriptmgr parameters" .PP config_list .RS 4 List of glob patters for finding config files\&. Example: .sp .if n \{\ .RS 4 .\} .nf config_list = ~/dbscripts/conf/*\&.ini, ~/random/conf/*\&.ini .fi .if n \{\ .RE .\} .RE .SS "Service section parameters" .PP cwd .RS 4 Working directory for script\&. .RE .PP args .RS 4 Arguments to give to script, in addition to \-d\&. .RE .PP script .RS 4 Path to script\&. Unless script is in PATH, full path should be given\&. .RE .PP disabled .RS 4 If this service should be ignored\&. .RE .SS "Example config file" .sp .if n \{\ .RS 4 .\} .nf [scriptmgr] job_name = scriptmgr_livesrv logfile = ~/log/%(job_name)s\&.log pidfile = ~/pid/%(job_name)s\&.pid .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf config_list = ~/scripts/conf/*\&.ini .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # defaults for all service sections [DEFAULT] cwd = ~/scripts .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf [table_dispatcher] script = table_dispatcher\&.py args = \-v .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf [cube_dispatcher] script = python2\&.4 cube_dispatcher\&.py disabled = 1 .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf [pgqadm] script = ~/scripts/pgqadm\&.py args = ticker .fi .if n \{\ .RE .\} .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE .sp Options specific to scriptmgr: .PP \-a, \-\-all .RS 4 Operate on all non\-disabled scripts\&. .RE skytools-2.1.13/doc/londiste.50000644000175000017500000000763711727600367015173 0ustar markomarko'\" t .\" Title: londiste .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "LONDISTE" "5" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" londiste \- PostgreSQL replication engine written in python .SH "SYNOPSIS" .sp .nf [londiste] job_name = asd .fi .SH "DESCRIPTION" .sp The londiste configuration file follow the famous \&.INI syntax\&. It contains only one section named londiste\&. .sp Most defaults values are reasonable ones\&. That means you can only edit provider_db, subscriber_db and pgq_queue_name and be done with londiste configuration\&. .SH "OPTIONS" .sp You can configure the following options into the londiste section\&. .PP job_name .RS 4 Each Skytools daemon process must have a unique job_name\&. Londiste uses it also as consumer name when subscribing to queue\&. .RE .PP provider_db .RS 4 Provider database connection string (DSN)\&. .RE .PP subscriber_db .RS 4 Subscriber database connection string (DSN)\&. .RE .PP pgq_queue_name .RS 4 Name of the queue to read from\&. Several subscribers can read from same queue\&. .RE .PP logfile .RS 4 Where to log londiste activity\&. .RE .PP pidfile .RS 4 Where to store the pid of the main londiste process, the replay one\&. .RE .PP lock_timeout .RS 4 Few operations take lock on provider (provider add/remove, compare, repair)\&. This parameter specifies timeout in seconds (float) how long a lock can be held\&. New in version 2\&.1\&.8\&. Default: 10 .RE .PP loop_delay .RS 4 How often to poll events from provider\&. In seconds (float)\&. Default: 1\&. .RE .PP pgq_lazy_fetch .RS 4 How many events to fetch at a time when processing a batch\&. Useful when you know a single transaction (maintenance UPDATE command, e\&.g\&.) will produce a lot of events in a single batch\&. When lazily fetching, a cursor is used so as to still process a single batch in a single transaction\&. Default: 0, always fetch all events of the batch, not using a cursor\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. Default: 0 .RE .SH "EXAMPLE" .sp .if n \{\ .RS 4 .\} .nf [londiste] job_name = test_to_subcriber .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf provider_db = dbname=provider port=6000 host=127\&.0\&.0\&.1 subscriber_db = dbname=subscriber port=6000 host=127\&.0\&.0\&.1 .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # it will be used as sql ident so no dots/spaces pgq_queue_name = londiste\&.replika .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf logfile = /tmp/%(job_name)s\&.log pidfile = /tmp/%(job_name)s\&.pid .fi .if n \{\ .RE .\} .SH "SEE ALSO" .sp londiste(1) skytools-2.1.13/doc/pgq-admin.txt0000644000175000017500000001526711670174255015677 0ustar markomarko= pgqadm(1) = == NAME == pgqadm - PgQ ticker and administration interface == SYNOPSIS == pgqadm.py [option] config.ini command [arguments] == DESCRIPTION == PgQ is Postgres based event processing system. It is part of SkyTools package that contains several useful implementations on this engine. Main function of PgQadm is to maintain and keep healthy both pgq internal tables and tables that store events. SkyTools is scripting framework for Postgres databases written in Python that provides several utilities and implements common database handling logic. Event - atomic piece of data created by Producers. In PgQ event is one record in one of tables that services that queue. Event record contains some system fields for PgQ and several data fileds filled by Producers. PgQ is neither checking nor enforcing event type. Event type is someting that consumer and produser must agree on. PgQ guarantees that each event is seen at least once but it is up to consumer to make sure that event is processed no more than once if that is needed. Batch - PgQ is designed for efficiency and high throughput so events are grouped into batches for bulk processing. Creating these batches is one of main tasks of PgQadm and there are several parameters for each queue that can be use to tune size and frequency of batches. Consumerss receive events in these batches and depending on business requirements process events separately or also in batches. Queue - Event are stored in queue tables i.e queues. Several producers can write into same queeu and several consumers can read from the queue. Events are kept in queue until all the consumers have seen them. We use table rotation to decrease hard disk io. Queue can contain any number of event types it is up to Producer and Consumer to agree on what types of events are passed and how they are encoded For example Londiste producer side can produce events for more tables tan consumer side needs so consumer subscribes only to those tables it needs and events for other tables are ignores. Producer - applicatione that pushes event into queue. Prodecer can be written in any langaage that is able to run stored procedures in Postgres. Consumer - application that reads events from queue. Consumers can be written in any language that can interact with Postgres. SkyTools package contains several useful consumers written in Python that can be used as they are or as good starting points to write more complex consumers. == QUICK-START == Basic PgQ setup and usage can be summarized by the following steps: 1. create the database 2. edit a PgQ ticker configuration file, say ticker.ini 3. install PgQ internal tables $ pgqadm.py ticker.ini install 4. launch the PgQ ticker on databse machine as daemon $ pgqadm.py -d ticker.ini ticker 5. create queue $ pgqadm.py ticker.ini create 6. register or run consumer to register it automatically $ pgqadm.py ticker.ini register 7. start producing events == CONFIG == [pgqadm] job_name = pgqadm_somedb db = dbname=somedb # how often to run maintenance [seconds] maint_delay = 600 # how often to check for activity [seconds] loop_delay = 0.1 logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid == COMMANDS == === ticker === Start ticking & maintenance process. Usually run as daemon with -d option. Must be running for PgQ to be functional and for consumers to see any events. === status === Show overview of registered queues and consumers and queue health. This command is used when you want to know what is happening inside PgQ. === install === Installs PgQ schema into database from config file. === create === Create queue tables into pgq schema. As soon as queue is created producers can start inserting events into it. But you must be aware that if there are no consumers on the queue the events are lost until consumer is registered. === drop === Drop queue and all it's consumers from PgQ. Queue tables are dropped and all the contents are lost forever so use with care as with most drop commands. === register === Register given consumer to listen to given queue. First batch seen by this consumer is the one completed after registration. Registration happens automatically when consumer is run first time so using this command is optional but may be needed when producers start producing events before consumer can be run. === unregister === Removes consumer from given queue. Note consumer must be stopped before issuing this command otherwise it automatically registers again. === config [ [= ... ]] === Show or change queue config. There are several parameters that can be set for each queue shown here with default values: queue_ticker_max_lag (2):: If no tick has happend during given number of seconds then one is generated just to keep queue lag in control. It may be increased if there is no need to deliver events fast. Not much room to decrease it :) queue_ticker_max_count (200):: Threshold number of events in filling batch that triggers tick. Can be increased to encourage PgQ to create larger batches or decreased to encourage faster ticking with smaller batches. queue_ticker_idle_period (60):: Number of seconds that can pass without ticking if no events are coming to queue. These empty ticks are used as keep alive signals for batch jobs and monitoring. queue_rotation_period (2 hours):: Interval of time that may pass before PgQ tries to rotate tables to free up space. Not PgQ can not rotate tables if there are long transactions in database like VACUUM or pg_dump. May be decreased if low on disk space or increased to keep longer history of old events. To small values might affect performance badly because postgres tends to do seq scans on small tables. Too big values may waste disk space. Looking at queue config. $ pgqadm.py mydb.ini config testqueue queue_ticker_max_lag = 3 queue_ticker_max_count = 500 queue_ticker_idle_period = 60 queue_rotation_period = 7200 $ pgqadm.py conf/pgqadm_myprovider.ini config testqueue queue_ticker_max_lag=10 queue_ticker_max_count=300 Change queue bazqueue config to: queue_ticker_max_lag='10', queue_ticker_max_count='300' $ == COMMON OPTIONS == -h, --help:: show help message -q, --quiet:: make program silent -v, --verbose:: make program verbose -d, --daemon:: go background -r, --reload:: reload config (send SIGHUP) -s, --stop:: stop program safely (send SIGINT) -k, --kill:: kill program immidiately (send SIGTERM) // vim:sw=2 et smarttab sts=2: skytools-2.1.13/doc/londiste.config.txt0000644000175000017500000000465011670174255017101 0ustar markomarko= londiste(5) = == NAME == londiste - PostgreSQL replication engine written in python == SYNOPSIS == [londiste] job_name = asd == DESCRIPTION == The londiste configuration file follow the famous .INI syntax. It contains only one section named londiste. Most defaults values are reasonable ones. That means you can only edit provider_db, subscriber_db and pgq_queue_name and be done with londiste configuration. == OPTIONS == You can configure the following options into the londiste section. job_name:: Each Skytools daemon process must have a unique job_name. Londiste uses it also as consumer name when subscribing to queue. provider_db:: Provider database connection string (DSN). subscriber_db:: Subscriber database connection string (DSN). pgq_queue_name:: Name of the queue to read from. Several subscribers can read from same queue. logfile:: Where to log londiste activity. pidfile:: Where to store the pid of the main londiste process, the replay one. lock_timeout:: Few operations take lock on provider (provider add/remove, compare, repair). This parameter specifies timeout in seconds (float) how long a lock can be held. New in version 2.1.8. Default: 10 loop_delay:: How often to poll events from provider. In seconds (float). Default: 1. pgq_lazy_fetch:: How many events to fetch at a time when processing a batch. Useful when you know a single transaction (maintenance +UPDATE+ command, e.g.) will produce a lot of events in a single batch. When lazily fetching, a cursor is used so as to still process a single batch in a single transaction. Default: 0, always fetch all events of the batch, not using a cursor. log_count:: Number of log files to keep. Default: 3 log_size:: Max size for one log file. File is rotated if max size is reached. Default: 10485760 (10M) use_skylog:: If set, search for `[./skylog.ini, ~/.skylog.ini, /etc/skylog.ini]`. If found then the file is used as config file for Pythons `logging` module. It allows setting up fully customizable logging setup. Default: 0 == EXAMPLE == [londiste] job_name = test_to_subcriber provider_db = dbname=provider port=6000 host=127.0.0.1 subscriber_db = dbname=subscriber port=6000 host=127.0.0.1 # it will be used as sql ident so no dots/spaces pgq_queue_name = londiste.replika logfile = /tmp/%(job_name)s.log pidfile = /tmp/%(job_name)s.pid == SEE ALSO == londiste(1) skytools-2.1.13/doc/fixman.py0000755000175000017500000000042511670174255015106 0ustar markomarko#! /usr/bin/env python import sys,re # hacks to force empty lines into manpage ln1 = r"\1\2" xml = sys.stdin.read() xml = re.sub(r"(\s*)(\s*)( .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "BULK_LOADER" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" bulk_loader \- PgQ consumer that loads urlencoded records to slow databases .SH "SYNOPSIS" .sp .nf bulk_loader\&.py [switches] config\&.ini .fi .SH "DESCRIPTION" .sp bulk_loader is PgQ consumer that reads url encoded records from source queue and writes them into tables according to configuration file\&. It is targeted to slow databases that cannot handle applying each row as separate statement\&. Originally written for BizgresMPP/greenplumDB which have very high per\-statement overhead, but can also be used to load regular PostgreSQL database that cannot manage regular replication\&. .sp Behaviour properties: \- reads urlencoded "logutriga" records\&. \- does not do partitioning, but allows optionally redirect table events\&. \- does not keep event order\&. \- always loads data with COPY, either directly to main table (INSERTs) or to temp tables (UPDATE/COPY) then applies from there\&. .sp Events are usually procuded by pgq\&.logutriga()\&. Logutriga adds all the data of the record into the event (also in case of updates and deletes)\&. .SH "QUICK-START" .sp Basic bulk_loader setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} pgq and logutriga must be installed in source databases\&. See pgqadm man page for details\&. target database must also have pgq_ext schema\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} edit a bulk_loader configuration file, say bulk_loader_sample\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} create source queue .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini create .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} Tune source queue to have big batches: .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini config ticker_max_count="10000" ticker_max_lag="10 minutes" ticker_idle_period="10 minutes" .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} create target database and tables in it\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 6.\h'+01'\c .\} .el \{\ .sp -1 .IP " 6." 4.2 .\} launch bulk_loader in daemon mode .sp .if n \{\ .RS 4 .\} .nf $ bulk_loader\&.py \-d bulk_loader_sample\&.ini .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 7.\h'+01'\c .\} .el \{\ .sp -1 .IP " 7." 4.2 .\} start producing events (create logutriga trggers on tables) CREATE OR REPLACE TRIGGER trig_bulk_replica AFTER INSERT OR UPDATE ON some_table FOR EACH ROW EXECUTE PROCEDURE pgq\&.logutriga(\fI\fR) .RE .SH "CONFIG" .SS "Common configuration parameters" .PP job_name .RS 4 Name for particulat job the script does\&. Script will log under this name to logdb/logserver\&. The name is also used as default for PgQ consumer name\&. It should be unique\&. .RE .PP pidfile .RS 4 Location for pid file\&. If not given, script is disallowed to daemonize\&. .RE .PP logfile .RS 4 Location for log file\&. .RE .PP loop_delay .RS 4 If continuisly running process, how long to sleep after each work loop, in seconds\&. Default: 1\&. .RE .PP connection_lifetime .RS 4 Close and reconnect older database connections\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. .RE .SS "Common PgQ consumer parameters" .PP pgq_queue_name .RS 4 Queue name to attach to\&. No default\&. .RE .PP pgq_consumer_id .RS 4 Consumers ID to use when registering\&. Default: %(job_name)s .RE .SS "Config options specific to bulk_loader" .PP src_db .RS 4 Connect string for source database where the queue resides\&. .RE .PP dst_db .RS 4 Connect string for target database where the tables should be created\&. .RE .PP remap_tables .RS 4 Optional parameter for table redirection\&. Contains comma\-separated list of : pairs\&. Eg: oldtable1:newtable1, oldtable2:newtable2\&. .RE .PP load_method .RS 4 Optional parameter for load method selection\&. Available options: .RE .PP 0 .RS 4 UPDATE as UPDATE from temp table\&. This is default\&. .RE .PP 1 .RS 4 UPDATE as DELETE+COPY from temp table\&. .RE .PP 2 .RS 4 merge INSERTs with UPDATEs, then do DELETE+COPY from temp table\&. .RE .SH "LOGUTRIGA EVENT FORMAT" .sp PgQ trigger function pgq\&.logutriga() sends table change event into queue in following format: .PP ev_type .RS 4 (op || ":" || pkey_fields)\&. Where op is either "I", "U" or "D", corresponging to insert, update or delete\&. And pkey_fields is comma\-separated list of primary key fields for table\&. Operation type is always present but pkey_fields list can be empty, if table has no primary keys\&. Example: I:col1,col2 .RE .PP ev_data .RS 4 Urlencoded record of data\&. It uses db\-specific urlecoding where existence of \fI=\fR is meaningful \- missing \fI=\fR means NULL, present \fI=\fR means literal value\&. Example: id=3&name=str&nullvalue&emptyvalue= .RE .PP ev_extra1 .RS 4 Fully qualified table name\&. .RE .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE skytools-2.1.13/doc/londiste.ref.txt0000644000175000017500000002213211670174255016403 0ustar markomarko = Londiste Reference = == Notes == === PgQ daemon === Londiste runs as a consumer on PgQ. Thus `pgqadm.py ticker` must be running on provider database. It is preferable to run it in same machine, because it needs low latency, but that is not a requirement. For monitoring you can use `pgqadm.py status` command. === Table Names === Londiste internally uses table names always fully schema-qualified. If table name without schema is given on command line, it just puts "public." in front of it, without looking at search_path. === PgQ events === ==== Table change event ==== Those events will be inserted by triggers on tables. * ev_type = 'I' / 'U' / 'D' * ev_data = partial SQL statement - the part between `[]` is removed: - `[ INSERT INTO table ] (column1, column2) values (value1, value2)` - `[ UPDATE table SET ] column2=value2 WHERE pkeycolumn1 = value1` - `[ DELETE FROM table WHERE ] pkeycolumn1 = value1` * ev_extra1 = table name with schema Such partial SQL format is used for 2 reasons - to conserve space and to make possible to redirect events to another table. ==== Registration change event ==== Those events will be inserted by `provider add` and `provider remove` commands. Then full registered tables list will be sent to the queue so subscribers can update their own registrations. * ev_type = 'T' * ev_data = comma-separated list of table names. Currently subscribers only remove tables that were removed from provider. In the future it's possible to make subscribers also automatically add tables that were added on provider. == log file == Londiste normal log consist just of statistics log-lines, key-value pairs between `{}`. Their meaning: * count: how many event was in batch. * ignored: how many of them was ignores - table not registered on subscriber or not yet in sync. * duration: how long the batch processing took. Example: {count: 110, duration: 0.88} == Commands for managing provider database == === provider install === londiste.py provider install Installs code into provider and subscriber database and creates queue. Equivalent to doing following by hand: CREATE LANGUAGE plpgsql; CREATE LANGUAGE plpython; \i .../contrib/txid.sql \i .../contrib/logtriga.sql \i .../contrib/pgq.sql \i .../contrib/londiste.sql select pgq.create_queue(queue name); Notes: * The schema/tables are installed under user Londiste is configured to run. If you prefer to run Londiste under non-admin user, they should also be installed by hand. === provider add === londiste.py provider add
... Registers table on provider database and adds trigger to the table that will send events to the queue. === provider remove === londiste.py provider remove
... Unregisters table on provider side and removes triggers on table. The event about table removal is also sent to the queue, so all subscriber unregister table from their end also. === provider tables === londiste.py provider tables Shows registered tables on provider side. === provider seqs === londiste.py provider seqs Shows registered sequences on provider side. == Commands for managing subscriber database == === subscriber install === londiste.py subscriber install Installs code into subscriber database. Equivalent to doing following by hand: CREATE LANGUAGE plpgsql; \i .../contrib/londiste.sql This will be done under Londiste user, if the tables should be owned by someone else, it needs to be done by hand. === subscriber add === londiste.py subscriber add
... [--excect-sync | --skip-truncate | --force] Registers table on subscriber side. Switches --expect-sync:: Table is tagged as in-sync so initial COPY is skipped. --skip-truncate:: When doing initial COPY, don't remove old data. --force:: Ignore table structure differences. === subscriber remove === londiste.py subscriber remove
... Unregisters the table from subscriber. No events will be applied to the table anymore. Actual table will not be touched. === subscriber resync === londiste.py subscriber resync
... Tags tables are "not synced." Later replay process will notice this and launch `copy` process to sync the table again. == Replication commands == === replay === The actual replication process. Should be run as daemon with `-d` switch, because it needs to be always running. It main task is to get a batches from PgQ and apply them in one transaction. Basic logic: * Get batch from PgQ queue on provider. See if it is already applied to subsciber, skip the batch in that case. * Management actions, can do transactions on subscriber: - Load table state from subscriber, to be up-to-date on registrations and `copy` processes running in parallel. - If a `copy` process wants to give table over to main process, wait until `copy` process catches-up. - If there is a table that is not synced and no `copy` process is already running, launch new `copy` process. - If there are sequences registered on subscriber, look latest state of them on provider and apply it to subscriber. * Event replay, all in one transaction on subscriber: - Apply events from the batch, only for tables that are registered on subscriber and are in sync. - Store tick_id on subscriber. === copy (internal) === Internal command for initial SYNC. Launched by `replay` if it notices that some tables are not in sync. The reason to do table copying in separate process is to avoid locking down main replay process for long time. When using either +-s+ or +-k+ to terminate a running londiste instance, londiste will check if a +COPY+ subprocess is running and kill it first, by sending it SIGTERM. When replay starts, it will check if a table is in a state which should be handled by a COPY subprocess, and if it's the case will ensure that such a process exists and run it it necessary. Basic logic: * Register on the same queue in parallel with different name. * One transaction on subscriber: - Drop constraints and indexes. - Truncate table. - COPY data in. - Restore constraints and indexes. - Tag the table as `catching-up`. * When catching-up, the `copy` process acts as regular `replay` process but just for one table. * When it reaches queue end, when no more batches are immidiately available, it hands the table over to main `replay` process. State changes between `replay` and `copy`: State | Owner | What is done ---------------------+--------+-------------------- NULL | replay | Changes state to "in-copy", launches londiste.py copy process, continues with it's work in-copy | copy | drops indexes, truncates, copies data in, restores indexes, changes state to "catching-up" catching-up | copy | replay events for that table only until no more batches (means current moment), | | change state to "wanna-sync:" and wait for state to change wanna-sync: | replay | catch up to given tick_id, change state to "do-sync:" and wait for state to change do-sync: | copy | catch up to given tick_id, both replay and copy must now be at same position. change state to "ok" and exit ok | replay | synced table, events can be applied Such state changes must guarantee that any process can die at any time and by just restarting it can continue where it left. "subscriber add" registers table with `NULL` state. "subscriber add --expect-sync" registers table with `ok` state. "subscriber resync" sets table state to `NULL`. == Utility commands == === repair === it tries to achieve a state where tables should be in sync and then compares them and writes out SQL statements that would fix differences. Syncing happens by locking provider tables against updates and then waiting unitl `replay` has applied all pending changes to subscriber database. As this is dangerous operation, it has hardwired limit of 10 seconds for locking. If `replay process does not catch up in that time, locks are releases and operation is canceled. Comparing happens by dumping out table from both sides, sorting them and then comparing line-by-line. As this is CPU and memory-hungry operation, good practice is to run the `repair` command on third machine, to avoid consuming resources on neither provider nor subscriber. === compare === it syncs tables like repair, but just runs SELECT count(*) on both sides, to get a little bit cheaper but also less precise way of checking if tables are in sync. == Config file == [londiste] job_name = test_to_subcriber # source database, where the queue resides provider_db = dbname=provider port=6000 host=127.0.0.1 # destination database subscriber_db = dbname=subscriber port=6000 host=127.0.0.1 # the queue where to listen on pgq_queue_name = londiste.replika # where to log logfile = ~/log/%(job_name)s.log # pidfile is used for avoiding duplicate processes pidfile = ~/pid/%(job_name)s.pid skytools-2.1.13/doc/queue_mover.10000644000175000017500000001346411727600400015661 0ustar markomarko'\" t .\" Title: queue_mover .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "QUEUE_MOVER" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" queue_mover \- PgQ consumer that copies data from one queue to another\&. .SH "SYNOPSIS" .sp .nf queue_mover\&.py [switches] config\&.ini .fi .SH "DESCRIPTION" .sp queue_mover is PgQ consumer that transports events from source queue into target queue\&. One use case is when events are produced in several databases then queue_mover is used to consolidate these events into single queue that can then be processed by consumers who need to handle theses events\&. For example in case of patitioned databases it\(cqs convenient to move events from each partition into one central queue database and then process them there\&. That way configuration and dependancies of partiton databases are simpler and more robust\&. Another use case is to move events from OLTP database to batch processing server\&. .sp Transactionality: events will be inserted as one transaction on target side\&. That means only batch_id needs to be tracked on target side\&. .SH "QUICK-START" .sp Basic PgQ setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} PgQ must be installed both in source and target databases\&. See pgqadm man page for details\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} Target database must also have pgq_ext schema installed\&. It is used to keep sync between two databases\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} Create a queue_mover configuration file, say qmover_sourceq_to_targetdb\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} create source and target queues .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py sourcedb_ticker\&.ini create $ pgqadm\&.py targetdb_ticker\&.ini create .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} launch queue mover in daemon mode .sp .if n \{\ .RS 4 .\} .nf $ queue_mover\&.py \-d qmover_sourceq_to_targetdb\&.ini .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 6.\h'+01'\c .\} .el \{\ .sp -1 .IP " 6." 4.2 .\} start producing and consuming events .RE .SH "CONFIG" .SS "Common configuration parameters" .PP job_name .RS 4 Name for particulat job the script does\&. Script will log under this name to logdb/logserver\&. The name is also used as default for PgQ consumer name\&. It should be unique\&. .RE .PP pidfile .RS 4 Location for pid file\&. If not given, script is disallowed to daemonize\&. .RE .PP logfile .RS 4 Location for log file\&. .RE .PP loop_delay .RS 4 If continuisly running process, how long to sleep after each work loop, in seconds\&. Default: 1\&. .RE .PP connection_lifetime .RS 4 Close and reconnect older database connections\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. .RE .SS "Common PgQ consumer parameters" .PP pgq_queue_name .RS 4 Queue name to attach to\&. No default\&. .RE .PP pgq_consumer_id .RS 4 Consumers ID to use when registering\&. Default: %(job_name)s .RE .SS "queue_mover parameters" .PP src_db .RS 4 Source database\&. .RE .PP dst_db .RS 4 Target database\&. .RE .PP dst_queue_name .RS 4 Target queue name\&. .RE .SS "Example config file" .sp .if n \{\ .RS 4 .\} .nf [queue_mover] job_name = eventlog_to_target_mover src_db = dbname=sourcedb dst_db = dbname=targetdb pgq_queue_name = eventlog dst_queue_name = copy_of_eventlog pidfile = log/%(job_name)s\&.pid logfile = pid/%(job_name)s\&.log .fi .if n \{\ .RE .\} .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE .SH "BUGS" .sp Event ID is not kept on target side\&. If needed is can be kept, then event_id seq at target side need to be increased by hand to inform ticker about new events\&. skytools-2.1.13/doc/common.switches.txt0000644000175000017500000000105511670174255017130 0ustar markomarko Following switches are common to all skytools.DBScript-based Python programs. -h, --help:: show help message and exit -q, --quiet:: make program silent -v, --verbose:: make program more verbose -d, --daemon:: make program go background Following switches are used to control already running process. The pidfile is read from config then signal is sent to process id specified there. -r, --reload:: reload config (send SIGHUP) -s, --stop:: stop program safely (send SIGINT) -k, --kill:: kill program immidiately (send SIGTERM) skytools-2.1.13/doc/queue_mover.txt0000644000175000017500000000440311670174255016344 0ustar markomarko = queue_mover(1) = == NAME == queue_mover - PgQ consumer that copies data from one queue to another. == SYNOPSIS == queue_mover.py [switches] config.ini == DESCRIPTION == queue_mover is PgQ consumer that transports events from source queue into target queue. One use case is when events are produced in several databases then queue_mover is used to consolidate these events into single queue that can then be processed by consumers who need to handle theses events. For example in case of patitioned databases it's convenient to move events from each partition into one central queue database and then process them there. That way configuration and dependancies of partiton databases are simpler and more robust. Another use case is to move events from OLTP database to batch processing server. Transactionality: events will be inserted as one transaction on target side. That means only batch_id needs to be tracked on target side. == QUICK-START == Basic PgQ setup and usage can be summarized by the following steps: 1. PgQ must be installed both in source and target databases. See pgqadm man page for details. 2. Target database must also have pgq_ext schema installed. It is used to keep sync between two databases. 3. Create a queue_mover configuration file, say qmover_sourceq_to_targetdb.ini 4. create source and target queues $ pgqadm.py sourcedb_ticker.ini create $ pgqadm.py targetdb_ticker.ini create 5. launch queue mover in daemon mode $ queue_mover.py -d qmover_sourceq_to_targetdb.ini 6. start producing and consuming events == CONFIG == include::common.config.txt[] === queue_mover parameters === src_db:: Source database. dst_db:: Target database. dst_queue_name:: Target queue name. === Example config file === [queue_mover] job_name = eventlog_to_target_mover src_db = dbname=sourcedb dst_db = dbname=targetdb pgq_queue_name = eventlog dst_queue_name = copy_of_eventlog pidfile = log/%(job_name)s.pid logfile = pid/%(job_name)s.log == COMMAND LINE SWITCHES == include::common.switches.txt[] == BUGS == Event ID is not kept on target side. If needed is can be kept, then event_id seq at target side need to be increased by hand to inform ticker about new events. skytools-2.1.13/doc/table_dispatcher.txt0000644000175000017500000000547111670174255017313 0ustar markomarko= table_dispatcher(1) = == NAME == table_dispatcher - PgQ consumer that is used to write source records into partitoned table. == SYNOPSIS == table_dispatcher.py [switches] config.ini == DESCRIPTION == table_dispatcher is PgQ consumer that reads url encoded records from source queue and writes them into partitioned tables according to configuration file. Used to partiton data. For example change log's that need to kept online only shortly can be written to daily tables and then dropped as they become irrelevant. Also allows to select which columns have to be written into target database Creates target tables according to configuration file as needed. == QUICK-START == Basic table_dispatcher setup and usage can be summarized by the following steps: 1. PgQ must be installed in source database. See pgqadm man page for details. Target database must have `pgq_ext` schema installed. 2. edit a table_dispatcher configuration file, say table_dispatcher_sample.ini 3. create source queue $ pgqadm.py ticker.ini create 4. launch table dispatcher in daemon mode $ table_dispatcher.py table_dispatcher_sample.ini -d 5. start producing events == CONFIG == include::common.config.txt[] === table_dispatcher parameters === src_db:: Source database. dst_db:: Target database. dest_table:: Where to put data. when partitioning, will be used as base name part_field:: date field with will be used for partitioning. part_template:: SQL code used to create partition tables. Various magic replacements are done there: _PKEY:: comma separated list of primery key columns. _PARENT:: schema-qualified parent table name. _DEST_TABLE:: schema-qualified partition table. _SCHEMA_TABLE:: same as _DEST_TABLE but dots replaced with "__", to allow use as index names. === Example config === [table_dispatcher] job_name = table_dispatcher_source_table_targetdb src_db = dbname=sourcedb dst_db = dbname=targetdb pgq_queue_name = sourceq logfile = log/%(job_name)s.log pidfile = pid/%(job_name)s.pid # where to put data. when partitioning, will be used as base name dest_table = orders # names of the fields that must be read from source records fields = id, order_date, customer_name # date field with will be used for partitioning part_field = order_date # template used for creating partition tables part_template = create table _DEST_TABLE () inherits (orders); alter table only _DEST_TABLE add constraint _DEST_TABLE_pkey primary key (id); grant select on _DEST_TABLE to group reporting; == COMMAND LINE SWITCHES == include::common.switches.txt[] == LOGUTRIGA EVENT FORMAT == include::common.logutriga.txt[] skytools-2.1.13/doc/table_dispatcher.10000644000175000017500000001645711727600377016644 0ustar markomarko'\" t .\" Title: table_dispatcher .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "TABLE_DISPATCHER" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" table_dispatcher \- PgQ consumer that is used to write source records into partitoned table\&. .SH "SYNOPSIS" .sp .nf table_dispatcher\&.py [switches] config\&.ini .fi .SH "DESCRIPTION" .sp table_dispatcher is PgQ consumer that reads url encoded records from source queue and writes them into partitioned tables according to configuration file\&. Used to partiton data\&. For example change log\(cqs that need to kept online only shortly can be written to daily tables and then dropped as they become irrelevant\&. Also allows to select which columns have to be written into target database Creates target tables according to configuration file as needed\&. .SH "QUICK-START" .sp Basic table_dispatcher setup and usage can be summarized by the following steps: .sp .RS 4 .ie n \{\ \h'-04' 1.\h'+01'\c .\} .el \{\ .sp -1 .IP " 1." 4.2 .\} PgQ must be installed in source database\&. See pgqadm man page for details\&. Target database must have pgq_ext schema installed\&. .RE .sp .RS 4 .ie n \{\ \h'-04' 2.\h'+01'\c .\} .el \{\ .sp -1 .IP " 2." 4.2 .\} edit a table_dispatcher configuration file, say table_dispatcher_sample\&.ini .RE .sp .RS 4 .ie n \{\ \h'-04' 3.\h'+01'\c .\} .el \{\ .sp -1 .IP " 3." 4.2 .\} create source queue .sp .if n \{\ .RS 4 .\} .nf $ pgqadm\&.py ticker\&.ini create .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 4.\h'+01'\c .\} .el \{\ .sp -1 .IP " 4." 4.2 .\} launch table dispatcher in daemon mode .sp .if n \{\ .RS 4 .\} .nf $ table_dispatcher\&.py table_dispatcher_sample\&.ini \-d .fi .if n \{\ .RE .\} .RE .sp .RS 4 .ie n \{\ \h'-04' 5.\h'+01'\c .\} .el \{\ .sp -1 .IP " 5." 4.2 .\} start producing events .RE .SH "CONFIG" .SS "Common configuration parameters" .PP job_name .RS 4 Name for particulat job the script does\&. Script will log under this name to logdb/logserver\&. The name is also used as default for PgQ consumer name\&. It should be unique\&. .RE .PP pidfile .RS 4 Location for pid file\&. If not given, script is disallowed to daemonize\&. .RE .PP logfile .RS 4 Location for log file\&. .RE .PP loop_delay .RS 4 If continuisly running process, how long to sleep after each work loop, in seconds\&. Default: 1\&. .RE .PP connection_lifetime .RS 4 Close and reconnect older database connections\&. .RE .PP log_count .RS 4 Number of log files to keep\&. Default: 3 .RE .PP log_size .RS 4 Max size for one log file\&. File is rotated if max size is reached\&. Default: 10485760 (10M) .RE .PP use_skylog .RS 4 If set, search for [\&./skylog\&.ini, ~/\&.skylog\&.ini, /etc/skylog\&.ini]\&. If found then the file is used as config file for Pythons logging module\&. It allows setting up fully customizable logging setup\&. .RE .SS "Common PgQ consumer parameters" .PP pgq_queue_name .RS 4 Queue name to attach to\&. No default\&. .RE .PP pgq_consumer_id .RS 4 Consumers ID to use when registering\&. Default: %(job_name)s .RE .SS "table_dispatcher parameters" .PP src_db .RS 4 Source database\&. .RE .PP dst_db .RS 4 Target database\&. .RE .PP dest_table .RS 4 Where to put data\&. when partitioning, will be used as base name .RE .PP part_field .RS 4 date field with will be used for partitioning\&. .RE .PP part_template .RS 4 SQL code used to create partition tables\&. Various magic replacements are done there: .RE .PP _PKEY .RS 4 comma separated list of primery key columns\&. .RE .PP _PARENT .RS 4 schema\-qualified parent table name\&. .RE .PP _DEST_TABLE .RS 4 schema\-qualified partition table\&. .RE .PP _SCHEMA_TABLE .RS 4 same as \fIDEST_TABLE but dots replaced with "_\fR", to allow use as index names\&. .RE .SS "Example config" .sp .if n \{\ .RS 4 .\} .nf [table_dispatcher] job_name = table_dispatcher_source_table_targetdb .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf src_db = dbname=sourcedb dst_db = dbname=targetdb .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf pgq_queue_name = sourceq .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf logfile = log/%(job_name)s\&.log pidfile = pid/%(job_name)s\&.pid .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # where to put data\&. when partitioning, will be used as base name dest_table = orders .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # names of the fields that must be read from source records fields = id, order_date, customer_name .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # date field with will be used for partitioning part_field = order_date .fi .if n \{\ .RE .\} .sp .if n \{\ .RS 4 .\} .nf # template used for creating partition tables part_template = create table _DEST_TABLE () inherits (orders); alter table only _DEST_TABLE add constraint _DEST_TABLE_pkey primary key (id); grant select on _DEST_TABLE to group reporting; .fi .if n \{\ .RE .\} .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE .SH "LOGUTRIGA EVENT FORMAT" .sp PgQ trigger function pgq\&.logutriga() sends table change event into queue in following format: .PP ev_type .RS 4 (op || ":" || pkey_fields)\&. Where op is either "I", "U" or "D", corresponging to insert, update or delete\&. And pkey_fields is comma\-separated list of primary key fields for table\&. Operation type is always present but pkey_fields list can be empty, if table has no primary keys\&. Example: I:col1,col2 .RE .PP ev_data .RS 4 Urlencoded record of data\&. It uses db\-specific urlecoding where existence of \fI=\fR is meaningful \- missing \fI=\fR means NULL, present \fI=\fR means literal value\&. Example: id=3&name=str&nullvalue&emptyvalue= .RE .PP ev_extra1 .RS 4 Fully qualified table name\&. .RE skytools-2.1.13/doc/skytools_upgrade.10000644000175000017500000000511411727600407016723 0ustar markomarko'\" t .\" Title: skytools_upgrade .\" Author: [FIXME: author] [see http://docbook.sf.net/el/author] .\" Generator: DocBook XSL Stylesheets v1.75.2 .\" Date: 03/13/2012 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "SKYTOOLS_UPGRADE" "1" "03/13/2012" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" ----------------------------------------------------------------- .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .\" http://bugs.debian.org/507673 .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" ----------------------------------------------------------------- .\" * set default formatting .\" ----------------------------------------------------------------- .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .\" ----------------------------------------------------------------- .\" * MAIN CONTENT STARTS HERE * .\" ----------------------------------------------------------------- .SH "NAME" skytools_upgrade \- utility for upgrading Skytools code in databases\&. .SH "SYNOPSIS" .sp .nf skytools_upgrade\&.py connstr [connstr \&.\&.] .fi .SH "DESCRIPTION" .sp It connects to given database, then looks for following schemas: .PP pgq .RS 4 Main PgQ code\&. .RE .PP pgq_ext .RS 4 PgQ batch/event tracking in remote database\&. .RE .PP londiste .RS 4 Londiste replication\&. .RE .sp If schema exists, its version is detected by querying \&.version() function under schema\&. If the function does not exists, there is some heiristics built in to differentiate between 2\&.1\&.4 and 2\&.1\&.5 version of ther schemas\&. .sp If detected that version is older that current, it is upgraded by applying upgrade scripts in order\&. .SH "COMMAND LINE SWITCHES" .sp Following switches are common to all skytools\&.DBScript\-based Python programs\&. .PP \-h, \-\-help .RS 4 show help message and exit .RE .PP \-q, \-\-quiet .RS 4 make program silent .RE .PP \-v, \-\-verbose .RS 4 make program more verbose .RE .PP \-d, \-\-daemon .RS 4 make program go background .RE .sp Following switches are used to control already running process\&. The pidfile is read from config then signal is sent to process id specified there\&. .PP \-r, \-\-reload .RS 4 reload config (send SIGHUP) .RE .PP \-s, \-\-stop .RS 4 stop program safely (send SIGINT) .RE .PP \-k, \-\-kill .RS 4 kill program immidiately (send SIGTERM) .RE skytools-2.1.13/doc/scriptmgr.txt0000644000175000017500000000434511670174255016027 0ustar markomarko= scriptmgr(1) = == NAME == scriptmgr - utility for controlling other skytools scripts. == SYNOPSIS == scriptmgr.py [switches] config.ini [-a | job_name ... ] == DESCRIPTION == scriptmgr is used to manage several scripts together. It discovers potential jobs based on config file glob expression. From config file it gets both job_name and service type (that is the main section name eg [cube_dispatcher]). For each service type there is subsection in the config how to handle it. Unknown services are ignored. == COMMANDS == === status === scriptmgr config.ini status Show status for all known jobs. === start === scriptmgr config.ini start -a scriptmgr config.ini start job_name1 job_name2 ... launch script(s) that are not running. === stop === scriptmgr config.ini stop -a scriptmgr config.ini stop job_name1 job_name2 ... stop script(s) that are running. === restart === scriptmgr config.ini restart -a scriptmgr config.ini restart job_name1 job_name2 ... restart scripts. === reload === scriptmgr config.ini reload -a scriptmgr config.ini reload job_name1 job_name2 ... Send SIGHUP to scripts that are running. == CONFIG == include::common.config.txt[] === scriptmgr parameters === config_list:: List of glob patters for finding config files. Example: config_list = ~/dbscripts/conf/*.ini, ~/random/conf/*.ini === Service section parameters === cwd:: Working directory for script. args:: Arguments to give to script, in addition to `-d`. script:: Path to script. Unless script is in PATH, full path should be given. disabled:: If this service should be ignored. === Example config file === [scriptmgr] job_name = scriptmgr_livesrv logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid config_list = ~/scripts/conf/*.ini # defaults for all service sections [DEFAULT] cwd = ~/scripts [table_dispatcher] script = table_dispatcher.py args = -v [cube_dispatcher] script = python2.4 cube_dispatcher.py disabled = 1 [pgqadm] script = ~/scripts/pgqadm.py args = ticker == COMMAND LINE SWITCHES == include::common.switches.txt[] Options specific to scriptmgr: -a, --all:: Operate on all non-disabled scripts. skytools-2.1.13/doc/config.txt0000644000175000017500000000041711670174255015256 0ustar markomarko == Common options == job_name pidfile logfile loop_delay connection_lifetime use_skylog == Common to PGQ scripts == pgq_queue_name pgq_consumer_id == Londiste == provider_db = subscriber_db = == PgqAdm == maint_delay = queue_refresh_period ticker_log_delay skytools-2.1.13/setup.py0000755000175000017500000000400311670174255014213 0ustar markomarko#! /usr/bin/env python import sys, os.path, re, glob from distutils.core import setup from distutils.extension import Extension # check if configure has run if not os.path.isfile('config.mak'): print "please run ./configure && make first" print "Note: setup.py is supposed to be run from Makefile" sys.exit(1) # load version buf = open("configure.ac","r").read(256) m = re.search("AC_INIT[(][^,]*,\s+([^)]*)[)]", buf) ac_ver = m.group(1) share_dup_files = [ 'sql/pgq/pgq.sql', 'sql/londiste/londiste.sql', 'sql/pgq_ext/pgq_ext.sql', 'sql/logtriga/logtriga.sql', ] if os.path.isfile('sql/txid/txid.sql'): share_dup_files.append('sql/txid/txid.sql') # run actual setup setup( name = "skytools", license = "BSD", version = ac_ver, maintainer = "Marko Kreen", maintainer_email = "marko.kreen@skype.net", url = "http://pgfoundry.org/projects/skytools/", package_dir = {'': 'python'}, packages = ['skytools', 'londiste', 'pgq'], scripts = ['python/londiste.py', 'python/pgqadm.py', 'python/walmgr.py', 'scripts/cube_dispatcher.py', 'scripts/queue_mover.py', 'scripts/table_dispatcher.py', 'scripts/bulk_loader.py', 'scripts/scriptmgr.py', 'scripts/queue_splitter.py', 'scripts/skytools_upgrade.py', ], data_files = [ ('share/doc/skytools/conf', [ 'python/conf/londiste.ini', 'python/conf/pgqadm.ini', 'python/conf/skylog.ini', 'python/conf/wal-master.ini', 'python/conf/wal-slave.ini', 'scripts/queue_mover.ini.templ', 'scripts/queue_splitter.ini.templ', 'scripts/cube_dispatcher.ini.templ', 'scripts/table_dispatcher.ini.templ', 'scripts/bulk_loader.ini.templ', 'scripts/scriptmgr.ini.templ', ]), ('share/skytools', share_dup_files), ('share/skytools/upgrade/final', glob.glob('upgrade/final/*.sql')), ], ext_modules=[Extension("skytools._cquoting", ['python/modules/cquoting.c'])], ) skytools-2.1.13/scripts/0000755000175000017500000000000011727601174014166 5ustar markomarkoskytools-2.1.13/scripts/cube_dispatcher.ini.templ0000644000175000017500000000106111670174255021133 0ustar markomarko[cube_dispatcher] job_name = some_queue_to_cube src_db = dbname=sourcedb_test dst_db = dbname=dataminedb_test pgq_queue_name = udata.some_queue logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid # how many rows are kept: keep_latest, keep_all mode = keep_latest # to_char() fmt for table suffix #dateformat = YYYY_MM_DD # following disables table suffixes: #dateformat = part_template = create table _DEST_TABLE (like _PARENT); alter table only _DEST_TABLE add primary key (_PKEY); skytools-2.1.13/scripts/catsql.py0000755000175000017500000000667111670174255016046 0ustar markomarko#! /usr/bin/env python """Prints out SQL files with psql command execution. Supported psql commands: \i, \cd, \q Others are skipped. Aditionally does some pre-processing for NDoc. NDoc is looks nice but needs some hand-holding. Bug: - function def end detection searches for 'as'/'is' but does not check word boundaries - finds them even in function name. That means in main conf, as/is must be disabled and $ ' added. This script can remove the unnecessary AS from output. Niceties: - Ndoc includes function def in output only if def is after comment. But for SQL functions its better to have it after def. This script can swap comment and def. - Optionally remove CREATE FUNCTION (OR REPLACE) from def to keep it shorter in doc. Note: - NDoc compares real function name and name in comment. if differ, it decides detection failed. """ import sys, os, re, getopt def usage(x): print "usage: catsql [--ndoc] FILE [FILE ...]" sys.exit(x) # NDoc specific changes cf_ndoc = 0 # compile regexes func_re = r"create\s+(or\s+replace\s+)?function\s+" func_rc = re.compile(func_re, re.I) comm_rc = re.compile(r"^\s*([#]\s*)?(?P--.*)", re.I) end_rc = re.compile(r"\b([;]|begin|declare|end)\b", re.I) as_rc = re.compile(r"\s+as\s+", re.I) cmd_rc = re.compile(r"^\\([a-z]*)(\s+.*)?", re.I) # conversion func def fix_func(ln): # if ndoc, replace AS with ' ' if cf_ndoc: return as_rc.sub(' ', ln) else: return ln # got function def def proc_func(f, ln): # remove CREATE OR REPLACE if cf_ndoc: ln = func_rc.sub('', ln) ln = fix_func(ln) pre_list = [ln] comm_list = [] n_comm = 0 while 1: ln = f.readline() if not ln: break com = None if cf_ndoc: com = comm_rc.search(ln) if cf_ndoc and com: pos = com.start('com') comm_list.append(ln[pos:]) elif end_rc.search(ln): break elif len(comm_list) > 0: break else: pre_list.append(fix_func(ln)) if len(comm_list) > 2: map(sys.stdout.write, comm_list) map(sys.stdout.write, pre_list) else: map(sys.stdout.write, pre_list) map(sys.stdout.write, comm_list) if ln: sys.stdout.write(fix_func(ln)) def cat_file(fn): sys.stdout.write("\n") f = open(fn) while 1: ln = f.readline() if not ln: break m = cmd_rc.search(ln) if m: cmd = m.group(1) if cmd == "i": # include a file fn2 = m.group(2).strip() cat_file(fn2) elif cmd == "q": # quit sys.exit(0) elif cmd == "cd": # chdir dir = m.group(2).strip() os.chdir(dir) else: # skip all others pass else: if func_rc.search(ln): # function header proc_func(f, ln) else: # normal sql sys.stdout.write(ln) sys.stdout.write("\n") def main(): global cf_ndoc try: opts, args = getopt.gnu_getopt(sys.argv[1:], 'h', ['ndoc']) except getopt.error, d: print d usage(1) for o, v in opts: if o == "-h": usage(0) elif o == "--ndoc": cf_ndoc = 1 for fn in args: cat_file(fn) if __name__ == '__main__': main() skytools-2.1.13/scripts/queue_splitter.py0000755000175000017500000000214411670174255017620 0ustar markomarko#! /usr/bin/env python # puts events into queue specified by field from 'queue_field' config parameter import sys, os, pgq, skytools class QueueSplitter(pgq.SerialConsumer): def __init__(self, args): pgq.SerialConsumer.__init__(self, "queue_splitter", "src_db", "dst_db", args) def process_remote_batch(self, db, batch_id, ev_list, dst_db): cache = {} queue_field = self.cf.get('queue_field', 'extra1') for ev in ev_list: row = [ev.type, ev.data, ev.extra1, ev.extra2, ev.extra3, ev.extra4, ev.time] queue = ev.__getattr__(queue_field) if queue not in cache: cache[queue] = [] cache[queue].append(row) ev.tag_done() # should match the composed row fields = ['type', 'data', 'extra1', 'extra2', 'extra3', 'extra4', 'time'] # now send them to right queues curs = dst_db.cursor() for queue, rows in cache.items(): pgq.bulk_insert_events(curs, rows, fields, queue) if __name__ == '__main__': script = QueueSplitter(sys.argv[1:]) script.start() skytools-2.1.13/scripts/queue_splitter.ini.templ0000644000175000017500000000043311670174255021063 0ustar markomarko[queue_splitter] job_name = queue_splitter_test src_db = dbname=sourcedb_test dst_db = dbname=destdb_test pgq_queue_name = source_queue logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid use_skylog = 0 skytools-2.1.13/scripts/skytools_upgrade.py0000755000175000017500000000447311670174255020153 0ustar markomarko#! /usr/bin/env python import sys, os, re, skytools ver_rx = r"(\d+)([.](\d+)([.](\d+))?)?" ver_rc = re.compile(ver_rx) def detect_londiste215(curs): return skytools.exists_table(curs, 'londiste.subscriber_pending_fkeys') version_list = [ ['pgq', '2.1.5', 'v2.1.5_pgq_core.sql', None], # those vers did not have version func ['pgq_ext', '2.1.5', 'v2.1.5_pgq_ext.sql', None], # ok to reapply ['londiste', '2.1.5', 'v2.1.5_londiste.sql', detect_londiste215], # not ok to reapply ['pgq_ext', '2.1.6', 'v2.1.6_pgq_ext.sql', None], ['londiste', '2.1.6', 'v2.1.6_londiste.sql', None], ['pgq', '2.1.7', 'v2.1.7_pgq_core.sql', None], ['londiste', '2.1.7', 'v2.1.7_londiste.sql', None], ['pgq', '2.1.8', 'v2.1.8_pgq_core.sql', None], ] def parse_ver(ver): m = ver_rc.match(ver) if not ver: return 0 v0 = int(m.group(1) or "0") v1 = int(m.group(3) or "0") v2 = int(m.group(5) or "0") return ((v0 * 100) + v1) * 100 + v2 def check_version(curs, schema, new_ver_str, recheck_func=None): funcname = "%s.version" % schema if not skytools.exists_function(curs, funcname, 0): if recheck_func is not None: return recheck_func(curs) else: return 0 q = "select %s()" % funcname curs.execute(q) old_ver_str = curs.fetchone()[0] new_ver = parse_ver(new_ver_str) old_ver = parse_ver(old_ver_str) return old_ver >= new_ver class DbUpgrade(skytools.DBScript): def upgrade(self, db): curs = db.cursor() for schema, ver, sql, recheck_fn in version_list: if not skytools.exists_schema(curs, schema): continue if check_version(curs, schema, ver, recheck_fn): continue fn = "upgrade/final/%s" % sql skytools.installer_apply_file(db, fn, self.log) def work(self): self.set_single_loop(1) # loop over hosts for cstr in self.args: db = self.get_database('db', connstr = cstr, autocommit = 1) self.upgrade(db) self.close_database('db') def load_config(self): return skytools.Config(self.service_name, None, user_defs = {'use_skylog': '0', 'job_name': 'db_upgrade'}) if __name__ == '__main__': script = DbUpgrade('db_upgrade', sys.argv[1:]) script.start() skytools-2.1.13/scripts/queue_mover.ini.templ0000644000175000017500000000047011670174255020346 0ustar markomarko[queue_mover] job_name = queue_mover_test src_db = dbname=sourcedb_test dst_db = dbname=dataminedb_test pgq_queue_name = source_queue dst_queue_name = dest_queue logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid use_skylog = 0 skytools-2.1.13/scripts/cube_dispatcher.py0000755000175000017500000001346711670174255017704 0ustar markomarko#! /usr/bin/env python # it accepts urlencoded rows for multiple tables from queue # and insert them into actual tables, with partitioning on tick time import sys, os, pgq, skytools DEF_CREATE = """ create table _DEST_TABLE (like _PARENT); alter table only _DEST_TABLE add primary key (_PKEY); """ class CubeDispatcher(pgq.SerialConsumer): def __init__(self, args): pgq.SerialConsumer.__init__(self, "cube_dispatcher", "src_db", "dst_db", args) self.dateformat = self.cf.get('dateformat', 'YYYY_MM_DD') self.part_template = self.cf.get('part_template', DEF_CREATE) mode = self.cf.get('mode', 'keep_latest') if mode == 'keep_latest': self.keep_latest = 1 elif mode == 'keep_all': self.keep_latest = 0 else: self.log.fatal('wrong mode setting') sys.exit(1) def get_part_date(self, batch_id): if not self.dateformat: return None # fetch and format batch date src_db = self.get_database('src_db') curs = src_db.cursor() q = 'select to_char(batch_end, %s) from pgq.get_batch_info(%s)' curs.execute(q, [self.dateformat, batch_id]) src_db.commit() return curs.fetchone()[0] def process_remote_batch(self, src_db, batch_id, ev_list, dst_db): # actual processing date_str = self.get_part_date(batch_id) self.dispatch(dst_db, ev_list, self.get_part_date(batch_id)) def dispatch(self, dst_db, ev_list, date_str): """Actual event processing.""" # get tables and sql tables = {} sql_list = [] for ev in ev_list: if date_str: tbl = "%s_%s" % (ev.extra1, date_str) else: tbl = ev.extra1 sql = self.make_sql(tbl, ev) sql_list.append(sql) if not tbl in tables: tables[tbl] = self.get_table_info(ev, tbl) ev.tag_done() # create tables if needed self.check_tables(dst_db, tables) # insert into data tables curs = dst_db.cursor() block = [] for sql in sql_list: self.log.debug(sql) block.append(sql) if len(block) > 100: curs.execute("\n".join(block)) block = [] if len(block) > 0: curs.execute("\n".join(block)) def get_table_info(self, ev, tbl): klist = [skytools.quote_ident(k) for k in ev.key_list.split(',')] inf = { 'parent': ev.extra1, 'table': tbl, 'key_list': ",".join(klist), } return inf def make_sql(self, tbl, ev): """Return SQL statement(s) for that event.""" # parse data data = skytools.db_urldecode(ev.data) # parse tbl info if ev.type.find(':') > 0: op, keys = ev.type.split(':') else: op = ev.type keys = ev.extra2 ev.key_list = keys key_list = keys.split(',') if self.keep_latest and len(key_list) == 0: raise Exception('No pkey on table %s' % tbl) # generate sql if op in ('I', 'U'): if self.keep_latest: sql = "%s %s" % (self.mk_delete_sql(tbl, key_list, data), self.mk_insert_sql(tbl, key_list, data)) else: sql = self.mk_insert_sql(tbl, key_list, data) elif op == "D": if not self.keep_latest: raise Exception('Delete op not supported if mode=keep_all') sql = self.mk_delete_sql(tbl, key_list, data) else: raise Exception('Unknown row op: %s' % op) return sql def mk_delete_sql(self, tbl, key_list, data): # generate delete command whe_list = [] for k in key_list: whe_list.append("%s = %s" % (skytools.quote_ident(k), skytools.quote_literal(data[k]))) whe_str = " and ".join(whe_list) return "delete from %s where %s;" % (skytools.quote_fqident(tbl), whe_str) def mk_insert_sql(self, tbl, key_list, data): # generate insert command col_list = [] val_list = [] for c, v in data.items(): col_list.append(skytools.quote_ident(c)) val_list.append(skytools.quote_literal(v)) col_str = ",".join(col_list) val_str = ",".join(val_list) return "insert into %s (%s) values (%s);" % ( skytools.quote_fqident(tbl), col_str, val_str) def check_tables(self, dcon, tables): """Checks that tables needed for copy are there. If not then creates them. Used by other procedures to ensure that table is there before they start inserting. The commits should not be dangerous, as we haven't done anything with cdr's yet, so they should still be in one TX. Although it would be nicer to have a lock for table creation. """ dcur = dcon.cursor() exist_map = {} for tbl, inf in tables.items(): if skytools.exists_table(dcur, tbl): continue sql = self.part_template sql = sql.replace('_DEST_TABLE', skytools.quote_fqident(inf['table'])) sql = sql.replace('_PARENT', skytools.quote_fqident(inf['parent'])) sql = sql.replace('_PKEY', inf['key_list']) # be similar to table_dispatcher schema_table = inf['table'].replace(".", "__") sql = sql.replace('_SCHEMA_TABLE', skytools.quote_ident(schema_table)) dcur.execute(sql) dcon.commit() self.log.info('%s: Created table %s' % (self.job_name, tbl)) if __name__ == '__main__': script = CubeDispatcher(sys.argv[1:]) script.start() skytools-2.1.13/scripts/scriptmgr.py0000755000175000017500000001646411670174255016572 0ustar markomarko#! /usr/bin/env python """Bulk start/stop of scripts. Reads a bunch of config files and maps them to scripts, then handles those. """ import sys, os, skytools, signal, glob, ConfigParser, time command_usage = """ %prog [options] INI CMD [subcmd args] commands: start [-a | jobname ..] start a job stop [-a | jobname ..] stop a job restart [-a | jobname ..] restart job(s) reload [-a | jobname ..] send reload signal status """ def job_sort_cmp(j1, j2): d1 = j1['service'] + j1['job_name'] d2 = j2['service'] + j2['job_name'] if d1 < d2: return -1 elif d1 > d2: return 1 else: return 0 class ScriptMgr(skytools.DBScript): def init_optparse(self, p = None): p = skytools.DBScript.init_optparse(self, p) p.add_option("-a", "--all", action="store_true", help="apply command to all jobs") p.set_usage(command_usage.strip()) return p def load_jobs(self): self.svc_list = [] self.svc_map = {} self.config_list = [] # load services svc_list = self.cf.sections() svc_list.remove(self.service_name) for svc_name in svc_list: cf = self.cf.clone(svc_name) disabled = cf.getboolean('disabled', 0) defscript = None if disabled: defscript = '/disabled' svc = { 'service': svc_name, 'script': cf.getfile('script', defscript), 'cwd': cf.getfile('cwd'), 'disabled': cf.getboolean('disabled', 0), 'args': cf.get('args', ''), } self.svc_list.append(svc) self.svc_map[svc_name] = svc # generate config list for tmp in self.cf.getlist('config_list'): tmp = os.path.expanduser(tmp) tmp = os.path.expandvars(tmp) for fn in glob.glob(tmp): self.config_list.append(fn) # read jobs for fn in self.config_list: raw = ConfigParser.SafeConfigParser({'job_name':'?', 'service_name':'?'}) raw.read(fn) # skip its own config if raw.has_section(self.service_name): continue got = 0 for sect in raw.sections(): if sect in self.svc_map: got = 1 self.add_job(fn, sect) if not got: self.log.warning('Cannot find service for %s' % fn) def add_job(self, cf_file, service_name): svc = self.svc_map[service_name] cf = skytools.Config(service_name, cf_file) disabled = svc['disabled'] if not disabled: disabled = cf.getboolean('disabled', 0) job = { 'disabled': disabled, 'config': cf_file, 'cwd': svc['cwd'], 'script': svc['script'], 'args': svc['args'], 'service': svc['service'], 'job_name': cf.get('job_name'), 'pidfile': cf.getfile('pidfile', ''), } self.job_list.append(job) self.job_map[job['job_name']] = job def cmd_status(self): for job in self.job_list: os.chdir(job['cwd']) cf = skytools.Config(job['service'], job['config']) pidfile = cf.getfile('pidfile', '') name = job['job_name'] svc = job['service'] if job['disabled']: name += " (disabled)" if not pidfile: print " pidfile? [%s] %s" % (svc, name) elif os.path.isfile(pidfile): print " OK [%s] %s" % (svc, name) else: print " STOPPED [%s] %s" % (svc, name) def cmd_info(self): for job in self.job_list: print job def cmd_start(self, job_name): if job_name not in self.job_map: self.log.error('Unknown job: '+job_name) return 1 job = self.job_map[job_name] if job['disabled']: self.log.info("Skipping %s" % job_name) return 0 self.log.info('Starting %s' % job_name) os.chdir(job['cwd']) pidfile = job['pidfile'] if not pidfile: self.log.warning("No pidfile for %s cannot launch") return 0 if os.path.isfile(pidfile): self.log.warning("Script %s seems running" % job_name) return 0 cmd = "%(script)s %(config)s %(args)s -d" % job res = os.system(cmd) self.log.debug(res) if res != 0: self.log.error('startup failed: %s' % job_name) return 1 else: return 0 def cmd_stop(self, job_name): if job_name not in self.job_map: self.log.error('Unknown job: '+job_name) return job = self.job_map[job_name] if job['disabled']: self.log.info("Skipping %s" % job_name) return self.log.info('Stopping %s' % job_name) self.signal_job(job, signal.SIGINT) def cmd_reload(self, job_name): if job_name not in self.job_map: self.log.error('Unknown job: '+job_name) return job = self.job_map[job_name] if job['disabled']: self.log.info("Skipping %s" % job_name) return self.log.info('Reloading %s' % job_name) self.signal_job(job, signal.SIGHUP) def signal_job(self, job, sig): os.chdir(job['cwd']) pidfile = job['pidfile'] if not pidfile: self.log.warning("No pidfile for %s (%s)" % (job['job_name'], job['config'])) return if os.path.isfile(pidfile): pid = int(open(pidfile).read()) try: os.kill(pid, sig) except Exception, det: self.log.warning("Signaling %s failed: %s" % (job['job_name'], str(det))) else: self.log.warning("Job %s not running" % job['job_name']) def work(self): self.set_single_loop(1) self.job_list = [] self.job_map = {} self.load_jobs() if len(self.args) < 2: print "need command" sys.exit(1) jobs = self.args[2:] if len(jobs) == 0 and self.options.all: for job in self.job_list: jobs.append(job['job_name']) self.job_list.sort(job_sort_cmp) cmd = self.args[1] if cmd == "status": self.cmd_status() return elif cmd == "info": self.cmd_info() return if len(jobs) == 0: print "no jobs given?" sys.exit(1) if cmd == "start": err = 0 for n in jobs: err += self.cmd_start(n) if err > 0: self.log.error('some scripts failed') sys.exit(1) elif cmd == "stop": for n in jobs: self.cmd_stop(n) elif cmd == "restart": for n in jobs: self.cmd_stop(n) time.sleep(2) self.cmd_start(n) elif cmd == "reload": for n in jobs: self.cmd_reload(n) else: print "unknown command:", cmd sys.exit(1) if __name__ == '__main__': script = ScriptMgr('scriptmgr', sys.argv[1:]) script.start() skytools-2.1.13/scripts/scriptmgr.ini.templ0000644000175000017500000000106011670174255020020 0ustar markomarko [scriptmgr] job_name = scriptmgr_cphdb5 config_list = ~/dbscripts/conf/*.ini, ~/random/conf/*.ini logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid #use_skylog = 1 # # defaults for services # [DEFAULT] cwd = ~/dbscripts args = -v # # service descriptions # [cube_dispatcher] script = cube_dispatcher.py [table_dispatcher] script = table_dispatcher.py [bulk_loader] script = bulk_loader.py [londiste] script = londiste.py args = replay [pgqadm] script = pgqadm.py args = ticker # # services to be ignored # [log_checker] disabled = 1 skytools-2.1.13/scripts/table_dispatcher.py0000755000175000017500000000766711670174255020062 0ustar markomarko#! /usr/bin/env python # it loads urlencoded rows for one trable from queue and inserts # them into actual tables, with optional partitioning import sys, os, pgq, skytools DEST_TABLE = "_DEST_TABLE" SCHEMA_TABLE = "_SCHEMA_TABLE" class TableDispatcher(pgq.SerialConsumer): def __init__(self, args): pgq.SerialConsumer.__init__(self, "table_dispatcher", "src_db", "dst_db", args) self.part_template = self.cf.get("part_template", '') self.dest_table = self.cf.get("dest_table") self.part_field = self.cf.get("part_field", '') self.part_method = self.cf.get("part_method", 'daily') if self.part_method not in ('daily', 'monthly'): raise Exception('bad part_method') if self.cf.get("fields", "*") == "*": self.field_map = None else: self.field_map = {} for fval in self.cf.getlist('fields'): tmp = fval.split(':') if len(tmp) == 1: self.field_map[tmp[0]] = tmp[0] else: self.field_map[tmp[0]] = tmp[1] def process_remote_batch(self, src_db, batch_id, ev_list, dst_db): # actual processing self.dispatch(dst_db, ev_list) def dispatch(self, dst_db, ev_list): """Generic dispatcher.""" # load data tables = {} for ev in ev_list: row = skytools.db_urldecode(ev.data) # guess dest table if self.part_field: if self.part_field == "_EVTIME": partval = str(ev.creation_date) else: partval = str(row[self.part_field]) partval = partval.split(' ')[0] date = partval.split('-') if self.part_method == 'monthly': date = date[:2] suffix = '_'.join(date) tbl = "%s_%s" % (self.dest_table, suffix) else: tbl = self.dest_table # map fields if self.field_map is None: dstrow = row else: dstrow = {} for k, v in self.field_map.items(): dstrow[v] = row[k] # add row into table if not tbl in tables: tables[tbl] = [dstrow] else: tables[tbl].append(dstrow) ev.tag_done() # create tables if needed self.check_tables(dst_db, tables) # insert into data tables curs = dst_db.cursor() for tbl, tbl_rows in tables.items(): skytools.magic_insert(curs, tbl, tbl_rows) def check_tables(self, dcon, tables): """Checks that tables needed for copy are there. If not then creates them. Used by other procedures to ensure that table is there before they start inserting. The commits should not be dangerous, as we haven't done anything with cdr's yet, so they should still be in one TX. Although it would be nicer to have a lock for table creation. """ dcur = dcon.cursor() exist_map = {} for tbl in tables.keys(): if not skytools.exists_table(dcur, tbl): if not self.part_template: raise Exception('Dest table does not exists and no way to create it.') sql = self.part_template sql = sql.replace(DEST_TABLE, skytools.quote_fqident(tbl)) # we do this to make sure that constraints for # tables who contain a schema will still work schema_table = tbl.replace(".", "__") sql = sql.replace(SCHEMA_TABLE, skytools.quote_ident(schema_table)) dcur.execute(sql) dcon.commit() self.log.info('%s: Created table %s' % (self.job_name, tbl)) if __name__ == '__main__': script = TableDispatcher(sys.argv[1:]) script.start() skytools-2.1.13/scripts/bulk_loader.py0000755000175000017500000003271611670174255017041 0ustar markomarko#! /usr/bin/env python """Bulkloader for slow databases (Bizgres). Idea is following: - Script reads from queue a batch of urlencoded row changes. Inserts/updates/deletes, maybe many per one row. - It creates 3 lists: ins_list, upd_list, del_list. If one row is changed several times, it keeps the latest. - Lists are processed in followin way: ins_list - COPY into main table upd_list - COPY into temp table, UPDATE from there del_list - COPY into temp table, DELETE from there - One side-effect is that total order of how rows appear changes, but per-row changes will be kept in order. The speedup from the COPY will happen only if the batches are large enough. So the ticks should happen only after couple of minutes. """ import sys, os, pgq, skytools from skytools import quote_ident, quote_fqident ## several methods for applying data # update as update METH_CORRECT = 0 # update as delete/copy METH_DELETE = 1 # merge ins_list and upd_list, do delete/copy METH_MERGED = 2 # no good method for temp table check before 8.2 USE_LONGLIVED_TEMP_TABLES = False def find_dist_fields(curs, fqtbl): if not skytools.exists_table(curs, "pg_catalog.mpp_distribution_policy"): return [] schema, name = fqtbl.split('.') q = "select a.attname"\ " from pg_class t, pg_namespace n, pg_attribute a,"\ " mpp_distribution_policy p"\ " where n.oid = t.relnamespace"\ " and p.localoid = t.oid"\ " and a.attrelid = t.oid"\ " and a.attnum = any(p.attrnums)"\ " and n.nspname = %s and t.relname = %s" curs.execute(q, [schema, name]) res = [] for row in curs.fetchall(): res.append(row[0]) return res def exists_temp_table(curs, tbl): # correct way, works only on 8.2 q = "select 1 from pg_class where relname = %s and relnamespace = pg_my_temp_schema()" # does not work with parallel case #q = """ #select 1 from pg_class t, pg_namespace n #where n.oid = t.relnamespace # and pg_table_is_visible(t.oid) # and has_schema_privilege(n.nspname, 'USAGE') # and has_table_privilege(n.nspname || '.' || t.relname, 'SELECT') # and substr(n.nspname, 1, 8) = 'pg_temp_' # and t.relname = %s; #""" curs.execute(q, [tbl]) tmp = curs.fetchall() return len(tmp) > 0 class TableCache: """Per-table data hander.""" def __init__(self, tbl): """Init per-batch table data cache.""" self.name = tbl self.ev_list = [] self.pkey_map = {} self.pkey_list = [] self.pkey_str = None self.col_list = None self.final_ins_list = [] self.final_upd_list = [] self.final_del_list = [] def add_event(self, ev): """Store new event.""" # op & data ev.op = ev.ev_type[0] ev.data = skytools.db_urldecode(ev.ev_data) # get pkey column names if self.pkey_str is None: if len(ev.ev_type) > 2: self.pkey_str = ev.ev_type.split(':')[1] else: self.pkey_str = ev.ev_extra2 if self.pkey_str: self.pkey_list = self.pkey_str.split(',') # get pkey value if self.pkey_str: pk_data = [] for k in self.pkey_list: pk_data.append(ev.data[k]) ev.pk_data = tuple(pk_data) elif ev.op == 'I': # fake pkey, just to get them spread out ev.pk_data = ev.id else: raise Exception('non-pk tables not supported: %s' % self.name) # get full column list, detect added columns if not self.col_list: self.col_list = ev.data.keys() elif self.col_list != ev.data.keys(): # ^ supposedly python guarantees same order in keys() # find new columns for c in ev.data.keys(): if c not in self.col_list: for oldev in self.ev_list: oldev.data[c] = None self.col_list = ev.data.keys() # add to list self.ev_list.append(ev) # keep all versions of row data if ev.pk_data in self.pkey_map: self.pkey_map[ev.pk_data].append(ev) else: self.pkey_map[ev.pk_data] = [ev] def finish(self): """Got all data, prepare for insertion.""" del_list = [] ins_list = [] upd_list = [] for ev_list in self.pkey_map.values(): # rewrite list of I/U/D events to # optional DELETE and optional INSERT/COPY command exists_before = -1 exists_after = 1 for ev in ev_list: if ev.op == "I": if exists_before < 0: exists_before = 0 exists_after = 1 elif ev.op == "U": if exists_before < 0: exists_before = 1 #exists_after = 1 # this shouldnt be needed elif ev.op == "D": if exists_before < 0: exists_before = 1 exists_after = 0 else: raise Exception('unknown event type: %s' % ev.op) # skip short-lived rows if exists_before == 0 and exists_after == 0: continue # take last event ev = ev_list[-1] # generate needed commands if exists_before and exists_after: upd_list.append(ev.data) elif exists_before: del_list.append(ev.data) elif exists_after: ins_list.append(ev.data) # reorder cols new_list = self.pkey_list[:] for k in self.col_list: if k not in self.pkey_list: new_list.append(k) self.col_list = new_list self.final_ins_list = ins_list self.final_upd_list = upd_list self.final_del_list = del_list class BulkLoader(pgq.SerialConsumer): def __init__(self, args): pgq.SerialConsumer.__init__(self, "bulk_loader", "src_db", "dst_db", args) def reload(self): pgq.SerialConsumer.reload(self) self.load_method = self.cf.getint("load_method", METH_CORRECT) if self.load_method not in (0,1,2): raise Exception("bad load_method") self.remap_tables = {} for map in self.cf.getlist("remap_tables", ''): tmp = map.split(':') tbl = tmp[0].strip() new = tmp[1].strip() self.remap_tables[tbl] = new def process_remote_batch(self, src_db, batch_id, ev_list, dst_db): """Content dispatcher.""" # add events to per-table caches tables = {} for ev in ev_list: tbl = ev.extra1 if not tbl in tables: tables[tbl] = TableCache(tbl) cache = tables[tbl] cache.add_event(ev) ev.tag_done() # then process them for tbl, cache in tables.items(): cache.finish() self.process_one_table(dst_db, tbl, cache) def process_one_table(self, dst_db, tbl, cache): del_list = cache.final_del_list ins_list = cache.final_ins_list upd_list = cache.final_upd_list col_list = cache.col_list real_update_count = len(upd_list) self.log.debug("process_one_table: %s (I/U/D = %d/%d/%d)" % ( tbl, len(ins_list), len(upd_list), len(del_list))) if tbl in self.remap_tables: old = tbl tbl = self.remap_tables[tbl] self.log.debug("Redirect %s to %s" % (old, tbl)) # hack to unbroke stuff if self.load_method == METH_MERGED: upd_list += ins_list ins_list = [] # check if interesting table curs = dst_db.cursor() if not skytools.exists_table(curs, tbl): self.log.warning("Ignoring events for table: %s" % tbl) return # fetch distribution fields dist_fields = find_dist_fields(curs, tbl) extra_fields = [] for fld in dist_fields: if fld not in cache.pkey_list: extra_fields.append(fld) self.log.debug("PKey fields: %s Extra fields: %s" % ( ",".join(cache.pkey_list), ",".join(extra_fields))) # create temp table temp = self.create_temp_table(curs, tbl) # where expr must have pkey and dist fields klist = [] for pk in cache.pkey_list + extra_fields: exp = "%s.%s = %s.%s" % (quote_fqident(tbl), quote_ident(pk), quote_fqident(temp), quote_ident(pk)) klist.append(exp) whe_expr = " and ".join(klist) # create del sql del_sql = "delete from only %s using %s where %s" % ( quote_fqident(tbl), quote_fqident(temp), whe_expr) # create update sql slist = [] key_fields = cache.pkey_list + extra_fields for col in cache.col_list: if col not in key_fields: exp = "%s = %s.%s" % (quote_ident(col), quote_fqident(temp), quote_ident(col)) slist.append(exp) upd_sql = "update only %s set %s from %s where %s" % ( quote_fqident(tbl), ", ".join(slist), quote_fqident(temp), whe_expr) # insert sql colstr = ",".join([quote_ident(c) for c in cache.col_list]) ins_sql = "insert into %s (%s) select %s from %s" % ( quote_fqident(tbl), colstr, colstr, quote_fqident(temp)) # process deleted rows if len(del_list) > 0: self.log.info("Deleting %d rows from %s" % (len(del_list), tbl)) # delete old rows q = "truncate %s" % quote_fqident(temp) self.log.debug(q) curs.execute(q) # copy rows self.log.debug("COPY %d rows into %s" % (len(del_list), temp)) skytools.magic_insert(curs, temp, del_list, col_list) # delete rows self.log.debug(del_sql) curs.execute(del_sql) self.log.debug("%s - %d" % (curs.statusmessage, curs.rowcount)) self.log.debug(curs.statusmessage) if len(del_list) != curs.rowcount: self.log.warning("Delete mismatch: expected=%s updated=%d" % (len(del_list), curs.rowcount)) # process updated rows if len(upd_list) > 0: self.log.info("Updating %d rows in %s" % (len(upd_list), tbl)) # delete old rows q = "truncate %s" % quote_fqident(temp) self.log.debug(q) curs.execute(q) # copy rows self.log.debug("COPY %d rows into %s" % (len(upd_list), temp)) skytools.magic_insert(curs, temp, upd_list, col_list) if self.load_method == METH_CORRECT: # update main table self.log.debug(upd_sql) curs.execute(upd_sql) self.log.debug(curs.statusmessage) # check count if len(upd_list) != curs.rowcount: self.log.warning("Update mismatch: expected=%s updated=%d" % (len(upd_list), curs.rowcount)) else: # delete from main table self.log.debug(del_sql) curs.execute(del_sql) self.log.debug(curs.statusmessage) # check count if real_update_count != curs.rowcount: self.log.warning("Update mismatch: expected=%s deleted=%d" % (real_update_count, curs.rowcount)) # insert into main table if 0: # does not work due bizgres bug self.log.debug(ins_sql) curs.execute(ins_sql) self.log.debug(curs.statusmessage) else: # copy again, into main table self.log.debug("COPY %d rows into %s" % (len(upd_list), tbl)) skytools.magic_insert(curs, tbl, upd_list, col_list) # process new rows if len(ins_list) > 0: self.log.info("Inserting %d rows into %s" % (len(ins_list), tbl)) skytools.magic_insert(curs, tbl, ins_list, col_list) # delete remaining rows if USE_LONGLIVED_TEMP_TABLES: q = "truncate %s" % quote_fqident(temp) else: # fscking problems with long-lived temp tables q = "drop table %s" % quote_fqident(temp) self.log.debug(q) curs.execute(q) def create_temp_table(self, curs, tbl): # create temp table for loading tempname = tbl.replace('.', '_') + "_loadertmp" # check if exists if USE_LONGLIVED_TEMP_TABLES: if exists_temp_table(curs, tempname): self.log.debug("Using existing temp table %s" % tempname) return tempname # bizgres crashes on delete rows arg = "on commit delete rows" arg = "on commit preserve rows" # create temp table for loading q = "create temp table %s (like %s) %s" % ( quote_fqident(tempname), quote_fqident(tbl), arg) self.log.debug("Creating temp table: %s" % q) curs.execute(q) return tempname if __name__ == '__main__': script = BulkLoader(sys.argv[1:]) script.start() skytools-2.1.13/scripts/bulk_loader.ini.templ0000644000175000017500000000065111670174255020276 0ustar markomarko[bulk_loader] job_name = bizgres_loader src_db = dbname=bulksrc dst_db = dbname=bulkdst pgq_queue_name = xx use_skylog = 1 logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid # 0 - apply UPDATE as UPDATE # 1 - apply UPDATE as DELETE+INSERT # 2 - merge INSERT/UPDATE, do DELETE+INSERT load_method = 0 # no hurry loop_delay = 10 # table renaming # remap_tables = skypein_cdr_closed:skypein_cdr, tbl1:tbl2 skytools-2.1.13/scripts/queue_mover.py0000755000175000017500000000160711670174255017105 0ustar markomarko#! /usr/bin/env python # this script simply mover events from one queue to another import sys, os, pgq, skytools class QueueMover(pgq.SerialConsumer): def __init__(self, args): pgq.SerialConsumer.__init__(self, "queue_mover", "src_db", "dst_db", args) self.dst_queue_name = self.cf.get("dst_queue_name") def process_remote_batch(self, db, batch_id, ev_list, dst_db): # load data rows = [] for ev in ev_list: data = [ev.type, ev.data, ev.extra1, ev.extra2, ev.extra3, ev.extra4, ev.time] rows.append(data) ev.tag_done() fields = ['type', 'data', 'extra1', 'extra2', 'extra3', 'extra4', 'time'] # insert data curs = dst_db.cursor() pgq.bulk_insert_events(curs, rows, fields, self.dst_queue_name) if __name__ == '__main__': script = QueueMover(sys.argv[1:]) script.start() skytools-2.1.13/scripts/table_dispatcher.ini.templ0000644000175000017500000000143511670174255021311 0ustar markomarko[udata_dispatcher] job_name = test_move src_db = dbname=sourcedb_test dst_db = dbname=dataminedb_test pgq_queue_name = OrderLog logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid # where to put data. when partitioning, will be used as base name dest_table = orders # date field with will be used for partitioning # special value: _EVTIME - event creation time part_column = start_date #fields = * #fields = id, name #fields = id:newid, name, bar:baz # template used for creating partition tables # _DEST_TABLE part_template = create table _DEST_TABLE () inherits (orders); alter table only _DEST_TABLE add constraint _DEST_TABLE_pkey primary key (id); grant select on _DEST_TABLE to group reporting; skytools-2.1.13/tests/0000755000175000017500000000000011727601174013641 5ustar markomarkoskytools-2.1.13/tests/scripts/0000755000175000017500000000000011727601174015330 5ustar markomarkoskytools-2.1.13/tests/scripts/stop.sh0000755000175000017500000000027411670174255016661 0ustar markomarko#! /bin/sh . ./env.sh cube_dispatcher.py -s conf/cube.ini table_dispatcher.py -s conf/table.ini queue_mover.py -s conf/mover.ini sleep 1 pgqadm.py -s conf/ticker.ini #killall python skytools-2.1.13/tests/scripts/run-tests.sh0000755000175000017500000000046011670174255017635 0ustar markomarko#! /bin/sh . ./env.sh ./gendb.sh pgqadm.py -d conf/ticker.ini ticker queue_mover.py -d conf/mover.ini cube_dispatcher.py -d conf/cube.ini table_dispatcher.py -d conf/table.ini sleep 1 psql scriptsrc < 0: return 1 if ok > 0: return 0 return 1 def resync_table(self, db, curs): self.log.info('trying to remove table') curs.execute("update londiste.subscriber_table"\ " set merge_state = null" " where table_name='public.data1'") db.commit() def run_compare(self): args = ["londiste.py", "conf/replic.ini", "compare"] err = os.spawnvp(os.P_WAIT, "londiste.py", args) self.log.info("Compare result=%d" % err) if __name__ == '__main__': script = Tester(sys.argv[1:]) script.start() skytools-2.1.13/tests/londiste/env.sh0000644000175000017500000000017711670174255016615 0ustar markomarko PYTHONPATH=../../python:$PYTHONPATH PATH=../../python:../../scripts:$PATH export PYTHONPATH PATH #. /opt/apps/pgsql-dev/env skytools-2.1.13/tests/londiste/data.sql0000644000175000017500000000336711670174255017127 0ustar markomarko set client_min_messages = 'warning'; create table data1 ( id serial primary key, data text ); cluster data1 using data1_pkey; create or replace function test_triga() returns trigger as $$ begin return new; end; $$ language plpgsql; create trigger xtriga after insert on data1 for each row execute procedure test_triga(); create unique index idx_data1_uq on data1 (data); create index idx_data1_rand on data1 (id, data); create table data2 ( id serial primary key, data text, ref1 integer references data1, constraint uq_data2 unique (data) ); create index idx_data2_rand on data2 (id, data); cluster data2 using idx_data2_rand; create sequence test_seq; select setval('test_seq', 50); create table expect_test ( dbname text primary key ); insert into expect_test values (current_database()); create table skip_test ( id serial not null, dbname text not null, primary key (id, dbname) ); insert into skip_test (dbname) values (current_database()); create table "Table" ( "I D" serial primary key, "table" text, "d1.ref" int4 references data1, constraint "Woof" unique ("table") ); create index "idx Table" on "Table" ("table", "I D"); create table inh_parent1 ( id serial primary key, data text, ref1 integer, constraint p1_uq_data unique (data) ); create table inh_parent2 ( id serial primary key, data text, ref1 integer, constraint p2_uq_data unique (data) ); create table inh_mid ( id serial primary key, data text, ref1 integer, constraint m_uq_data unique (data) ) inherits (inh_parent1, inh_parent2); create table inh_child ( id serial primary key, data text, ref1 integer, constraint c_uq_data unique (data) ) inherits (inh_mid); skytools-2.1.13/tests/quoting/0000755000175000017500000000000011727601174015327 5ustar markomarkoskytools-2.1.13/tests/quoting/Makefile0000644000175000017500000000007511670174255016773 0ustar markomarko test: PYTHONPATH=`echo ../../build/lib.*` \ ./regtest.py skytools-2.1.13/tests/quoting/regtest.py0000755000175000017500000000525711670174255017374 0ustar markomarko#! /usr/bin/env python import sys, time import skytools.psycopgwrapper import skytools._cquoting, skytools._pyquoting # create a DictCursor row class fake_cursor: index = {'id': 0, 'data': 1} description = ['x', 'x'] dbrow = skytools.psycopgwrapper._CompatRow(fake_cursor()) dbrow[0] = '123' dbrow[1] = 'value' def regtest(name, func, cases): bad = 0 for dat, res in cases: res2 = func(dat) if res != res2: print "failure: %s(%s) = %s (expected %s)" % (name, repr(dat), repr(res2), repr(res)) bad += 1 if bad: print "%-20s: failed" % name else: print "%-20s: OK" % name sql_literal = [ [None, "null"], ["", "''"], ["a'b", "'a''b'"], [r"a\'b", r"E'a\\''b'"], [1, "'1'"], [True, "'True'"], ] regtest("quote_literal/c", skytools._cquoting.quote_literal, sql_literal) regtest("quote_literal/py", skytools._pyquoting.quote_literal, sql_literal) sql_copy = [ [None, "\\N"], ["", ""], ["a'\tb", "a'\\tb"], [r"a\'b", r"a\\'b"], [1, "1"], [True, "True"], [u"qwe", "qwe"], ] regtest("quote_copy/c", skytools._cquoting.quote_copy, sql_copy) regtest("quote_copy/py", skytools._pyquoting.quote_copy, sql_copy) sql_bytea_raw = [ [None, None], ["", ""], ["a'\tb", "a'\\011b"], [r"a\'b", r"a\\'b"], ["\t\344", r"\011\344"], ] regtest("quote_bytea_raw/c", skytools._cquoting.quote_bytea_raw, sql_bytea_raw) regtest("quote_bytea_raw/py", skytools._pyquoting.quote_bytea_raw, sql_bytea_raw) sql_ident = [ ["", ""], ["a'\t\\\"b", '"a\'\t\\""b"'], ['abc_19', 'abc_19'], ['from', '"from"'], ['0foo', '"0foo"'], ['mixCase', '"mixCase"'], ] regtest("quote_ident", skytools.quote_ident, sql_ident) t_urlenc = [ [{}, ""], [{'a': 1}, "a=1"], [{'a': None}, "a"], [{'qwe': 1, u'zz': u"qwe"}, "qwe=1&zz=qwe"], [{'a': '\000%&'}, "a=%00%25%26"], [dbrow, 'data=value&id=123'], ] regtest("db_urlencode/c", skytools._cquoting.db_urlencode, t_urlenc) regtest("db_urlencode/py", skytools._pyquoting.db_urlencode, t_urlenc) t_urldec = [ ["", {}], ["a=b&c", {'a': 'b', 'c': None}], ["&&b=f&&", {'b': 'f'}], [u"abc=qwe", {'abc': 'qwe'}], ["b=", {'b': ''}], ["b=%00%45", {'b': '\x00E'}], ] regtest("db_urldecode/c", skytools._cquoting.db_urldecode, t_urldec) regtest("db_urldecode/py", skytools._pyquoting.db_urldecode, t_urldec) t_unesc = [ ["", ""], ["\\N", "N"], ["abc", "abc"], [u"abc", "abc"], [r"\0\000\001\01\1", "\0\000\001\001\001"], [r"a\001b\tc\r\n", "a\001b\tc\r\n"], ] regtest("unescape/c", skytools._cquoting.unescape, t_unesc) regtest("unescape/py", skytools._pyquoting.unescape, t_unesc) skytools-2.1.13/tests/skylog/0000755000175000017500000000000011727601174015151 5ustar markomarkoskytools-2.1.13/tests/skylog/logtest.py0000755000175000017500000000056511670174255017217 0ustar markomarko#! /usr/bin/env python import sys, os, skytools import skytools.skylog class LogTest(skytools.DBScript): def work(self): self.log.error('test error') self.log.warning('test warning') self.log.info('test info') self.log.debug('test debug') if __name__ == '__main__': script = LogTest('log_test', sys.argv[1:]) script.start() skytools-2.1.13/tests/skylog/runtest.sh0000755000175000017500000000007211670174255017215 0ustar markomarko#! /bin/sh . ../env.sh exec ./logtest.py test.ini "$@" skytools-2.1.13/tests/skylog/test.ini0000644000175000017500000000006111670174255016630 0ustar markomarko[log_test] loop_delay = 5 logfile = xtest.log skytools-2.1.13/tests/skylog/skylog.ini0000644000175000017500000000252211670174255017165 0ustar markomarko; notes: ; - 'args' is mandatory in [handler_*] sections ; - in lists there must not be spaces ; ; top-level config ; ; list of all loggers [loggers] keys=root ; root logger sees everything. there can be per-job configs by ; specifing loggers with job_name of the script ; list of all handlers [handlers] keys=stderr,logdb,logsrv,logfile ; list of all formatters [formatters] keys=short,long,none ; ; map specific loggers to specifig handlers ; [logger_root] level=DEBUG handlers=stderr,logdb,logsrv,logfile ;,logfile ;logdb,logsrv,logfile ; ; configure formatters ; [formatter_short] format=%(asctime)s %(levelname)s %(message)s datefmt=%H:%M [formatter_long] format=%(asctime)s %(process)s %(levelname)s %(message)s [formatter_none] format=%(message)s ; ; configure handlers ; ; file. args: stream [handler_stderr] class=StreamHandler args=(sys.stderr,) formatter=short ; log into db. args: conn_string [handler_logdb] class=skylog.LogDBHandler args=("host=127.0.0.1 port=5432 user=marko dbname=logdb",) formatter=none level=INFO ; JSON messages over UDP. args: host, port [handler_logsrv] class=skylog.UdpLogServerHandler args=('127.0.0.1', 6666) formatter=none ; rotating logfile. args: filename, maxsize, maxcount [handler_logfile] class=skylog.EasyRotatingFileHandler args=('~/log/%(job_name)s.log', 100*1024*1024, 3) formatter=long skytools-2.1.13/tests/env.sh0000644000175000017500000000014411670174255014766 0ustar markomarko PYTHONPATH=../../python:$PYTHONPATH PATH=../../python:../../scripts:$PATH export PYTHONPATH PATH skytools-2.1.13/tests/upgrade/0000755000175000017500000000000011727601174015270 5ustar markomarkoskytools-2.1.13/tests/upgrade/run-tests.sh0000755000175000017500000000065111670174255017577 0ustar markomarko#! /bin/sh . ../env.sh ./gendb.sh rm -rf upgrade cp -rp ../../upgrade . skytools_upgrade.py "dbname=upgradedb" ./gendb.sh psql -q upgradedb -f upgrade/final/v2.1.5_pgq_core.sql psql -q upgradedb -f upgrade/final/v2.1.5_pgq_ext.sql psql -q upgradedb -f upgrade/final/v2.1.5_londiste.sql echo "update from 2.1.5 to 2.1.6" skytools_upgrade.py "dbname=upgradedb" echo " no update" skytools_upgrade.py "dbname=upgradedb" skytools-2.1.13/tests/upgrade/gendb.sh0000755000175000017500000000071411670174255016712 0ustar markomarko#! /bin/sh . ../env.sh old=./sql db=upgradedb echo "creating database: $db" dropdb $db sleep 1 createdb $db sver=`psql -At $db -c "show server_version" | sed 's/\([0-9]*[.][0-9]*\).*/\1/'` echo "server version: $sver" psql -q $db -c "create language plpgsql" psql -q $db -c "create language plpythonu" psql -q $db -f $old/v2.1.4_txid82.sql psql -q $db -f $old/v2.1.4_pgq.sql psql -q $db -f $old/v2.1.4_pgq_ext.sql psql -q $db -f $old/v2.1.4_londiste.sql skytools-2.1.13/tests/upgrade/sql/0000755000175000017500000000000011727601174016067 5ustar markomarkoskytools-2.1.13/tests/upgrade/sql/v2.1.4_pgq.sql0000644000175000017500000021763111670174255020323 0ustar markomarko -- ---------------------------------------------------------------------- -- Section: Internal Tables -- -- Overview: -- pgq.queue - Queue configuration -- pgq.consumer - Consumer names -- pgq.subscription - Consumer registrations -- pgq.tick - Per-queue snapshots (ticks) -- pgq.event_* - Data tables -- pgq.retry_queue - Events to be retried later -- pgq.failed_queue - Events whose processing failed -- -- Its basically generalized and simplified Slony-I structure: -- sl_node - pgq.consumer -- sl_set - pgq.queue -- sl_subscriber + sl_confirm - pgq.subscription -- sl_event - pgq.tick -- sl_setsync - pgq_ext.completed_* -- sl_log_* - slony1 has per-cluster data tables, -- pgq has per-queue data tables. -- ---------------------------------------------------------------------- set client_min_messages = 'warning'; -- drop schema if exists pgq cascade; create schema pgq; grant usage on schema pgq to public; -- ---------------------------------------------------------------------- -- Table: pgq.consumer -- -- Name to id lookup for consumers -- -- Columns: -- co_id - consumer's id for internal usage -- co_name - consumer's id for external usage -- ---------------------------------------------------------------------- create table pgq.consumer ( co_id serial, co_name text not null default 'fooz', constraint consumer_pkey primary key (co_id), constraint consumer_name_uq UNIQUE (co_name) ); -- ---------------------------------------------------------------------- -- Table: pgq.queue -- -- Information about available queues -- -- Columns: -- queue_id - queue id for internal usage -- queue_name - queue name visible outside -- queue_data - parent table for actual data tables -- queue_switch_step1 - tx when rotation happened -- queue_switch_step2 - tx after rotation was committed -- queue_switch_time - time when switch happened -- queue_ticker_max_count - batch should not contain more events -- queue_ticker_max_lag - events should not age more -- queue_ticker_idle_period - how often to tick when no events happen -- ---------------------------------------------------------------------- create table pgq.queue ( queue_id serial, queue_name text not null, queue_ntables integer not null default 3, queue_cur_table integer not null default 0, queue_rotation_period interval not null default '2 hours', queue_switch_step1 bigint not null default get_current_txid(), queue_switch_step2 bigint default get_current_txid(), queue_switch_time timestamptz not null default now(), queue_external_ticker boolean not null default false, queue_ticker_max_count integer not null default 500, queue_ticker_max_lag interval not null default '3 seconds', queue_ticker_idle_period interval not null default '1 minute', queue_data_pfx text not null, queue_event_seq text not null, queue_tick_seq text not null, constraint queue_pkey primary key (queue_id), constraint queue_name_uq unique (queue_name) ); -- ---------------------------------------------------------------------- -- Table: pgq.tick -- -- Snapshots for event batching -- -- Columns: -- tick_queue - queue id whose tick it is -- tick_id - ticks id (per-queue) -- tick_time - time when tick happened -- tick_snapshot -- ---------------------------------------------------------------------- create table pgq.tick ( tick_queue int4 not null, tick_id bigint not null, tick_time timestamptz not null default now(), tick_snapshot txid_snapshot not null default get_current_snapshot(), constraint tick_pkey primary key (tick_queue, tick_id), constraint tick_queue_fkey foreign key (tick_queue) references pgq.queue (queue_id) ); -- ---------------------------------------------------------------------- -- Sequence: pgq.batch_id_seq -- -- Sequence for batch id's. -- ---------------------------------------------------------------------- create sequence pgq.batch_id_seq; -- ---------------------------------------------------------------------- -- Table: pgq.subscription -- -- Consumer registration on a queue -- -- Columns: -- -- sub_id - subscription id for internal usage -- sub_queue - queue id -- sub_consumer - consumer's id -- sub_tick - last tick the consumer processed -- sub_batch - shortcut for queue_id/consumer_id/tick_id -- sub_next_tick - -- ---------------------------------------------------------------------- create table pgq.subscription ( sub_id serial not null, sub_queue int4 not null, sub_consumer int4 not null, sub_last_tick bigint not null, sub_active timestamptz not null default now(), sub_batch bigint, sub_next_tick bigint, constraint subscription_pkey primary key (sub_id), constraint sub_queue_fkey foreign key (sub_queue) references pgq.queue (queue_id), constraint sub_consumer_fkey foreign key (sub_consumer) references pgq.consumer (co_id) ); -- ---------------------------------------------------------------------- -- Table: pgq.event_template -- -- Parent table for all event tables -- -- Columns: -- ev_id - event's id, supposed to be unique per queue -- ev_time - when the event was inserted -- ev_txid - transaction id which inserted the event -- ev_owner - subscription id that wanted to retry this -- ev_retry - how many times the event has been retried, NULL for new events -- ev_type - consumer/producer can specify what the data fields contain -- ev_data - data field -- ev_extra1 - extra data field -- ev_extra2 - extra data field -- ev_extra3 - extra data field -- ev_extra4 - extra data field -- ---------------------------------------------------------------------- create table pgq.event_template ( ev_id bigint not null, ev_time timestamptz not null, ev_txid bigint not null default get_current_txid(), ev_owner int4, ev_retry int4, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text ); -- ---------------------------------------------------------------------- -- Table: pgq.retry_queue -- -- Events to be retried -- -- Columns: -- ev_retry_after - time when it should be re-inserted to main queue -- ---------------------------------------------------------------------- create table pgq.retry_queue ( ev_retry_after timestamptz not null, like pgq.event_template, constraint rq_pkey primary key (ev_owner, ev_id), constraint rq_owner_fkey foreign key (ev_owner) references pgq.subscription (sub_id) ); alter table pgq.retry_queue alter column ev_owner set not null; alter table pgq.retry_queue alter column ev_txid drop not null; create index rq_retry_idx on pgq.retry_queue (ev_retry_after); -- ---------------------------------------------------------------------- -- Table: pgq.failed_queue -- -- Events whose processing failed -- -- Columns: -- ev_failed_reason - consumer's excuse for not processing -- ev_failed_time - when it was tagged failed -- ---------------------------------------------------------------------- create table pgq.failed_queue ( ev_failed_reason text, ev_failed_time timestamptz not null, -- all event fields like pgq.event_template, constraint fq_pkey primary key (ev_owner, ev_id), constraint fq_owner_fkey foreign key (ev_owner) references pgq.subscription (sub_id) ); alter table pgq.failed_queue alter column ev_owner set not null; alter table pgq.failed_queue alter column ev_txid drop not null; create type pgq.ret_queue_info as ( queue_name text, queue_ntables integer, queue_cur_table integer, queue_rotation_period interval, queue_switch_time timestamptz, queue_external_ticker boolean, queue_ticker_max_count integer, queue_ticker_max_lag interval, queue_ticker_idle_period interval, ticker_lag interval ); create type pgq.ret_consumer_info as ( queue_name text, consumer_name text, lag interval, last_seen interval, last_tick bigint, current_batch bigint, next_tick bigint ); create type pgq.ret_batch_info as ( queue_name text, consumer_name text, batch_start timestamptz, batch_end timestamptz, prev_tick_id bigint, tick_id bigint, lag interval ); create type pgq.ret_batch_event as ( ev_id bigint, ev_time timestamptz, ev_txid bigint, ev_retry int4, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text ); -- Section: Internal Functions -- Group: Low-level event handling create or replace function pgq.batch_event_sql(x_batch_id bigint) returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.batch_event_sql(1) -- Creates SELECT statement that fetches events for this batch. -- -- Parameters: -- x_batch_id - ID of a active batch. -- -- Returns: -- SQL statement. -- ---------------------------------------------------------------------- -- ---------------------------------------------------------------------- -- Algorithm description: -- Given 2 snapshots, sn1 and sn2 with sn1 having xmin1, xmax1 -- and sn2 having xmin2, xmax2 create expression that filters -- right txid's from event table. -- -- Simplest solution would be -- > WHERE ev_txid >= xmin1 AND ev_txid <= xmax2 -- > AND NOT txid_in_snapshot(ev_txid, sn1) -- > AND txid_in_snapshot(ev_txid, sn2) -- -- The simple solution has a problem with long transactions (xmin1 very low). -- All the batches that happen when the long tx is active will need -- to scan all events in that range. Here is 2 optimizations used: -- -- 1) Use [xmax1..xmax2] for range scan. That limits the range to -- txids that actually happened between two snapshots. For txids -- in the range [xmin1..xmax1] look which ones were actually -- committed between snapshots and search for them using exact -- values using IN (..) list. -- -- 2) As most TX are short, there could be lot of them that were -- just below xmax1, but were committed before xmax2. So look -- if there are ID's near xmax1 and lower the range to include -- them, thus decresing size of IN (..) list. -- ---------------------------------------------------------------------- declare rec record; sql text; tbl text; arr text; part text; select_fields text; retry_expr text; batch record; begin select s.sub_last_tick, s.sub_next_tick, s.sub_id, s.sub_queue, get_snapshot_xmax(last.tick_snapshot) as tx_start, get_snapshot_xmax(cur.tick_snapshot) as tx_end, last.tick_snapshot as last_snapshot, cur.tick_snapshot as cur_snapshot into batch from pgq.subscription s, pgq.tick last, pgq.tick cur where s.sub_batch = x_batch_id and last.tick_queue = s.sub_queue and last.tick_id = s.sub_last_tick and cur.tick_queue = s.sub_queue and cur.tick_id = s.sub_next_tick; if not found then raise exception 'batch not found'; end if; -- load older transactions arr := ''; for rec in -- active tx-es in prev_snapshot that were committed in cur_snapshot select id1 from get_snapshot_active(batch.last_snapshot) id1 left join get_snapshot_active(batch.cur_snapshot) id2 on (id1 = id2) where id2 is null order by 1 desc loop -- try to avoid big IN expression, so try to include nearby -- tx'es into range if batch.tx_start - 100 <= rec.id1 then batch.tx_start := rec.id1; else if arr = '' then arr := rec.id1; else arr := arr || ',' || rec.id1; end if; end if; end loop; -- must match pgq.event_template select_fields := 'select ev_id, ev_time, ev_txid, ev_retry, ev_type,' || ' ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4'; retry_expr := ' and (ev_owner is null or ev_owner = ' || batch.sub_id || ')'; -- now generate query that goes over all potential tables sql := ''; for rec in select xtbl from pgq.batch_event_tables(x_batch_id) xtbl loop tbl := rec.xtbl; -- this gets newer queries that definitely are not in prev_snapshot part := select_fields || ' from pgq.tick cur, pgq.tick last, ' || tbl || ' ev ' || ' where cur.tick_id = ' || batch.sub_next_tick || ' and cur.tick_queue = ' || batch.sub_queue || ' and last.tick_id = ' || batch.sub_last_tick || ' and last.tick_queue = ' || batch.sub_queue || ' and ev.ev_txid >= ' || batch.tx_start || ' and ev.ev_txid <= ' || batch.tx_end || ' and txid_in_snapshot(ev.ev_txid, cur.tick_snapshot)' || ' and not txid_in_snapshot(ev.ev_txid, last.tick_snapshot)' || retry_expr; -- now include older tx-es, that were ongoing -- at the time of prev_snapshot if arr <> '' then part := part || ' union all ' || select_fields || ' from ' || tbl || ' ev ' || ' where ev.ev_txid in (' || arr || ')' || retry_expr; end if; if sql = '' then sql := part; else sql := sql || ' union all ' || part; end if; end loop; if sql = '' then raise exception 'could not construct sql for batch %', x_batch_id; end if; return sql || ' order by 1'; end; $$ language plpgsql; -- no perms needed create or replace function pgq.batch_event_tables(x_batch_id bigint) returns setof text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.batch_event_tables(1) -- -- Returns set of table names where this batch events may reside. -- -- Parameters: -- x_batch_id - ID of a active batch. -- ---------------------------------------------------------------------- declare nr integer; tbl text; use_prev integer; use_next integer; batch record; begin select get_snapshot_xmin(last.tick_snapshot) as tx_min, -- absolute minimum get_snapshot_xmax(cur.tick_snapshot) as tx_max, -- absolute maximum q.queue_data_pfx, q.queue_ntables, q.queue_cur_table, q.queue_switch_step1, q.queue_switch_step2 into batch from pgq.tick last, pgq.tick cur, pgq.subscription s, pgq.queue q where cur.tick_id = s.sub_next_tick and cur.tick_queue = s.sub_queue and last.tick_id = s.sub_last_tick and last.tick_queue = s.sub_queue and s.sub_batch = x_batch_id and q.queue_id = s.sub_queue; if not found then raise exception 'Cannot find data for batch %', x_batch_id; end if; -- if its definitely not in one or other, look into both if batch.tx_max < batch.queue_switch_step1 then use_prev := 1; use_next := 0; elsif batch.queue_switch_step2 is not null and (batch.tx_min > batch.queue_switch_step2) then use_prev := 0; use_next := 1; else use_prev := 1; use_next := 1; end if; if use_prev then nr := batch.queue_cur_table - 1; if nr < 0 then nr := batch.queue_ntables - 1; end if; tbl := batch.queue_data_pfx || '_' || nr; return next tbl; end if; if use_next then tbl := batch.queue_data_pfx || '_' || batch.queue_cur_table; return next tbl; end if; return; end; $$ language plpgsql; -- no perms needed create or replace function pgq.event_retry_raw( x_queue text, x_consumer text, x_retry_after timestamptz, x_ev_id bigint, x_ev_time timestamptz, x_ev_retry integer, x_ev_type text, x_ev_data text, x_ev_extra1 text, x_ev_extra2 text, x_ev_extra3 text, x_ev_extra4 text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry_raw(12) -- -- Allows full control over what goes to retry queue. -- -- Parameters: -- x_queue - name of the queue -- x_consumer - name of the consumer -- x_retry_after - when the event should be processed again -- x_ev_id - event id -- x_ev_time - creation time -- x_ev_retry - retry count -- x_ev_type - user data -- x_ev_data - user data -- x_ev_extra1 - user data -- x_ev_extra2 - user data -- x_ev_extra3 - user data -- x_ev_extra4 - user data -- -- Returns: -- Event ID. -- ---------------------------------------------------------------------- declare q record; id bigint; begin select sub_id, queue_event_seq into q from pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue and co_name = x_consumer and sub_consumer = co_id and sub_queue = queue_id; if not found then raise exception 'consumer not registered'; end if; id := x_ev_id; if id is null then id := nextval(q.queue_event_seq); end if; insert into pgq.retry_queue (ev_retry_after, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) values (x_retry_after, x_ev_id, x_ev_time, q.sub_id, x_ev_retry, x_ev_type, x_ev_data, x_ev_extra1, x_ev_extra2, x_ev_extra3, x_ev_extra4); return id; end; $$ language plpgsql security definer; create or replace function pgq.insert_event_raw( queue_name text, ev_id bigint, ev_time timestamptz, ev_owner integer, ev_retry integer, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text) returns bigint as $$ # -- ---------------------------------------------------------------------- # -- Function: pgq.insert_event_raw(11) # -- # -- Actual event insertion. Used also by retry queue maintenance. # -- # -- Parameters: # -- queue_name - Name of the queue # -- ev_id - Event ID. If NULL, will be taken from seq. # -- ev_time - Event creation time. # -- ev_type - user data # -- ev_data - user data # -- ev_extra1 - user data # -- ev_extra2 - user data # -- ev_extra3 - user data # -- ev_extra4 - user data # -- # -- Returns: # -- Event ID. # -- ---------------------------------------------------------------------- # load args queue_name = args[0] ev_id = args[1] ev_time = args[2] ev_owner = args[3] ev_retry = args[4] ev_type = args[5] ev_data = args[6] ev_extra1 = args[7] ev_extra2 = args[8] ev_extra3 = args[9] ev_extra4 = args[10] if not "cf_plan" in SD: # get current event table q = "select queue_data_pfx, queue_cur_table, queue_event_seq "\ " from pgq.queue where queue_name = $1" SD["cf_plan"] = plpy.prepare(q, ["text"]) # get next id q = "select nextval($1) as id" SD["seq_plan"] = plpy.prepare(q, ["text"]) # get queue config res = plpy.execute(SD["cf_plan"], [queue_name]) if len(res) != 1: plpy.error("Unknown event queue: %s" % (queue_name)) tbl_prefix = res[0]["queue_data_pfx"] cur_nr = res[0]["queue_cur_table"] id_seq = res[0]["queue_event_seq"] # get id - bump seq even if id is given res = plpy.execute(SD['seq_plan'], [id_seq]) if ev_id is None: ev_id = res[0]["id"] # create plan for insertion ins_plan = None ins_key = "ins.%s" % (queue_name) if ins_key in SD: nr, ins_plan = SD[ins_key] if nr != cur_nr: ins_plan = None if ins_plan == None: q = "insert into %s_%s (ev_id, ev_time, ev_owner, ev_retry,"\ " ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4)"\ " values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)" % ( tbl_prefix, cur_nr) types = ["int8", "timestamptz", "int4", "int4", "text", "text", "text", "text", "text", "text"] ins_plan = plpy.prepare(q, types) SD[ins_key] = (cur_nr, ins_plan) # insert the event plpy.execute(ins_plan, [ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4]) # done return ev_id $$ language plpythonu; -- event inserting needs no special perms -- Group: Ticker create or replace function pgq.ticker(i_queue_name text, i_tick_id bigint) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.ticker(2) -- -- Insert a tick with a particular tick_id. -- -- For external tickers. -- -- Parameters: -- i_queue_name - Name of the queue -- i_tick_id - Id of new tick. -- -- Returns: -- Tick id. -- ---------------------------------------------------------------------- begin insert into pgq.tick (tick_queue, tick_id) select queue_id, i_tick_id from pgq.queue where queue_name = i_queue_name and queue_external_ticker; if not found then raise exception 'queue not found'; end if; return i_tick_id; end; $$ language plpgsql security definer; -- unsure about access create or replace function pgq.ticker(i_queue_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.ticker(1) -- -- Insert a tick with a tick_id from sequence. -- -- For pgqadm usage. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- Tick id. -- ---------------------------------------------------------------------- declare res bigint; ext boolean; seq text; q record; begin select queue_id, queue_tick_seq, queue_external_ticker into q from pgq.queue where queue_name = i_queue_name; if not found then raise exception 'no such queue'; end if; if q.queue_external_ticker then raise exception 'This queue has external tick source.'; end if; insert into pgq.tick (tick_queue, tick_id) values (q.queue_id, nextval(q.queue_tick_seq)); res = currval(q.queue_tick_seq); return res; end; $$ language plpgsql security definer; -- unsure about access create or replace function pgq.ticker() returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.ticker(0) -- -- Creates ticks for all queues which dont have external ticker. -- -- Returns: -- Number of queues that were processed. -- ---------------------------------------------------------------------- declare res bigint; begin select count(pgq.ticker(queue_name)) into res from pgq.queue where not queue_external_ticker; return res; end; $$ language plpgsql security definer; -- Group: Periodic maintenence create or replace function pgq.maint_retry_events() returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_retry_events(0) -- -- Moves retry events back to main queue. -- -- It moves small amount at a time. It should be called -- until it returns 0 -- -- Parameters: -- arg - desc -- -- Returns: -- Number of events processed. -- ---------------------------------------------------------------------- declare cnt integer; rec record; begin cnt := 0; for rec in select pgq.insert_event_raw(queue_name, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4), ev_owner, ev_id from pgq.retry_queue, pgq.queue, pgq.subscription where ev_retry_after <= current_timestamp and sub_id = ev_owner and queue_id = sub_queue order by ev_retry_after limit 10 loop cnt := cnt + 1; delete from pgq.retry_queue where ev_owner = rec.ev_owner and ev_id = rec.ev_id; end loop; return cnt; end; $$ language plpgsql; -- need admin access create or replace function pgq.maint_rotate_tables_step1(i_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_rotate_tables_step1(1) -- -- Rotate tables for one queue. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare badcnt integer; cf record; nr integer; tbl text; begin -- check if needed and load record select * from pgq.queue into cf where queue_name = i_queue_name and queue_rotation_period is not null and queue_switch_step2 is not null and queue_switch_time + queue_rotation_period < current_timestamp for update; if not found then return 0; end if; -- check if any consumer is on previous table select coalesce(count(*), 0) into badcnt from pgq.subscription, pgq.tick where get_snapshot_xmin(tick_snapshot) < cf.queue_switch_step2 and sub_queue = cf.queue_id and tick_queue = cf.queue_id and tick_id = (select tick_id from pgq.tick where tick_id < sub_last_tick and tick_queue = sub_queue order by tick_queue desc, tick_id desc limit 1); if badcnt > 0 then return 0; end if; -- all is fine, calc next table number nr := cf.queue_cur_table + 1; if nr = cf.queue_ntables then nr := 0; end if; tbl := cf.queue_data_pfx || '_' || nr; -- there may be long lock on the table from pg_dump, -- detect it and skip rotate then begin execute 'lock table ' || tbl || ' nowait'; execute 'truncate ' || tbl; exception when lock_not_available then raise warning 'truncate of % failed, skipping rotate', tbl; return 0; end; -- remember the moment update pgq.queue set queue_cur_table = nr, queue_switch_time = current_timestamp, queue_switch_step1 = get_current_txid(), queue_switch_step2 = NULL where queue_id = cf.queue_id; -- clean ticks - avoid partial batches delete from pgq.tick where tick_queue = cf.queue_id and get_snapshot_xmin(tick_snapshot) < cf.queue_switch_step2; return 1; end; $$ language plpgsql; -- need admin access -- ---------------------------------------------------------------------- -- Function: pgq.maint_rotate_tables_step2(0) -- -- It tag rotation as finished where needed. It should be -- called in separate transaction than pgq.maint_rotate_tables_step1() -- ---------------------------------------------------------------------- create or replace function pgq.maint_rotate_tables_step2() returns integer as $$ -- visibility tracking. this should run in separate -- tranaction than step1 begin update pgq.queue set queue_switch_step2 = get_current_txid() where queue_switch_step2 is null; return 1; end; $$ language plpgsql; -- need admin access create or replace function pgq.maint_tables_to_vacuum() returns setof text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_tables_to_vacuum(0) -- -- Returns list of tablenames that need frequent vacuuming. -- -- The goal is to avoid hardcoding them into maintenance process. -- -- Returns: -- List of table names. -- ---------------------------------------------------------------------- begin return next 'pgq.subscription'; return next 'pgq.consumer'; return next 'pgq.queue'; return next 'pgq.tick'; return next 'pgq.retry_queue'; -- vacuum also txid.epoch, if exists perform 1 from pg_class t, pg_namespace n where t.relname = 'epoch' and n.nspname = 'txid' and n.oid = t.relnamespace; if found then return next 'txid.epoch'; end if; return; end; $$ language plpgsql; -- Group: Random utility functions create or replace function pgq.grant_perms(x_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.grant_perms(1) -- -- Make event tables readable by public. -- -- Parameters: -- x_queue_name - Name of the queue. -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare q record; i integer; begin select * from pgq.queue into q where queue_name = x_queue_name; if not found then raise exception 'Queue not found'; end if; execute 'grant select, update on ' || q.queue_event_seq || ',' || q.queue_tick_seq || ' to public'; execute 'grant select on ' || q.queue_data_pfx || ' to public'; for i in 0 .. q.queue_ntables - 1 loop execute 'grant select, insert on ' || q.queue_data_pfx || '_' || i || ' to public'; end loop; return 1; end; $$ language plpgsql security definer; create or replace function pgq.force_tick(i_queue_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.force_tick(2) -- -- Simulate lots of events happening to force ticker to tick. -- -- Should be called in loop, with some delay until last tick -- changes or too much time is passed. -- -- Such function is needed because paraller calls o ticker() are -- dangerous, and cannot be protected with locks as snapshot -- is taken before. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- Currently last tick id. -- ---------------------------------------------------------------------- declare q record; t record; begin -- bump seq and get queue id select queue_id, setval(queue_event_seq, nextval(queue_event_seq) + queue_ticker_max_count * 2) as tmp into q from pgq.queue where queue_name = i_queue_name and not queue_external_ticker; if not found then raise exception 'queue not found or ticks not allowed'; end if; -- return last tick id select tick_id into t from pgq.tick where tick_queue = q.queue_id order by tick_queue desc, tick_id desc limit 1; return t.tick_id; end; $$ language plpgsql security definer; -- Section: Public Functions -- Group: Queue creation create or replace function pgq.create_queue(i_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.create_queue(1) -- -- Creates new queue with given name. -- -- Returns: -- 0 - queue already exists -- 1 - queue created -- ---------------------------------------------------------------------- declare tblpfx text; tblname text; idxpfx text; idxname text; sql text; id integer; tick_seq text; ev_seq text; n_tables integer; begin if i_queue_name is null then raise exception 'Invalid NULL value'; end if; -- check if exists perform 1 from pgq.queue where queue_name = i_queue_name; if found then return 0; end if; -- insert event id := nextval('pgq.queue_queue_id_seq'); tblpfx := 'pgq.event_' || id; idxpfx := 'event_' || id; tick_seq := 'pgq.event_' || id || '_tick_seq'; ev_seq := 'pgq.event_' || id || '_id_seq'; insert into pgq.queue (queue_id, queue_name, queue_data_pfx, queue_event_seq, queue_tick_seq) values (id, i_queue_name, tblpfx, ev_seq, tick_seq); select queue_ntables into n_tables from pgq.queue where queue_id = id; -- create seqs execute 'CREATE SEQUENCE ' || tick_seq; execute 'CREATE SEQUENCE ' || ev_seq; -- create data tables execute 'CREATE TABLE ' || tblpfx || ' () ' || ' INHERITS (pgq.event_template)'; for i in 0 .. (n_tables - 1) loop tblname := tblpfx || '_' || i; idxname := idxpfx || '_' || i; execute 'CREATE TABLE ' || tblname || ' () ' || ' INHERITS (' || tblpfx || ')'; execute 'ALTER TABLE ' || tblname || ' ALTER COLUMN ev_id ' || ' SET DEFAULT nextval(' || quote_literal(ev_seq) || ')'; execute 'create index ' || idxname || '_txid_idx on ' || tblname || ' (ev_txid)'; end loop; perform pgq.grant_perms(i_queue_name); perform pgq.ticker(i_queue_name); return 1; end; $$ language plpgsql security definer; create or replace function pgq.drop_queue(x_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.drop_queue(1) -- -- Drop queue and all associated tables. -- No consumers must be listening on the queue. -- -- ---------------------------------------------------------------------- declare tblname text; q record; num integer; begin -- check ares if x_queue_name is null then raise exception 'Invalid NULL value'; end if; -- check if exists select * into q from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'No such event queue'; end if; -- check if no consumers select count(*) into num from pgq.subscription where sub_queue = q.queue_id; if num > 0 then raise exception 'cannot drop queue, consumers still attached'; end if; -- drop data tables for i in 0 .. (q.queue_ntables - 1) loop tblname := q.queue_data_pfx || '_' || i; execute 'DROP TABLE ' || tblname; end loop; execute 'DROP TABLE ' || q.queue_data_pfx; -- delete ticks delete from pgq.tick where tick_queue = q.queue_id; -- drop seqs -- FIXME: any checks needed here? execute 'DROP SEQUENCE ' || q.queue_tick_seq; execute 'DROP SEQUENCE ' || q.queue_event_seq; -- delete event delete from pgq.queue where queue_name = x_queue_name; return 1; end; $$ language plpgsql security definer; -- Group: Event publishing create or replace function pgq.insert_event(queue_name text, ev_type text, ev_data text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.insert_event(3) -- -- Insert a event into queue. -- -- Parameters: -- queue_name - Name of the queue -- ev_type - User-specified type for the event -- ev_data - User data for the event -- -- Returns: -- Event ID -- ---------------------------------------------------------------------- begin return pgq.insert_event(queue_name, ev_type, ev_data, null, null, null, null); end; $$ language plpgsql; -- event inserting needs no special perms create or replace function pgq.insert_event( queue_name text, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.insert_event(7) -- -- Insert a event into queue with all the extra fields. -- -- Parameters: -- queue_name - Name of the queue -- ev_type - User-specified type for the event -- ev_data - User data for the event -- ev_extra1 - Extra data field for the event -- ev_extra2 - Extra data field for the event -- ev_extra3 - Extra data field for the event -- ev_extra4 - Extra data field for the event -- -- Returns: -- Event ID -- ---------------------------------------------------------------------- begin return pgq.insert_event_raw(queue_name, null, now(), null, null, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4); end; $$ language plpgsql; -- event inserting needs no special perms create or replace function pgq.current_event_table(x_queue_name text) returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.current_event_table(1) -- -- Return active event table for particular queue. -- -- Note: -- The result is valid only during current transaction. -- -- Parameters: -- x_queue_name - Queue name. -- ---------------------------------------------------------------------- declare res text; begin select queue_data_pfx || '_' || queue_cur_table into res from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'Event queue not found'; end if; return res; end; $$ language plpgsql; -- no perms needed -- Group: Subscribing to queue create or replace function pgq.register_consumer( x_queue_name text, x_consumer_id text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.register_consumer(2) -- -- Subscribe consumer on a queue. -- -- From this moment forward, consumer will see all events in the queue. -- -- Parameters: -- x_queue_name - Name of queue -- x_consumer_name - Name of consumer -- -- Returns: -- 0 - if already registered -- 1 - if new registration -- ---------------------------------------------------------------------- begin return pgq.register_consumer(x_queue_name, x_consumer_id, NULL); end; $$ language plpgsql; -- no perms needed create or replace function pgq.register_consumer( x_queue_name text, x_consumer_name text, x_tick_pos bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.register_consumer(3) -- -- Extended registration, allows to specify tick_id. -- -- Note: -- For usage in special situations. -- -- Parameters: -- x_queue_name - Name of a queue -- x_consumer_name - Name of consumer -- x_tick_pos - Tick ID -- -- Returns: -- 0/1 whether consumer has already registered. -- ---------------------------------------------------------------------- declare tmp text; last_tick bigint; x_queue_id integer; x_consumer_id integer; queue integer; sub record; begin select queue_id into x_queue_id from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'Event queue not created yet'; end if; -- get consumer and create if new select co_id into x_consumer_id from pgq.consumer where co_name = x_consumer_name; if not found then insert into pgq.consumer (co_name) values (x_consumer_name); x_consumer_id := currval('pgq.consumer_co_id_seq'); end if; -- if particular tick was requested, check if it exists if x_tick_pos is not null then perform 1 from pgq.tick where tick_queue = x_queue_id and tick_id = x_tick_pos; if not found then raise exception 'cannot reposition, tick not found: %', x_tick_pos; end if; end if; -- check if already registered select sub_last_tick, sub_batch into sub from pgq.subscription where sub_consumer = x_consumer_id and sub_queue = x_queue_id; if found then if x_tick_pos is not null then if sub.sub_batch is not null then raise exception 'reposition while active not allowed'; end if; -- update tick pos if requested update pgq.subscription set sub_last_tick = x_tick_pos where sub_consumer = x_consumer_id and sub_queue = x_queue_id; end if; -- already registered return 0; end if; -- new registration if x_tick_pos is null then -- start from current tick select tick_id into last_tick from pgq.tick where tick_queue = x_queue_id order by tick_queue desc, tick_id desc limit 1; if not found then raise exception 'No ticks for this queue. Please run ticker on database.'; end if; else last_tick := x_tick_pos; end if; -- register insert into pgq.subscription (sub_queue, sub_consumer, sub_last_tick) values (x_queue_id, x_consumer_id, last_tick); return 1; end; $$ language plpgsql security definer; create or replace function pgq.unregister_consumer( x_queue_name text, x_consumer_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.unregister_consumer(2) -- -- Unsubscriber consumer from the queue. Also consumer's failed -- and retry events are deleted. -- -- Parameters: -- x_queue_name - Name of the queue -- x_consumer_name - Name of the consumer -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare x_sub_id integer; begin select sub_id into x_sub_id from pgq.subscription, pgq.consumer, pgq.queue where sub_queue = queue_id and sub_consumer = co_id and queue_name = x_queue_name and co_name = x_consumer_name; if not found then raise exception 'consumer not registered on queue'; end if; delete from pgq.retry_queue where ev_owner = x_sub_id; delete from pgq.failed_queue where ev_owner = x_sub_id; delete from pgq.subscription where sub_id = x_sub_id; return 1; end; $$ language plpgsql security definer; -- Group: Batch processing create or replace function pgq.next_batch(x_queue_name text, x_consumer_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.next_batch(2) -- -- Makes next block of events active. -- -- If it returns NULL, there is no events available in queue. -- Consumer should sleep a bith then. -- -- Parameters: -- x_queue_name - Name of the queue -- x_consumer_name - Name of the consumer -- -- Returns: -- Batch ID or NULL if there are no more events available. -- ---------------------------------------------------------------------- declare next_tick bigint; next_batch bigint; errmsg text; sub record; begin select sub_queue, sub_id, sub_last_tick, sub_batch into sub from pgq.queue q, pgq.consumer c, pgq.subscription s where q.queue_name = x_queue_name and c.co_name = x_consumer_name and s.sub_queue = q.queue_id and s.sub_consumer = c.co_id; if not found then errmsg := 'Not subscriber to queue: ' || coalesce(x_queue_name, 'NULL') || '/' || coalesce(x_consumer_name, 'NULL'); raise exception '%', errmsg; end if; -- has already active batch if sub.sub_batch is not null then return sub.sub_batch; end if; -- find next tick select tick_id into next_tick from pgq.tick where tick_id > sub.sub_last_tick and tick_queue = sub.sub_queue order by tick_queue asc, tick_id asc limit 1; if not found then -- nothing to do return null; end if; -- get next batch next_batch := nextval('pgq.batch_id_seq'); update pgq.subscription set sub_batch = next_batch, sub_next_tick = next_tick, sub_active = now() where sub_id = sub.sub_id; return next_batch; end; $$ language plpgsql security definer; create or replace function pgq.get_batch_events(x_batch_id bigint) returns setof pgq.ret_batch_event as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_batch_events(1) -- -- Get all events in batch. -- -- Parameters: -- x_batch_id - ID of active batch. -- -- Returns: -- List of events. -- ---------------------------------------------------------------------- declare rec pgq.ret_batch_event%rowtype; sql text; begin sql := pgq.batch_event_sql(x_batch_id); for rec in execute sql loop return next rec; end loop; return; end; $$ language plpgsql; -- no perms needed create or replace function pgq.event_failed( x_batch_id bigint, x_event_id bigint, x_reason text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_failed(3) -- -- Copies the event to failed queue. Can be looked later. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - Event id -- x_reason - Text to associate with event. -- -- Returns: -- 0 if event was already in queue, 1 otherwise. -- ---------------------------------------------------------------------- begin insert into pgq.failed_queue (ev_failed_reason, ev_failed_time, ev_id, ev_time, ev_txid, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) select x_reason, now(), ev_id, ev_time, NULL, sub_id, coalesce(ev_retry, 0), ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.get_batch_events(x_batch_id), pgq.subscription where sub_batch = x_batch_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; -- dont worry if the event is already in queue exception when unique_violation then return 0; end; $$ language plpgsql security definer; create or replace function pgq.event_retry( x_batch_id bigint, x_event_id bigint, x_retry_time timestamptz) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry(3) -- -- Put the event into retry queue, to be processed later again. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - event id -- x_retry_time - Time when the event should be put back into queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- begin insert into pgq.retry_queue (ev_retry_after, ev_id, ev_time, ev_txid, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) select x_retry_time, ev_id, ev_time, NULL, sub_id, coalesce(ev_retry, 0) + 1, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.get_batch_events(x_batch_id), pgq.subscription where sub_batch = x_batch_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; -- dont worry if the event is already in queue exception when unique_violation then return 0; end; $$ language plpgsql security definer; create or replace function pgq.event_retry( x_batch_id bigint, x_event_id bigint, x_retry_seconds integer) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry(3) -- -- Put the event into retry queue, to be processed later again. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - event id -- x_retry_seconds - Time when the event should be put back into queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare new_retry timestamptz; begin new_retry := current_timestamp + ((x_retry_seconds || ' seconds')::interval); return pgq.event_retry(x_batch_id, x_event_id, new_retry); end; $$ language plpgsql security definer; create or replace function pgq.finish_batch( x_batch_id bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.finish_batch(1) -- -- Closes a batch. No more operations can be done with events -- of this batch. -- -- Parameters: -- x_batch_id - id of batch. -- -- Returns: -- If batch 1 if batch was found, 0 otherwise. -- ---------------------------------------------------------------------- begin update pgq.subscription set sub_active = now(), sub_last_tick = sub_next_tick, sub_next_tick = null, sub_batch = null where sub_batch = x_batch_id; if not found then raise warning 'finish_batch: batch % not found', x_batch_id; return 0; end if; return 1; end; $$ language plpgsql security definer; -- Group: General info functions create or replace function pgq.get_queue_info() returns setof pgq.ret_queue_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_queue_info(0) -- -- Get info about all queues. -- -- Returns: -- List of pgq.ret_queue_info records. -- ---------------------------------------------------------------------- declare q record; ret pgq.ret_queue_info%rowtype; begin for q in select queue_name from pgq.queue order by 1 loop select * into ret from pgq.get_queue_info(q.queue_name); return next ret; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.get_queue_info(qname text) returns pgq.ret_queue_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_queue_info(1) -- -- Get info about particular queue. -- -- Returns: -- One pgq.ret_queue_info record. -- ---------------------------------------------------------------------- declare ret pgq.ret_queue_info%rowtype; begin select queue_name, queue_ntables, queue_cur_table, queue_rotation_period, queue_switch_time, queue_external_ticker, queue_ticker_max_count, queue_ticker_max_lag, queue_ticker_idle_period, (select current_timestamp - tick_time from pgq.tick where tick_queue = queue_id order by tick_queue desc, tick_id desc limit 1 ) as ticker_lag into ret from pgq.queue where queue_name = qname; return ret; end; $$ language plpgsql security definer; ------------------------------------------------------------------------- create or replace function pgq.get_consumer_info() returns setof pgq.ret_consumer_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_consumer_info(0) -- -- Returns info about all consumers on all queues. -- -- Returns: -- See pgq.get_consumer_info(2) -- ---------------------------------------------------------------------- declare ret pgq.ret_consumer_info%rowtype; i record; begin for i in select queue_name from pgq.queue order by 1 loop for ret in select * from pgq.get_consumer_info(i.queue_name) loop return next ret; end loop; end loop; return; end; $$ language plpgsql security definer; ------------------------------------------------------------------------- create or replace function pgq.get_consumer_info(x_queue_name text) returns setof pgq.ret_consumer_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_consumer_info(1) -- -- Returns info about consumers on one particular queue. -- -- Parameters: -- x_queue_name - Queue name -- -- Returns: -- See pgq.get_consumer_info(2) -- ---------------------------------------------------------------------- declare ret pgq.ret_consumer_info%rowtype; tmp record; begin for tmp in select queue_name, co_name from pgq.queue, pgq.consumer, pgq.subscription where queue_id = sub_queue and co_id = sub_consumer and queue_name = x_queue_name order by 1, 2 loop for ret in select * from pgq.get_consumer_info(tmp.queue_name, tmp.co_name) loop return next ret; end loop; end loop; return; end; $$ language plpgsql security definer; ------------------------------------------------------------------------ create or replace function pgq.get_consumer_info( x_queue_name text, x_consumer_name text) returns setof pgq.ret_consumer_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_consumer_info(2) -- -- Get info about particular consumer on particular queue. -- -- Parameters: -- x_queue_name - name of a queue. -- x_consumer_name - name of a consumer -- -- Returns: -- queue_name - Queue name -- consumer_name - Consumer name -- lag - How old are events the consumer is processing -- last_seen - When the consumer seen by pgq -- last_tick - Tick ID of last processed tick -- current_batch - Current batch ID, if one is active or NULL -- next_tick - If batch is active, then its final tick. -- ---------------------------------------------------------------------- declare ret pgq.ret_consumer_info%rowtype; begin for ret in select queue_name, co_name, current_timestamp - tick_time as lag, current_timestamp - sub_active as last_seen, sub_last_tick as last_tick, sub_batch as current_batch, sub_next_tick as next_tick from pgq.subscription, pgq.tick, pgq.queue, pgq.consumer where tick_id = sub_last_tick and queue_id = sub_queue and tick_queue = sub_queue and co_id = sub_consumer and queue_name = x_queue_name and co_name = x_consumer_name order by 1,2 loop return next ret; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.version() returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.version(0) -- -- Returns verison string for pgq. ATM its SkyTools version -- that is only bumped when PGQ database code changes. -- ---------------------------------------------------------------------- begin return '2.1.4'; end; $$ language plpgsql; create or replace function pgq.get_batch_info(x_batch_id bigint) returns pgq.ret_batch_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_batch_info(1) -- -- Returns detailed info about a batch. -- -- Parameters: -- x_batch_id - id of a active batch. -- -- Returns: -- Info -- ---------------------------------------------------------------------- declare ret pgq.ret_batch_info%rowtype; begin select queue_name, co_name, prev.tick_time as batch_start, cur.tick_time as batch_end, sub_last_tick, sub_next_tick, current_timestamp - cur.tick_time as lag into ret from pgq.subscription, pgq.tick cur, pgq.tick prev, pgq.queue, pgq.consumer where sub_batch = x_batch_id and prev.tick_id = sub_last_tick and prev.tick_queue = sub_queue and cur.tick_id = sub_next_tick and cur.tick_queue = sub_queue and queue_id = sub_queue and co_id = sub_consumer; return ret; end; $$ language plpgsql security definer; -- Group: Failed queue browsing create or replace function pgq.failed_event_list( x_queue_name text, x_consumer_name text) returns setof pgq.failed_queue as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_list(2) -- -- Get list of all failed events for one consumer. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- -- Returns: -- List of failed events. -- ---------------------------------------------------------------------- declare rec pgq.failed_queue%rowtype; begin for rec in select fq.* from pgq.failed_queue fq, pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id and ev_owner = sub_id order by ev_id loop return next rec; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_list( x_queue_name text, x_consumer_name text, x_count integer, x_offset integer) returns setof pgq.failed_queue as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_list(4) -- -- Get list of failed events, from offset and specific count. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- x_count - Max amount of events to fetch -- x_offset - From this offset -- -- Returns: -- List of failed events. -- ---------------------------------------------------------------------- declare rec pgq.failed_queue%rowtype; begin for rec in select fq.* from pgq.failed_queue fq, pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id and ev_owner = sub_id order by ev_id limit x_count offset x_offset loop return next rec; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_count( x_queue_name text, x_consumer_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_count(2) -- -- Get size of failed event queue. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- -- Returns: -- Number of failed events in failed event queue. -- ---------------------------------------------------------------------- declare ret integer; begin select count(1) into ret from pgq.failed_queue, pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue_name and co_name = x_consumer_name and sub_queue = queue_id and sub_consumer = co_id and ev_owner = sub_id; return ret; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_delete( x_queue_name text, x_consumer_name text, x_event_id bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_delete(3) -- -- Delete specific event from failed event queue. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- x_event_id - Event ID -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare x_sub_id integer; begin select sub_id into x_sub_id from pgq.subscription, pgq.consumer, pgq.queue where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id; if not found then raise exception 'no such queue/consumer'; end if; delete from pgq.failed_queue where ev_owner = x_sub_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_retry( x_queue_name text, x_consumer_name text, x_event_id bigint) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_retry(3) -- -- Insert specific event from failed queue to main queue. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- x_event_id - Event ID -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare ret bigint; x_sub_id integer; begin select sub_id into x_sub_id from pgq.subscription, pgq.consumer, pgq.queue where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id; if not found then raise exception 'no such queue/consumer'; end if; select pgq.insert_event_raw(x_queue_name, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) into ret from pgq.failed_queue, pgq.consumer, pgq.queue where ev_owner = x_sub_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; perform pgq.failed_event_delete(x_queue_name, x_consumer_name, x_event_id); return ret; end; $$ language plpgsql security definer; -- Section: Public Triggers -- Group: Trigger Functions create or replace function pgq.logutriga() returns trigger as $$ # -- ---------------------------------------------------------------------- # -- Function: pgq.logutriga() # -- # -- Trigger function that puts row data urlencoded into queue. # -- # -- Trigger parameters: # -- arg1 - queue name # -- arg2 - optionally 'SKIP' # -- # -- Queue event fields: # -- ev_type - I/U/D # -- ev_data - column values urlencoded # -- ev_extra1 - table name # -- ev_extra2 - primary key columns # -- # -- Regular listen trigger example: # -- > CREATE TRIGGER triga_nimi AFTER INSERT OR UPDATE ON customer # -- > FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('qname'); # -- # -- Redirect trigger example: # -- > CREATE TRIGGER triga_nimi AFTER INSERT OR UPDATE ON customer # -- > FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('qname', 'SKIP'); # -- ---------------------------------------------------------------------- # this triger takes 1 or 2 args: # queue_name - destination queue # option return code (OK, SKIP) SKIP means op won't happen # copy-paste of db_urlencode from skytools.quoting from urllib import quote_plus def db_urlencode(dict): elem_list = [] for k, v in dict.items(): if v is None: elem = quote_plus(str(k)) else: elem = quote_plus(str(k)) + '=' + quote_plus(str(v)) elem_list.append(elem) return '&'.join(elem_list) # load args queue_name = TD['args'][0] if len(TD['args']) > 1: ret_code = TD['args'][1] else: ret_code = 'OK' table_oid = TD['relid'] # on first call init plans if not 'init_done' in SD: # find table name q = "SELECT n.nspname || '.' || c.relname AS table_name"\ " FROM pg_namespace n, pg_class c"\ " WHERE n.oid = c.relnamespace AND c.oid = $1" SD['name_plan'] = plpy.prepare(q, ['oid']) # find key columns q = "SELECT k.attname FROM pg_index i, pg_attribute k"\ " WHERE i.indrelid = $1 AND k.attrelid = i.indexrelid"\ " AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped"\ " ORDER BY k.attnum" SD['key_plan'] = plpy.prepare(q, ['oid']) # insert data q = "SELECT pgq.insert_event($1, $2, $3, $4, $5, null, null)" SD['ins_plan'] = plpy.prepare(q, ['text', 'text', 'text', 'text', 'text']) # shorter tags SD['op_map'] = {'INSERT': 'I', 'UPDATE': 'U', 'DELETE': 'D'} # remember init SD['init_done'] = 1 # load & cache table data if table_oid in SD: tbl_name, tbl_keys = SD[table_oid] else: res = plpy.execute(SD['name_plan'], [table_oid]) tbl_name = res[0]['table_name'] res = plpy.execute(SD['key_plan'], [table_oid]) tbl_keys = ",".join(map(lambda x: x['attname'], res)) SD[table_oid] = (tbl_name, tbl_keys) # prepare args if TD['event'] == 'DELETE': data = db_urlencode(TD['old']) else: data = db_urlencode(TD['new']) # insert event plpy.execute(SD['ins_plan'], [ queue_name, SD['op_map'][TD['event']], data, tbl_name, tbl_keys]) # done return ret_code $$ language plpythonu; -- listen trigger: -- create trigger triga_nimi after insert or update on customer -- for each row execute procedure pgq.sqltriga('qname'); -- redirect trigger: -- create trigger triga_nimi after insert or update on customer -- for each row execute procedure pgq.sqltriga('qname', 'ret=SKIP'); create or replace function pgq.sqltriga() returns trigger as $$ # -- ---------------------------------------------------------------------- # -- Function: pgq.sqltriga() # -- # -- Trigger function that puts row data in partial SQL form into queue. # -- # -- Parameters: # -- arg1 - queue name # -- arg2 - optional urlencoded options # -- # -- Extra options: # -- # -- ret - return value for function OK/SKIP # -- pkey - override pkey fields, can be functions # -- ignore - comma separated field names to ignore # -- # -- Queue event fields: # -- ev_type - I/U/D # -- ev_data - partial SQL statement # -- ev_extra1 - table name # -- # -- ---------------------------------------------------------------------- # this triger takes 1 or 2 args: # queue_name - destination queue # args - urlencoded dict of options: # ret - return value: OK/SKIP # pkey - comma-separated col names or funcs on cols # simple: pkey=user,orderno # hashed: pkey=user,hashtext(user) # ignore - comma-separated col names to ignore # on first call init stuff if not 'init_done' in SD: # find table name plan q = "SELECT n.nspname || '.' || c.relname AS table_name"\ " FROM pg_namespace n, pg_class c"\ " WHERE n.oid = c.relnamespace AND c.oid = $1" SD['name_plan'] = plpy.prepare(q, ['oid']) # find key columns plan q = "SELECT k.attname FROM pg_index i, pg_attribute k"\ " WHERE i.indrelid = $1 AND k.attrelid = i.indexrelid"\ " AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped"\ " ORDER BY k.attnum" SD['key_plan'] = plpy.prepare(q, ['oid']) # data insertion q = "SELECT pgq.insert_event($1, $2, $3, $4, null, null, null)" SD['ins_plan'] = plpy.prepare(q, ['text', 'text', 'text', 'text']) # shorter tags SD['op_map'] = {'INSERT': 'I', 'UPDATE': 'U', 'DELETE': 'D'} # quoting from psycopg import QuotedString def quote(s): if s is None: return "null" s = str(s) return str(QuotedString(s)) s = s.replace('\\', '\\\\') s = s.replace("'", "''") return "'%s'" % s # TableInfo class import re, urllib class TableInfo: func_rc = re.compile("([^(]+) [(] ([^)]+) [)]", re.I | re.X) def __init__(self, table_oid, options_txt): res = plpy.execute(SD['name_plan'], [table_oid]) self.name = res[0]['table_name'] self.parse_options(options_txt) self.load_pkey() def recheck(self, options_txt): if self.options_txt == options_txt: return self.parse_options(options_txt) self.load_pkey() def parse_options(self, options_txt): self.options = {'ret': 'OK'} if options_txt: for s in options_txt.split('&'): k, v = s.split('=', 1) self.options[k] = urllib.unquote_plus(v) self.options_txt = options_txt def load_pkey(self): self.pkey_list = [] if not 'pkey' in self.options: res = plpy.execute(SD['key_plan'], [table_oid]) for krow in res: col = krow['attname'] expr = col + "=%s" self.pkey_list.append( (col, expr) ) else: for a_pk in self.options['pkey'].split(','): m = self.func_rc.match(a_pk) if m: col = m.group(2) fn = m.group(1) expr = "%s(%s) = %s(%%s)" % (fn, col, fn) else: # normal case col = a_pk expr = col + "=%s" self.pkey_list.append( (col, expr) ) if len(self.pkey_list) == 0: plpy.error('sqltriga needs primary key on table') def get_insert_stmt(self, new): col_list = [] val_list = [] for k, v in new.items(): col_list.append(k) val_list.append(quote(v)) return "(%s) values (%s)" % (",".join(col_list), ",".join(val_list)) def get_update_stmt(self, old, new): chg_list = [] for k, v in new.items(): ov = old[k] if v == ov: continue chg_list.append("%s=%s" % (k, quote(v))) if len(chg_list) == 0: pk = self.pkey_list[0][0] chg_list.append("%s=%s" % (pk, quote(new[pk]))) return "%s where %s" % (",".join(chg_list), self.get_pkey_expr(new)) def get_pkey_expr(self, data): exp_list = [] for col, exp in self.pkey_list: exp_list.append(exp % quote(data[col])) return " and ".join(exp_list) SD['TableInfo'] = TableInfo # cache some functions def proc_insert(tbl): return tbl.get_insert_stmt(TD['new']) def proc_update(tbl): return tbl.get_update_stmt(TD['old'], TD['new']) def proc_delete(tbl): return tbl.get_pkey_expr(TD['old']) SD['event_func'] = { 'I': proc_insert, 'U': proc_update, 'D': proc_delete, } # remember init SD['init_done'] = 1 # load args table_oid = TD['relid'] queue_name = TD['args'][0] if len(TD['args']) > 1: options_str = TD['args'][1] else: options_str = '' # load & cache table data if table_oid in SD: tbl = SD[table_oid] tbl.recheck(options_str) else: tbl = SD['TableInfo'](table_oid, options_str) SD[table_oid] = tbl # generate payload op = SD['op_map'][TD['event']] data = SD['event_func'][op](tbl) # insert event plpy.execute(SD['ins_plan'], [queue_name, op, data, tbl.name]) # done return tbl.options['ret'] $$ language plpythonu; skytools-2.1.13/tests/upgrade/sql/v2.1.4_pgq_ext.sql0000644000175000017500000000704611670174255021200 0ustar markomarko set client_min_messages = 'warning'; set default_with_oids = 'off'; create schema pgq_ext; grant usage on schema pgq_ext to public; -- -- batch tracking -- create table pgq_ext.completed_batch ( consumer_id text not null, last_batch_id bigint not null, primary key (consumer_id) ); -- -- event tracking -- create table pgq_ext.completed_event ( consumer_id text not null, batch_id bigint not null, event_id bigint not null, primary key (consumer_id, batch_id, event_id) ); create table pgq_ext.partial_batch ( consumer_id text not null, cur_batch_id bigint not null, primary key (consumer_id) ); -- -- tick tracking for SerialConsumer() -- no access functions provided here -- create table pgq_ext.completed_tick ( consumer_id text not null, last_tick_id bigint not null, primary key (consumer_id) ); create or replace function pgq_ext.is_batch_done( a_consumer text, a_batch_id bigint) returns boolean as $$ declare res boolean; begin select last_batch_id = a_batch_id into res from pgq_ext.completed_batch where consumer_id = a_consumer; if not found then return false; end if; return res; end; $$ language plpgsql security definer; create or replace function pgq_ext.set_batch_done( a_consumer text, a_batch_id bigint) returns boolean as $$ begin if pgq_ext.is_batch_done(a_consumer, a_batch_id) then return false; end if; if a_batch_id > 0 then update pgq_ext.completed_batch set last_batch_id = a_batch_id where consumer_id = a_consumer; if not found then insert into pgq_ext.completed_batch (consumer_id, last_batch_id) values (a_consumer, a_batch_id); end if; end if; return true; end; $$ language plpgsql security definer; create or replace function pgq_ext.is_event_done( a_consumer text, a_batch_id bigint, a_event_id bigint) returns boolean as $$ declare res bigint; begin perform 1 from pgq_ext.completed_event where consumer_id = a_consumer and batch_id = a_batch_id and event_id = a_event_id; return found; end; $$ language plpgsql security definer; create or replace function pgq_ext.set_event_done( a_consumer text, a_batch_id bigint, a_event_id bigint) returns boolean as $$ declare old_batch bigint; begin -- check if done perform 1 from pgq_ext.completed_event where consumer_id = a_consumer and batch_id = a_batch_id and event_id = a_event_id; if found then return false; end if; -- if batch changed, do cleanup select cur_batch_id into old_batch from pgq_ext.partial_batch where consumer_id = a_consumer; if not found then -- first time here insert into pgq_ext.partial_batch (consumer_id, cur_batch_id) values (a_consumer, a_batch_id); elsif old_batch <> a_batch_id then -- batch changed, that means old is finished on queue db -- thus the tagged events are not needed anymore delete from pgq_ext.completed_event where consumer_id = a_consumer and batch_id = old_batch; -- remember current one update pgq_ext.partial_batch set cur_batch_id = a_batch_id where consumer_id = a_consumer; end if; -- tag as done insert into pgq_ext.completed_event (consumer_id, batch_id, event_id) values (a_consumer, a_batch_id, a_event_id); return true; end; $$ language plpgsql security definer; skytools-2.1.13/tests/upgrade/sql/v2.1.4_londiste.sql0000644000175000017500000005105611670174255021352 0ustar markomarkoset default_with_oids = 'off'; create schema londiste; grant usage on schema londiste to public; create table londiste.provider_table ( nr serial not null, queue_name text not null, table_name text not null, trigger_name text, primary key (queue_name, table_name) ); create table londiste.provider_seq ( nr serial not null, queue_name text not null, seq_name text not null, primary key (queue_name, seq_name) ); create table londiste.completed ( consumer_id text not null, last_tick_id bigint not null, primary key (consumer_id) ); create table londiste.link ( source text not null, dest text not null, primary key (source), unique (dest) ); create table londiste.subscriber_table ( nr serial not null, queue_name text not null, table_name text not null, snapshot text, merge_state text, trigger_name text, skip_truncate bool, primary key (queue_name, table_name) ); create table londiste.subscriber_seq ( nr serial not null, queue_name text not null, seq_name text not null, primary key (queue_name, seq_name) ); create type londiste.ret_provider_table_list as ( table_name text, trigger_name text ); create type londiste.ret_subscriber_table as ( table_name text, merge_state text, snapshot text, trigger_name text, skip_truncate bool ); create or replace function londiste.deny_trigger() returns trigger as $$ if 'undeny' in GD: return 'OK' plpy.error('Changes no allowed on this table') $$ language plpythonu; create or replace function londiste.disable_deny_trigger(i_allow boolean) returns boolean as $$ if args[0]: GD['undeny'] = 1 return True else: if 'undeny' in GD: del GD['undeny'] return False $$ language plpythonu; create or replace function londiste.find_column_types(tbl text) returns text as $$ declare res text; col record; tbl_oid oid; begin tbl_oid := londiste.find_table_oid(tbl); res := ''; for col in SELECT CASE WHEN k.attname IS NOT NULL THEN 'k' ELSE 'v' END AS type FROM pg_attribute a LEFT JOIN ( SELECT k.attname FROM pg_index i, pg_attribute k WHERE i.indrelid = tbl_oid AND k.attrelid = i.indexrelid AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped ) k ON (k.attname = a.attname) WHERE a.attrelid = tbl_oid AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum loop res := res || col.type; end loop; return res; end; $$ language plpgsql; create or replace function londiste.find_rel_oid(tbl text, kind text) returns oid as $$ declare res oid; pos integer; schema text; name text; begin pos := position('.' in tbl); if pos > 0 then schema := substring(tbl for pos - 1); name := substring(tbl from pos + 1); else schema := 'public'; name := tbl; end if; select c.oid into res from pg_namespace n, pg_class c where c.relnamespace = n.oid and c.relkind = kind and n.nspname = schema and c.relname = name; if not found then if kind = 'r' then raise exception 'table not found'; elsif kind = 'S' then raise exception 'seq not found'; else raise exception 'weird relkind'; end if; end if; return res; end; $$ language plpgsql; create or replace function londiste.find_table_oid(tbl text) returns oid as $$ begin return londiste.find_rel_oid(tbl, 'r'); end; $$ language plpgsql; create or replace function londiste.find_seq_oid(tbl text) returns oid as $$ begin return londiste.find_rel_oid(tbl, 'S'); end; $$ language plpgsql; create or replace function londiste.get_last_tick(i_consumer text) returns bigint as $$ declare res bigint; begin select last_tick_id into res from londiste.completed where consumer_id = i_consumer; return res; end; $$ language plpgsql security definer; create or replace function londiste.link_source(i_dst_name text) returns text as $$ declare res text; begin select source into res from londiste.link where dest = i_dst_name; return res; end; $$ language plpgsql security definer; create or replace function londiste.link_dest(i_source_name text) returns text as $$ declare res text; begin select dest into res from londiste.link where source = i_source_name; return res; end; $$ language plpgsql security definer; create or replace function londiste.cmp_list(list1 text, a_queue text, a_table text, a_field text) returns boolean as $$ declare sql text; tmp record; list2 text; begin sql := 'select ' || a_field || ' as name from ' || a_table || ' where queue_name = ' || quote_literal(a_queue) || ' order by 1'; list2 := ''; for tmp in execute sql loop if list2 = '' then list2 := tmp.name; else list2 := list2 || ',' || tmp.name; end if; end loop; return list1 = list2; end; $$ language plpgsql; create or replace function londiste.link(i_source_name text, i_dest_name text, prov_tick_id bigint, prov_tbl_list text, prov_seq_list text) returns text as $$ declare tmp text; list text; tick_seq text; external boolean; last_tick bigint; begin -- check if all matches if not londiste.cmp_list(prov_tbl_list, i_source_name, 'londiste.subscriber_table', 'table_name') then raise exception 'not all tables copied into subscriber'; end if; if not londiste.cmp_list(prov_seq_list, i_source_name, 'londiste.subscriber_seq', 'seq_name') then raise exception 'not all seqs copied into subscriber'; end if; if not londiste.cmp_list(prov_seq_list, i_dest_name, 'londiste.provider_table', 'table_name') then raise exception 'linked provider queue does not have all tables'; end if; if not londiste.cmp_list(prov_seq_list, i_dest_name, 'londiste.provider_seq', 'seq_name') then raise exception 'linked provider queue does not have all seqs'; end if; -- check pgq select queue_external_ticker, queue_tick_seq into external, tick_seq from pgq.queue where queue_name = i_dest_name; if not found then raise exception 'dest queue does not exist'; end if; if external then raise exception 'dest queue has already external_ticker turned on?'; end if; if nextval(tick_seq) >= prov_tick_id then raise exception 'dest queue ticks larger'; end if; update pgq.queue set queue_external_ticker = true where queue_name = i_dest_name; insert into londiste.link (source, dest) values (i_source_name, i_dest_name); return null; end; $$ language plpgsql security definer; create or replace function londiste.link_del(i_source_name text, i_dest_name text) returns text as $$ begin delete from londiste.link where source = i_source_name and dest = i_dest_name; if not found then raise exception 'no suck link'; end if; return null; end; $$ language plpgsql security definer; create or replace function londiste.provider_add_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin -- check if linked queue link := londiste.link_source(i_queue_name); if link is not null then raise exception 'Linked queue, cannot modify'; end if; perform 1 from pg_class where oid = londiste.find_seq_oid(i_seq_name); if not found then raise exception 'seq not found'; end if; insert into londiste.provider_seq (queue_name, seq_name) values (i_queue_name, i_seq_name); perform londiste.provider_notify_change(i_queue_name); return 0; end; $$ language plpgsql security definer; create or replace function londiste.provider_add_table( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; sql text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; if position('k' in i_col_types) < 1 then raise exception 'need key column'; end if; if position('.' in i_table_name) < 1 then raise exception 'need fully-qualified table name'; end if; select queue_name into tgname from pgq.queue where queue_name = i_queue_name; if not found then raise exception 'no such event queue'; end if; tgname := i_queue_name || '_logger'; tgname := replace(lower(tgname), '.', '_'); insert into londiste.provider_table (queue_name, table_name, trigger_name) values (i_queue_name, i_table_name, tgname); perform londiste.provider_create_trigger( i_queue_name, i_table_name, i_col_types); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_add_table( i_queue_name text, i_table_name text ) returns integer as $$ begin return londiste.provider_add_table(i_queue_name, i_table_name, londiste.find_column_types(i_table_name)); end; $$ language plpgsql; create or replace function londiste.provider_create_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; sql text; begin select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; sql := 'select pgq.insert_event(' || quote_literal(i_queue_name) || ', $1, $2, ' || quote_literal(i_table_name) || ', NULL, NULL, NULL)'; execute 'create trigger ' || tgname || ' after insert or update or delete on ' || i_table_name || ' for each row execute procedure logtriga($arg1$' || i_col_types || '$arg1$, $arg2$' || sql || '$arg2$)'; return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_get_seq_list(i_queue_name text) returns setof text as $$ declare rec record; begin for rec in select seq_name from londiste.provider_seq where queue_name = i_queue_name order by nr loop return next rec.seq_name; end loop; return; end; $$ language plpgsql security definer; create or replace function londiste.provider_get_table_list(i_queue text) returns setof londiste.ret_provider_table_list as $$ declare rec londiste.ret_provider_table_list%rowtype; begin for rec in select table_name, trigger_name from londiste.provider_table where queue_name = i_queue order by nr loop return next rec; end loop; return; end; $$ language plpgsql security definer; create or replace function londiste.provider_notify_change(i_queue_name text) returns integer as $$ declare res text; tbl record; begin res := ''; for tbl in select table_name from londiste.provider_table where queue_name = i_queue_name order by nr loop if res = '' then res := tbl.table_name; else res := res || ',' || tbl.table_name; end if; end loop; perform pgq.insert_event(i_queue_name, 'T', res); return 1; end; $$ language plpgsql; create or replace function londiste.provider_refresh_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare t_name text; tbl_oid oid; begin select trigger_name into t_name from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; tbl_oid := londiste.find_table_oid(i_table_name); perform 1 from pg_trigger where tgrelid = tbl_oid and tgname = t_name; if found then execute 'drop trigger ' || t_name || ' on ' || i_table_name; end if; perform londiste.provider_create_trigger(i_queue_name, i_table_name, i_col_types); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_refresh_trigger( i_queue_name text, i_table_name text ) returns integer strict as $$ begin return londiste.provider_refresh_trigger(i_queue_name, i_table_name, londiste.find_column_types(i_table_name)); end; $$ language plpgsql security definer; create or replace function londiste.provider_remove_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin -- check if linked queue link := londiste.link_source(i_queue_name); if link is not null then raise exception 'Linked queue, cannot modify'; end if; delete from londiste.provider_seq where queue_name = i_queue_name and seq_name = i_seq_name; if not found then raise exception 'seq not attached'; end if; perform londiste.provider_notify_change(i_queue_name); return 0; end; $$ language plpgsql security definer; create or replace function londiste.provider_remove_table( i_queue_name text, i_table_name text ) returns integer as $$ declare tgname text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'no such table registered'; end if; execute 'drop trigger ' || tgname || ' on ' || i_table_name; delete from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; return 1; end; $$ language plpgsql security definer; create or replace function londiste.set_last_tick( i_consumer text, i_tick_id bigint) returns integer as $$ begin update londiste.completed set last_tick_id = i_tick_id where consumer_id = i_consumer; if not found then insert into londiste.completed (consumer_id, last_tick_id) values (i_consumer, i_tick_id); end if; return 1; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_add_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin insert into londiste.subscriber_seq (queue_name, seq_name) values (i_queue_name, i_seq_name); -- update linked queue if needed link := londiste.link_dest(i_queue_name); if link is not null then insert into londiste.provider_seq (queue_name, seq_name) values (link, i_seq_name); perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_add_table( i_queue_name text, i_table text) returns integer as $$ begin insert into londiste.subscriber_table (queue_name, table_name) values (i_queue_name, i_table); -- linked queue is updated, when the table is copied return 0; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_get_seq_list(i_queue_name text) returns setof text as $$ declare rec record; begin for rec in select seq_name from londiste.subscriber_seq where queue_name = i_queue_name order by nr loop return next rec.seq_name; end loop; return; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_get_table_list(i_queue_name text) returns setof londiste.ret_subscriber_table as $$ declare rec londiste.ret_subscriber_table%rowtype; begin for rec in select table_name, merge_state, snapshot, trigger_name, skip_truncate from londiste.subscriber_table where queue_name = i_queue_name order by nr loop return next rec; end loop; return; end; $$ language plpgsql security definer; -- compat create or replace function londiste.get_table_state(i_queue text) returns setof londiste.subscriber_table as $$ declare rec londiste.subscriber_table%rowtype; begin for rec in select * from londiste.subscriber_table where queue_name = i_queue order by nr loop return next rec; end loop; return; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_remove_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin delete from londiste.subscriber_seq where queue_name = i_queue_name and seq_name = i_seq_name; if not found then raise exception 'no such seq?'; end if; -- update linked queue if needed link := londiste.link_dest(i_queue_name); if link is not null then delete from londiste.provider_seq where queue_name = link and seq_name = i_seq_name; perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_remove_table( i_queue_name text, i_table text) returns integer as $$ declare link text; begin delete from londiste.subscriber_table where queue_name = i_queue_name and table_name = i_table; if not found then raise exception 'no such table'; end if; -- sync link link := londiste.link_dest(i_queue_name); if link is not null then delete from londiste.provider_table where queue_name = link and table_name = i_table; perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql; create or replace function londiste.subscriber_set_skip_truncate( i_queue text, i_table text, i_value bool) returns integer as $$ begin update londiste.subscriber_table set skip_truncate = i_value where queue_name = i_queue and table_name = i_table; if not found then raise exception 'table not found'; end if; return 1; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_set_table_state( i_queue_name text, i_table_name text, i_snapshot text, i_merge_state text) returns integer as $$ declare link text; ok integer; begin update londiste.subscriber_table set snapshot = i_snapshot, merge_state = i_merge_state, -- reset skip_snapshot when table is copied over skip_truncate = case when i_merge_state = 'ok' then null else skip_truncate end where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'no such table'; end if; -- sync link state also link := londiste.link_dest(i_queue_name); if link then select * from londiste.provider_table where queue_name = linkdst and table_name = i_table_name; if found then if i_merge_state is null or i_merge_state <> 'ok' then delete from londiste.provider_table where queue_name = link and table_name = i_table_name; perform londiste.notify_change(link); end if; else if i_merge_state = 'ok' then insert into londiste.provider_table (queue_name, table_name) values (link, i_table_name); perform londiste.notify_change(link); end if; end if; end if; return 1; end; $$ language plpgsql security definer; create or replace function londiste.set_table_state( i_queue_name text, i_table_name text, i_snapshot text, i_merge_state text) returns integer as $$ begin return londiste.subscriber_set_table_state(i_queue_name, i_table_name, i_snapshot, i_merge_state); end; $$ language plpgsql security definer; skytools-2.1.13/tests/upgrade/sql/v2.1.4_txid82.sql0000644000175000017500000000365411670174255020654 0ustar markomarko-- ---------- -- txid.sql -- -- SQL script for loading the transaction ID compatible datatype -- -- Copyright (c) 2003-2004, PostgreSQL Global Development Group -- Author: Jan Wieck, Afilias USA INC. -- -- ---------- set client_min_messages = 'warning'; CREATE DOMAIN txid AS bigint CHECK (value > 0); -- -- A special transaction snapshot data type for faster visibility checks -- CREATE OR REPLACE FUNCTION txid_snapshot_in(cstring) RETURNS txid_snapshot AS '$libdir/txid' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION txid_snapshot_out(txid_snapshot) RETURNS cstring AS '$libdir/txid' LANGUAGE C IMMUTABLE STRICT; -- -- The data type itself -- CREATE TYPE txid_snapshot ( INPUT = txid_snapshot_in, OUTPUT = txid_snapshot_out, INTERNALLENGTH = variable, STORAGE = extended, ALIGNMENT = double ); CREATE OR REPLACE FUNCTION get_current_txid() RETURNS bigint AS '$libdir/txid', 'txid_current' LANGUAGE C SECURITY DEFINER; CREATE OR REPLACE FUNCTION get_current_snapshot() RETURNS txid_snapshot AS '$libdir/txid', 'txid_current_snapshot' LANGUAGE C SECURITY DEFINER; CREATE OR REPLACE FUNCTION get_snapshot_xmin(txid_snapshot) RETURNS bigint AS '$libdir/txid', 'txid_snapshot_xmin' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION get_snapshot_xmax(txid_snapshot) RETURNS bigint AS '$libdir/txid', 'txid_snapshot_xmax' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION get_snapshot_active(txid_snapshot) RETURNS setof bigint AS '$libdir/txid', 'txid_snapshot_active' LANGUAGE C IMMUTABLE STRICT; -- -- Special comparision functions used by the remote worker -- for sync chunk selection -- CREATE OR REPLACE FUNCTION txid_in_snapshot(bigint, txid_snapshot) RETURNS boolean AS '$libdir/txid', 'txid_in_snapshot' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION txid_not_in_snapshot(bigint, txid_snapshot) RETURNS boolean AS '$libdir/txid', 'txid_not_in_snapshot' LANGUAGE C IMMUTABLE STRICT; skytools-2.1.13/tests/walmgr/0000755000175000017500000000000011727601174015132 5ustar markomarkoskytools-2.1.13/tests/walmgr/conf.slave/0000755000175000017500000000000011727601174017170 5ustar markomarkoskytools-2.1.13/tests/walmgr/conf.slave/postgresql.conf0000644000175000017500000000115011670174255022241 0ustar markomarko#listen_addresses = 'localhost' # what IP address(es) to listen on; # comma-separated list of addresses; # defaults to 'localhost', '*' = all port = 7201 max_connections = 100 #superuser_reserved_connections = 2 unix_socket_directory = '/tmp/waltest' shared_buffers = 1000 # min 16 or max_connections*2, 8KB each # These settings are initialized by initdb -- they might be changed lc_messages = 'C' # locale for system error message # strings lc_monetary = 'C' # locale for monetary formatting lc_numeric = 'C' # locale for number formatting lc_time = 'C' # locale for time formatting skytools-2.1.13/tests/walmgr/conf.slave/pg_ident.conf0000644000175000017500000000266411670174255021642 0ustar markomarko# PostgreSQL Ident Authentication Maps # ==================================== # # Refer to the PostgreSQL Administrator's Guide, chapter "Client # Authentication" for a complete description. A short synopsis # follows. # # This file controls PostgreSQL ident-based authentication. It maps # ident user names (typically Unix user names) to their corresponding # PostgreSQL user names. Records are of the form: # # MAPNAME IDENT-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. IDENT-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The # existence of a record specifies that IDENT-USERNAME may connect as # PG-USERNAME. Multiple maps may be specified in this file and used # by pg_hba.conf. # # This file is read on server startup and when the postmaster receives # a SIGHUP signal. If you edit the file on a running system, you have # to SIGHUP the postmaster for the changes to take effect. You can use # "pg_ctl reload" to do that. # Put your actual configuration here # ---------------------------------- # # No map names are defined in the default configuration. If all ident # user names and PostgreSQL user names are the same, you don't need # this file. Instead, use the special map name "sameuser" in # pg_hba.conf. # MAPNAME IDENT-USERNAME PG-USERNAME skytools-2.1.13/tests/walmgr/conf.slave/pg_hba.conf0000644000175000017500000000650411670174255021266 0ustar markomarko# PostgreSQL Client Authentication Configuration File # =================================================== # # Refer to the PostgreSQL Administrator's Guide, chapter "Client # Authentication" for a complete description. A short synopsis # follows. # # This file controls: which hosts are allowed to connect, how clients # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # # local DATABASE USER METHOD [OPTION] # host DATABASE USER CIDR-ADDRESS METHOD [OPTION] # hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION] # hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION] # # (The uppercase items must be replaced by actual values.) # # The first field is the connection type: "local" is a Unix-domain socket, # "host" is either a plain or SSL-encrypted TCP/IP socket, "hostssl" is an # SSL-encrypted TCP/IP socket, and "hostnossl" is a plain TCP/IP socket. # # DATABASE can be "all", "sameuser", "samerole", a database name, or # a comma-separated list thereof. # # USER can be "all", a user name, a group name prefixed with "+", or # a comma-separated list thereof. In both the DATABASE and USER fields # you can also write a file name prefixed with "@" to include names from # a separate file. # # CIDR-ADDRESS specifies the set of hosts the record matches. # It is made up of an IP address and a CIDR mask that is an integer # (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that specifies # the number of significant bits in the mask. Alternatively, you can write # an IP address and netmask in separate columns to specify the set of hosts. # # METHOD can be "trust", "reject", "md5", "crypt", "password", # "krb5", "ident", or "pam". Note that "password" sends passwords # in clear text; "md5" is preferred since it sends encrypted passwords. # # OPTION is the ident map or the name of the PAM service, depending on METHOD. # # Database and user names containing spaces, commas, quotes and other special # characters must be quoted. Quoting one of the keywords "all", "sameuser" or # "samerole" makes the name lose its special character, and just match a # database or username with that name. # # This file is read on server startup and when the postmaster receives # a SIGHUP signal. If you edit the file on a running system, you have # to SIGHUP the postmaster for the changes to take effect. You can use # "pg_ctl reload" to do that. # Put your actual configuration here # ---------------------------------- # # If you want to allow non-local connections, you need to add more # "host" records. In that case you will also need to make PostgreSQL listen # on a non-local interface via the listen_addresses configuration parameter, # or via the -i or -h command line switches. # # CAUTION: Configuring the system for local "trust" authentication allows # any local user to connect as any PostgreSQL user, including the database # superuser. If you do not trust all your local users, use another # authentication method. # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust skytools-2.1.13/tests/walmgr/conf.master/0000755000175000017500000000000011727601174017351 5ustar markomarkoskytools-2.1.13/tests/walmgr/conf.master/postgresql.conf0000644000175000017500000000073511670174255022432 0ustar markomarko# - Connection Settings - #port = 5432 port = 7200 unix_socket_directory = '/tmp/waltest' archive_mode = on #archive_command = '' # command to use to archive a logfile # segment # These settings are initialized by initdb -- they might be changed lc_messages = 'C' # locale for system error message # strings lc_monetary = 'C' # locale for monetary formatting lc_numeric = 'C' # locale for number formatting lc_time = 'C' # locale for time formatting skytools-2.1.13/tests/walmgr/conf.master/pg_ident.conf0000644000175000017500000000266411670174255022023 0ustar markomarko# PostgreSQL Ident Authentication Maps # ==================================== # # Refer to the PostgreSQL Administrator's Guide, chapter "Client # Authentication" for a complete description. A short synopsis # follows. # # This file controls PostgreSQL ident-based authentication. It maps # ident user names (typically Unix user names) to their corresponding # PostgreSQL user names. Records are of the form: # # MAPNAME IDENT-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. IDENT-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The # existence of a record specifies that IDENT-USERNAME may connect as # PG-USERNAME. Multiple maps may be specified in this file and used # by pg_hba.conf. # # This file is read on server startup and when the postmaster receives # a SIGHUP signal. If you edit the file on a running system, you have # to SIGHUP the postmaster for the changes to take effect. You can use # "pg_ctl reload" to do that. # Put your actual configuration here # ---------------------------------- # # No map names are defined in the default configuration. If all ident # user names and PostgreSQL user names are the same, you don't need # this file. Instead, use the special map name "sameuser" in # pg_hba.conf. # MAPNAME IDENT-USERNAME PG-USERNAME skytools-2.1.13/tests/walmgr/conf.master/pg_hba.conf0000644000175000017500000000650411670174255021447 0ustar markomarko# PostgreSQL Client Authentication Configuration File # =================================================== # # Refer to the PostgreSQL Administrator's Guide, chapter "Client # Authentication" for a complete description. A short synopsis # follows. # # This file controls: which hosts are allowed to connect, how clients # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # # local DATABASE USER METHOD [OPTION] # host DATABASE USER CIDR-ADDRESS METHOD [OPTION] # hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION] # hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION] # # (The uppercase items must be replaced by actual values.) # # The first field is the connection type: "local" is a Unix-domain socket, # "host" is either a plain or SSL-encrypted TCP/IP socket, "hostssl" is an # SSL-encrypted TCP/IP socket, and "hostnossl" is a plain TCP/IP socket. # # DATABASE can be "all", "sameuser", "samerole", a database name, or # a comma-separated list thereof. # # USER can be "all", a user name, a group name prefixed with "+", or # a comma-separated list thereof. In both the DATABASE and USER fields # you can also write a file name prefixed with "@" to include names from # a separate file. # # CIDR-ADDRESS specifies the set of hosts the record matches. # It is made up of an IP address and a CIDR mask that is an integer # (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that specifies # the number of significant bits in the mask. Alternatively, you can write # an IP address and netmask in separate columns to specify the set of hosts. # # METHOD can be "trust", "reject", "md5", "crypt", "password", # "krb5", "ident", or "pam". Note that "password" sends passwords # in clear text; "md5" is preferred since it sends encrypted passwords. # # OPTION is the ident map or the name of the PAM service, depending on METHOD. # # Database and user names containing spaces, commas, quotes and other special # characters must be quoted. Quoting one of the keywords "all", "sameuser" or # "samerole" makes the name lose its special character, and just match a # database or username with that name. # # This file is read on server startup and when the postmaster receives # a SIGHUP signal. If you edit the file on a running system, you have # to SIGHUP the postmaster for the changes to take effect. You can use # "pg_ctl reload" to do that. # Put your actual configuration here # ---------------------------------- # # If you want to allow non-local connections, you need to add more # "host" records. In that case you will also need to make PostgreSQL listen # on a non-local interface via the listen_addresses configuration parameter, # or via the -i or -h command line switches. # # CAUTION: Configuring the system for local "trust" authentication allows # any local user to connect as any PostgreSQL user, including the database # superuser. If you do not trust all your local users, use another # authentication method. # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust skytools-2.1.13/tests/walmgr/run-test.sh0000755000175000017500000000463211670174255017261 0ustar markomarko#! /bin/sh set -e . ../env.sh tmp=/tmp/waltest src=$PWD walmgr=$src/../../python/walmgr.py test -f $tmp/data.master/postmaster.pid \ && kill `head -1 $tmp/data.master/postmaster.pid` || true test -f $tmp/data.slave/postmaster.pid \ && kill `head -1 $tmp/data.slave/postmaster.pid` || true rm -rf $tmp mkdir -p $tmp cd $tmp LANG=C PATH=/usr/lib/postgresql/8.3/bin:$PATH export PATH LANG mkdir log slave slave/logs.complete slave/logs.partial # # Prepare configs # ### wal.master.ini ### cat > wal.master.ini < wal.slave.ini < rc.slave < log/initdb.log 2>&1 cp $src/conf.master/*.conf data.master/ pg_ctl -D data.master -l log/pg.master.log start sleep 4 createdb -h /tmp/waltest -p 7200 echo '####' $walmgr $tmp/wal.master.ini setup $walmgr wal.master.ini setup echo '####' $walmgr $tmp/wal.master.ini backup $walmgr wal.master.ini backup psql -c "create table t as select * from now()" -p 7200 -h /tmp/waltest echo '####' $walmgr $tmp/wal.slave.ini restore $walmgr $tmp/wal.slave.ini restore sleep 10 echo '####' $walmgr $tmp/wal.master.ini sync $walmgr wal.master.ini sync sleep 4 echo '####' $walmgr $tmp/wal.slave.ini boot $walmgr $tmp/wal.slave.ini boot sleep 20 psql -c "select * from t" -p 7201 -h /tmp/waltest pg_ctl -D data.master stop pg_ctl -D data.slave stop skytools-2.1.13/debian/0000755000175000017500000000000011727601174013721 5ustar markomarkoskytools-2.1.13/debian/changelog0000644000175000017500000000555711727573647015624 0ustar markomarkoskytools (2.1.13) unstable; urgency=low * v2.1.13 -- Marko Kreen Tue, 13 Mar 2012 09:30:38 +0200 skytools (2.1.12) unstable; urgency=low * v2.1.12 -- Marko Kreen Wed, 10 Nov 2010 15:23:59 +0200 skytools (2.1.12rc2) unstable; urgency=low * v2.1.12rc2 -- Marko Kreen Tue, 05 Oct 2010 11:43:38 +0300 skytools (2.1.12rc1) unstable; urgency=low * v2.1.12rc1 -- Marko Kreen Tue, 21 Sep 2010 07:26:22 -0700 skytools (2.1.11) unstable; urgency=low * v2.1.11 -- Marko Kreen Wed, 03 Feb 2010 18:28:57 +0200 skytools (2.1.11rc1) unstable; urgency=low * v2.1.11rc1 -- Marko Kreen Fri, 30 Oct 2009 18:05:58 +0200 skytools (2.1.10) unstable; urgency=low * v2.1.10 -- Marko Kreen Mon, 31 Aug 2009 16:44:53 +0300 skytools (2.1.10rc1) unstable; urgency=low * v2.1.10rc1 -- Marko Kreen Mon, 17 Aug 2009 15:52:16 +0300 skytools (2.1.9) unstable; urgency=low * v2.1.9 -- Marko Kreen Fri, 13 Mar 2009 15:39:18 +0200 skytools (2.1.9rc1) unstable; urgency=low * v2.1.9rc1 -- Marko Kreen Thu, 26 Feb 2009 14:50:49 +0200 skytools (2.1.8) unstable; urgency=low * v2.1.8 -- Marko Kreen Sun, 12 Oct 2008 13:29:09 +0300 skytools (2.1.8rc1) unstable; urgency=low * v2.1.8rc1 -- Marko Kreen Mon, 22 Sep 2008 16:31:27 +0300 skytools (2.1.7) unstable; urgency=low * v2.1.7 -- Marko Kreen Wed, 28 May 2008 17:04:32 +0300 skytools (2.1.6) unstable; urgency=low * Final release -- Marko Kreen Sat, 05 Apr 2008 16:45:11 +0300 skytools (2.1.6rc3) unstable; urgency=low * quoting/parsing fixes ? walmgr fix -- Marko Kreen Wed, 12 Mar 2008 15:43:39 +0200 skytools (2.1.6rc2) unstable; urgency=low * Bugfix release. -- Marko Kreen Fri, 07 Dec 2007 16:12:27 +0200 skytools (2.1.5) unstable; urgency=low * New public release. -- Marko Kreen Mon, 19 Nov 2007 15:32:41 +0200 skytools (2.1.4) unstable; urgency=low * Upgrade walmgr, some fixes. -- Marko Kreen Fri, 13 Apr 2007 11:08:41 +0300 skytools (2.1.3) unstable; urgency=low * brown paper bag -- Marko Kreen Tue, 10 Apr 2007 11:55:47 +0300 skytools (2.1.2) unstable; urgency=low * more bugfixes -- Marko Kreen Mon, 09 Apr 2007 17:56:35 +0300 skytools (2.1.1) unstable; urgency=low * bugfixes -- Marko Kreen Tue, 03 Apr 2007 15:03:28 +0300 skytools (2.1) unstable; urgency=low * cleanup -- Marko Kreen Fri, 02 Feb 2007 12:38:17 +0200 skytools-2.1.13/debian/packages.in0000644000175000017500000000324511670174255016035 0ustar markomarko## debian/packages for skytools Source: skytools Section: contrib/misc Priority: extra Maintainer: Marko Kreen Standards-Version: 3.6.2 Description: PostgreSQL Copyright: BSD Copyright 2006 Marko Kreen Build: sh PATH=/usr/lib/postgresql/PGVER/bin:$PATH \ ./configure --prefix=/usr --with-pgconfig=/usr/lib/postgresql/PGVER/bin/pg_config --with-python=pythonPYVER PATH=/usr/lib/postgresql/PGVER/bin:$PATH \ make DESTDIR=$ROOT Clean: sh PATH=/usr/lib/postgresql/PGVER/bin:$PATH \ make distclean || make clean || true Build-Depends: python-dev, postgresql-server-dev-PGVER Package: skytools Architecture: any Depends: python-psycopg2 | pythonPYVER-psycopg2 | python-psycopg | pythonPYVER-psycopg, skytools-modules-9.1 |skytools-modules-9.0 | skytools-modules-8.4 | skytools-modules-8.3 | skytools-modules-8.2 | skytools-modules-8.1 | skytools-modules-8.0, rsync, [] Description: Skype database tools - Python parts . londiste - replication pgqadm - generic event queue walmgr - failover server scripts Install: sh PATH=/usr/lib/postgresql/PGVER/bin:$PATH \ make python-install DESTDIR=$ROOT prefix=/usr `make -s debfix` Package: skytools-modules-PGVER Architecture: any Depends: postgresql-PGVER, [] Conflicts: postgresql-extras-PGVER Description: Extra modules for PostgreSQL It includes various extra modules for PostgreSQL: . txid - 8-byte transaction id's logtriga - Trigger function to log change in SQL format. logutriga - Trigger function to log change in urlencoded format. londiste - Database parts of replication engine. pgq - Generic queue in database. Install: sh PATH=/usr/lib/postgresql/PGVER/bin:$PATH \ make modules-install DESTDIR=$ROOT skytools-2.1.13/configure0000755000175000017500000034655111727404462014425 0ustar markomarko#! /bin/sh # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.67 for skytools 2.1.13. # # # Copyright (C) 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001, # 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 Free Software # Foundation, Inc. # # # This configure script is free software; the Free Software Foundation # gives unlimited permission to copy, distribute and modify it. ## -------------------- ## ## M4sh Initialization. ## ## -------------------- ## # Be more Bourne compatible DUALCASE=1; export DUALCASE # for MKS sh if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then : emulate sh NULLCMD=: # Pre-4.2 versions of Zsh do word splitting on ${1+"$@"}, which # is contrary to our usage. Disable this feature. alias -g '${1+"$@"}'='"$@"' setopt NO_GLOB_SUBST else case `(set -o) 2>/dev/null` in #( *posix*) : set -o posix ;; #( *) : ;; esac fi as_nl=' ' export as_nl # Printing a long string crashes Solaris 7 /usr/bin/printf. as_echo='\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo$as_echo # Prefer a ksh shell builtin over an external printf program on Solaris, # but without wasting forks for bash or zsh. if test -z "$BASH_VERSION$ZSH_VERSION" \ && (test "X`print -r -- $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='print -r --' as_echo_n='print -rn --' elif (test "X`printf %s $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='printf %s\n' as_echo_n='printf %s' else if test "X`(/usr/ucb/echo -n -n $as_echo) 2>/dev/null`" = "X-n $as_echo"; then as_echo_body='eval /usr/ucb/echo -n "$1$as_nl"' as_echo_n='/usr/ucb/echo -n' else as_echo_body='eval expr "X$1" : "X\\(.*\\)"' as_echo_n_body='eval arg=$1; case $arg in #( *"$as_nl"*) expr "X$arg" : "X\\(.*\\)$as_nl"; arg=`expr "X$arg" : ".*$as_nl\\(.*\\)"`;; esac; expr "X$arg" : "X\\(.*\\)" | tr -d "$as_nl" ' export as_echo_n_body as_echo_n='sh -c $as_echo_n_body as_echo' fi export as_echo_body as_echo='sh -c $as_echo_body as_echo' fi # The user is always right. if test "${PATH_SEPARATOR+set}" != set; then PATH_SEPARATOR=: (PATH='/bin;/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 && { (PATH='/bin:/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 || PATH_SEPARATOR=';' } fi # IFS # We need space, tab and new line, in precisely that order. Quoting is # there to prevent editors from complaining about space-tab. # (If _AS_PATH_WALK were called with IFS unset, it would disable word # splitting by setting IFS to empty value.) IFS=" "" $as_nl" # Find who we are. Look in the path if we contain no directory separator. case $0 in #(( *[\\/]* ) as_myself=$0 ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. test -r "$as_dir/$0" && as_myself=$as_dir/$0 && break done IFS=$as_save_IFS ;; esac # We did not find ourselves, most probably we were run as `sh COMMAND' # in which case we are not to be found in the path. if test "x$as_myself" = x; then as_myself=$0 fi if test ! -f "$as_myself"; then $as_echo "$as_myself: error: cannot find myself; rerun with an absolute file name" >&2 exit 1 fi # Unset variables that we do not need and which cause bugs (e.g. in # pre-3.0 UWIN ksh). But do not cause bugs in bash 2.01; the "|| exit 1" # suppresses any "Segmentation fault" message there. '((' could # trigger a bug in pdksh 5.2.14. for as_var in BASH_ENV ENV MAIL MAILPATH do eval test x\${$as_var+set} = xset \ && ( (unset $as_var) || exit 1) >/dev/null 2>&1 && unset $as_var || : done PS1='$ ' PS2='> ' PS4='+ ' # NLS nuisances. LC_ALL=C export LC_ALL LANGUAGE=C export LANGUAGE # CDPATH. (unset CDPATH) >/dev/null 2>&1 && unset CDPATH if test "x$CONFIG_SHELL" = x; then as_bourne_compatible="if test -n \"\${ZSH_VERSION+set}\" && (emulate sh) >/dev/null 2>&1; then : emulate sh NULLCMD=: # Pre-4.2 versions of Zsh do word splitting on \${1+\"\$@\"}, which # is contrary to our usage. Disable this feature. alias -g '\${1+\"\$@\"}'='\"\$@\"' setopt NO_GLOB_SUBST else case \`(set -o) 2>/dev/null\` in #( *posix*) : set -o posix ;; #( *) : ;; esac fi " as_required="as_fn_return () { (exit \$1); } as_fn_success () { as_fn_return 0; } as_fn_failure () { as_fn_return 1; } as_fn_ret_success () { return 0; } as_fn_ret_failure () { return 1; } exitcode=0 as_fn_success || { exitcode=1; echo as_fn_success failed.; } as_fn_failure && { exitcode=1; echo as_fn_failure succeeded.; } as_fn_ret_success || { exitcode=1; echo as_fn_ret_success failed.; } as_fn_ret_failure && { exitcode=1; echo as_fn_ret_failure succeeded.; } if ( set x; as_fn_ret_success y && test x = \"\$1\" ); then : else exitcode=1; echo positional parameters were not saved. fi test x\$exitcode = x0 || exit 1" as_suggested=" as_lineno_1=";as_suggested=$as_suggested$LINENO;as_suggested=$as_suggested" as_lineno_1a=\$LINENO as_lineno_2=";as_suggested=$as_suggested$LINENO;as_suggested=$as_suggested" as_lineno_2a=\$LINENO eval 'test \"x\$as_lineno_1'\$as_run'\" != \"x\$as_lineno_2'\$as_run'\" && test \"x\`expr \$as_lineno_1'\$as_run' + 1\`\" = \"x\$as_lineno_2'\$as_run'\"' || exit 1" if (eval "$as_required") 2>/dev/null; then : as_have_required=yes else as_have_required=no fi if test x$as_have_required = xyes && (eval "$as_suggested") 2>/dev/null; then : else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR as_found=false for as_dir in /bin$PATH_SEPARATOR/usr/bin$PATH_SEPARATOR$PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. as_found=: case $as_dir in #( /*) for as_base in sh bash ksh sh5; do # Try only shells that exist, to save several forks. as_shell=$as_dir/$as_base if { test -f "$as_shell" || test -f "$as_shell.exe"; } && { $as_echo "$as_bourne_compatible""$as_required" | as_run=a "$as_shell"; } 2>/dev/null; then : CONFIG_SHELL=$as_shell as_have_required=yes if { $as_echo "$as_bourne_compatible""$as_suggested" | as_run=a "$as_shell"; } 2>/dev/null; then : break 2 fi fi done;; esac as_found=false done $as_found || { if { test -f "$SHELL" || test -f "$SHELL.exe"; } && { $as_echo "$as_bourne_compatible""$as_required" | as_run=a "$SHELL"; } 2>/dev/null; then : CONFIG_SHELL=$SHELL as_have_required=yes fi; } IFS=$as_save_IFS if test "x$CONFIG_SHELL" != x; then : # We cannot yet assume a decent shell, so we have to provide a # neutralization value for shells without unset; and this also # works around shells that cannot unset nonexistent variables. BASH_ENV=/dev/null ENV=/dev/null (unset BASH_ENV) >/dev/null 2>&1 && unset BASH_ENV ENV export CONFIG_SHELL exec "$CONFIG_SHELL" "$as_myself" ${1+"$@"} fi if test x$as_have_required = xno; then : $as_echo "$0: This script requires a shell more modern than all" $as_echo "$0: the shells that I found on your system." if test x${ZSH_VERSION+set} = xset ; then $as_echo "$0: In particular, zsh $ZSH_VERSION has bugs and should" $as_echo "$0: be upgraded to zsh 4.3.4 or later." else $as_echo "$0: Please tell bug-autoconf@gnu.org about your system, $0: including any error possibly output before this $0: message. Then install a modern shell, or manually run $0: the script under such a shell if you do have one." fi exit 1 fi fi fi SHELL=${CONFIG_SHELL-/bin/sh} export SHELL # Unset more variables known to interfere with behavior of common tools. CLICOLOR_FORCE= GREP_OPTIONS= unset CLICOLOR_FORCE GREP_OPTIONS ## --------------------- ## ## M4sh Shell Functions. ## ## --------------------- ## # as_fn_unset VAR # --------------- # Portably unset VAR. as_fn_unset () { { eval $1=; unset $1;} } as_unset=as_fn_unset # as_fn_set_status STATUS # ----------------------- # Set $? to STATUS, without forking. as_fn_set_status () { return $1 } # as_fn_set_status # as_fn_exit STATUS # ----------------- # Exit the shell with STATUS, even in a "trap 0" or "set -e" context. as_fn_exit () { set +e as_fn_set_status $1 exit $1 } # as_fn_exit # as_fn_mkdir_p # ------------- # Create "$as_dir" as a directory, including parents if necessary. as_fn_mkdir_p () { case $as_dir in #( -*) as_dir=./$as_dir;; esac test -d "$as_dir" || eval $as_mkdir_p || { as_dirs= while :; do case $as_dir in #( *\'*) as_qdir=`$as_echo "$as_dir" | sed "s/'/'\\\\\\\\''/g"`;; #'( *) as_qdir=$as_dir;; esac as_dirs="'$as_qdir' $as_dirs" as_dir=`$as_dirname -- "$as_dir" || $as_expr X"$as_dir" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$as_dir" : 'X\(//\)[^/]' \| \ X"$as_dir" : 'X\(//\)$' \| \ X"$as_dir" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$as_dir" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` test -d "$as_dir" && break done test -z "$as_dirs" || eval "mkdir $as_dirs" } || test -d "$as_dir" || as_fn_error $? "cannot create directory $as_dir" } # as_fn_mkdir_p # as_fn_append VAR VALUE # ---------------------- # Append the text in VALUE to the end of the definition contained in VAR. Take # advantage of any shell optimizations that allow amortized linear growth over # repeated appends, instead of the typical quadratic growth present in naive # implementations. if (eval "as_var=1; as_var+=2; test x\$as_var = x12") 2>/dev/null; then : eval 'as_fn_append () { eval $1+=\$2 }' else as_fn_append () { eval $1=\$$1\$2 } fi # as_fn_append # as_fn_arith ARG... # ------------------ # Perform arithmetic evaluation on the ARGs, and store the result in the # global $as_val. Take advantage of shells that can avoid forks. The arguments # must be portable across $(()) and expr. if (eval "test \$(( 1 + 1 )) = 2") 2>/dev/null; then : eval 'as_fn_arith () { as_val=$(( $* )) }' else as_fn_arith () { as_val=`expr "$@" || test $? -eq 1` } fi # as_fn_arith # as_fn_error STATUS ERROR [LINENO LOG_FD] # ---------------------------------------- # Output "`basename $0`: error: ERROR" to stderr. If LINENO and LOG_FD are # provided, also output the error to LOG_FD, referencing LINENO. Then exit the # script with STATUS, using 1 if that was 0. as_fn_error () { as_status=$1; test $as_status -eq 0 && as_status=1 if test "$4"; then as_lineno=${as_lineno-"$3"} as_lineno_stack=as_lineno_stack=$as_lineno_stack $as_echo "$as_me:${as_lineno-$LINENO}: error: $2" >&$4 fi $as_echo "$as_me: error: $2" >&2 as_fn_exit $as_status } # as_fn_error if expr a : '\(a\)' >/dev/null 2>&1 && test "X`expr 00001 : '.*\(...\)'`" = X001; then as_expr=expr else as_expr=false fi if (basename -- /) >/dev/null 2>&1 && test "X`basename -- / 2>&1`" = "X/"; then as_basename=basename else as_basename=false fi if (as_dir=`dirname -- /` && test "X$as_dir" = X/) >/dev/null 2>&1; then as_dirname=dirname else as_dirname=false fi as_me=`$as_basename -- "$0" || $as_expr X/"$0" : '.*/\([^/][^/]*\)/*$' \| \ X"$0" : 'X\(//\)$' \| \ X"$0" : 'X\(/\)' \| . 2>/dev/null || $as_echo X/"$0" | sed '/^.*\/\([^/][^/]*\)\/*$/{ s//\1/ q } /^X\/\(\/\/\)$/{ s//\1/ q } /^X\/\(\/\).*/{ s//\1/ q } s/.*/./; q'` # Avoid depending upon Character Ranges. as_cr_letters='abcdefghijklmnopqrstuvwxyz' as_cr_LETTERS='ABCDEFGHIJKLMNOPQRSTUVWXYZ' as_cr_Letters=$as_cr_letters$as_cr_LETTERS as_cr_digits='0123456789' as_cr_alnum=$as_cr_Letters$as_cr_digits as_lineno_1=$LINENO as_lineno_1a=$LINENO as_lineno_2=$LINENO as_lineno_2a=$LINENO eval 'test "x$as_lineno_1'$as_run'" != "x$as_lineno_2'$as_run'" && test "x`expr $as_lineno_1'$as_run' + 1`" = "x$as_lineno_2'$as_run'"' || { # Blame Lee E. McMahon (1931-1989) for sed's syntax. :-) sed -n ' p /[$]LINENO/= ' <$as_myself | sed ' s/[$]LINENO.*/&-/ t lineno b :lineno N :loop s/[$]LINENO\([^'$as_cr_alnum'_].*\n\)\(.*\)/\2\1\2/ t loop s/-\n.*// ' >$as_me.lineno && chmod +x "$as_me.lineno" || { $as_echo "$as_me: error: cannot create $as_me.lineno; rerun with a POSIX shell" >&2; as_fn_exit 1; } # Don't try to exec as it changes $[0], causing all sort of problems # (the dirname of $[0] is not the place where we might find the # original and so on. Autoconf is especially sensitive to this). . "./$as_me.lineno" # Exit status is that of the last command. exit } ECHO_C= ECHO_N= ECHO_T= case `echo -n x` in #((((( -n*) case `echo 'xy\c'` in *c*) ECHO_T=' ';; # ECHO_T is single tab character. xy) ECHO_C='\c';; *) echo `echo ksh88 bug on AIX 6.1` > /dev/null ECHO_T=' ';; esac;; *) ECHO_N='-n';; esac rm -f conf$$ conf$$.exe conf$$.file if test -d conf$$.dir; then rm -f conf$$.dir/conf$$.file else rm -f conf$$.dir mkdir conf$$.dir 2>/dev/null fi if (echo >conf$$.file) 2>/dev/null; then if ln -s conf$$.file conf$$ 2>/dev/null; then as_ln_s='ln -s' # ... but there are two gotchas: # 1) On MSYS, both `ln -s file dir' and `ln file dir' fail. # 2) DJGPP < 2.04 has no symlinks; `ln -s' creates a wrapper executable. # In both cases, we have to default to `cp -p'. ln -s conf$$.file conf$$.dir 2>/dev/null && test ! -f conf$$.exe || as_ln_s='cp -p' elif ln conf$$.file conf$$ 2>/dev/null; then as_ln_s=ln else as_ln_s='cp -p' fi else as_ln_s='cp -p' fi rm -f conf$$ conf$$.exe conf$$.dir/conf$$.file conf$$.file rmdir conf$$.dir 2>/dev/null if mkdir -p . 2>/dev/null; then as_mkdir_p='mkdir -p "$as_dir"' else test -d ./-p && rmdir ./-p as_mkdir_p=false fi if test -x / >/dev/null 2>&1; then as_test_x='test -x' else if ls -dL / >/dev/null 2>&1; then as_ls_L_option=L else as_ls_L_option= fi as_test_x=' eval sh -c '\'' if test -d "$1"; then test -d "$1/."; else case $1 in #( -*)set "./$1";; esac; case `ls -ld'$as_ls_L_option' "$1" 2>/dev/null` in #(( ???[sx]*):;;*)false;;esac;fi '\'' sh ' fi as_executable_p=$as_test_x # Sed expression to map a string onto a valid CPP name. as_tr_cpp="eval sed 'y%*$as_cr_letters%P$as_cr_LETTERS%;s%[^_$as_cr_alnum]%_%g'" # Sed expression to map a string onto a valid variable name. as_tr_sh="eval sed 'y%*+%pp%;s%[^_$as_cr_alnum]%_%g'" test -n "$DJDIR" || exec 7<&0 &1 # Name of the host. # hostname on some systems (SVR3.2, old GNU/Linux) returns a bogus exit status, # so uname gets run too. ac_hostname=`(hostname || uname -n) 2>/dev/null | sed 1q` # # Initializations. # ac_default_prefix=/usr/local ac_clean_files= ac_config_libobj_dir=. LIBOBJS= cross_compiling=no subdirs= MFLAGS= MAKEFLAGS= # Identity of this package. PACKAGE_NAME='skytools' PACKAGE_TARNAME='skytools' PACKAGE_VERSION='2.1.13' PACKAGE_STRING='skytools 2.1.13' PACKAGE_BUGREPORT='' PACKAGE_URL='' ac_unique_file="python/pgqadm.py" ac_subst_vars='LTLIBOBJS LIBOBJS OBJEXT EXEEXT ac_ct_CC CPPFLAGS LDFLAGS CFLAGS CC XMLTO ASCIIDOC MAKE PG_CONFIG PYTHON target_alias host_alias build_alias LIBS ECHO_T ECHO_N ECHO_C DEFS mandir localedir libdir psdir pdfdir dvidir htmldir infodir docdir oldincludedir includedir localstatedir sharedstatedir sysconfdir datadir datarootdir libexecdir sbindir bindir program_transform_name prefix exec_prefix PACKAGE_URL PACKAGE_BUGREPORT PACKAGE_STRING PACKAGE_VERSION PACKAGE_TARNAME PACKAGE_NAME PATH_SEPARATOR SHELL' ac_subst_files='' ac_user_opts=' enable_option_checking with_python with_pgconfig with_asciidoc ' ac_precious_vars='build_alias host_alias target_alias CC CFLAGS LDFLAGS LIBS CPPFLAGS' # Initialize some variables set by options. ac_init_help= ac_init_version=false ac_unrecognized_opts= ac_unrecognized_sep= # The variables have the same names as the options, with # dashes changed to underlines. cache_file=/dev/null exec_prefix=NONE no_create= no_recursion= prefix=NONE program_prefix=NONE program_suffix=NONE program_transform_name=s,x,x, silent= site= srcdir= verbose= x_includes=NONE x_libraries=NONE # Installation directory options. # These are left unexpanded so users can "make install exec_prefix=/foo" # and all the variables that are supposed to be based on exec_prefix # by default will actually change. # Use braces instead of parens because sh, perl, etc. also accept them. # (The list follows the same order as the GNU Coding Standards.) bindir='${exec_prefix}/bin' sbindir='${exec_prefix}/sbin' libexecdir='${exec_prefix}/libexec' datarootdir='${prefix}/share' datadir='${datarootdir}' sysconfdir='${prefix}/etc' sharedstatedir='${prefix}/com' localstatedir='${prefix}/var' includedir='${prefix}/include' oldincludedir='/usr/include' docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' infodir='${datarootdir}/info' htmldir='${docdir}' dvidir='${docdir}' pdfdir='${docdir}' psdir='${docdir}' libdir='${exec_prefix}/lib' localedir='${datarootdir}/locale' mandir='${datarootdir}/man' ac_prev= ac_dashdash= for ac_option do # If the previous option needs an argument, assign it. if test -n "$ac_prev"; then eval $ac_prev=\$ac_option ac_prev= continue fi case $ac_option in *=?*) ac_optarg=`expr "X$ac_option" : '[^=]*=\(.*\)'` ;; *=) ac_optarg= ;; *) ac_optarg=yes ;; esac # Accept the important Cygnus configure options, so we can diagnose typos. case $ac_dashdash$ac_option in --) ac_dashdash=yes ;; -bindir | --bindir | --bindi | --bind | --bin | --bi) ac_prev=bindir ;; -bindir=* | --bindir=* | --bindi=* | --bind=* | --bin=* | --bi=*) bindir=$ac_optarg ;; -build | --build | --buil | --bui | --bu) ac_prev=build_alias ;; -build=* | --build=* | --buil=* | --bui=* | --bu=*) build_alias=$ac_optarg ;; -cache-file | --cache-file | --cache-fil | --cache-fi \ | --cache-f | --cache- | --cache | --cach | --cac | --ca | --c) ac_prev=cache_file ;; -cache-file=* | --cache-file=* | --cache-fil=* | --cache-fi=* \ | --cache-f=* | --cache-=* | --cache=* | --cach=* | --cac=* | --ca=* | --c=*) cache_file=$ac_optarg ;; --config-cache | -C) cache_file=config.cache ;; -datadir | --datadir | --datadi | --datad) ac_prev=datadir ;; -datadir=* | --datadir=* | --datadi=* | --datad=*) datadir=$ac_optarg ;; -datarootdir | --datarootdir | --datarootdi | --datarootd | --dataroot \ | --dataroo | --dataro | --datar) ac_prev=datarootdir ;; -datarootdir=* | --datarootdir=* | --datarootdi=* | --datarootd=* \ | --dataroot=* | --dataroo=* | --dataro=* | --datar=*) datarootdir=$ac_optarg ;; -disable-* | --disable-*) ac_useropt=`expr "x$ac_option" : 'x-*disable-\(.*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid feature name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "enable_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--disable-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval enable_$ac_useropt=no ;; -docdir | --docdir | --docdi | --doc | --do) ac_prev=docdir ;; -docdir=* | --docdir=* | --docdi=* | --doc=* | --do=*) docdir=$ac_optarg ;; -dvidir | --dvidir | --dvidi | --dvid | --dvi | --dv) ac_prev=dvidir ;; -dvidir=* | --dvidir=* | --dvidi=* | --dvid=* | --dvi=* | --dv=*) dvidir=$ac_optarg ;; -enable-* | --enable-*) ac_useropt=`expr "x$ac_option" : 'x-*enable-\([^=]*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid feature name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "enable_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--enable-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval enable_$ac_useropt=\$ac_optarg ;; -exec-prefix | --exec_prefix | --exec-prefix | --exec-prefi \ | --exec-pref | --exec-pre | --exec-pr | --exec-p | --exec- \ | --exec | --exe | --ex) ac_prev=exec_prefix ;; -exec-prefix=* | --exec_prefix=* | --exec-prefix=* | --exec-prefi=* \ | --exec-pref=* | --exec-pre=* | --exec-pr=* | --exec-p=* | --exec-=* \ | --exec=* | --exe=* | --ex=*) exec_prefix=$ac_optarg ;; -gas | --gas | --ga | --g) # Obsolete; use --with-gas. with_gas=yes ;; -help | --help | --hel | --he | -h) ac_init_help=long ;; -help=r* | --help=r* | --hel=r* | --he=r* | -hr*) ac_init_help=recursive ;; -help=s* | --help=s* | --hel=s* | --he=s* | -hs*) ac_init_help=short ;; -host | --host | --hos | --ho) ac_prev=host_alias ;; -host=* | --host=* | --hos=* | --ho=*) host_alias=$ac_optarg ;; -htmldir | --htmldir | --htmldi | --htmld | --html | --htm | --ht) ac_prev=htmldir ;; -htmldir=* | --htmldir=* | --htmldi=* | --htmld=* | --html=* | --htm=* \ | --ht=*) htmldir=$ac_optarg ;; -includedir | --includedir | --includedi | --included | --include \ | --includ | --inclu | --incl | --inc) ac_prev=includedir ;; -includedir=* | --includedir=* | --includedi=* | --included=* | --include=* \ | --includ=* | --inclu=* | --incl=* | --inc=*) includedir=$ac_optarg ;; -infodir | --infodir | --infodi | --infod | --info | --inf) ac_prev=infodir ;; -infodir=* | --infodir=* | --infodi=* | --infod=* | --info=* | --inf=*) infodir=$ac_optarg ;; -libdir | --libdir | --libdi | --libd) ac_prev=libdir ;; -libdir=* | --libdir=* | --libdi=* | --libd=*) libdir=$ac_optarg ;; -libexecdir | --libexecdir | --libexecdi | --libexecd | --libexec \ | --libexe | --libex | --libe) ac_prev=libexecdir ;; -libexecdir=* | --libexecdir=* | --libexecdi=* | --libexecd=* | --libexec=* \ | --libexe=* | --libex=* | --libe=*) libexecdir=$ac_optarg ;; -localedir | --localedir | --localedi | --localed | --locale) ac_prev=localedir ;; -localedir=* | --localedir=* | --localedi=* | --localed=* | --locale=*) localedir=$ac_optarg ;; -localstatedir | --localstatedir | --localstatedi | --localstated \ | --localstate | --localstat | --localsta | --localst | --locals) ac_prev=localstatedir ;; -localstatedir=* | --localstatedir=* | --localstatedi=* | --localstated=* \ | --localstate=* | --localstat=* | --localsta=* | --localst=* | --locals=*) localstatedir=$ac_optarg ;; -mandir | --mandir | --mandi | --mand | --man | --ma | --m) ac_prev=mandir ;; -mandir=* | --mandir=* | --mandi=* | --mand=* | --man=* | --ma=* | --m=*) mandir=$ac_optarg ;; -nfp | --nfp | --nf) # Obsolete; use --without-fp. with_fp=no ;; -no-create | --no-create | --no-creat | --no-crea | --no-cre \ | --no-cr | --no-c | -n) no_create=yes ;; -no-recursion | --no-recursion | --no-recursio | --no-recursi \ | --no-recurs | --no-recur | --no-recu | --no-rec | --no-re | --no-r) no_recursion=yes ;; -oldincludedir | --oldincludedir | --oldincludedi | --oldincluded \ | --oldinclude | --oldinclud | --oldinclu | --oldincl | --oldinc \ | --oldin | --oldi | --old | --ol | --o) ac_prev=oldincludedir ;; -oldincludedir=* | --oldincludedir=* | --oldincludedi=* | --oldincluded=* \ | --oldinclude=* | --oldinclud=* | --oldinclu=* | --oldincl=* | --oldinc=* \ | --oldin=* | --oldi=* | --old=* | --ol=* | --o=*) oldincludedir=$ac_optarg ;; -prefix | --prefix | --prefi | --pref | --pre | --pr | --p) ac_prev=prefix ;; -prefix=* | --prefix=* | --prefi=* | --pref=* | --pre=* | --pr=* | --p=*) prefix=$ac_optarg ;; -program-prefix | --program-prefix | --program-prefi | --program-pref \ | --program-pre | --program-pr | --program-p) ac_prev=program_prefix ;; -program-prefix=* | --program-prefix=* | --program-prefi=* \ | --program-pref=* | --program-pre=* | --program-pr=* | --program-p=*) program_prefix=$ac_optarg ;; -program-suffix | --program-suffix | --program-suffi | --program-suff \ | --program-suf | --program-su | --program-s) ac_prev=program_suffix ;; -program-suffix=* | --program-suffix=* | --program-suffi=* \ | --program-suff=* | --program-suf=* | --program-su=* | --program-s=*) program_suffix=$ac_optarg ;; -program-transform-name | --program-transform-name \ | --program-transform-nam | --program-transform-na \ | --program-transform-n | --program-transform- \ | --program-transform | --program-transfor \ | --program-transfo | --program-transf \ | --program-trans | --program-tran \ | --progr-tra | --program-tr | --program-t) ac_prev=program_transform_name ;; -program-transform-name=* | --program-transform-name=* \ | --program-transform-nam=* | --program-transform-na=* \ | --program-transform-n=* | --program-transform-=* \ | --program-transform=* | --program-transfor=* \ | --program-transfo=* | --program-transf=* \ | --program-trans=* | --program-tran=* \ | --progr-tra=* | --program-tr=* | --program-t=*) program_transform_name=$ac_optarg ;; -pdfdir | --pdfdir | --pdfdi | --pdfd | --pdf | --pd) ac_prev=pdfdir ;; -pdfdir=* | --pdfdir=* | --pdfdi=* | --pdfd=* | --pdf=* | --pd=*) pdfdir=$ac_optarg ;; -psdir | --psdir | --psdi | --psd | --ps) ac_prev=psdir ;; -psdir=* | --psdir=* | --psdi=* | --psd=* | --ps=*) psdir=$ac_optarg ;; -q | -quiet | --quiet | --quie | --qui | --qu | --q \ | -silent | --silent | --silen | --sile | --sil) silent=yes ;; -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb) ac_prev=sbindir ;; -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \ | --sbi=* | --sb=*) sbindir=$ac_optarg ;; -sharedstatedir | --sharedstatedir | --sharedstatedi \ | --sharedstated | --sharedstate | --sharedstat | --sharedsta \ | --sharedst | --shareds | --shared | --share | --shar \ | --sha | --sh) ac_prev=sharedstatedir ;; -sharedstatedir=* | --sharedstatedir=* | --sharedstatedi=* \ | --sharedstated=* | --sharedstate=* | --sharedstat=* | --sharedsta=* \ | --sharedst=* | --shareds=* | --shared=* | --share=* | --shar=* \ | --sha=* | --sh=*) sharedstatedir=$ac_optarg ;; -site | --site | --sit) ac_prev=site ;; -site=* | --site=* | --sit=*) site=$ac_optarg ;; -srcdir | --srcdir | --srcdi | --srcd | --src | --sr) ac_prev=srcdir ;; -srcdir=* | --srcdir=* | --srcdi=* | --srcd=* | --src=* | --sr=*) srcdir=$ac_optarg ;; -sysconfdir | --sysconfdir | --sysconfdi | --sysconfd | --sysconf \ | --syscon | --sysco | --sysc | --sys | --sy) ac_prev=sysconfdir ;; -sysconfdir=* | --sysconfdir=* | --sysconfdi=* | --sysconfd=* | --sysconf=* \ | --syscon=* | --sysco=* | --sysc=* | --sys=* | --sy=*) sysconfdir=$ac_optarg ;; -target | --target | --targe | --targ | --tar | --ta | --t) ac_prev=target_alias ;; -target=* | --target=* | --targe=* | --targ=* | --tar=* | --ta=* | --t=*) target_alias=$ac_optarg ;; -v | -verbose | --verbose | --verbos | --verbo | --verb) verbose=yes ;; -version | --version | --versio | --versi | --vers | -V) ac_init_version=: ;; -with-* | --with-*) ac_useropt=`expr "x$ac_option" : 'x-*with-\([^=]*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid package name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "with_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--with-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval with_$ac_useropt=\$ac_optarg ;; -without-* | --without-*) ac_useropt=`expr "x$ac_option" : 'x-*without-\(.*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid package name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "with_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--without-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval with_$ac_useropt=no ;; --x) # Obsolete; use --with-x. with_x=yes ;; -x-includes | --x-includes | --x-include | --x-includ | --x-inclu \ | --x-incl | --x-inc | --x-in | --x-i) ac_prev=x_includes ;; -x-includes=* | --x-includes=* | --x-include=* | --x-includ=* | --x-inclu=* \ | --x-incl=* | --x-inc=* | --x-in=* | --x-i=*) x_includes=$ac_optarg ;; -x-libraries | --x-libraries | --x-librarie | --x-librari \ | --x-librar | --x-libra | --x-libr | --x-lib | --x-li | --x-l) ac_prev=x_libraries ;; -x-libraries=* | --x-libraries=* | --x-librarie=* | --x-librari=* \ | --x-librar=* | --x-libra=* | --x-libr=* | --x-lib=* | --x-li=* | --x-l=*) x_libraries=$ac_optarg ;; -*) as_fn_error $? "unrecognized option: \`$ac_option' Try \`$0 --help' for more information" ;; *=*) ac_envvar=`expr "x$ac_option" : 'x\([^=]*\)='` # Reject names that are not valid shell variable names. case $ac_envvar in #( '' | [0-9]* | *[!_$as_cr_alnum]* ) as_fn_error $? "invalid variable name: \`$ac_envvar'" ;; esac eval $ac_envvar=\$ac_optarg export $ac_envvar ;; *) # FIXME: should be removed in autoconf 3.0. $as_echo "$as_me: WARNING: you should use --build, --host, --target" >&2 expr "x$ac_option" : ".*[^-._$as_cr_alnum]" >/dev/null && $as_echo "$as_me: WARNING: invalid host type: $ac_option" >&2 : ${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option} ;; esac done if test -n "$ac_prev"; then ac_option=--`echo $ac_prev | sed 's/_/-/g'` as_fn_error $? "missing argument to $ac_option" fi if test -n "$ac_unrecognized_opts"; then case $enable_option_checking in no) ;; fatal) as_fn_error $? "unrecognized options: $ac_unrecognized_opts" ;; *) $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2 ;; esac fi # Check all directory arguments for consistency. for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \ datadir sysconfdir sharedstatedir localstatedir includedir \ oldincludedir docdir infodir htmldir dvidir pdfdir psdir \ libdir localedir mandir do eval ac_val=\$$ac_var # Remove trailing slashes. case $ac_val in */ ) ac_val=`expr "X$ac_val" : 'X\(.*[^/]\)' \| "X$ac_val" : 'X\(.*\)'` eval $ac_var=\$ac_val;; esac # Be sure to have absolute directory names. case $ac_val in [\\/$]* | ?:[\\/]* ) continue;; NONE | '' ) case $ac_var in *prefix ) continue;; esac;; esac as_fn_error $? "expected an absolute directory name for --$ac_var: $ac_val" done # There might be people who depend on the old broken behavior: `$host' # used to hold the argument of --host etc. # FIXME: To remove some day. build=$build_alias host=$host_alias target=$target_alias # FIXME: To remove some day. if test "x$host_alias" != x; then if test "x$build_alias" = x; then cross_compiling=maybe $as_echo "$as_me: WARNING: if you wanted to set the --build type, don't use --host. If a cross compiler is detected then cross compile mode will be used" >&2 elif test "x$build_alias" != "x$host_alias"; then cross_compiling=yes fi fi ac_tool_prefix= test -n "$host_alias" && ac_tool_prefix=$host_alias- test "$silent" = yes && exec 6>/dev/null ac_pwd=`pwd` && test -n "$ac_pwd" && ac_ls_di=`ls -di .` && ac_pwd_ls_di=`cd "$ac_pwd" && ls -di .` || as_fn_error $? "working directory cannot be determined" test "X$ac_ls_di" = "X$ac_pwd_ls_di" || as_fn_error $? "pwd does not report name of working directory" # Find the source files, if location was not specified. if test -z "$srcdir"; then ac_srcdir_defaulted=yes # Try the directory containing this script, then the parent directory. ac_confdir=`$as_dirname -- "$as_myself" || $as_expr X"$as_myself" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$as_myself" : 'X\(//\)[^/]' \| \ X"$as_myself" : 'X\(//\)$' \| \ X"$as_myself" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$as_myself" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` srcdir=$ac_confdir if test ! -r "$srcdir/$ac_unique_file"; then srcdir=.. fi else ac_srcdir_defaulted=no fi if test ! -r "$srcdir/$ac_unique_file"; then test "$ac_srcdir_defaulted" = yes && srcdir="$ac_confdir or .." as_fn_error $? "cannot find sources ($ac_unique_file) in $srcdir" fi ac_msg="sources are in $srcdir, but \`cd $srcdir' does not work" ac_abs_confdir=`( cd "$srcdir" && test -r "./$ac_unique_file" || as_fn_error $? "$ac_msg" pwd)` # When building in place, set srcdir=. if test "$ac_abs_confdir" = "$ac_pwd"; then srcdir=. fi # Remove unnecessary trailing slashes from srcdir. # Double slashes in file names in object file debugging info # mess up M-x gdb in Emacs. case $srcdir in */) srcdir=`expr "X$srcdir" : 'X\(.*[^/]\)' \| "X$srcdir" : 'X\(.*\)'`;; esac for ac_var in $ac_precious_vars; do eval ac_env_${ac_var}_set=\${${ac_var}+set} eval ac_env_${ac_var}_value=\$${ac_var} eval ac_cv_env_${ac_var}_set=\${${ac_var}+set} eval ac_cv_env_${ac_var}_value=\$${ac_var} done # # Report the --help message. # if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF \`configure' configures skytools 2.1.13 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... To assign environment variables (e.g., CC, CFLAGS...), specify them as VAR=VALUE. See below for descriptions of some of the useful variables. Defaults for the options are specified in brackets. Configuration: -h, --help display this help and exit --help=short display options specific to this package --help=recursive display the short help of all the included packages -V, --version display version information and exit -q, --quiet, --silent do not print \`checking ...' messages --cache-file=FILE cache test results in FILE [disabled] -C, --config-cache alias for \`--cache-file=config.cache' -n, --no-create do not create output files --srcdir=DIR find the sources in DIR [configure dir or \`..'] Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [$ac_default_prefix] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX] By default, \`make install' will install all the files in \`$ac_default_prefix/bin', \`$ac_default_prefix/lib' etc. You can specify an installation prefix other than \`$ac_default_prefix' using \`--prefix', for instance \`--prefix=\$HOME'. For better control, use the options below. Fine tuning of the installation directories: --bindir=DIR user executables [EPREFIX/bin] --sbindir=DIR system admin executables [EPREFIX/sbin] --libexecdir=DIR program executables [EPREFIX/libexec] --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR] --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] --docdir=DIR documentation root [DATAROOTDIR/doc/skytools] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] --psdir=DIR ps documentation [DOCDIR] _ACEOF cat <<\_ACEOF _ACEOF fi if test -n "$ac_init_help"; then case $ac_init_help in short | recursive ) echo "Configuration of skytools 2.1.13:";; esac cat <<\_ACEOF Optional Packages: --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --with-python=PYTHON name of the Python executable (default: python) --with-pgconfig=PG_CONFIG path to pg_config (default: pg_config) --with-asciidoc[=prog] path to asciidoc 8.2 (default: asciidoc) Some influential environment variables: CC C compiler command CFLAGS C compiler flags LDFLAGS linker flags, e.g. -L if you have libraries in a nonstandard directory LIBS libraries to pass to the linker, e.g. -l CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I if you have headers in a nonstandard directory Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. Report bugs to the package provider. _ACEOF ac_status=$? fi if test "$ac_init_help" = "recursive"; then # If there are subdirs, report their specific --help. for ac_dir in : $ac_subdirs_all; do test "x$ac_dir" = x: && continue test -d "$ac_dir" || { cd "$srcdir" && ac_pwd=`pwd` && srcdir=. && test -d "$ac_dir"; } || continue ac_builddir=. case "$ac_dir" in .) ac_dir_suffix= ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_dir_suffix=/`$as_echo "$ac_dir" | sed 's|^\.[\\/]||'` # A ".." for each directory in $ac_dir_suffix. ac_top_builddir_sub=`$as_echo "$ac_dir_suffix" | sed 's|/[^\\/]*|/..|g;s|/||'` case $ac_top_builddir_sub in "") ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_top_build_prefix=$ac_top_builddir_sub/ ;; esac ;; esac ac_abs_top_builddir=$ac_pwd ac_abs_builddir=$ac_pwd$ac_dir_suffix # for backward compatibility: ac_top_builddir=$ac_top_build_prefix case $srcdir in .) # We are building in place. ac_srcdir=. ac_top_srcdir=$ac_top_builddir_sub ac_abs_top_srcdir=$ac_pwd ;; [\\/]* | ?:[\\/]* ) # Absolute name. ac_srcdir=$srcdir$ac_dir_suffix; ac_top_srcdir=$srcdir ac_abs_top_srcdir=$srcdir ;; *) # Relative name. ac_srcdir=$ac_top_build_prefix$srcdir$ac_dir_suffix ac_top_srcdir=$ac_top_build_prefix$srcdir ac_abs_top_srcdir=$ac_pwd/$srcdir ;; esac ac_abs_srcdir=$ac_abs_top_srcdir$ac_dir_suffix cd "$ac_dir" || { ac_status=$?; continue; } # Check for guested configure. if test -f "$ac_srcdir/configure.gnu"; then echo && $SHELL "$ac_srcdir/configure.gnu" --help=recursive elif test -f "$ac_srcdir/configure"; then echo && $SHELL "$ac_srcdir/configure" --help=recursive else $as_echo "$as_me: WARNING: no configuration information is in $ac_dir" >&2 fi || ac_status=$? cd "$ac_pwd" || { ac_status=$?; break; } done fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF skytools configure 2.1.13 generated by GNU Autoconf 2.67 Copyright (C) 2010 Free Software Foundation, Inc. This configure script is free software; the Free Software Foundation gives unlimited permission to copy, distribute and modify it. _ACEOF exit fi ## ------------------------ ## ## Autoconf initialization. ## ## ------------------------ ## # ac_fn_c_try_compile LINENO # -------------------------- # Try to compile conftest.$ac_ext, and return whether this succeeded. ac_fn_c_try_compile () { as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack rm -f conftest.$ac_objext if { { ac_try="$ac_compile" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_compile") 2>conftest.err ac_status=$? if test -s conftest.err; then grep -v '^ *+' conftest.err >conftest.er1 cat conftest.er1 >&5 mv -f conftest.er1 conftest.err fi $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; } && { test -z "$ac_c_werror_flag" || test ! -s conftest.err } && test -s conftest.$ac_objext; then : ac_retval=0 else $as_echo "$as_me: failed program was:" >&5 sed 's/^/| /' conftest.$ac_ext >&5 ac_retval=1 fi eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;} as_fn_set_status $ac_retval } # ac_fn_c_try_compile # ac_fn_c_try_link LINENO # ----------------------- # Try to link conftest.$ac_ext, and return whether this succeeded. ac_fn_c_try_link () { as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack rm -f conftest.$ac_objext conftest$ac_exeext if { { ac_try="$ac_link" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_link") 2>conftest.err ac_status=$? if test -s conftest.err; then grep -v '^ *+' conftest.err >conftest.er1 cat conftest.er1 >&5 mv -f conftest.er1 conftest.err fi $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; } && { test -z "$ac_c_werror_flag" || test ! -s conftest.err } && test -s conftest$ac_exeext && { test "$cross_compiling" = yes || $as_test_x conftest$ac_exeext }; then : ac_retval=0 else $as_echo "$as_me: failed program was:" >&5 sed 's/^/| /' conftest.$ac_ext >&5 ac_retval=1 fi # Delete the IPA/IPO (Inter Procedural Analysis/Optimization) information # created by the PGI compiler (conftest_ipa8_conftest.oo), as it would # interfere with the next link command; also delete a directory that is # left behind by Apple's compiler. We do this before executing the actions. rm -rf conftest.dSYM conftest_ipa8_conftest.oo eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;} as_fn_set_status $ac_retval } # ac_fn_c_try_link # ac_fn_c_check_func LINENO FUNC VAR # ---------------------------------- # Tests whether FUNC exists, setting the cache variable VAR accordingly ac_fn_c_check_func () { as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5 $as_echo_n "checking for $2... " >&6; } if eval "test \"\${$3+set}\"" = set; then : $as_echo_n "(cached) " >&6 else cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ /* Define $2 to an innocuous variant, in case declares $2. For example, HP-UX 11i declares gettimeofday. */ #define $2 innocuous_$2 /* System header to define __stub macros and hopefully few prototypes, which can conflict with char $2 (); below. Prefer to if __STDC__ is defined, since exists even on freestanding compilers. */ #ifdef __STDC__ # include #else # include #endif #undef $2 /* Override any GCC internal prototype to avoid an error. Use char because int might match the return type of a GCC builtin and then its argument prototype would still apply. */ #ifdef __cplusplus extern "C" #endif char $2 (); /* The GNU C library defines this for functions which it implements to always fail with ENOSYS. Some functions are actually named something starting with __ and the normal name is an alias. */ #if defined __stub_$2 || defined __stub___$2 choke me #endif int main () { return $2 (); ; return 0; } _ACEOF if ac_fn_c_try_link "$LINENO"; then : eval "$3=yes" else eval "$3=no" fi rm -f core conftest.err conftest.$ac_objext \ conftest$ac_exeext conftest.$ac_ext fi eval ac_res=\$$3 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5 $as_echo "$ac_res" >&6; } eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;} } # ac_fn_c_check_func cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. It was created by skytools $as_me 2.1.13, which was generated by GNU Autoconf 2.67. Invocation command line was $ $0 $@ _ACEOF exec 5>>config.log { cat <<_ASUNAME ## --------- ## ## Platform. ## ## --------- ## hostname = `(hostname || uname -n) 2>/dev/null | sed 1q` uname -m = `(uname -m) 2>/dev/null || echo unknown` uname -r = `(uname -r) 2>/dev/null || echo unknown` uname -s = `(uname -s) 2>/dev/null || echo unknown` uname -v = `(uname -v) 2>/dev/null || echo unknown` /usr/bin/uname -p = `(/usr/bin/uname -p) 2>/dev/null || echo unknown` /bin/uname -X = `(/bin/uname -X) 2>/dev/null || echo unknown` /bin/arch = `(/bin/arch) 2>/dev/null || echo unknown` /usr/bin/arch -k = `(/usr/bin/arch -k) 2>/dev/null || echo unknown` /usr/convex/getsysinfo = `(/usr/convex/getsysinfo) 2>/dev/null || echo unknown` /usr/bin/hostinfo = `(/usr/bin/hostinfo) 2>/dev/null || echo unknown` /bin/machine = `(/bin/machine) 2>/dev/null || echo unknown` /usr/bin/oslevel = `(/usr/bin/oslevel) 2>/dev/null || echo unknown` /bin/universe = `(/bin/universe) 2>/dev/null || echo unknown` _ASUNAME as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. $as_echo "PATH: $as_dir" done IFS=$as_save_IFS } >&5 cat >&5 <<_ACEOF ## ----------- ## ## Core tests. ## ## ----------- ## _ACEOF # Keep a trace of the command line. # Strip out --no-create and --no-recursion so they do not pile up. # Strip out --silent because we don't want to record it for future runs. # Also quote any args containing shell meta-characters. # Make two passes to allow for proper duplicate-argument suppression. ac_configure_args= ac_configure_args0= ac_configure_args1= ac_must_keep_next=false for ac_pass in 1 2 do for ac_arg do case $ac_arg in -no-create | --no-c* | -n | -no-recursion | --no-r*) continue ;; -q | -quiet | --quiet | --quie | --qui | --qu | --q \ | -silent | --silent | --silen | --sile | --sil) continue ;; *\'*) ac_arg=`$as_echo "$ac_arg" | sed "s/'/'\\\\\\\\''/g"` ;; esac case $ac_pass in 1) as_fn_append ac_configure_args0 " '$ac_arg'" ;; 2) as_fn_append ac_configure_args1 " '$ac_arg'" if test $ac_must_keep_next = true; then ac_must_keep_next=false # Got value, back to normal. else case $ac_arg in *=* | --config-cache | -C | -disable-* | --disable-* \ | -enable-* | --enable-* | -gas | --g* | -nfp | --nf* \ | -q | -quiet | --q* | -silent | --sil* | -v | -verb* \ | -with-* | --with-* | -without-* | --without-* | --x) case "$ac_configure_args0 " in "$ac_configure_args1"*" '$ac_arg' "* ) continue ;; esac ;; -* ) ac_must_keep_next=true ;; esac fi as_fn_append ac_configure_args " '$ac_arg'" ;; esac done done { ac_configure_args0=; unset ac_configure_args0;} { ac_configure_args1=; unset ac_configure_args1;} # When interrupted or exit'd, cleanup temporary files, and complete # config.log. We remove comments because anyway the quotes in there # would cause problems or look ugly. # WARNING: Use '\'' to represent an apostrophe within the trap. # WARNING: Do not start the trap code with a newline, due to a FreeBSD 4.0 bug. trap 'exit_status=$? # Save into config.log some information that might help in debugging. { echo $as_echo "## ---------------- ## ## Cache variables. ## ## ---------------- ##" echo # The following way of writing the cache mishandles newlines in values, ( for ac_var in `(set) 2>&1 | sed -n '\''s/^\([a-zA-Z_][a-zA-Z0-9_]*\)=.*/\1/p'\''`; do eval ac_val=\$$ac_var case $ac_val in #( *${as_nl}*) case $ac_var in #( *_cv_*) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: cache variable $ac_var contains a newline" >&5 $as_echo "$as_me: WARNING: cache variable $ac_var contains a newline" >&2;} ;; esac case $ac_var in #( _ | IFS | as_nl) ;; #( BASH_ARGV | BASH_SOURCE) eval $ac_var= ;; #( *) { eval $ac_var=; unset $ac_var;} ;; esac ;; esac done (set) 2>&1 | case $as_nl`(ac_space='\'' '\''; set) 2>&1` in #( *${as_nl}ac_space=\ *) sed -n \ "s/'\''/'\''\\\\'\'''\''/g; s/^\\([_$as_cr_alnum]*_cv_[_$as_cr_alnum]*\\)=\\(.*\\)/\\1='\''\\2'\''/p" ;; #( *) sed -n "/^[_$as_cr_alnum]*_cv_[_$as_cr_alnum]*=/p" ;; esac | sort ) echo $as_echo "## ----------------- ## ## Output variables. ## ## ----------------- ##" echo for ac_var in $ac_subst_vars do eval ac_val=\$$ac_var case $ac_val in *\'\''*) ac_val=`$as_echo "$ac_val" | sed "s/'\''/'\''\\\\\\\\'\'''\''/g"`;; esac $as_echo "$ac_var='\''$ac_val'\''" done | sort echo if test -n "$ac_subst_files"; then $as_echo "## ------------------- ## ## File substitutions. ## ## ------------------- ##" echo for ac_var in $ac_subst_files do eval ac_val=\$$ac_var case $ac_val in *\'\''*) ac_val=`$as_echo "$ac_val" | sed "s/'\''/'\''\\\\\\\\'\'''\''/g"`;; esac $as_echo "$ac_var='\''$ac_val'\''" done | sort echo fi if test -s confdefs.h; then $as_echo "## ----------- ## ## confdefs.h. ## ## ----------- ##" echo cat confdefs.h echo fi test "$ac_signal" != 0 && $as_echo "$as_me: caught signal $ac_signal" $as_echo "$as_me: exit $exit_status" } >&5 rm -f core *.core core.conftest.* && rm -f -r conftest* confdefs* conf$$* $ac_clean_files && exit $exit_status ' 0 for ac_signal in 1 2 13 15; do trap 'ac_signal='$ac_signal'; as_fn_exit 1' $ac_signal done ac_signal=0 # confdefs.h avoids OS command line length limits that DEFS can exceed. rm -f -r conftest* confdefs.h $as_echo "/* confdefs.h */" > confdefs.h # Predefined preprocessor variables. cat >>confdefs.h <<_ACEOF #define PACKAGE_NAME "$PACKAGE_NAME" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_TARNAME "$PACKAGE_TARNAME" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_VERSION "$PACKAGE_VERSION" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_STRING "$PACKAGE_STRING" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_BUGREPORT "$PACKAGE_BUGREPORT" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_URL "$PACKAGE_URL" _ACEOF # Let the site file select an alternate cache file if it wants to. # Prefer an explicitly selected file to automatically selected ones. ac_site_file1=NONE ac_site_file2=NONE if test -n "$CONFIG_SITE"; then # We do not want a PATH search for config.site. case $CONFIG_SITE in #(( -*) ac_site_file1=./$CONFIG_SITE;; */*) ac_site_file1=$CONFIG_SITE;; *) ac_site_file1=./$CONFIG_SITE;; esac elif test "x$prefix" != xNONE; then ac_site_file1=$prefix/share/config.site ac_site_file2=$prefix/etc/config.site else ac_site_file1=$ac_default_prefix/share/config.site ac_site_file2=$ac_default_prefix/etc/config.site fi for ac_site_file in "$ac_site_file1" "$ac_site_file2" do test "x$ac_site_file" = xNONE && continue if test /dev/null != "$ac_site_file" && test -r "$ac_site_file"; then { $as_echo "$as_me:${as_lineno-$LINENO}: loading site script $ac_site_file" >&5 $as_echo "$as_me: loading site script $ac_site_file" >&6;} sed 's/^/| /' "$ac_site_file" >&5 . "$ac_site_file" \ || { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "failed to load site script $ac_site_file See \`config.log' for more details" "$LINENO" 5 ; } fi done if test -r "$cache_file"; then # Some versions of bash will fail to source /dev/null (special files # actually), so we avoid doing that. DJGPP emulates it as a regular file. if test /dev/null != "$cache_file" && test -f "$cache_file"; then { $as_echo "$as_me:${as_lineno-$LINENO}: loading cache $cache_file" >&5 $as_echo "$as_me: loading cache $cache_file" >&6;} case $cache_file in [\\/]* | ?:[\\/]* ) . "$cache_file";; *) . "./$cache_file";; esac fi else { $as_echo "$as_me:${as_lineno-$LINENO}: creating cache $cache_file" >&5 $as_echo "$as_me: creating cache $cache_file" >&6;} >$cache_file fi # Check that the precious variables saved in the cache have kept the same # value. ac_cache_corrupted=false for ac_var in $ac_precious_vars; do eval ac_old_set=\$ac_cv_env_${ac_var}_set eval ac_new_set=\$ac_env_${ac_var}_set eval ac_old_val=\$ac_cv_env_${ac_var}_value eval ac_new_val=\$ac_env_${ac_var}_value case $ac_old_set,$ac_new_set in set,) { $as_echo "$as_me:${as_lineno-$LINENO}: error: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&5 $as_echo "$as_me: error: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&2;} ac_cache_corrupted=: ;; ,set) { $as_echo "$as_me:${as_lineno-$LINENO}: error: \`$ac_var' was not set in the previous run" >&5 $as_echo "$as_me: error: \`$ac_var' was not set in the previous run" >&2;} ac_cache_corrupted=: ;; ,);; *) if test "x$ac_old_val" != "x$ac_new_val"; then # differences in whitespace do not lead to failure. ac_old_val_w=`echo x $ac_old_val` ac_new_val_w=`echo x $ac_new_val` if test "$ac_old_val_w" != "$ac_new_val_w"; then { $as_echo "$as_me:${as_lineno-$LINENO}: error: \`$ac_var' has changed since the previous run:" >&5 $as_echo "$as_me: error: \`$ac_var' has changed since the previous run:" >&2;} ac_cache_corrupted=: else { $as_echo "$as_me:${as_lineno-$LINENO}: warning: ignoring whitespace changes in \`$ac_var' since the previous run:" >&5 $as_echo "$as_me: warning: ignoring whitespace changes in \`$ac_var' since the previous run:" >&2;} eval $ac_var=\$ac_old_val fi { $as_echo "$as_me:${as_lineno-$LINENO}: former value: \`$ac_old_val'" >&5 $as_echo "$as_me: former value: \`$ac_old_val'" >&2;} { $as_echo "$as_me:${as_lineno-$LINENO}: current value: \`$ac_new_val'" >&5 $as_echo "$as_me: current value: \`$ac_new_val'" >&2;} fi;; esac # Pass precious variables to config.status. if test "$ac_new_set" = set; then case $ac_new_val in *\'*) ac_arg=$ac_var=`$as_echo "$ac_new_val" | sed "s/'/'\\\\\\\\''/g"` ;; *) ac_arg=$ac_var=$ac_new_val ;; esac case " $ac_configure_args " in *" '$ac_arg' "*) ;; # Avoid dups. Use of quotes ensures accuracy. *) as_fn_append ac_configure_args " '$ac_arg'" ;; esac fi done if $ac_cache_corrupted; then { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} { $as_echo "$as_me:${as_lineno-$LINENO}: error: changes in the environment can compromise the build" >&5 $as_echo "$as_me: error: changes in the environment can compromise the build" >&2;} as_fn_error $? "run \`make distclean' and/or \`rm $cache_file' and start over" "$LINENO" 5 fi ## -------------------- ## ## Main body of script. ## ## -------------------- ## ac_ext=c ac_cpp='$CPP $CPPFLAGS' ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5' ac_compiler_gnu=$ac_cv_c_compiler_gnu # Check whether --with-python was given. if test "${with_python+set}" = set; then : withval=$with_python; { $as_echo "$as_me:${as_lineno-$LINENO}: checking for python" >&5 $as_echo_n "checking for python... " >&6; } PYTHON=$withval { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PYTHON" >&5 $as_echo "$PYTHON" >&6; } else for ac_prog in python do # Extract the first word of "$ac_prog", so it can be a program name with args. set dummy $ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_path_PYTHON+set}" = set; then : $as_echo_n "(cached) " >&6 else case $PYTHON in [\\/]* | ?:[\\/]*) ac_cv_path_PYTHON="$PYTHON" # Let the user override the test with a path. ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_path_PYTHON="$as_dir/$ac_word$ac_exec_ext" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS ;; esac fi PYTHON=$ac_cv_path_PYTHON if test -n "$PYTHON"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PYTHON" >&5 $as_echo "$PYTHON" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$PYTHON" && break done fi test -n "$PYTHON" || as_fn_error $? "Cannot continue without Python" "$LINENO" 5 # Check whether --with-pgconfig was given. if test "${with_pgconfig+set}" = set; then : withval=$with_pgconfig; { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pg_config" >&5 $as_echo_n "checking for pg_config... " >&6; } PG_CONFIG=$withval { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PG_CONFIG" >&5 $as_echo "$PG_CONFIG" >&6; } else for ac_prog in pg_config do # Extract the first word of "$ac_prog", so it can be a program name with args. set dummy $ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_path_PG_CONFIG+set}" = set; then : $as_echo_n "(cached) " >&6 else case $PG_CONFIG in [\\/]* | ?:[\\/]*) ac_cv_path_PG_CONFIG="$PG_CONFIG" # Let the user override the test with a path. ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_path_PG_CONFIG="$as_dir/$ac_word$ac_exec_ext" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS ;; esac fi PG_CONFIG=$ac_cv_path_PG_CONFIG if test -n "$PG_CONFIG"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PG_CONFIG" >&5 $as_echo "$PG_CONFIG" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$PG_CONFIG" && break done fi test -n "$PG_CONFIG" || as_fn_error $? "Cannot continue without pg_config" "$LINENO" 5 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for GNU make" >&5 $as_echo_n "checking for GNU make... " >&6; } if test ! -n "$MAKE"; then for a in make gmake gnumake; do if "$a" --version 2>&1 | grep GNU > /dev/null; then MAKE="$a" break fi done fi if test -n "$MAKE"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $MAKE" >&5 $as_echo "$MAKE" >&6; } else as_fn_error $? "GNU make is not found" "$LINENO" 5 fi # Check whether --with-asciidoc was given. if test "${with_asciidoc+set}" = set; then : withval=$with_asciidoc; if test "$withval" = "yes"; then for ac_prog in $ASCIIDOC asciidoc do # Extract the first word of "$ac_prog", so it can be a program name with args. set dummy $ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_ASCIIDOC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$ASCIIDOC"; then ac_cv_prog_ASCIIDOC="$ASCIIDOC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_ASCIIDOC="$ac_prog" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi ASCIIDOC=$ac_cv_prog_ASCIIDOC if test -n "$ASCIIDOC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ASCIIDOC" >&5 $as_echo "$ASCIIDOC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$ASCIIDOC" && break done test -n "$ASCIIDOC" || ASCIIDOC=no else { $as_echo "$as_me:${as_lineno-$LINENO}: checking for asciidoc" >&5 $as_echo_n "checking for asciidoc... " >&6; } ASCIIDOC=$withval { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ASCIIDOC" >&5 $as_echo "$ASCIIDOC" >&6; } fi else ASCIIDOC="no" fi if test "$ASCIIDOC" != "no"; then { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether asciidoc version >= 8.2" >&5 $as_echo_n "checking whether asciidoc version >= 8.2... " >&6; } ver=`$ASCIIDOC --version 2>&1 | sed -e 's/asciidoc //'` case "$ver" in [0-7].*|8.[01]|8.[01].*) { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ver, too old" >&5 $as_echo "$ver, too old" >&6; } ASCIIDOC="no" ;; *) { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ver, ok" >&5 $as_echo "$ver, ok" >&6; } ;; esac fi if test "$ASCIIDOC" != "no"; then for ac_prog in $XMLTO xmlto do # Extract the first word of "$ac_prog", so it can be a program name with args. set dummy $ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_XMLTO+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$XMLTO"; then ac_cv_prog_XMLTO="$XMLTO" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_XMLTO="$ac_prog" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi XMLTO=$ac_cv_prog_XMLTO if test -n "$XMLTO"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $XMLTO" >&5 $as_echo "$XMLTO" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$XMLTO" && break done test -n "$XMLTO" || XMLTO=no else XMLTO="no" fi # when in cvs/git tree, turn asciidoc/xmlto unconditionally on if test -d .git -o -d CVS; then if test "$ASCIIDOC" = "no"; then echo "*** Building from CVS/GIT requires asciidoc, enabling it ***" ASCIIDOC="asciidoc" fi if test "$XMLTO" = "no"; then echo "*** Building from CVS/GIT requires xmlto, enabling it ***" XMLTO="xmlto" fi fi ac_ext=c ac_cpp='$CPP $CPPFLAGS' ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5' ac_compiler_gnu=$ac_cv_c_compiler_gnu if test -n "$ac_tool_prefix"; then # Extract the first word of "${ac_tool_prefix}gcc", so it can be a program name with args. set dummy ${ac_tool_prefix}gcc; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_CC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$CC"; then ac_cv_prog_CC="$CC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_CC="${ac_tool_prefix}gcc" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi CC=$ac_cv_prog_CC if test -n "$CC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $CC" >&5 $as_echo "$CC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi fi if test -z "$ac_cv_prog_CC"; then ac_ct_CC=$CC # Extract the first word of "gcc", so it can be a program name with args. set dummy gcc; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_ac_ct_CC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$ac_ct_CC"; then ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_ac_ct_CC="gcc" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi ac_ct_CC=$ac_cv_prog_ac_ct_CC if test -n "$ac_ct_CC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_ct_CC" >&5 $as_echo "$ac_ct_CC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi if test "x$ac_ct_CC" = x; then CC="" else case $cross_compiling:$ac_tool_warned in yes:) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 $as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} ac_tool_warned=yes ;; esac CC=$ac_ct_CC fi else CC="$ac_cv_prog_CC" fi if test -z "$CC"; then if test -n "$ac_tool_prefix"; then # Extract the first word of "${ac_tool_prefix}cc", so it can be a program name with args. set dummy ${ac_tool_prefix}cc; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_CC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$CC"; then ac_cv_prog_CC="$CC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_CC="${ac_tool_prefix}cc" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi CC=$ac_cv_prog_CC if test -n "$CC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $CC" >&5 $as_echo "$CC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi fi fi if test -z "$CC"; then # Extract the first word of "cc", so it can be a program name with args. set dummy cc; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_CC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$CC"; then ac_cv_prog_CC="$CC" # Let the user override the test. else ac_prog_rejected=no as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then if test "$as_dir/$ac_word$ac_exec_ext" = "/usr/ucb/cc"; then ac_prog_rejected=yes continue fi ac_cv_prog_CC="cc" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS if test $ac_prog_rejected = yes; then # We found a bogon in the path, so make sure we never use it. set dummy $ac_cv_prog_CC shift if test $# != 0; then # We chose a different compiler from the bogus one. # However, it has the same basename, so the bogon will be chosen # first if we set CC to just the basename; use the full file name. shift ac_cv_prog_CC="$as_dir/$ac_word${1+' '}$@" fi fi fi fi CC=$ac_cv_prog_CC if test -n "$CC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $CC" >&5 $as_echo "$CC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi fi if test -z "$CC"; then if test -n "$ac_tool_prefix"; then for ac_prog in cl.exe do # Extract the first word of "$ac_tool_prefix$ac_prog", so it can be a program name with args. set dummy $ac_tool_prefix$ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_CC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$CC"; then ac_cv_prog_CC="$CC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_CC="$ac_tool_prefix$ac_prog" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi CC=$ac_cv_prog_CC if test -n "$CC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $CC" >&5 $as_echo "$CC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$CC" && break done fi if test -z "$CC"; then ac_ct_CC=$CC for ac_prog in cl.exe do # Extract the first word of "$ac_prog", so it can be a program name with args. set dummy $ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if test "${ac_cv_prog_ac_ct_CC+set}" = set; then : $as_echo_n "(cached) " >&6 else if test -n "$ac_ct_CC"; then ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if { test -f "$as_dir/$ac_word$ac_exec_ext" && $as_test_x "$as_dir/$ac_word$ac_exec_ext"; }; then ac_cv_prog_ac_ct_CC="$ac_prog" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi ac_ct_CC=$ac_cv_prog_ac_ct_CC if test -n "$ac_ct_CC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_ct_CC" >&5 $as_echo "$ac_ct_CC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$ac_ct_CC" && break done if test "x$ac_ct_CC" = x; then CC="" else case $cross_compiling:$ac_tool_warned in yes:) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 $as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} ac_tool_warned=yes ;; esac CC=$ac_ct_CC fi fi fi test -z "$CC" && { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "no acceptable C compiler found in \$PATH See \`config.log' for more details" "$LINENO" 5 ; } # Provide some information about the compiler. $as_echo "$as_me:${as_lineno-$LINENO}: checking for C compiler version" >&5 set X $ac_compile ac_compiler=$2 for ac_option in --version -v -V -qversion; do { { ac_try="$ac_compiler $ac_option >&5" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_compiler $ac_option >&5") 2>conftest.err ac_status=$? if test -s conftest.err; then sed '10a\ ... rest of stderr output deleted ... 10q' conftest.err >conftest.er1 cat conftest.er1 >&5 fi rm -f conftest.er1 conftest.err $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; } done cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ int main () { ; return 0; } _ACEOF ac_clean_files_save=$ac_clean_files ac_clean_files="$ac_clean_files a.out a.out.dSYM a.exe b.out" # Try to create an executable without -o first, disregard a.out. # It will help us diagnose broken compilers, and finding out an intuition # of exeext. { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether the C compiler works" >&5 $as_echo_n "checking whether the C compiler works... " >&6; } ac_link_default=`$as_echo "$ac_link" | sed 's/ -o *conftest[^ ]*//'` # The possible output files: ac_files="a.out conftest.exe conftest a.exe a_out.exe b.out conftest.*" ac_rmfiles= for ac_file in $ac_files do case $ac_file in *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb | *.xSYM | *.bb | *.bbg | *.map | *.inf | *.dSYM | *.o | *.obj ) ;; * ) ac_rmfiles="$ac_rmfiles $ac_file";; esac done rm -f $ac_rmfiles if { { ac_try="$ac_link_default" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_link_default") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then : # Autoconf-2.13 could set the ac_cv_exeext variable to `no'. # So ignore a value of `no', otherwise this would lead to `EXEEXT = no' # in a Makefile. We should not override ac_cv_exeext if it was cached, # so that the user can short-circuit this test for compilers unknown to # Autoconf. for ac_file in $ac_files '' do test -f "$ac_file" || continue case $ac_file in *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb | *.xSYM | *.bb | *.bbg | *.map | *.inf | *.dSYM | *.o | *.obj ) ;; [ab].out ) # We found the default executable, but exeext='' is most # certainly right. break;; *.* ) if test "${ac_cv_exeext+set}" = set && test "$ac_cv_exeext" != no; then :; else ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'` fi # We set ac_cv_exeext here because the later test for it is not # safe: cross compilers may not add the suffix if given an `-o' # argument, so we may need to know it at that point already. # Even if this section looks crufty: it has the advantage of # actually working. break;; * ) break;; esac done test "$ac_cv_exeext" = no && ac_cv_exeext= else ac_file='' fi if test -z "$ac_file"; then : { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } $as_echo "$as_me: failed program was:" >&5 sed 's/^/| /' conftest.$ac_ext >&5 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error 77 "C compiler cannot create executables See \`config.log' for more details" "$LINENO" 5 ; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; } fi { $as_echo "$as_me:${as_lineno-$LINENO}: checking for C compiler default output file name" >&5 $as_echo_n "checking for C compiler default output file name... " >&6; } { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_file" >&5 $as_echo "$ac_file" >&6; } ac_exeext=$ac_cv_exeext rm -f -r a.out a.out.dSYM a.exe conftest$ac_cv_exeext b.out ac_clean_files=$ac_clean_files_save { $as_echo "$as_me:${as_lineno-$LINENO}: checking for suffix of executables" >&5 $as_echo_n "checking for suffix of executables... " >&6; } if { { ac_try="$ac_link" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_link") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then : # If both `conftest.exe' and `conftest' are `present' (well, observable) # catch `conftest.exe'. For instance with Cygwin, `ls conftest' will # work properly (i.e., refer to `conftest.exe'), while it won't with # `rm'. for ac_file in conftest.exe conftest conftest.*; do test -f "$ac_file" || continue case $ac_file in *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb | *.xSYM | *.bb | *.bbg | *.map | *.inf | *.dSYM | *.o | *.obj ) ;; *.* ) ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'` break;; * ) break;; esac done else { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "cannot compute suffix of executables: cannot compile and link See \`config.log' for more details" "$LINENO" 5 ; } fi rm -f conftest conftest$ac_cv_exeext { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_exeext" >&5 $as_echo "$ac_cv_exeext" >&6; } rm -f conftest.$ac_ext EXEEXT=$ac_cv_exeext ac_exeext=$EXEEXT cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ #include int main () { FILE *f = fopen ("conftest.out", "w"); return ferror (f) || fclose (f) != 0; ; return 0; } _ACEOF ac_clean_files="$ac_clean_files conftest.out" # Check that the compiler produces executables we can run. If not, either # the compiler is broken, or we cross compile. { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether we are cross compiling" >&5 $as_echo_n "checking whether we are cross compiling... " >&6; } if test "$cross_compiling" != yes; then { { ac_try="$ac_link" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_link") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; } if { ac_try='./conftest$ac_cv_exeext' { { case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_try") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; }; then cross_compiling=no else if test "$cross_compiling" = maybe; then cross_compiling=yes else { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "cannot run C compiled programs. If you meant to cross compile, use \`--host'. See \`config.log' for more details" "$LINENO" 5 ; } fi fi fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $cross_compiling" >&5 $as_echo "$cross_compiling" >&6; } rm -f conftest.$ac_ext conftest$ac_cv_exeext conftest.out ac_clean_files=$ac_clean_files_save { $as_echo "$as_me:${as_lineno-$LINENO}: checking for suffix of object files" >&5 $as_echo_n "checking for suffix of object files... " >&6; } if test "${ac_cv_objext+set}" = set; then : $as_echo_n "(cached) " >&6 else cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ int main () { ; return 0; } _ACEOF rm -f conftest.o conftest.obj if { { ac_try="$ac_compile" case "(($ac_try" in *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; *) ac_try_echo=$ac_try;; esac eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\"" $as_echo "$ac_try_echo"; } >&5 (eval "$ac_compile") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then : for ac_file in conftest.o conftest.obj conftest.*; do test -f "$ac_file" || continue; case $ac_file in *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb | *.xSYM | *.bb | *.bbg | *.map | *.inf | *.dSYM ) ;; *) ac_cv_objext=`expr "$ac_file" : '.*\.\(.*\)'` break;; esac done else $as_echo "$as_me: failed program was:" >&5 sed 's/^/| /' conftest.$ac_ext >&5 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "cannot compute suffix of object files: cannot compile See \`config.log' for more details" "$LINENO" 5 ; } fi rm -f conftest.$ac_cv_objext conftest.$ac_ext fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_objext" >&5 $as_echo "$ac_cv_objext" >&6; } OBJEXT=$ac_cv_objext ac_objext=$OBJEXT { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether we are using the GNU C compiler" >&5 $as_echo_n "checking whether we are using the GNU C compiler... " >&6; } if test "${ac_cv_c_compiler_gnu+set}" = set; then : $as_echo_n "(cached) " >&6 else cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ int main () { #ifndef __GNUC__ choke me #endif ; return 0; } _ACEOF if ac_fn_c_try_compile "$LINENO"; then : ac_compiler_gnu=yes else ac_compiler_gnu=no fi rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext ac_cv_c_compiler_gnu=$ac_compiler_gnu fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_c_compiler_gnu" >&5 $as_echo "$ac_cv_c_compiler_gnu" >&6; } if test $ac_compiler_gnu = yes; then GCC=yes else GCC= fi ac_test_CFLAGS=${CFLAGS+set} ac_save_CFLAGS=$CFLAGS { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -g" >&5 $as_echo_n "checking whether $CC accepts -g... " >&6; } if test "${ac_cv_prog_cc_g+set}" = set; then : $as_echo_n "(cached) " >&6 else ac_save_c_werror_flag=$ac_c_werror_flag ac_c_werror_flag=yes ac_cv_prog_cc_g=no CFLAGS="-g" cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ int main () { ; return 0; } _ACEOF if ac_fn_c_try_compile "$LINENO"; then : ac_cv_prog_cc_g=yes else CFLAGS="" cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ int main () { ; return 0; } _ACEOF if ac_fn_c_try_compile "$LINENO"; then : else ac_c_werror_flag=$ac_save_c_werror_flag CFLAGS="-g" cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ int main () { ; return 0; } _ACEOF if ac_fn_c_try_compile "$LINENO"; then : ac_cv_prog_cc_g=yes fi rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext fi rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext fi rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext ac_c_werror_flag=$ac_save_c_werror_flag fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_prog_cc_g" >&5 $as_echo "$ac_cv_prog_cc_g" >&6; } if test "$ac_test_CFLAGS" = set; then CFLAGS=$ac_save_CFLAGS elif test $ac_cv_prog_cc_g = yes; then if test "$GCC" = yes; then CFLAGS="-g -O2" else CFLAGS="-g" fi else if test "$GCC" = yes; then CFLAGS="-O2" else CFLAGS= fi fi { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $CC option to accept ISO C89" >&5 $as_echo_n "checking for $CC option to accept ISO C89... " >&6; } if test "${ac_cv_prog_cc_c89+set}" = set; then : $as_echo_n "(cached) " >&6 else ac_cv_prog_cc_c89=no ac_save_CC=$CC cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ #include #include #include #include /* Most of the following tests are stolen from RCS 5.7's src/conf.sh. */ struct buf { int x; }; FILE * (*rcsopen) (struct buf *, struct stat *, int); static char *e (p, i) char **p; int i; { return p[i]; } static char *f (char * (*g) (char **, int), char **p, ...) { char *s; va_list v; va_start (v,p); s = g (p, va_arg (v,int)); va_end (v); return s; } /* OSF 4.0 Compaq cc is some sort of almost-ANSI by default. It has function prototypes and stuff, but not '\xHH' hex character constants. These don't provoke an error unfortunately, instead are silently treated as 'x'. The following induces an error, until -std is added to get proper ANSI mode. Curiously '\x00'!='x' always comes out true, for an array size at least. It's necessary to write '\x00'==0 to get something that's true only with -std. */ int osf4_cc_array ['\x00' == 0 ? 1 : -1]; /* IBM C 6 for AIX is almost-ANSI by default, but it replaces macro parameters inside strings and character constants. */ #define FOO(x) 'x' int xlc6_cc_array[FOO(a) == 'x' ? 1 : -1]; int test (int i, double x); struct s1 {int (*f) (int a);}; struct s2 {int (*f) (double a);}; int pairnames (int, char **, FILE *(*)(struct buf *, struct stat *, int), int, int); int argc; char **argv; int main () { return f (e, argv, 0) != argv[0] || f (e, argv, 1) != argv[1]; ; return 0; } _ACEOF for ac_arg in '' -qlanglvl=extc89 -qlanglvl=ansi -std \ -Ae "-Aa -D_HPUX_SOURCE" "-Xc -D__EXTENSIONS__" do CC="$ac_save_CC $ac_arg" if ac_fn_c_try_compile "$LINENO"; then : ac_cv_prog_cc_c89=$ac_arg fi rm -f core conftest.err conftest.$ac_objext test "x$ac_cv_prog_cc_c89" != "xno" && break done rm -f conftest.$ac_ext CC=$ac_save_CC fi # AC_CACHE_VAL case "x$ac_cv_prog_cc_c89" in x) { $as_echo "$as_me:${as_lineno-$LINENO}: result: none needed" >&5 $as_echo "none needed" >&6; } ;; xno) { $as_echo "$as_me:${as_lineno-$LINENO}: result: unsupported" >&5 $as_echo "unsupported" >&6; } ;; *) CC="$CC $ac_cv_prog_cc_c89" { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_prog_cc_c89" >&5 $as_echo "$ac_cv_prog_cc_c89" >&6; } ;; esac if test "x$ac_cv_prog_cc_c89" != xno; then : fi ac_ext=c ac_cpp='$CPP $CPPFLAGS' ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5' ac_compiler_gnu=$ac_cv_c_compiler_gnu for ac_func in unsetenv do : ac_fn_c_check_func "$LINENO" "unsetenv" "ac_cv_func_unsetenv" if test "x$ac_cv_func_unsetenv" = x""yes; then : cat >>confdefs.h <<_ACEOF #define HAVE_UNSETENV 1 _ACEOF fi done ac_config_files="$ac_config_files config.mak" cat >confcache <<\_ACEOF # This file is a shell script that caches the results of configure # tests run on this system so they can be shared between configure # scripts and configure runs, see configure's option --config-cache. # It is not useful on other systems. If it contains results you don't # want to keep, you may remove or edit it. # # config.status only pays attention to the cache file if you give it # the --recheck option to rerun configure. # # `ac_cv_env_foo' variables (set or unset) will be overridden when # loading this file, other *unset* `ac_cv_foo' will be assigned the # following values. _ACEOF # The following way of writing the cache mishandles newlines in values, # but we know of no workaround that is simple, portable, and efficient. # So, we kill variables containing newlines. # Ultrix sh set writes to stderr and can't be redirected directly, # and sets the high bit in the cache file unless we assign to the vars. ( for ac_var in `(set) 2>&1 | sed -n 's/^\([a-zA-Z_][a-zA-Z0-9_]*\)=.*/\1/p'`; do eval ac_val=\$$ac_var case $ac_val in #( *${as_nl}*) case $ac_var in #( *_cv_*) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: cache variable $ac_var contains a newline" >&5 $as_echo "$as_me: WARNING: cache variable $ac_var contains a newline" >&2;} ;; esac case $ac_var in #( _ | IFS | as_nl) ;; #( BASH_ARGV | BASH_SOURCE) eval $ac_var= ;; #( *) { eval $ac_var=; unset $ac_var;} ;; esac ;; esac done (set) 2>&1 | case $as_nl`(ac_space=' '; set) 2>&1` in #( *${as_nl}ac_space=\ *) # `set' does not quote correctly, so add quotes: double-quote # substitution turns \\\\ into \\, and sed turns \\ into \. sed -n \ "s/'/'\\\\''/g; s/^\\([_$as_cr_alnum]*_cv_[_$as_cr_alnum]*\\)=\\(.*\\)/\\1='\\2'/p" ;; #( *) # `set' quotes correctly as required by POSIX, so do not add quotes. sed -n "/^[_$as_cr_alnum]*_cv_[_$as_cr_alnum]*=/p" ;; esac | sort ) | sed ' /^ac_cv_env_/b end t clear :clear s/^\([^=]*\)=\(.*[{}].*\)$/test "${\1+set}" = set || &/ t end s/^\([^=]*\)=\(.*\)$/\1=${\1=\2}/ :end' >>confcache if diff "$cache_file" confcache >/dev/null 2>&1; then :; else if test -w "$cache_file"; then test "x$cache_file" != "x/dev/null" && { $as_echo "$as_me:${as_lineno-$LINENO}: updating cache $cache_file" >&5 $as_echo "$as_me: updating cache $cache_file" >&6;} cat confcache >$cache_file else { $as_echo "$as_me:${as_lineno-$LINENO}: not updating unwritable cache $cache_file" >&5 $as_echo "$as_me: not updating unwritable cache $cache_file" >&6;} fi fi rm -f confcache test "x$prefix" = xNONE && prefix=$ac_default_prefix # Let make expand exec_prefix. test "x$exec_prefix" = xNONE && exec_prefix='${prefix}' # Transform confdefs.h into DEFS. # Protect against shell expansion while executing Makefile rules. # Protect against Makefile macro expansion. # # If the first sed substitution is executed (which looks for macros that # take arguments), then branch to the quote section. Otherwise, # look for a macro that doesn't take arguments. ac_script=' :mline /\\$/{ N s,\\\n,, b mline } t clear :clear s/^[ ]*#[ ]*define[ ][ ]*\([^ (][^ (]*([^)]*)\)[ ]*\(.*\)/-D\1=\2/g t quote s/^[ ]*#[ ]*define[ ][ ]*\([^ ][^ ]*\)[ ]*\(.*\)/-D\1=\2/g t quote b any :quote s/[ `~#$^&*(){}\\|;'\''"<>?]/\\&/g s/\[/\\&/g s/\]/\\&/g s/\$/$$/g H :any ${ g s/^\n// s/\n/ /g p } ' DEFS=`sed -n "$ac_script" confdefs.h` ac_libobjs= ac_ltlibobjs= U= for ac_i in : $LIBOBJS; do test "x$ac_i" = x: && continue # 1. Remove the extension, and $U if already installed. ac_script='s/\$U\././;s/\.o$//;s/\.obj$//' ac_i=`$as_echo "$ac_i" | sed "$ac_script"` # 2. Prepend LIBOBJDIR. When used with automake>=1.10 LIBOBJDIR # will be set to the directory where LIBOBJS objects are built. as_fn_append ac_libobjs " \${LIBOBJDIR}$ac_i\$U.$ac_objext" as_fn_append ac_ltlibobjs " \${LIBOBJDIR}$ac_i"'$U.lo' done LIBOBJS=$ac_libobjs LTLIBOBJS=$ac_ltlibobjs : ${CONFIG_STATUS=./config.status} ac_write_fail=0 ac_clean_files_save=$ac_clean_files ac_clean_files="$ac_clean_files $CONFIG_STATUS" { $as_echo "$as_me:${as_lineno-$LINENO}: creating $CONFIG_STATUS" >&5 $as_echo "$as_me: creating $CONFIG_STATUS" >&6;} as_write_fail=0 cat >$CONFIG_STATUS <<_ASEOF || as_write_fail=1 #! $SHELL # Generated by $as_me. # Run this file to recreate the current configuration. # Compiler output produced by configure, useful for debugging # configure, is in config.log if it exists. debug=false ac_cs_recheck=false ac_cs_silent=false SHELL=\${CONFIG_SHELL-$SHELL} export SHELL _ASEOF cat >>$CONFIG_STATUS <<\_ASEOF || as_write_fail=1 ## -------------------- ## ## M4sh Initialization. ## ## -------------------- ## # Be more Bourne compatible DUALCASE=1; export DUALCASE # for MKS sh if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then : emulate sh NULLCMD=: # Pre-4.2 versions of Zsh do word splitting on ${1+"$@"}, which # is contrary to our usage. Disable this feature. alias -g '${1+"$@"}'='"$@"' setopt NO_GLOB_SUBST else case `(set -o) 2>/dev/null` in #( *posix*) : set -o posix ;; #( *) : ;; esac fi as_nl=' ' export as_nl # Printing a long string crashes Solaris 7 /usr/bin/printf. as_echo='\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo$as_echo # Prefer a ksh shell builtin over an external printf program on Solaris, # but without wasting forks for bash or zsh. if test -z "$BASH_VERSION$ZSH_VERSION" \ && (test "X`print -r -- $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='print -r --' as_echo_n='print -rn --' elif (test "X`printf %s $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='printf %s\n' as_echo_n='printf %s' else if test "X`(/usr/ucb/echo -n -n $as_echo) 2>/dev/null`" = "X-n $as_echo"; then as_echo_body='eval /usr/ucb/echo -n "$1$as_nl"' as_echo_n='/usr/ucb/echo -n' else as_echo_body='eval expr "X$1" : "X\\(.*\\)"' as_echo_n_body='eval arg=$1; case $arg in #( *"$as_nl"*) expr "X$arg" : "X\\(.*\\)$as_nl"; arg=`expr "X$arg" : ".*$as_nl\\(.*\\)"`;; esac; expr "X$arg" : "X\\(.*\\)" | tr -d "$as_nl" ' export as_echo_n_body as_echo_n='sh -c $as_echo_n_body as_echo' fi export as_echo_body as_echo='sh -c $as_echo_body as_echo' fi # The user is always right. if test "${PATH_SEPARATOR+set}" != set; then PATH_SEPARATOR=: (PATH='/bin;/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 && { (PATH='/bin:/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 || PATH_SEPARATOR=';' } fi # IFS # We need space, tab and new line, in precisely that order. Quoting is # there to prevent editors from complaining about space-tab. # (If _AS_PATH_WALK were called with IFS unset, it would disable word # splitting by setting IFS to empty value.) IFS=" "" $as_nl" # Find who we are. Look in the path if we contain no directory separator. case $0 in #(( *[\\/]* ) as_myself=$0 ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. test -r "$as_dir/$0" && as_myself=$as_dir/$0 && break done IFS=$as_save_IFS ;; esac # We did not find ourselves, most probably we were run as `sh COMMAND' # in which case we are not to be found in the path. if test "x$as_myself" = x; then as_myself=$0 fi if test ! -f "$as_myself"; then $as_echo "$as_myself: error: cannot find myself; rerun with an absolute file name" >&2 exit 1 fi # Unset variables that we do not need and which cause bugs (e.g. in # pre-3.0 UWIN ksh). But do not cause bugs in bash 2.01; the "|| exit 1" # suppresses any "Segmentation fault" message there. '((' could # trigger a bug in pdksh 5.2.14. for as_var in BASH_ENV ENV MAIL MAILPATH do eval test x\${$as_var+set} = xset \ && ( (unset $as_var) || exit 1) >/dev/null 2>&1 && unset $as_var || : done PS1='$ ' PS2='> ' PS4='+ ' # NLS nuisances. LC_ALL=C export LC_ALL LANGUAGE=C export LANGUAGE # CDPATH. (unset CDPATH) >/dev/null 2>&1 && unset CDPATH # as_fn_error STATUS ERROR [LINENO LOG_FD] # ---------------------------------------- # Output "`basename $0`: error: ERROR" to stderr. If LINENO and LOG_FD are # provided, also output the error to LOG_FD, referencing LINENO. Then exit the # script with STATUS, using 1 if that was 0. as_fn_error () { as_status=$1; test $as_status -eq 0 && as_status=1 if test "$4"; then as_lineno=${as_lineno-"$3"} as_lineno_stack=as_lineno_stack=$as_lineno_stack $as_echo "$as_me:${as_lineno-$LINENO}: error: $2" >&$4 fi $as_echo "$as_me: error: $2" >&2 as_fn_exit $as_status } # as_fn_error # as_fn_set_status STATUS # ----------------------- # Set $? to STATUS, without forking. as_fn_set_status () { return $1 } # as_fn_set_status # as_fn_exit STATUS # ----------------- # Exit the shell with STATUS, even in a "trap 0" or "set -e" context. as_fn_exit () { set +e as_fn_set_status $1 exit $1 } # as_fn_exit # as_fn_unset VAR # --------------- # Portably unset VAR. as_fn_unset () { { eval $1=; unset $1;} } as_unset=as_fn_unset # as_fn_append VAR VALUE # ---------------------- # Append the text in VALUE to the end of the definition contained in VAR. Take # advantage of any shell optimizations that allow amortized linear growth over # repeated appends, instead of the typical quadratic growth present in naive # implementations. if (eval "as_var=1; as_var+=2; test x\$as_var = x12") 2>/dev/null; then : eval 'as_fn_append () { eval $1+=\$2 }' else as_fn_append () { eval $1=\$$1\$2 } fi # as_fn_append # as_fn_arith ARG... # ------------------ # Perform arithmetic evaluation on the ARGs, and store the result in the # global $as_val. Take advantage of shells that can avoid forks. The arguments # must be portable across $(()) and expr. if (eval "test \$(( 1 + 1 )) = 2") 2>/dev/null; then : eval 'as_fn_arith () { as_val=$(( $* )) }' else as_fn_arith () { as_val=`expr "$@" || test $? -eq 1` } fi # as_fn_arith if expr a : '\(a\)' >/dev/null 2>&1 && test "X`expr 00001 : '.*\(...\)'`" = X001; then as_expr=expr else as_expr=false fi if (basename -- /) >/dev/null 2>&1 && test "X`basename -- / 2>&1`" = "X/"; then as_basename=basename else as_basename=false fi if (as_dir=`dirname -- /` && test "X$as_dir" = X/) >/dev/null 2>&1; then as_dirname=dirname else as_dirname=false fi as_me=`$as_basename -- "$0" || $as_expr X/"$0" : '.*/\([^/][^/]*\)/*$' \| \ X"$0" : 'X\(//\)$' \| \ X"$0" : 'X\(/\)' \| . 2>/dev/null || $as_echo X/"$0" | sed '/^.*\/\([^/][^/]*\)\/*$/{ s//\1/ q } /^X\/\(\/\/\)$/{ s//\1/ q } /^X\/\(\/\).*/{ s//\1/ q } s/.*/./; q'` # Avoid depending upon Character Ranges. as_cr_letters='abcdefghijklmnopqrstuvwxyz' as_cr_LETTERS='ABCDEFGHIJKLMNOPQRSTUVWXYZ' as_cr_Letters=$as_cr_letters$as_cr_LETTERS as_cr_digits='0123456789' as_cr_alnum=$as_cr_Letters$as_cr_digits ECHO_C= ECHO_N= ECHO_T= case `echo -n x` in #((((( -n*) case `echo 'xy\c'` in *c*) ECHO_T=' ';; # ECHO_T is single tab character. xy) ECHO_C='\c';; *) echo `echo ksh88 bug on AIX 6.1` > /dev/null ECHO_T=' ';; esac;; *) ECHO_N='-n';; esac rm -f conf$$ conf$$.exe conf$$.file if test -d conf$$.dir; then rm -f conf$$.dir/conf$$.file else rm -f conf$$.dir mkdir conf$$.dir 2>/dev/null fi if (echo >conf$$.file) 2>/dev/null; then if ln -s conf$$.file conf$$ 2>/dev/null; then as_ln_s='ln -s' # ... but there are two gotchas: # 1) On MSYS, both `ln -s file dir' and `ln file dir' fail. # 2) DJGPP < 2.04 has no symlinks; `ln -s' creates a wrapper executable. # In both cases, we have to default to `cp -p'. ln -s conf$$.file conf$$.dir 2>/dev/null && test ! -f conf$$.exe || as_ln_s='cp -p' elif ln conf$$.file conf$$ 2>/dev/null; then as_ln_s=ln else as_ln_s='cp -p' fi else as_ln_s='cp -p' fi rm -f conf$$ conf$$.exe conf$$.dir/conf$$.file conf$$.file rmdir conf$$.dir 2>/dev/null # as_fn_mkdir_p # ------------- # Create "$as_dir" as a directory, including parents if necessary. as_fn_mkdir_p () { case $as_dir in #( -*) as_dir=./$as_dir;; esac test -d "$as_dir" || eval $as_mkdir_p || { as_dirs= while :; do case $as_dir in #( *\'*) as_qdir=`$as_echo "$as_dir" | sed "s/'/'\\\\\\\\''/g"`;; #'( *) as_qdir=$as_dir;; esac as_dirs="'$as_qdir' $as_dirs" as_dir=`$as_dirname -- "$as_dir" || $as_expr X"$as_dir" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$as_dir" : 'X\(//\)[^/]' \| \ X"$as_dir" : 'X\(//\)$' \| \ X"$as_dir" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$as_dir" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` test -d "$as_dir" && break done test -z "$as_dirs" || eval "mkdir $as_dirs" } || test -d "$as_dir" || as_fn_error $? "cannot create directory $as_dir" } # as_fn_mkdir_p if mkdir -p . 2>/dev/null; then as_mkdir_p='mkdir -p "$as_dir"' else test -d ./-p && rmdir ./-p as_mkdir_p=false fi if test -x / >/dev/null 2>&1; then as_test_x='test -x' else if ls -dL / >/dev/null 2>&1; then as_ls_L_option=L else as_ls_L_option= fi as_test_x=' eval sh -c '\'' if test -d "$1"; then test -d "$1/."; else case $1 in #( -*)set "./$1";; esac; case `ls -ld'$as_ls_L_option' "$1" 2>/dev/null` in #(( ???[sx]*):;;*)false;;esac;fi '\'' sh ' fi as_executable_p=$as_test_x # Sed expression to map a string onto a valid CPP name. as_tr_cpp="eval sed 'y%*$as_cr_letters%P$as_cr_LETTERS%;s%[^_$as_cr_alnum]%_%g'" # Sed expression to map a string onto a valid variable name. as_tr_sh="eval sed 'y%*+%pp%;s%[^_$as_cr_alnum]%_%g'" exec 6>&1 ## ----------------------------------- ## ## Main body of $CONFIG_STATUS script. ## ## ----------------------------------- ## _ASEOF test $as_write_fail = 0 && chmod +x $CONFIG_STATUS || ac_write_fail=1 cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # Save the log message, to keep $0 and so on meaningful, and to # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" This file was extended by skytools $as_me 2.1.13, which was generated by GNU Autoconf 2.67. Invocation command line was CONFIG_FILES = $CONFIG_FILES CONFIG_HEADERS = $CONFIG_HEADERS CONFIG_LINKS = $CONFIG_LINKS CONFIG_COMMANDS = $CONFIG_COMMANDS $ $0 $@ on `(hostname || uname -n) 2>/dev/null | sed 1q` " _ACEOF case $ac_config_files in *" "*) set x $ac_config_files; shift; ac_config_files=$*;; esac cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 # Files that config.status was made for. config_files="$ac_config_files" _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 ac_cs_usage="\ \`$as_me' instantiates files and other configuration actions from templates according to the current configuration. Unless the files and actions are specified as TAGs, all are instantiated by default. Usage: $0 [OPTION]... [TAG]... -h, --help print this help, then exit -V, --version print version number and configuration settings, then exit --config print configuration, then exit -q, --quiet, --silent do not print progress messages -d, --debug don't remove temporary files --recheck update $as_me by reconfiguring in the same conditions --file=FILE[:TEMPLATE] instantiate the configuration file FILE Configuration files: $config_files Report bugs to the package provider." _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ skytools config.status 2.1.13 configured by $0, generated by GNU Autoconf 2.67, with options \\"\$ac_cs_config\\" Copyright (C) 2010 Free Software Foundation, Inc. This config.status script is free software; the Free Software Foundation gives unlimited permission to copy, distribute and modify it." ac_pwd='$ac_pwd' srcdir='$srcdir' test -n "\$AWK" || AWK=awk _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # The default lists apply if the user does not specify any file. ac_need_defaults=: while test $# != 0 do case $1 in --*=?*) ac_option=`expr "X$1" : 'X\([^=]*\)='` ac_optarg=`expr "X$1" : 'X[^=]*=\(.*\)'` ac_shift=: ;; --*=) ac_option=`expr "X$1" : 'X\([^=]*\)='` ac_optarg= ac_shift=: ;; *) ac_option=$1 ac_optarg=$2 ac_shift=shift ;; esac case $ac_option in # Handling of the options. -recheck | --recheck | --rechec | --reche | --rech | --rec | --re | --r) ac_cs_recheck=: ;; --version | --versio | --versi | --vers | --ver | --ve | --v | -V ) $as_echo "$ac_cs_version"; exit ;; --config | --confi | --conf | --con | --co | --c ) $as_echo "$ac_cs_config"; exit ;; --debug | --debu | --deb | --de | --d | -d ) debug=: ;; --file | --fil | --fi | --f ) $ac_shift case $ac_optarg in *\'*) ac_optarg=`$as_echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"` ;; '') as_fn_error $? "missing file argument" ;; esac as_fn_append CONFIG_FILES " '$ac_optarg'" ac_need_defaults=false;; --he | --h | --help | --hel | -h ) $as_echo "$ac_cs_usage"; exit ;; -q | -quiet | --quiet | --quie | --qui | --qu | --q \ | -silent | --silent | --silen | --sile | --sil | --si | --s) ac_cs_silent=: ;; # This is an error. -*) as_fn_error $? "unrecognized option: \`$1' Try \`$0 --help' for more information." ;; *) as_fn_append ac_config_targets " $1" ac_need_defaults=false ;; esac shift done ac_configure_extra_args= if $ac_cs_silent; then exec 6>/dev/null ac_configure_extra_args="$ac_configure_extra_args --silent" fi _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 if \$ac_cs_recheck; then set X '$SHELL' '$0' $ac_configure_args \$ac_configure_extra_args --no-create --no-recursion shift \$as_echo "running CONFIG_SHELL=$SHELL \$*" >&6 CONFIG_SHELL='$SHELL' export CONFIG_SHELL exec "\$@" fi _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 exec 5>>config.log { echo sed 'h;s/./-/g;s/^.../## /;s/...$/ ##/;p;x;p;x' <<_ASBOX ## Running $as_me. ## _ASBOX $as_echo "$ac_log" } >&5 _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # Handling of arguments. for ac_config_target in $ac_config_targets do case $ac_config_target in "config.mak") CONFIG_FILES="$CONFIG_FILES config.mak" ;; *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5 ;; esac done # If the user did not use the arguments to specify the items to instantiate, # then the envvar interface is used. Set only those that are not. # We use the long form for the default assignment because of an extremely # bizarre bug on SunOS 4.1.3. if $ac_need_defaults; then test "${CONFIG_FILES+set}" = set || CONFIG_FILES=$config_files fi # Have a temporary directory for convenience. Make it in the build tree # simply because there is no reason against having it here, and in addition, # creating and moving files from /tmp can sometimes cause problems. # Hook for its removal unless debugging. # Note that there is a small window in which the directory will not be cleaned: # after its creation but before its name has been assigned to `$tmp'. $debug || { tmp= trap 'exit_status=$? { test -z "$tmp" || test ! -d "$tmp" || rm -fr "$tmp"; } && exit $exit_status ' 0 trap 'as_fn_exit 1' 1 2 13 15 } # Create a (secure) tmp directory for tmp files. { tmp=`(umask 077 && mktemp -d "./confXXXXXX") 2>/dev/null` && test -n "$tmp" && test -d "$tmp" } || { tmp=./conf$$-$RANDOM (umask 077 && mkdir "$tmp") } || as_fn_error $? "cannot create a temporary directory in ." "$LINENO" 5 # Set up the scripts for CONFIG_FILES section. # No need to generate them if there are no CONFIG_FILES. # This happens for instance with `./config.status config.h'. if test -n "$CONFIG_FILES"; then ac_cr=`echo X | tr X '\015'` # On cygwin, bash can eat \r inside `` if the user requested igncr. # But we know of no other shell where ac_cr would be empty at this # point, so we can use a bashism as a fallback. if test "x$ac_cr" = x; then eval ac_cr=\$\'\\r\' fi ac_cs_awk_cr=`$AWK 'BEGIN { print "a\rb" }' /dev/null` if test "$ac_cs_awk_cr" = "a${ac_cr}b"; then ac_cs_awk_cr='\\r' else ac_cs_awk_cr=$ac_cr fi echo 'BEGIN {' >"$tmp/subs1.awk" && _ACEOF { echo "cat >conf$$subs.awk <<_ACEOF" && echo "$ac_subst_vars" | sed 's/.*/&!$&$ac_delim/' && echo "_ACEOF" } >conf$$subs.sh || as_fn_error $? "could not make $CONFIG_STATUS" "$LINENO" 5 ac_delim_num=`echo "$ac_subst_vars" | grep -c '^'` ac_delim='%!_!# ' for ac_last_try in false false false false false :; do . ./conf$$subs.sh || as_fn_error $? "could not make $CONFIG_STATUS" "$LINENO" 5 ac_delim_n=`sed -n "s/.*$ac_delim\$/X/p" conf$$subs.awk | grep -c X` if test $ac_delim_n = $ac_delim_num; then break elif $ac_last_try; then as_fn_error $? "could not make $CONFIG_STATUS" "$LINENO" 5 else ac_delim="$ac_delim!$ac_delim _$ac_delim!! " fi done rm -f conf$$subs.sh cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>"\$tmp/subs1.awk" <<\\_ACAWK && _ACEOF sed -n ' h s/^/S["/; s/!.*/"]=/ p g s/^[^!]*!// :repl t repl s/'"$ac_delim"'$// t delim :nl h s/\(.\{148\}\)..*/\1/ t more1 s/["\\]/\\&/g; s/^/"/; s/$/\\n"\\/ p n b repl :more1 s/["\\]/\\&/g; s/^/"/; s/$/"\\/ p g s/.\{148\}// t nl :delim h s/\(.\{148\}\)..*/\1/ t more2 s/["\\]/\\&/g; s/^/"/; s/$/"/ p b :more2 s/["\\]/\\&/g; s/^/"/; s/$/"\\/ p g s/.\{148\}// t delim ' >$CONFIG_STATUS || ac_write_fail=1 rm -f conf$$subs.awk cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 _ACAWK cat >>"\$tmp/subs1.awk" <<_ACAWK && for (key in S) S_is_set[key] = 1 FS = "" } { line = $ 0 nfields = split(line, field, "@") substed = 0 len = length(field[1]) for (i = 2; i < nfields; i++) { key = field[i] keylen = length(key) if (S_is_set[key]) { value = S[key] line = substr(line, 1, len) "" value "" substr(line, len + keylen + 3) len += length(value) + length(field[++i]) substed = 1 } else len += 1 + keylen } print line } _ACAWK _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 if sed "s/$ac_cr//" < /dev/null > /dev/null 2>&1; then sed "s/$ac_cr\$//; s/$ac_cr/$ac_cs_awk_cr/g" else cat fi < "$tmp/subs1.awk" > "$tmp/subs.awk" \ || as_fn_error $? "could not setup config files machinery" "$LINENO" 5 _ACEOF # VPATH may cause trouble with some makes, so we remove sole $(srcdir), # ${srcdir} and @srcdir@ entries from VPATH if srcdir is ".", strip leading and # trailing colons and then remove the whole line if VPATH becomes empty # (actually we leave an empty line to preserve line numbers). if test "x$srcdir" = x.; then ac_vpsub='/^[ ]*VPATH[ ]*=[ ]*/{ h s/// s/^/:/ s/[ ]*$/:/ s/:\$(srcdir):/:/g s/:\${srcdir}:/:/g s/:@srcdir@:/:/g s/^:*// s/:*$// x s/\(=[ ]*\).*/\1/ G s/\n// s/^[^=]*=[ ]*$// }' fi cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 fi # test -n "$CONFIG_FILES" eval set X " :F $CONFIG_FILES " shift for ac_tag do case $ac_tag in :[FHLC]) ac_mode=$ac_tag; continue;; esac case $ac_mode$ac_tag in :[FHL]*:*);; :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5 ;; :[FH]-) ac_tag=-:-;; :[FH]*) ac_tag=$ac_tag:$ac_tag.in;; esac ac_save_IFS=$IFS IFS=: set x $ac_tag IFS=$ac_save_IFS shift ac_file=$1 shift case $ac_mode in :L) ac_source=$1;; :[FH]) ac_file_inputs= for ac_f do case $ac_f in -) ac_f="$tmp/stdin";; *) # Look for the file first in the build tree, then in the source tree # (if the path is not absolute). The absolute path cannot be DOS-style, # because $ac_f cannot contain `:'. test -f "$ac_f" || case $ac_f in [\\/$]*) false;; *) test -f "$srcdir/$ac_f" && ac_f="$srcdir/$ac_f";; esac || as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5 ;; esac case $ac_f in *\'*) ac_f=`$as_echo "$ac_f" | sed "s/'/'\\\\\\\\''/g"`;; esac as_fn_append ac_file_inputs " '$ac_f'" done # Let's still pretend it is `configure' which instantiates (i.e., don't # use $as_me), people would be surprised to read: # /* config.h. Generated by config.status. */ configure_input='Generated from '` $as_echo "$*" | sed 's|^[^:]*/||;s|:[^:]*/|, |g' `' by configure.' if test x"$ac_file" != x-; then configure_input="$ac_file. $configure_input" { $as_echo "$as_me:${as_lineno-$LINENO}: creating $ac_file" >&5 $as_echo "$as_me: creating $ac_file" >&6;} fi # Neutralize special characters interpreted by sed in replacement strings. case $configure_input in #( *\&* | *\|* | *\\* ) ac_sed_conf_input=`$as_echo "$configure_input" | sed 's/[\\\\&|]/\\\\&/g'`;; #( *) ac_sed_conf_input=$configure_input;; esac case $ac_tag in *:-:* | *:-) cat >"$tmp/stdin" \ || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;; esac ;; esac ac_dir=`$as_dirname -- "$ac_file" || $as_expr X"$ac_file" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$ac_file" : 'X\(//\)[^/]' \| \ X"$ac_file" : 'X\(//\)$' \| \ X"$ac_file" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$ac_file" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` as_dir="$ac_dir"; as_fn_mkdir_p ac_builddir=. case "$ac_dir" in .) ac_dir_suffix= ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_dir_suffix=/`$as_echo "$ac_dir" | sed 's|^\.[\\/]||'` # A ".." for each directory in $ac_dir_suffix. ac_top_builddir_sub=`$as_echo "$ac_dir_suffix" | sed 's|/[^\\/]*|/..|g;s|/||'` case $ac_top_builddir_sub in "") ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_top_build_prefix=$ac_top_builddir_sub/ ;; esac ;; esac ac_abs_top_builddir=$ac_pwd ac_abs_builddir=$ac_pwd$ac_dir_suffix # for backward compatibility: ac_top_builddir=$ac_top_build_prefix case $srcdir in .) # We are building in place. ac_srcdir=. ac_top_srcdir=$ac_top_builddir_sub ac_abs_top_srcdir=$ac_pwd ;; [\\/]* | ?:[\\/]* ) # Absolute name. ac_srcdir=$srcdir$ac_dir_suffix; ac_top_srcdir=$srcdir ac_abs_top_srcdir=$srcdir ;; *) # Relative name. ac_srcdir=$ac_top_build_prefix$srcdir$ac_dir_suffix ac_top_srcdir=$ac_top_build_prefix$srcdir ac_abs_top_srcdir=$ac_pwd/$srcdir ;; esac ac_abs_srcdir=$ac_abs_top_srcdir$ac_dir_suffix case $ac_mode in :F) # # CONFIG_FILE # _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # If the template does not know about datarootdir, expand it. # FIXME: This hack should be removed a few years after 2.60. ac_datarootdir_hack=; ac_datarootdir_seen= ac_sed_dataroot=' /datarootdir/ { p q } /@datadir@/p /@docdir@/p /@infodir@/p /@localedir@/p /@mandir@/p' case `eval "sed -n \"\$ac_sed_dataroot\" $ac_file_inputs"` in *datarootdir*) ac_datarootdir_seen=yes;; *@datadir@*|*@docdir@*|*@infodir@*|*@localedir@*|*@mandir@*) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file_inputs seems to ignore the --datarootdir setting" >&5 $as_echo "$as_me: WARNING: $ac_file_inputs seems to ignore the --datarootdir setting" >&2;} _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_datarootdir_hack=' s&@datadir@&$datadir&g s&@docdir@&$docdir&g s&@infodir@&$infodir&g s&@localedir@&$localedir&g s&@mandir@&$mandir&g s&\\\${datarootdir}&$datarootdir&g' ;; esac _ACEOF # Neutralize VPATH when `$srcdir' = `.'. # Shell code in configure.ac might set extrasub. # FIXME: do we really want to maintain this feature? cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_sed_extra="$ac_vpsub $extrasub _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 :t /@[a-zA-Z_][a-zA-Z_0-9]*@/!b s|@configure_input@|$ac_sed_conf_input|;t t s&@top_builddir@&$ac_top_builddir_sub&;t t s&@top_build_prefix@&$ac_top_build_prefix&;t t s&@srcdir@&$ac_srcdir&;t t s&@abs_srcdir@&$ac_abs_srcdir&;t t s&@top_srcdir@&$ac_top_srcdir&;t t s&@abs_top_srcdir@&$ac_abs_top_srcdir&;t t s&@builddir@&$ac_builddir&;t t s&@abs_builddir@&$ac_abs_builddir&;t t s&@abs_top_builddir@&$ac_abs_top_builddir&;t t $ac_datarootdir_hack " eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$tmp/subs.awk" >$tmp/out \ || as_fn_error $? "could not create $ac_file" "$LINENO" 5 test -z "$ac_datarootdir_hack$ac_datarootdir_seen" && { ac_out=`sed -n '/\${datarootdir}/p' "$tmp/out"`; test -n "$ac_out"; } && { ac_out=`sed -n '/^[ ]*datarootdir[ ]*:*=/p' "$tmp/out"`; test -z "$ac_out"; } && { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file contains a reference to the variable \`datarootdir' which seems to be undefined. Please make sure it is defined" >&5 $as_echo "$as_me: WARNING: $ac_file contains a reference to the variable \`datarootdir' which seems to be undefined. Please make sure it is defined" >&2;} rm -f "$tmp/stdin" case $ac_file in -) cat "$tmp/out" && rm -f "$tmp/out";; *) rm -f "$ac_file" && mv "$tmp/out" "$ac_file";; esac \ || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;; esac done # for ac_tag as_fn_exit 0 _ACEOF ac_clean_files=$ac_clean_files_save test $ac_write_fail = 0 || as_fn_error $? "write failure creating $CONFIG_STATUS" "$LINENO" 5 # configure is writing to config.log, and then calls config.status. # config.status does its own redirection, appending to config.log. # Unfortunately, on DOS this fails, as config.log is still kept open # by configure, so config.status won't be able to write to it; its # output is simply discarded. So we exec the FD to /dev/null, # effectively closing config.log, so it can be properly (re)opened and # appended to by config.status. When coming back to configure, we # need to make the FD available again. if test "$no_create" != yes; then ac_cs_success=: ac_config_status_args= test "$silent" = yes && ac_config_status_args="$ac_config_status_args --quiet" exec 5>/dev/null $SHELL $CONFIG_STATUS $ac_config_status_args || ac_cs_success=false exec 5>>config.log # Use ||, not &&, to avoid exiting from the if with $? = 1, which # would make configure fail if this is the last instruction. $ac_cs_success || as_fn_exit 1 fi if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: unrecognized options: $ac_unrecognized_opts" >&5 $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;} fi skytools-2.1.13/source.cfg0000644000175000017500000000120311670174255014456 0ustar markomarko# what to include in source distribution # MANIFEST.in for Python Distutils include Makefile COPYRIGHT README NEWS config.mak.in configure configure.ac source.cfg recursive-include sql *.sql Makefile *.out *.in *.[ch] README* *.in recursive-include python/conf *.ini recursive-include scripts *.py *.templ recursive-include debian changelog packages.in recursive-include doc Makefile *.py *.txt *.[1-9] include python/skytools/installer_config.py.in prune python/skytools/installer_config.py recursive-include upgrade *.sql Makefile recursive-include tests *.conf *.sh *.ini *.py Makefile data.sql install.sql v2*.sql *.conf prune fix*.sql skytools-2.1.13/Makefile0000644000175000017500000000701011670174255014137 0ustar markomarko -include config.mak PYTHON ?= python pyver = $(shell $(PYTHON) -V 2>&1 | sed 's/^[^ ]* \([0-9]*\.[0-9]*\).*/\1/') SUBDIRS = sql doc all: python-all modules-all modules-all: config.mak $(MAKE) -C sql all python-all: config.mak $(PYTHON) setup.py build clean: $(PYTHON) setup.py clean $(MAKE) -C sql clean $(MAKE) -C doc clean rm -rf build find python -name '*.py[oc]' -print | xargs rm -f rm -f python/skytools/installer_config.py rm -rf tests/londiste/sys rm -rf tests/londiste/file_logs rm -rf tests/londiste/fix.* rm -rf tests/scripts/sys install: python-install modules-install installcheck: $(MAKE) -C sql installcheck modules-install: config.mak $(MAKE) -C sql install DESTDIR=$(DESTDIR) test \! -d compat || $(MAKE) -C compat $@ DESTDIR=$(DESTDIR) python-install: config.mak modules-all $(PYTHON) setup.py install --prefix=$(prefix) --root=$(DESTDIR)/ $(BROKEN_PYTHON) $(MAKE) -C doc DESTDIR=$(DESTDIR) install python-install python-all: python/skytools/installer_config.py python/skytools/installer_config.py: python/skytools/installer_config.py.in config.mak sed -e 's!@SQLDIR@!$(SQLDIR)!g' -e 's!@PACKAGE_VERSION@!$(PACKAGE_VERSION)!g' $< > $@ realclean: $(MAKE) -C doc $@ $(MAKE) distclean distclean: clean for dir in $(SUBDIRS); do $(MAKE) -C $$dir $@ || exit 1; done $(MAKE) -C doc $@ rm -rf source.list dist skytools-* find python -name '*.pyc' | xargs rm -f rm -rf dist build rm -rf autom4te.cache config.log config.status config.mak deb80: ./configure --with-pgconfig=/usr/lib/postgresql/8.0/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/8.0/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b deb81: ./configure --with-pgconfig=/usr/lib/postgresql/8.1/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/8.1/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b deb82: ./configure --with-pgconfig=/usr/lib/postgresql/8.2/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/8.2/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b deb83: ./configure --with-pgconfig=/usr/lib/postgresql/8.3/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/8.3/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b deb84: ./configure --with-pgconfig=/usr/lib/postgresql/8.4/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/8.4/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b deb90: ./configure --with-pgconfig=/usr/lib/postgresql/9.0/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/9.0/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b deb91: ./configure --with-pgconfig=/usr/lib/postgresql/9.1/bin/pg_config --with-python=$(PYTHON) sed -e s/PGVER/9.1/g -e s/PYVER/$(pyver)/g < debian/packages.in > debian/packages yada rebuild debuild -uc -us -b tgz: config.mak clean $(MAKE) -C doc man $(PYTHON) setup.py sdist -t source.cfg -m source.list debclean: distclean rm -rf debian/tmp-* debian/build* debian/control debian/packages-tmp* rm -f debian/files debian/rules debian/sub* debian/packages boot: configure configure: configure.ac autoconf # workaround for Debian's broken python debfix: $(PYTHON) setup.py install --help | grep -q install-layout \ && echo BROKEN_PYTHON=--install-layout=deb || echo 'WORKING_PYTHON=found' .PHONY: all clean distclean install deb debclean tgz .PHONY: python-all python-clean python-install skytools-2.1.13/COPYRIGHT0000644000175000017500000000143211670174255013774 0ustar markomarkoSkyTools - tool collection for PostgreSQL Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. skytools-2.1.13/README0000644000175000017500000000265611670174255013372 0ustar markomarko= SkyTools - tools for PostgreSQL = This is a package of tools in use in Skype for replication and failover. Also it includes a generic queuing mechanism PgQ and utility library for Python scripts. == Overview == It contains following tools: === PgQ === This is the queue machanism we use. Consists of PL/pgsql, and C code in database, with Python framework on top of it. It is based on snapshot based event handling ideas from Slony-I, written for general usage. Features: - There can be several queues in database. - There can be several producers than can insert into any queue. - There can be several consumers on one queue and all consumers see all events. Documentation: - PgQ admin tool (pgqadm) usage: doc/pgq-admin.txt - PgQ SQL API overview: doc/pgq-sql.txt - PgQ SQL reference: http://skytools.projects.postgresql.org/pgq/ === Londiste === Replication tool written in Python, using PgQ as event transport. Features: - Tables can be added one-by-one into set. - Initial COPY for one table does not block event replay for other tables. - Can compare tables on both sides. Documentation: - Londiste script usage: doc/londiste.cmdline.txt (also available as `man 1 londiste`) - Londiste configuration: doc/londiste.config.txt (also available as `man 5 londiste`) - Londiste reference: doc/londiste.ref.txt === walmgr === This script will setup WAL archiving, does initial backup and runtime WAL archive and restore. skytools-2.1.13/configure.ac0000644000175000017500000000473711727404444015001 0ustar markomarkodnl Process this file with autoconf to produce a configure script. AC_INIT(skytools, 2.1.13) AC_CONFIG_SRCDIR(python/pgqadm.py) dnl Find Python interpreter AC_ARG_WITH(python, [ --with-python=PYTHON name of the Python executable (default: python)], [ AC_MSG_CHECKING(for python) PYTHON=$withval AC_MSG_RESULT($PYTHON)], [ AC_PATH_PROGS(PYTHON, python) ]) test -n "$PYTHON" || AC_MSG_ERROR([Cannot continue without Python]) dnl Find PostgreSQL pg_config AC_ARG_WITH(pgconfig, [ --with-pgconfig=PG_CONFIG path to pg_config (default: pg_config)], [ AC_MSG_CHECKING(for pg_config) PG_CONFIG=$withval AC_MSG_RESULT($PG_CONFIG)], [ AC_PATH_PROGS(PG_CONFIG, pg_config) ]) test -n "$PG_CONFIG" || AC_MSG_ERROR([Cannot continue without pg_config]) dnl Find GNU make AC_MSG_CHECKING(for GNU make) if test ! -n "$MAKE"; then for a in make gmake gnumake; do if "$a" --version 2>&1 | grep GNU > /dev/null; then MAKE="$a" break fi done fi if test -n "$MAKE"; then AC_MSG_RESULT($MAKE) else AC_MSG_ERROR([GNU make is not found]) fi AC_SUBST(MAKE) dnl asciidoc >= 8.2 AC_ARG_WITH(asciidoc, [ --with-asciidoc[[=prog]] path to asciidoc 8.2 (default: asciidoc)], [ if test "$withval" = "yes"; then AC_CHECK_PROGS(ASCIIDOC, [$ASCIIDOC asciidoc]) test -n "$ASCIIDOC" || ASCIIDOC=no else AC_MSG_CHECKING(for asciidoc) ASCIIDOC=$withval AC_MSG_RESULT($ASCIIDOC) fi ], [ ASCIIDOC="no" ]) if test "$ASCIIDOC" != "no"; then AC_MSG_CHECKING([whether asciidoc version >= 8.2]) ver=`$ASCIIDOC --version 2>&1 | sed -e 's/asciidoc //'` case "$ver" in dnl hack to make possible to use [, ] in regex changequote({, })dnl [0-7].*|8.[01]|8.[01].*) changequote([, ])dnl AC_MSG_RESULT([$ver, too old]) ASCIIDOC="no" ;; *) AC_MSG_RESULT([$ver, ok]) ;; esac fi dnl check for xmlto, but only if asciidoc is found if test "$ASCIIDOC" != "no"; then AC_CHECK_PROGS(XMLTO, [$XMLTO xmlto]) test -n "$XMLTO" || XMLTO=no else XMLTO="no" fi # when in cvs/git tree, turn asciidoc/xmlto unconditionally on if test -d .git -o -d CVS; then if test "$ASCIIDOC" = "no"; then echo "*** Building from CVS/GIT requires asciidoc, enabling it ***" ASCIIDOC="asciidoc" fi if test "$XMLTO" = "no"; then echo "*** Building from CVS/GIT requires xmlto, enabling it ***" XMLTO="xmlto" fi fi dnl Postres headers on Solaris define incompat unsetenv without that AC_CHECK_FUNCS(unsetenv) dnl Write result AC_CONFIG_FILES([config.mak]) AC_OUTPUT skytools-2.1.13/python/0000755000175000017500000000000011727601174014020 5ustar markomarkoskytools-2.1.13/python/londiste.py0000755000175000017500000001176711670174255016234 0ustar markomarko#! /usr/bin/env python """Londiste launcher. """ import sys, os, optparse, signal, skytools # python 2.3 will try londiste.py first... import sys, os.path if os.path.exists(os.path.join(sys.path[0], 'londiste.py')) \ and not os.path.isdir(os.path.join(sys.path[0], 'londiste')): del sys.path[0] from londiste import * __all__ = ['Londiste'] command_usage = """ %prog [options] INI CMD [subcmd args] commands: replay replay events to subscriber provider install installs modules, creates queue provider add TBL ... add table to queue provider remove TBL ... remove table from queue provider tables show all tables on provider provider add-seq SEQ ... add sequence to provider provider remove-seq SEQ ... remove sequence from provider provider seqs show all sequences on provider subscriber install installs schema subscriber add TBL ... add table to subscriber subscriber remove TBL ... remove table from subscriber subscriber add-seq SEQ ... add table to subscriber subscriber remove-seq SEQ ... remove table from subscriber subscriber tables list tables subscriber has attached to subscriber seqs list sequences subscriber is interested subscriber missing list tables subscriber has not yet attached to subscriber check compare table structure on both sides subscriber resync TBL ... do full copy again subscriber fkeys [pending|active] show fkeys on tables subscriber triggers [pending|active] show triggers on tables subscriber restore-triggers TBL [TGNAME ..] restore pending triggers subscriber register register consumer on provider's queue subscriber unregister unregister consumer on provider's queue compare [TBL ...] compare table contents on both sides repair [TBL ...] repair data on subscriber copy [internal command - copy table logic] """ class Londiste(skytools.DBScript): def __init__(self, args): skytools.DBScript.__init__(self, 'londiste', args) if self.options.rewind or self.options.reset: self.script = Replicator(args) return if len(self.args) < 2: print "need command" sys.exit(1) cmd = self.args[1] if cmd =="provider": script = ProviderSetup(args) elif cmd == "subscriber": script = SubscriberSetup(args) elif cmd == "replay": method = self.cf.get('method', 'direct') if method == 'direct': script = Replicator(args) elif method == 'file_write': script = FileWrite(args) elif method == 'file_write': script = FileWrite(args) else: print "unknown method, quitting" sys.exit(1) elif cmd == "copy": script = CopyTable(args) elif cmd == "compare": script = Comparator(args) elif cmd == "repair": script = Repairer(args) elif cmd == "upgrade": script = UpgradeV2(args) else: print "Unknown command '%s', use --help for help" % cmd sys.exit(1) self.script = script def start(self): self.script.start() def init_optparse(self, parser=None): p = skytools.DBScript.init_optparse(self, parser) p.set_usage(command_usage.strip()) g = optparse.OptionGroup(p, "expert options") g.add_option("--all", action="store_true", help = "add: include all possible tables") g.add_option("--force", action="store_true", help = "add: ignore table differences, repair: ignore lag") g.add_option("--expect-sync", action="store_true", dest="expect_sync", help = "add: no copy needed", default=False) g.add_option("--skip-truncate", action="store_true", dest="skip_truncate", help = "add: keep old data", default=False) g.add_option("--rewind", action="store_true", help = "replay: sync queue pos with subscriber") g.add_option("--reset", action="store_true", help = "replay: forget queue pos on subscriber") p.add_option_group(g) return p def send_signal(self, sig): """ Londiste can launch other process for copy, so manages it here """ if sig in (signal.SIGTERM, signal.SIGINT): # kill copy process if it exists before stopping copy_pidfile = self.pidfile + ".copy" if os.path.isfile(copy_pidfile): self.log.info("Signaling running COPY first") skytools.signal_pidfile(copy_pidfile, signal.SIGTERM) # now resort to DBScript send_signal() skytools.DBScript.send_signal(self, sig) if __name__ == '__main__': script = Londiste(sys.argv[1:]) script.start() skytools-2.1.13/python/walmgr.py0000755000175000017500000020551511670174255015700 0ustar markomarko#! /usr/bin/env python """WALShipping manager. walmgr INI COMMAND [-n] Master commands: setup Configure PostgreSQL for WAL archiving sync Copies in-progress WALs to slave syncdaemon Daemon mode for regular syncing stop Stop archiving - de-configure PostgreSQL periodic Run periodic command if configured. Slave commands: boot Stop playback, accept queries pause Just wait, don't play WAL-s continue Start playing WAL-s again Common commands: listbackups List backups. backup Copies all master data to slave. Will keep backup history if slave keep_backups is set. EXPERIMENTAL: If run on slave, creates backup from in-recovery slave data. restore [set][dst] Stop postmaster, move new data dir to right location and start postmaster in playback mode. Optionally use [set] as the backupset name to restore. In this case the directory is copied, not moved. Internal commands: xarchive archive one WAL file (master) xrestore restore one WAL file (slave) xlock Obtain backup lock (master) xrelease Release backup lock (master) xrotate Rotate backup sets, expire and archive oldest if necessary. xpurgewals Remove WAL files not needed for backup (slave) Switches: -n no action, just print commands """ import os, sys, re, signal, time, traceback import errno, glob, ConfigParser, shutil, subprocess import skytools MASTER = 1 SLAVE = 0 XLOG_SEGMENT_SIZE = 16 * 1024**2 def usage(err): if err > 0: print >>sys.stderr, __doc__ else: print __doc__ sys.exit(err) def die(err,msg): print >> sys.stderr, msg sys.exit(err) def yesno(prompt): """Ask a Yes/No question""" while True: sys.stderr.write(prompt + " ") sys.stderr.flush() answer = sys.stdin.readline() if not answer: return False answer = answer.strip().lower() if answer in ('yes','y'): return True if answer in ('no','n'): return False sys.stderr.write("Please answer yes or no.\n") def copy_conf(src, dst): """Copy config file or symlink. Does _not_ overwrite target. """ if os.path.isdir(dst): dst = os.path.join(dst, os.path.basename(src)) if os.path.exists(dst): return False if os.path.islink(src): linkdst = os.readlink(src) os.symlink(linkdst, dst) elif os.path.isfile(src): shutil.copy2(src, dst) else: raise Exception("Unsupported file type: %s" % src) return True class WalChunk: """Represents a chunk of WAL used in record based shipping""" def __init__(self,filename,pos=0,bytes=0): self.filename = filename self.pos = pos self.bytes = bytes self.start_time = time.time() self.sync_count = 0 self.sync_time = 0.0 def __str__(self): return "%s @ %d +%d" % (self.filename, self.pos, self.bytes) class PgControlData: """Contents of pg_controldata""" def __init__(self, bin_dir, data_dir, findRestartPoint): """Collect last checkpoint information from pg_controldata output""" self.xlogid = None self.xrecoff = None self.timeline = None self.wal_size = None self.wal_name = None self.cluster_state = None self.is_shutdown = False self.pg_version = 0 self.is_valid = False try: pg_controldata = os.path.join(bin_dir, "pg_controldata") pipe = subprocess.Popen([ pg_controldata, data_dir ], stdout=subprocess.PIPE) except OSError: # don't complain if we cannot execute it return matches = 0 for line in pipe.stdout.readlines(): if findRestartPoint: m = re.match("^Latest checkpoint's REDO location:\s+([0-9A-F]+)/([0-9A-F]+)", line) else: m = re.match("^Latest checkpoint location:\s+([0-9A-F]+)/([0-9A-F]+)", line) if m: matches += 1 self.xlogid = int(m.group(1), 16) self.xrecoff = int(m.group(2), 16) m = re.match("^Latest checkpoint's TimeLineID:\s+(\d+)", line) if m: matches += 1 self.timeline = int(m.group(1)) m = re.match("^Bytes per WAL segment:\s+(\d+)", line) if m: matches += 1 self.wal_size = int(m.group(1)) m = re.match("^pg_control version number:\s+(\d+)", line) if m: matches += 1 self.pg_version = int(m.group(1)) m = re.match("^Database cluster state:\s+(.*$)", line) if m: matches += 1 self.cluster_state = m.group(1) self.is_shutdown = (self.cluster_state == "shut down") # ran successfully and we got our needed matches if pipe.wait() == 0 and matches == 5: self.wal_name = "%08X%08X%08X" % \ (self.timeline, self.xlogid, self.xrecoff / self.wal_size) self.is_valid = True class BackupLabel: """Backup label contents""" def __init__(self, backupdir): """Initialize a new BackupLabel from existing file""" filename = os.path.join(backupdir, "backup_label") self.first_wal = None self.start_time = None self.label_string = None if not os.path.exists(filename): return for line in open(filename): m = re.match('^START WAL LOCATION: [^\s]+ \(file ([0-9A-Z]+)\)$', line) if m: self.first_wal = m.group(1) m = re.match('^START TIME:\s(.*)$', line) if m: self.start_time = m.group(1) m = re.match('^LABEL: (.*)$', line) if m: self.label_string = m.group(1) class PostgresConfiguration: """Postgres configuration manipulation""" def __init__(self, walmgr, cf_file): """load the configuration from master_config""" self.walmgr = walmgr self.log = walmgr.log self.cf_file = cf_file self.cf_buf = open(self.cf_file, "r").read() def archive_mode(self): """Return value for specified parameter""" # see if explicitly set m = re.search("^\s*archive_mode\s*=\s*'?([a-zA-Z01]+)'?\s*#?.*$", self.cf_buf, re.M | re.I) if m: return m.group(1) # also, it could be commented out as initdb leaves it # it'd probably be best to check from the database ... m = re.search("^#archive_mode\s*=.*$", self.cf_buf, re.M | re.I) if m: return "off" return None def wal_level(self): """Return value for specified parameter""" # see if explicitly set m = re.search("^\s*wal_level\s*=\s*'?([a-z_]+)'?\s*#?.*$", self.cf_buf, re.M | re.I) if m: return m.group(1) # also, it could be commented out as initdb leaves it # it'd probably be best to check from the database ... m = re.search("^#wal_level\s*=.*$", self.cf_buf, re.M | re.I) if m: return "minimal" return None def modify(self, cf_params): """Change the configuration parameters supplied in cf_params""" for (param, value) in cf_params.iteritems(): r_active = re.compile("^\s*%s\s*=\s*([^\s#]*).*$" % param, re.M) r_disabled = re.compile("^\s*#\s*%s\s*=.*$" % param, re.M) cf_full = "%s = '%s'" % (param, value) m = r_active.search(self.cf_buf) if m: old_val = m.group(1) self.log.debug("found parameter %s with value '%s'" % (param, old_val)) self.cf_buf = "%s%s%s" % (self.cf_buf[:m.start()], cf_full, self.cf_buf[m.end():]) else: m = r_disabled.search(self.cf_buf) if m: self.log.debug("found disabled parameter %s" % param) self.cf_buf = "%s\n%s%s" % (self.cf_buf[:m.end()], cf_full, self.cf_buf[m.end():]) else: # not found, append to the end self.log.debug("found no value") self.cf_buf = "%s\n%s\n\n" % (self.cf_buf, cf_full) def write(self): """Write the configuration back to file""" cf_old = self.cf_file + ".old" cf_new = self.cf_file + ".new" if self.walmgr.not_really: cf_new = "/tmp/postgresql.conf.new" open(cf_new, "w").write(self.cf_buf) self.log.info("Showing diff") os.system("diff -u %s %s" % (self.cf_file, cf_new)) self.log.info("Done diff") os.remove(cf_new) return # polite method does not work, as usually not enough perms for it open(self.cf_file, "w").write(self.cf_buf) class WalMgr(skytools.DBScript): def init_optparse(self, parser=None): p = skytools.DBScript.init_optparse(self, parser) p.set_usage(__doc__.strip()) p.add_option("-n", "--not-really", action="store_true", dest="not_really", help = "Don't actually do anything.", default=False) return p def __init__(self, args): if len(args) == 1 and args[0] == '--version': skytools.DBScript.__init__(self, 'wal-master', args) if len(args) < 2: # need at least config file and command usage(1) # determine the role of the node from provided configuration cf = ConfigParser.ConfigParser() cf.read(args[0]) for (self.wtype, self.service_name) in [ (MASTER, "wal-master"), (SLAVE, "wal-slave") ]: if cf.has_section(self.service_name): break else: print >> sys.stderr, "Invalid config file: %s" % args[0] sys.exit(1) skytools.DBScript.__init__(self, self.service_name, args) self.set_single_loop(1) self.not_really = self.options.not_really self.pg_backup = 0 self.walchunk = None if len(self.args) < 2: usage(1) self.cfgfile = self.args[0] self.cmd = self.args[1] self.args = self.args[2:] self.script = os.path.abspath(sys.argv[0]) cmdtab = { 'setup': self.walmgr_setup, 'stop': self.master_stop, 'backup': self.run_backup, 'listbackups': self.list_backups, 'restore': self.restore_database, 'periodic': self.master_periodic, 'sync': self.master_sync, 'syncdaemon': self.master_syncdaemon, 'pause': self.slave_pause, 'continue': self.slave_continue, 'boot': self.slave_boot, 'xlock': self.slave_lock_backups_exit, 'xrelease': self.slave_resume_backups, 'xrotate': self.slave_rotate_backups, 'xpurgewals': self.slave_purge_wals, 'xarchive': self.master_xarchive, 'xrestore': self.xrestore, 'xpartialsync': self.slave_append_partial, } if self.cmd not in ('sync', 'syncdaemon'): # don't let pidfile interfere with normal operations, but # disallow concurrent syncing self.pidfile = None if not cmdtab.has_key(self.cmd): usage(1) self.work = cmdtab[self.cmd] def assert_valid_role(self,role): if self.wtype != role: self.log.warning("Action not available on current node.") sys.exit(1) def pg_start_backup(self, code): q = "select pg_start_backup('FullBackup')" self.log.info("Execute SQL: %s; [%s]" % (q, self.cf.get("master_db"))) if self.not_really: self.pg_backup = 1 return db = self.get_database("master_db") db.cursor().execute(q) db.commit() self.close_database("master_db") self.pg_backup = 1 def pg_stop_backup(self): if not self.pg_backup: return q = "select pg_stop_backup()" self.log.debug("Execute SQL: %s; [%s]" % (q, self.cf.get("master_db"))) if self.not_really: return db = self.get_database("master_db") db.cursor().execute(q) db.commit() self.close_database("master_db") def signal_postmaster(self, data_dir, sgn): pidfile = os.path.join(data_dir, "postmaster.pid") if not os.path.isfile(pidfile): self.log.info("postmaster is not running (pidfile not present)") return False buf = open(pidfile, "r").readline() pid = int(buf.strip()) self.log.debug("Signal %d to process %d" % (sgn, pid)) if sgn == 0 or not self.not_really: try: os.kill(pid, sgn) except OSError, ex: if ex.errno == errno.ESRCH: self.log.info("postmaster is not running (no process at indicated PID)") return False else: raise return True def exec_rsync(self,args,die_on_error=False): cmdline = [ "rsync", "-a", "--quiet" ] if self.cf.getint("compression", 0) > 0: cmdline.append("-z") cmdline += args cmd = "' '".join(cmdline) self.log.debug("Execute rsync cmd: '%s'" % (cmd)) if self.not_really: return 0 res = os.spawnvp(os.P_WAIT, cmdline[0], cmdline) if res == 24: self.log.info("Some files vanished, but thats OK") res = 0 elif res != 0: self.log.fatal("rsync exec failed, res=%d" % res) if die_on_error: sys.exit(1) return res def exec_big_rsync(self, args): if self.exec_rsync(args) != 0: self.log.fatal("Big rsync failed") self.pg_stop_backup() sys.exit(1) def rsync_log_directory(self, source_dir, dst_loc): """rsync a pg_log or pg_xlog directory - ignore most of the directory contents, and pay attention to symlinks """ keep_symlinks = self.cf.getint("keep_symlinks", 1) subdir = os.path.basename(source_dir) if not os.path.exists(source_dir): self.log.info("%s does not exist, skipping" % subdir) return cmdline = [] # if this is a symlink, copy it's target first if os.path.islink(source_dir) and keep_symlinks: self.log.info('%s is a symlink, attempting to create link target' % subdir) # expand the link link = os.readlink(source_dir) if not link.startswith("/"): link = os.path.join(os.getcwd(), link) link_target = os.path.join(link, "") remote_target = "%s:%s" % (self.slave_host(), link_target) options = [ "--include=archive_status", "--exclude=/**" ] if self.exec_rsync( options + [ link_target, remote_target ]): # unable to create the link target, just convert the links # to directories in PGDATA self.log.warning('Unable to create symlinked %s on target, copying' % subdir) cmdline += [ "--copy-unsafe-links" ] cmdline += [ "--exclude=pg_log/*" ] cmdline += [ "--exclude=pg_xlog/archive_status/*" ] cmdline += [ "--include=pg_xlog/archive_status" ] cmdline += [ "--exclude=pg_xlog/*" ] self.exec_big_rsync(cmdline + [ source_dir, dst_loc ]) def exec_cmd(self, cmdline,allow_error=False): cmd = "' '".join(cmdline) self.log.debug("Execute cmd: '%s'" % (cmd)) if self.not_really: return #res = os.spawnvp(os.P_WAIT, cmdline[0], cmdline) process = subprocess.Popen(cmdline,stdout=subprocess.PIPE) output=process.communicate() res = process.returncode if res != 0 and not allow_error: self.log.fatal("exec failed, res=%d (%s)" % (res, repr(cmdline))) sys.exit(1) return (res,output[0]) def exec_system(self, cmdline): self.log.debug("Execute cmd: '%s'" % (cmdline)) if self.not_really: return 0 return os.WEXITSTATUS(os.system(cmdline)) def chdir(self, loc): self.log.debug("chdir: '%s'" % (loc)) if self.not_really: return try: os.chdir(loc) except os.error: self.log.fatal("CHDir failed") self.pg_stop_backup() sys.exit(1) def get_last_complete(self): """Get the name of last xarchived segment.""" data_dir = self.cf.get("master_data") fn = os.path.join(data_dir, ".walshipping.last") try: last = open(fn, "r").read().strip() return last except: self.log.info("Failed to read %s" % fn) return None def set_last_complete(self, last): """Set the name of last xarchived segment.""" data_dir = self.cf.get("master_data") fn = os.path.join(data_dir, ".walshipping.last") fn_tmp = fn + ".new" try: f = open(fn_tmp, "w") f.write(last) f.close() os.rename(fn_tmp, fn) except: self.log.fatal("Cannot write to %s" % fn) def master_stop(self): """Deconfigure archiving, attempt to stop syncdaemon""" data_dir = self.cf.get("master_data") restart_cmd = self.cf.get("master_restart_cmd", "") self.assert_valid_role(MASTER) self.log.info("Disabling WAL archiving") self.master_configure_archiving(False, restart_cmd) # if we have a restart command, then use it, otherwise signal if restart_cmd: self.log.info("Restarting postmaster") self.exec_system(restart_cmd) else: self.log.info("Sending SIGHUP to postmaster") self.signal_postmaster(data_dir, signal.SIGHUP) # stop any running syncdaemons pidfile = self.cf.get("pidfile", "") if os.path.exists(pidfile): self.log.info('Pidfile %s exists, attempting to stop syncdaemon.' % pidfile) self.exec_cmd([self.script, self.cfgfile, "syncdaemon", "-s"]) self.log.info("Done") def master_configure_archiving(self, enable_archiving, can_restart): """Turn the archiving on or off""" cf = PostgresConfiguration(self, self.cf.get("master_config")) curr_archive_mode = cf.archive_mode() curr_wal_level = cf.wal_level() need_restart_warning = False if enable_archiving: # enable archiving cf_file = os.path.abspath(self.cf.filename) xarchive = "%s %s %s" % (self.script, cf_file, "xarchive %p %f") cf_params = { "archive_command": xarchive } if curr_archive_mode is not None: # archive mode specified in config, turn it on self.log.debug("found 'archive_mode' in config -- enabling it") cf_params["archive_mode"] = "on" if curr_archive_mode.lower() not in ('1', 'on', 'true') and not can_restart: need_restart_warning = True if curr_wal_level is not None and curr_wal_level != 'hot_standby': # wal level set in config, enable it wal_level = self.cf.getboolean("hot_standby", False) and "hot_standby" or "archive" self.log.debug("found 'wal_level' in config -- setting to '%s'" % wal_level) cf_params["wal_level"] = wal_level if curr_wal_level not in ("archive", "hot_standby") and not can_restart: need_restart_warning = True if need_restart_warning: self.log.warning("database must be restarted to enable archiving") else: # disable archiving cf_params = dict() if can_restart: # can restart, disable archive mode and set wal_level to minimal cf_params['archive_command'] = '' if curr_archive_mode: cf_params['archive_mode'] = 'off' if curr_wal_level: cf_params['wal_level'] = 'minimal' cf_params['max_wal_senders'] = '0' else: # not possible to change archive_mode or wal_level (requires restart), # so we just set the archive_command to /bin/true to avoid WAL pileup. self.log.warning("database must be restarted to disable archiving") self.log.info("Setting archive_command to /bin/true to avoid WAL pileup") cf_params['archive_command'] = '/bin/true' self.log.debug("modifying configuration: %s" % cf_params) cf.modify(cf_params) cf.write() def slave_deconfigure_archiving(self, cf_file): """Disable archiving for the slave. This is done by setting archive_command to a trivial command, so that archiving can be re-enabled without restarting postgres. Needed when slave is booted with postgresql.conf from master.""" self.log.debug("Disable archiving in %s" % cf_file) cf = PostgresConfiguration(self, cf_file) cf_params = { "archive_command": "/bin/true" } self.log.debug("modifying configuration: %s" % cf_params) cf.modify(cf_params) cf.write() def remote_mkdir(self, remdir): tmp = remdir.split(":", 1) if len(tmp) < 1: raise Exception("cannot find pathname") elif len(tmp) < 2: self.exec_cmd([ "mkdir", "-p", tmp[0] ]) else: host, path = tmp cmdline = ["ssh", "-nT", host, "mkdir", "-p", path] self.exec_cmd(cmdline) def slave_host(self): """Extract the slave hostname""" try: slave = self.cf.get("slave") host, path = slave.split(":", 1) except: raise Exception("invalid value for 'slave' in %s" % self.cfgfile) return host def remote_walmgr(self, command, stdin_disabled = True,allow_error=False): """Pass a command to slave WalManager""" sshopt = "-T" if stdin_disabled: sshopt += "n" slave_config = self.cf.get("slave_config") if not slave_config: raise Exception("slave_config not specified in %s" % self.cfgfile) try: slave = self.cf.get("slave") host, path = slave.split(":", 1) except: raise Exception("invalid value for 'slave' in %s" % self.cfgfile) cmdline = [ "ssh", sshopt, host, self.script, slave_config, command ] if self.not_really: self.log.info("remote_walmgr: %s" % command) else: return self.exec_cmd(cmdline,allow_error) def walmgr_setup(self): if self.wtype == MASTER: self.log.info("Configuring WAL archiving") data_dir = self.cf.get("master_data") restart_cmd = self.cf.get("master_restart_cmd", "") self.master_configure_archiving(True, restart_cmd) # if we have a restart command, then use it, otherwise signal if restart_cmd: self.log.info("Restarting postmaster") self.exec_system(restart_cmd) else: self.log.info("Sending SIGHUP to postmaster") self.signal_postmaster(data_dir, signal.SIGHUP) # ask slave to init self.remote_walmgr("setup") self.log.info("Done") else: # create slave directory structure def mkdir(dir): if not os.path.exists(dir): self.log.debug("Creating directory %s" % dir) os.mkdir(dir) mkdir(self.cf.get("slave")) mkdir(self.cf.get("completed_wals")) mkdir(self.cf.get("partial_wals")) mkdir(self.cf.get("full_backup")) cf_backup = self.cf.get("config_backup", "") if cf_backup: mkdir(cf_backup) def master_periodic(self): """ Run periodic command on master node. We keep time using .walshipping.last file, so this has to be run before set_last_complete() """ self.assert_valid_role(MASTER) try: command_interval = self.cf.getint("command_interval", 0) periodic_command = self.cf.get("periodic_command", "") if periodic_command: check_file = os.path.join(self.cf.get("master_data"), ".walshipping.periodic") elapsed = 0 if os.path.isfile(check_file): elapsed = time.time() - os.stat(check_file).st_mtime self.log.info("Running periodic command: %s" % periodic_command) if not elapsed or elapsed > command_interval: if not self.not_really: rc = os.WEXITSTATUS(self.exec_system(periodic_command)) if rc != 0: self.log.error("Periodic command exited with status %d" % rc) # dont update timestamp - try again next time else: open(check_file,"w").write("1") else: self.log.debug("%d seconds elapsed, not enough to run periodic." % elapsed) except Exception, det: self.log.error("Failed to run periodic command: %s" % str(det)) def master_backup(self): """ Copy master data directory to slave. 1. Obtain backup lock on slave. 2. Rotate backups on slave 3. Perform backup as usual 4. Purge unneeded WAL-s from slave 5. Release backup lock """ self.remote_xlock() errors = False try: self.pg_start_backup("FullBackup") self.remote_walmgr("xrotate") data_dir = self.cf.get("master_data") dst_loc = self.cf.get("full_backup") if dst_loc[-1] != "/": dst_loc += "/" master_spc_dir = os.path.join(data_dir, "pg_tblspc") slave_spc_dir = dst_loc + "tmpspc" # copy data self.chdir(data_dir) cmdline = [ "--delete", "--exclude", ".*", "--exclude", "*.pid", "--exclude", "*.opts", "--exclude", "*.conf", "--exclude", "pg_xlog", "--exclude", "pg_tblspc", "--exclude", "pg_log", "--copy-unsafe-links", ".", dst_loc] self.exec_big_rsync(cmdline) # copy tblspc first, to test if os.path.isdir(master_spc_dir): self.log.info("Checking tablespaces") list = os.listdir(master_spc_dir) if len(list) > 0: self.remote_mkdir(slave_spc_dir) for tblspc in list: if tblspc[0] == ".": continue tfn = os.path.join(master_spc_dir, tblspc) if not os.path.islink(tfn): self.log.info("Suspicious pg_tblspc entry: "+tblspc) continue spc_path = os.path.realpath(tfn) self.log.info("Got tablespace %s: %s" % (tblspc, spc_path)) dstfn = slave_spc_dir + "/" + tblspc try: os.chdir(spc_path) except Exception, det: self.log.warning("Broken link:" + str(det)) continue cmdline = [ "--delete", "--exclude", ".*", "--copy-unsafe-links", ".", dstfn] self.exec_big_rsync(cmdline) # copy the pg_log and pg_xlog directories, these may be # symlinked to nonstandard location, so pay attention self.rsync_log_directory(os.path.join(data_dir, "pg_log"), dst_loc) self.rsync_log_directory(os.path.join(data_dir, "pg_xlog"), dst_loc) # copy config files conf_dst_loc = self.cf.get("config_backup", "") if conf_dst_loc: master_conf_dir = os.path.dirname(self.cf.get("master_config")) self.log.info("Backup conf files from %s" % master_conf_dir) self.chdir(master_conf_dir) cmdline = [ "--include", "*.conf", "--exclude", "*", ".", conf_dst_loc] self.exec_big_rsync(cmdline) self.remote_walmgr("xpurgewals") except Exception, e: self.log.error(e) errors = True finally: try: self.pg_stop_backup() except: pass try: self.remote_walmgr("xrelease") except: pass if not errors: self.log.info("Full backup successful") else: self.log.error("Full backup failed.") def slave_backup(self): """ Create backup on slave host. 1. Obtain backup lock 2. Pause WAL apply 3. Wait for WAL apply to complete (look at PROGRESS file) 4. Rotate old backups 5. Copy data directory to data.master 6. Create backup label and history file. 7. Purge unneeded WAL-s 8. Resume WAL apply 9. Release backup lock """ self.assert_valid_role(SLAVE) if self.slave_lock_backups() != 0: self.log.error("Cannot obtain backup lock.") sys.exit(1) try: self.slave_pause(waitcomplete=1) try: self.slave_rotate_backups() src = self.cf.get("slave_data") dst = self.cf.get("full_backup") start_time = time.localtime() cmdline = ["cp", "-a", src, dst ] self.log.info("Executing %s" % " ".join(cmdline)) if not self.not_really: self.exec_cmd(cmdline) stop_time = time.localtime() # Obtain the last restart point information ctl = PgControlData(self.cf.get("slave_bin", ""), dst, False) # TODO: The newly created backup directory probably still contains # backup_label.old and recovery.conf files. Remove these. if not ctl.is_valid: self.log.warning("Unable to determine last restart point, backup_label not created.") else: # Write backup label and history file backup_label = \ """START WAL LOCATION: %(xlogid)X/%(xrecoff)X (file %(wal_name)s) CHECKPOINT LOCATION: %(xlogid)X/%(xrecoff)X START TIME: %(start_time)s LABEL: SlaveBackup" """ backup_history = \ """START WAL LOCATION: %(xlogid)X/%(xrecoff)X (file %(wal_name)s) STOP WAL LOCATION: %(xlogid)X/%(xrecoff)X (file %(wal_name)s) CHECKPOINT LOCATION: %(xlogid)X/%(xrecoff)X START TIME: %(start_time)s LABEL: SlaveBackup" STOP TIME: %(stop_time)s """ label_params = { "xlogid": ctl.xlogid, "xrecoff": ctl.xrecoff, "wal_name": ctl.wal_name, "start_time": time.strftime("%Y-%m-%d %H:%M:%S %Z", start_time), "stop_time": time.strftime("%Y-%m-%d %H:%M:%S %Z", stop_time), } # Write the label filename = os.path.join(dst, "backup_label") if self.not_really: self.log.info("Writing backup label to %s" % filename) else: lf = open(filename, "w") lf.write(backup_label % label_params) lf.close() # Now the history histfile = "%s.%08X.backup" % (ctl.wal_name, ctl.xrecoff % ctl.wal_size) completed_wals = self.cf.get("completed_wals") filename = os.path.join(completed_wals, histfile) if os.path.exists(filename): self.log.warning("%s: already exists, refusing to overwrite." % filename) else: if self.not_really: self.log.info("Writing backup history to %s" % filename) else: lf = open(filename, "w") lf.write(backup_history % label_params) lf.close() self.slave_purge_wals() finally: self.slave_continue() finally: self.slave_resume_backups() def run_backup(self): if self.wtype == MASTER: self.master_backup() else: self.slave_backup() def master_xarchive(self): """Copy a complete WAL segment to slave.""" self.assert_valid_role(MASTER) if len(self.args) < 2: die(1, "usage: xarchive srcpath srcname") srcpath = self.args[0] srcname = self.args[1] start_time = time.time() self.log.debug("%s: start copy", srcname) self.master_periodic() self.set_last_complete(srcname) dst_loc = self.cf.get("completed_wals") if dst_loc[-1] != "/": dst_loc += "/" # copy data self.exec_rsync([ srcpath, dst_loc ], True) self.log.debug("%s: done", srcname) end_time = time.time() self.stat_add('count', 1) self.stat_add('duration', end_time - start_time) self.send_stats() def slave_append_partial(self): """ Read 'bytes' worth of data from stdin, append to the partial log file starting from 'offset'. On error it is assumed that master restarts from zero. The resulting file is always padded to XLOG_SEGMENT_SIZE bytes to simplify recovery. """ def fail(message): self.log.error("Slave: %s: %s" % (filename, message)) sys.exit(1) self.assert_valid_role(SLAVE) if len(self.args) < 3: die(1, "usage: xpartialsync ") filename = self.args[0] offset = int(self.args[1]) bytes = int(self.args[2]) data = sys.stdin.read(bytes) if len(data) != bytes: fail("not enough data, expected %d, got %d" % (bytes, len(data))) chunk = WalChunk(filename, offset, bytes) self.log.debug("Slave: adding to %s" % chunk) name = os.path.join(self.cf.get("partial_wals"), filename) try: xlog = open(name, (offset == 0) and "w+" or "r+") except: fail("unable to open partial WAL: %s" % name) xlog.seek(offset) xlog.write(data) # padd the file to 16MB boundary, use sparse files padsize = XLOG_SEGMENT_SIZE - xlog.tell() if padsize > 0: xlog.seek(XLOG_SEGMENT_SIZE-1) xlog.write('\0') xlog.close() def master_send_partial(self, xlog_dir, chunk, daemon_mode): """ Send the partial log chunk to slave. Use SSH with input redirection for the copy, consider other options if the overhead becomes visible. """ try: xlog = open(os.path.join(xlog_dir, chunk.filename)) except IOError, det: self.log.warning("Cannot access file %s" % chunk.filename) return xlog.seek(chunk.pos) # Fork the sync process childpid = os.fork() syncstart = time.time() if childpid == 0: os.dup2(xlog.fileno(), sys.stdin.fileno()) try: self.remote_walmgr("xpartialsync %s %d %d" % (chunk.filename, chunk.pos, chunk.bytes), False) except: os._exit(1) os._exit(0) chunk.sync_time += (time.time() - syncstart) status = os.waitpid(childpid, 0) rc = os.WEXITSTATUS(status[1]) if rc == 0: log = daemon_mode and self.log.debug or self.log.info log("sent to slave: %s" % chunk) chunk.pos += chunk.bytes chunk.sync_count += 1 else: # Start from zero after an error chunk.pos = 0 self.log.error("xpartialsync exited with status %d, restarting from zero." % rc) time.sleep(5) def master_syncdaemon(self): self.assert_valid_role(MASTER) self.set_single_loop(0) self.master_sync(True) def master_sync(self, daemon_mode=False): """ Copy partial WAL segments to slave. On 8.2 set use_xlog_functions=1 in config file - this enables record based walshipping. On 8.0 the only option is to sync files. If daemon_mode is specified it never switches from record based shipping to file based shipping. """ self.assert_valid_role(MASTER) use_xlog_functions = self.cf.getint("use_xlog_functions", False) data_dir = self.cf.get("master_data") xlog_dir = os.path.join(data_dir, "pg_xlog") master_bin = self.cf.get("master_bin", "") dst_loc = os.path.join(self.cf.get("partial_wals"), "") db = None if use_xlog_functions: try: db = self.get_database("master_db", autocommit=1) except: self.log.warning("Database unavailable, record based log shipping not possible.") if daemon_mode: return if db: cur = db.cursor() cur.execute("select file_name, file_offset from pg_xlogfile_name_offset(pg_current_xlog_location())") (file_name, file_offs) = cur.fetchone() if not self.walchunk or self.walchunk.filename != file_name: # Switched to new WAL segment. Don't bother to copy the last bits - it # will be obsoleted by the archive_command. if self.walchunk and self.walchunk.sync_count > 0: self.log.info("Switched in %d seconds, %f sec in %d interim syncs, avg %f" % (time.time() - self.walchunk.start_time, self.walchunk.sync_time, self.walchunk.sync_count, self.walchunk.sync_time / self.walchunk.sync_count)) self.walchunk = WalChunk(file_name, 0, file_offs) else: self.walchunk.bytes = file_offs - self.walchunk.pos if self.walchunk.bytes > 0: self.master_send_partial(xlog_dir, self.walchunk, daemon_mode) else: files = os.listdir(xlog_dir) files.sort() last = self.get_last_complete() if last: self.log.info("%s: last complete" % last) else: self.log.info("last complete not found, copying all") # obtain the last checkpoint wal name, this can be used for # limiting the amount of WAL files to copy if the database # has been cleanly shut down ctl = PgControlData(master_bin, data_dir, False) checkpoint_wal = None if ctl.is_valid: if not ctl.is_shutdown: # cannot rely on the checkpoint wal, should use some other method self.log.info("Database state is not 'shut down', copying all") else: # ok, the database is shut down, we can use last checkpoint wal checkpoint_wal = ctl.wal_name self.log.info("last checkpoint wal: %s" % checkpoint_wal) else: self.log.info("Unable to obtain control file information, copying all") for fn in files: # check if interesting file if len(fn) < 10: continue if fn[0] < "0" or fn[0] > '9': continue if fn.find(".") > 0: continue # check if too old if last: dot = last.find(".") if dot > 0: xlast = last[:dot] if fn < xlast: continue else: if fn <= last: continue # check if too new if checkpoint_wal and fn > checkpoint_wal: continue # got interesting WAL xlog = os.path.join(xlog_dir, fn) # copy data if self.exec_rsync([xlog, dst_loc]) != 0: self.log.error('Cannot sync %s' % xlog) break else: self.log.info("Partial copy done") def xrestore(self): if len(self.args) < 2: die(1, "usage: xrestore srcname dstpath [last restartpoint wal]") srcname = self.args[0] dstpath = self.args[1] lstname = None if len(self.args) > 2: lstname = self.args[2] if self.wtype == MASTER: self.master_xrestore(srcname, dstpath) else: self.slave_xrestore_unsafe(srcname, dstpath, os.getppid(), lstname) def slave_xrestore(self, srcname, dstpath): loop = 1 ppid = os.getppid() while loop: try: self.slave_xrestore_unsafe(srcname, dstpath, ppid) loop = 0 except SystemExit, d: sys.exit(1) except Exception, d: exc, msg, tb = sys.exc_info() self.log.fatal("xrestore %s crashed: %s: '%s' (%s: %s)" % ( srcname, str(exc), str(msg).rstrip(), str(tb), repr(traceback.format_tb(tb)))) time.sleep(10) self.log.info("Re-exec: %s", repr(sys.argv)) os.execv(sys.argv[0], sys.argv) def master_xrestore(self, srcname, dstpath): """ Restore the xlog file from slave. """ paths = [ self.cf.get("completed_wals"), self.cf.get("partial_wals") ] self.log.info("Restore %s to %s" % (srcname, dstpath)) for src in paths: self.log.debug("Looking in %s" % src) srcfile = os.path.join(src, srcname) if self.exec_rsync([srcfile, dstpath]) == 0: return self.log.warning("Could not restore file %s" % srcname) def is_parent_alive(self, parent_pid): if os.getppid() != parent_pid or parent_pid <= 1: return False return True def slave_xrestore_unsafe(self, srcname, dstpath, parent_pid, lstname = None): srcdir = self.cf.get("completed_wals") partdir = self.cf.get("partial_wals") pausefile = os.path.join(srcdir, "PAUSE") stopfile = os.path.join(srcdir, "STOP") prgrfile = os.path.join(srcdir, "PROGRESS") srcfile = os.path.join(srcdir, srcname) partfile = os.path.join(partdir, srcname) # if we are using streaming replication, exit immediately # if the srcfile is not here yet primary_conninfo = self.cf.get("primary_conninfo", "") if primary_conninfo and not os.path.isfile(srcfile): self.log.info("%s: not found (ignored)" % srcname) sys.exit(1) # assume that postgres has processed the WAL file and is # asking for next - hence work not in progress anymore if os.path.isfile(prgrfile): os.remove(prgrfile) # loop until srcfile or stopfile appears while 1: if os.path.isfile(pausefile): self.log.info("pause requested, sleeping") time.sleep(20) continue if os.path.isfile(srcfile): self.log.info("%s: Found" % srcname) break # ignore .history files unused, ext = os.path.splitext(srcname) if ext == ".history": self.log.info("%s: not found, ignoring" % srcname) sys.exit(1) # if stopping, include also partial wals if os.path.isfile(stopfile): if os.path.isfile(partfile): self.log.info("%s: found partial" % srcname) srcfile = partfile break else: self.log.info("%s: not found, stopping" % srcname) sys.exit(1) # nothing to do, just in case check if parent is alive if not self.is_parent_alive(parent_pid): self.log.warning("Parent dead, quitting") sys.exit(1) # nothing to do, sleep self.log.debug("%s: not found, sleeping" % srcname) time.sleep(1) # got one, copy it cmdline = ["cp", srcfile, dstpath] self.exec_cmd(cmdline) if self.cf.getint("keep_backups", 0) == 0: # cleanup only if we don't keep backup history, keep the files needed # to roll forward from last restart point. If the restart point is not # handed to us (i.e 8.3 or later), then calculate it ourselves. # Note that historic WAL files are removed during backup rotation if lstname == None: lstname = self.last_restart_point(srcname) self.log.debug("calculated restart point: %s" % lstname) else: self.log.debug("using supplied restart point: %s" % lstname) self.log.debug("%s: copy done, cleanup" % srcname) self.slave_cleanup(lstname) # create a PROGRESS file to notify that postgres is processing the WAL open(prgrfile, "w").write("1") # it would be nice to have apply time too self.stat_add('count', 1) self.send_stats() def restore_database(self): """Restore the database from backup If setname is specified, the contents of that backup set directory are restored instead of "full_backup". Also copy is used instead of rename to restore the directory (unless a pg_xlog directory has been specified). Restore to altdst if specified. Complain if it exists. """ setname = len(self.args) > 0 and self.args[0] or None altdst = len(self.args) > 1 and self.args[1] or None if self.wtype == SLAVE: data_dir = self.cf.get("slave_data") stop_cmd = self.cf.get("slave_stop_cmd", "") start_cmd = self.cf.get("slave_start_cmd") pidfile = os.path.join(data_dir, "postmaster.pid") else: if not setname or not altdst: die(1, "Source and target directories must be specified if running on master node.") data_dir = altdst stop_cmd = None pidfile = None if setname: full_dir = os.path.join(self.cf.get("slave"), setname) else: full_dir = self.cf.get("full_backup") # stop postmaster if ordered if stop_cmd and os.path.isfile(pidfile): self.log.info("Stopping postmaster: " + stop_cmd) self.exec_system(stop_cmd) time.sleep(3) # is it dead? if pidfile and os.path.isfile(pidfile): self.log.info("Pidfile exists, checking if process is running.") if self.signal_postmaster(data_dir, 0): self.log.fatal("Postmaster still running. Cannot continue.") sys.exit(1) # find name for data backup i = 0 while 1: bak = "%s.%d" % (data_dir.rstrip("/"), i) if not os.path.isdir(bak): break i += 1 if self.wtype == MASTER: print >>sys.stderr, "About to restore to directory %s. The postgres cluster should be shut down." % data_dir if not yesno("Is postgres shut down on %s ?" % data_dir): die(1, "Shut it down and try again.") if self.wtype == SLAVE: createbackup = True elif os.path.isdir(data_dir): createbackup = yesno("Create backup of %s?" % data_dir) else: # nothing to back up createbackup = False # see if we have to make a backup of the data directory backup_datadir = self.cf.getboolean('backup_datadir', True) if os.path.isdir(data_dir) and not backup_datadir: self.log.warning('backup_datadir is disabled, deleting old data dir') shutil.rmtree(data_dir) if not setname and os.path.isdir(data_dir) and backup_datadir: # compatibility mode - restore without a set name and data directory exists self.log.warning("Data directory already exists, moving it out of the way.") createbackup = True # move old data away if createbackup and os.path.isdir(data_dir): self.log.info("Move %s to %s" % (data_dir, bak)) if not self.not_really: os.rename(data_dir, bak) # move new data, copy if setname specified self.log.info("%s %s to %s" % (setname and "Copy" or "Move", full_dir, data_dir)) if self.cf.get('slave_pg_xlog', ''): link_xlog_dir = True exclude_pg_xlog = '--exclude=pg_xlog' else: link_xlog_dir = False exclude_pg_xlog = '' if not self.not_really: if not setname and not link_xlog_dir: os.rename(full_dir, data_dir) else: rsync_args=["--delete", "--no-relative", "--exclude=pg_xlog/*"] if exclude_pg_xlog: rsync_args.append(exclude_pg_xlog) rsync_args += [os.path.join(full_dir, ""), data_dir] self.exec_rsync(rsync_args, True) if link_xlog_dir: os.symlink(self.cf.get('slave_pg_xlog'), "%s/pg_xlog" % data_dir) if (self.wtype == MASTER and createbackup and os.path.isdir(bak)): # restore original xlog files to data_dir/pg_xlog # symlinked directories are dereferenced self.exec_cmd(["cp", "-rL", "%s/pg_xlog/" % full_dir, "%s/pg_xlog" % data_dir ]) else: # create an archive_status directory xlog_dir = os.path.join(data_dir, "pg_xlog") archive_path = os.path.join(xlog_dir, "archive_status") if not os.path.exists(archive_path): os.mkdir(archive_path, 0700) else: data_dir = full_dir # copy configuration files to rotated backup directory if createbackup and os.path.isdir(bak): for cf in ('postgresql.conf', 'pg_hba.conf', 'pg_ident.conf'): cfsrc = os.path.join(bak, cf) cfdst = os.path.join(data_dir, cf) if os.path.exists(cfdst): self.log.info("Already exists: %s" % cfdst) elif os.path.exists(cfsrc): self.log.debug("Copy %s to %s" % (cfsrc, cfdst)) if not self.not_really: copy_conf(cfsrc, cfdst) # re-link tablespaces spc_dir = os.path.join(data_dir, "pg_tblspc") tmp_dir = os.path.join(data_dir, "tmpspc") if not os.path.isdir(spc_dir): # 8.3 requires its existence os.mkdir(spc_dir) if os.path.isdir(tmp_dir): self.log.info("Linking tablespaces to temporary location") # don't look into spc_dir, thus allowing # user to move them before. re-link only those # that are still in tmp_dir list = os.listdir(tmp_dir) list.sort() for d in list: if d[0] == ".": continue link_loc = os.path.abspath(os.path.join(spc_dir, d)) link_dst = os.path.abspath(os.path.join(tmp_dir, d)) self.log.info("Linking tablespace %s to %s" % (d, link_dst)) if not self.not_really: if os.path.islink(link_loc): os.remove(link_loc) os.symlink(link_dst, link_loc) # write recovery.conf rconf = os.path.join(data_dir, "recovery.conf") cf_file = os.path.abspath(self.cf.filename) # determine if we can use %r in restore_command ctl = PgControlData(self.cf.get("slave_bin", ""), data_dir, True) if ctl.pg_version > 830: self.log.debug('pg_version is %s, adding %%r to restore command' % ctl.pg_version) restore_command = 'xrestore %f "%p" %r' else: if not ctl.is_valid: self.log.warning('unable to run pg_controldata, assuming pre 8.3 environment') else: self.log.debug('using pg_controldata to determine restart points') restore_command = 'xrestore %f "%p"' conf = "restore_command = '%s %s %s'\n" % (self.script, cf_file, restore_command) # do we have streaming replication (hot standby) primary_conninfo = self.cf.get("primary_conninfo", "") if primary_conninfo: conf += "standby_mode = 'on'\n" conf += "trigger_file = '%s'\n" % os.path.join(self.cf.get("completed_wals"), "STOP") conf += "primary_conninfo = '%s'\n" % primary_conninfo self.log.info("Write %s" % rconf) if self.not_really: print conf else: f = open(rconf, "w") f.write(conf) f.close() # remove stopfile on slave if self.wtype == SLAVE: stopfile = os.path.join(self.cf.get("completed_wals"), "STOP") if os.path.isfile(stopfile): self.log.info("Removing stopfile: "+stopfile) if not self.not_really: os.remove(stopfile) # attempt to restore configuration. Note that we cannot # postpone this to boot time, as the configuration is needed # to start postmaster. self.slave_restore_config() # run database in recovery mode self.log.info("Starting postmaster: " + start_cmd) self.exec_system(start_cmd) else: self.log.info("Data files restored, recovery.conf created.") self.log.info("postgresql.conf and additional WAL files may need to be restored manually.") def slave_restore_config(self): """Restore the configuration files if target directory specified.""" self.assert_valid_role(SLAVE) cf_source_dir = self.cf.get("config_backup", "") cf_target_dir = self.cf.get("slave_config_dir", "") if not cf_source_dir: self.log.info("Configuration backup location not specified.") return if not cf_target_dir: self.log.info("Configuration directory not specified, config files not restored.") return if not os.path.exists(cf_target_dir): self.log.warning("Configuration directory does not exist: %s" % cf_target_dir) return self.log.info("Restoring configuration files") for cf in ('postgresql.conf', 'pg_hba.conf', 'pg_ident.conf'): cfsrc = os.path.join(cf_source_dir, cf) cfdst = os.path.join(cf_target_dir, cf) if not os.path.isfile(cfsrc): self.log.warning("Missing configuration file backup: %s" % cf) continue self.log.debug("Copy %s to %s" % (cfsrc, cfdst)) if not self.not_really: copy_conf(cfsrc, cfdst) if cf == 'postgresql.conf': self.slave_deconfigure_archiving(cfdst) def slave_boot(self): self.assert_valid_role(SLAVE) srcdir = self.cf.get("completed_wals") datadir = self.cf.get("slave_data") stopfile = os.path.join(srcdir, "STOP") if self.not_really: self.log.info("Writing STOP file: %s" % stopfile) else: open(stopfile, "w").write("1") self.log.info("Stopping recovery mode") def slave_pause(self, waitcomplete=0): """Pause the WAL apply, wait until last file applied if needed""" self.assert_valid_role(SLAVE) srcdir = self.cf.get("completed_wals") pausefile = os.path.join(srcdir, "PAUSE") if not self.not_really: open(pausefile, "w").write("1") else: self.log.info("Writing PAUSE file: %s" % pausefile) self.log.info("Pausing recovery mode") # wait for log apply to complete if waitcomplete: prgrfile = os.path.join(srcdir, "PROGRESS") stopfile = os.path.join(srcdir, "STOP") if os.path.isfile(stopfile): self.log.warning("Recovery is stopped, backup is invalid if the database is open.") return while os.path.isfile(prgrfile): self.log.info("Waiting for WAL processing to complete ...") if self.not_really: return time.sleep(1) def slave_continue(self): self.assert_valid_role(SLAVE) srcdir = self.cf.get("completed_wals") pausefile = os.path.join(srcdir, "PAUSE") if os.path.isfile(pausefile): if not self.not_really: os.remove(pausefile) self.log.info("Continuing with recovery") else: self.log.info("Recovery not paused?") def slave_lock_backups_exit(self): """Exit with lock acquired status""" self.assert_valid_role(SLAVE) sys.exit(self.slave_lock_backups()) def slave_lock_backups(self): """Create lock file to deny other concurrent backups""" srcdir = self.cf.get("completed_wals") lockfile = os.path.join(srcdir, "BACKUPLOCK") if os.path.isfile(lockfile): self.log.warning("Somebody already has the backup lock.") lockfilehandle = open(lockfile,"r") pidstring = lockfilehandle.read(); try: pid = int(pidstring) print("%d",pid) except ValueError: self.log.error("lock file does not contain a pid:" + pidstring) return 1 if not self.not_really: open(lockfile, "w").write(self.args[0]) self.log.info("Backup lock obtained.") return 0 def slave_resume_backups(self): """Remove backup lock file, allow other backups to run""" self.assert_valid_role(SLAVE) srcdir = self.cf.get("completed_wals") lockfile = os.path.join(srcdir, "BACKUPLOCK") if os.path.isfile(lockfile): if not self.not_really: os.remove(lockfile) self.log.info("Backup lock released.") else: self.log.info("Backup lock not held.") def list_backups(self): """List available backups. On master this just calls slave listbackups via SSH""" if self.wtype == MASTER: self.remote_walmgr("listbackups") else: backups = self.get_backup_list(self.cf.get("full_backup")) if backups: print "\nList of backups:\n" print "%-15s %-24s %-11s %-24s" % \ ("Backup set", "Timestamp", "Label", "First WAL") print "%s %s %s %s" % (15*'-', 24*'-', 11*'-',24*'-') for backup in backups: lbl = BackupLabel(backup) print "%-15s %-24.24s %-11.11s %-24s" % \ (os.path.basename(backup), lbl.start_time, lbl.label_string, lbl.first_wal) print else: print "\nNo backups found.\n" def get_first_walname(self,backupdir): """Returns the name of the first needed WAL segment for backupset""" label = BackupLabel(backupdir) if not label.first_wal: self.log.error("WAL name not found at %s" % backupdir) return None return label.first_wal def last_restart_point(self,walname): """ Determine the WAL file of the last restart point (recovery checkpoint). For 8.3 this could be done with %r parameter to restore_command, for 8.2 we need to consult control file (parse pg_controldata output). """ slave_data = self.cf.get("slave_data") backup_label = os.path.join(slave_data, "backup_label") if os.path.exists(backup_label): # Label file still exists, use it for determining the restart point lbl = BackupLabel(slave_data) self.log.debug("Last restart point from backup_label: %s" % lbl.first_wal) return lbl.first_wal ctl = PgControlData(self.cf.get("slave_bin", ""), ".", True) if not ctl.is_valid: # No restart point information, use the given wal name self.log.warning("Unable to determine last restart point") return walname self.log.debug("Last restart point: %s" % ctl.wal_name) return ctl.wal_name def order_backupdirs(self,prefix,a,b): """Compare the backup directory indexes numerically""" prefix = os.path.abspath(prefix) a_indx = a[len(prefix)+1:] if not a_indx: a_indx = -1 b_indx = b[len(prefix)+1:] if not b_indx: b_indx = -1 return cmp(int(a_indx), int(b_indx)) def get_backup_list(self,dst_loc): """Return the list of backup directories""" dirlist = glob.glob(os.path.abspath(dst_loc) + "*") dirlist.sort(lambda x,y: self.order_backupdirs(dst_loc, x,y)) backupdirs = [ dir for dir in dirlist if os.path.isdir(dir) and os.path.isfile(os.path.join(dir, "backup_label")) or os.path.isfile(os.path.join(dir, "backup_label.old"))] return backupdirs def slave_purge_wals(self): """ Remove WAL files not needed for recovery """ self.assert_valid_role(SLAVE) backups = self.get_backup_list(self.cf.get("full_backup")) if backups: lastwal = self.get_first_walname(backups[-1]) if lastwal: self.log.info("First useful WAL file is: %s" % lastwal) self.slave_cleanup(lastwal) else: self.log.debug("No WAL-s to clean up.") def slave_rotate_backups(self): """ Rotate backups by increasing backup directory suffixes. Note that since we also have to make room for next backup, we actually have keep_backups - 1 backups available after this. Unneeded WAL files are not removed here, handled by xpurgewals command instead. """ self.assert_valid_role(SLAVE) dst_loc = self.cf.get("full_backup") maxbackups = self.cf.getint("keep_backups", 0) archive_command = self.cf.get("archive_command", "") backupdirs = self.get_backup_list(dst_loc) if not backupdirs or maxbackups < 1: self.log.debug("Nothing to rotate") # remove expired backups while len(backupdirs) >= maxbackups and len(backupdirs) > 0: last = backupdirs.pop() # if archive_command is set, run it before removing the directory # Resume only if archive command succeeds. if archive_command: cmdline = archive_command.replace("$BACKUPDIR", last) self.log.info("Executing archive_command: " + cmdline) rc = self.exec_system(cmdline) if rc != 0: self.log.error("Backup archiving returned %d, exiting!" % rc) sys.exit(1) self.log.info("Removing expired backup directory: %s" % last) if self.not_really: continue cmdline = [ "rm", "-r", last ] self.exec_cmd(cmdline) # bump the suffixes if base directory exists if os.path.isdir(dst_loc): backupdirs.sort(lambda x,y: self.order_backupdirs(dst_loc, y,x)) for dir in backupdirs: (name, index) = os.path.splitext(dir) if not re.match('\.[0-9]+$', index): name = name + index index = 0 else: index = int(index[1:])+1 self.log.debug("Rename %s to %s.%s" % (dir, name, index)) if self.not_really: continue os.rename(dir, "%s.%s" % (name,index)) def slave_cleanup(self, last_applied): completed_wals = self.cf.get("completed_wals") partial_wals = self.cf.get("partial_wals") self.log.debug("cleaning completed wals before %s" % last_applied) self.del_wals(completed_wals, last_applied) if os.path.isdir(partial_wals): self.log.debug("cleaning partial wals before %s" % last_applied) self.del_wals(partial_wals, last_applied) else: self.log.warning("partial_wals dir does not exist: %s" % partial_wals) self.log.debug("cleaning done") def del_wals(self, path, last): dot = last.find(".") if dot > 0: last = last[:dot] list = os.listdir(path) list.sort() cur_last = None n = len(list) for i in range(n): fname = list[i] full = os.path.join(path, fname) if fname[0] < "0" or fname[0] > "9": continue if not fname.startswith(last[0:8]): # only look at WAL segments in a same timeline continue ok_del = 0 if fname < last: self.log.debug("deleting %s" % full) if not self.not_really: os.remove(full) cur_last = fname return cur_last def remote_xlock(self): ret = self.remote_walmgr("xlock " + str(os.getpid()),allow_error=True) if ret[0] != 0: # lock failed. try: lock_pid = int(ret[1]) if os.kill(lock_pid,0): #process exists. self.log.error("lock already obtained") else: self.remote_walmgr("xrelease") ret = self.remote_walmgr("xlock " + pid(),allow_error=True) if ret[0] != 0: self.log.error("unable to obtain lock") except ValueError: self.log.error("error obtaining lock") if __name__ == "__main__": script = WalMgr(sys.argv[1:]) script.start() skytools-2.1.13/python/londiste/0000755000175000017500000000000011727601174015641 5ustar markomarkoskytools-2.1.13/python/londiste/file_write.py0000644000175000017500000000331311670174255020346 0ustar markomarko """Writes events into file.""" import sys, os, skytools from cStringIO import StringIO from playback import * __all__ = ['FileWrite'] class FileWrite(Replicator): """Writes events into file. Incomplete implementation. """ last_successful_batch = None def load_state(self, batch_id): # maybe check if batch exists on filesystem? self.cur_tick = self.cur_batch_info['tick_id'] self.prev_tick = self.cur_batch_info['prev_tick_id'] return 1 def process_batch(self, db, batch_id, ev_list): pass def save_state(self, do_commit): # nothing to save pass def sync_tables(self, dst_db): # nothing to sync return 1 def interesting(self, ev): # wants all of them return 1 def handle_data_event(self, ev): fmt = self.sql_command[ev.type] sql = fmt % (ev.ev_extra1, ev.data) row = "%s -- txid:%d" % (sql, ev.txid) self.sql_list.append(row) ev.tag_done() def handle_system_event(self, ev): row = "-- sysevent:%s txid:%d data:%s" % ( ev.type, ev.txid, ev.data) self.sql_list.append(row) ev.tag_done() def flush_sql(self): self.sql_list.insert(0, "-- tick:%d prev:%s" % ( self.cur_tick, self.prev_tick)) self.sql_list.append("-- end_tick:%d\n" % self.cur_tick) # store result dir = self.cf.get("file_dst") fn = os.path.join(dir, "tick_%010d.sql" % self.cur_tick) f = open(fn, "w") buf = "\n".join(self.sql_list) f.write(buf) f.close() if __name__ == '__main__': script = Replicator(sys.argv[1:]) script.start() skytools-2.1.13/python/londiste/repair.py0000644000175000017500000002215211670174255017501 0ustar markomarko """Repair data on subscriber. Walks tables by primary key and searcher missing inserts/updates/deletes. """ import sys, os, time, skytools try: import subprocess have_subprocess = True except ImportError: have_subprocess = False from syncer import Syncer __all__ = ['Repairer'] def unescape(s): return skytools.unescape_copy(s) def get_pkey_list(curs, tbl): """Get list of pkey fields in right order.""" oid = skytools.get_table_oid(curs, tbl) q = """SELECT k.attname FROM pg_index i, pg_attribute k WHERE i.indrelid = %s AND k.attrelid = i.indexrelid AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped ORDER BY k.attnum""" curs.execute(q, [oid]) list = [] for row in curs.fetchall(): list.append(row[0]) return list def get_column_list(curs, tbl): """Get list of columns in right order.""" oid = skytools.get_table_oid(curs, tbl) q = """SELECT a.attname FROM pg_attribute a WHERE a.attrelid = %s AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum""" curs.execute(q, [oid]) list = [] for row in curs.fetchall(): list.append(row[0]) return list class Repairer(Syncer): """Walks tables in primary key order and checks if data matches.""" def process_sync(self, tbl, src_db, dst_db): """Actual comparision.""" src_curs = src_db.cursor() dst_curs = dst_db.cursor() self.log.info('Checking %s' % tbl) self.common_fields = [] self.pkey_list = [] copy_tbl = self.gen_copy_tbl(tbl, src_curs, dst_curs) dump_src = tbl + ".src" dump_dst = tbl + ".dst" self.log.info("Dumping src table: %s" % tbl) self.dump_table(tbl, copy_tbl, src_curs, dump_src) src_db.commit() self.log.info("Dumping dst table: %s" % tbl) self.dump_table(tbl, copy_tbl, dst_curs, dump_dst) dst_db.commit() self.log.info("Sorting src table: %s" % tbl) # check if sort supports -S if have_subprocess: p = subprocess.Popen(["sort", "--version"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) s_ver = p.communicate()[0] del p else: s_ver = os.popen4("sort --version")[1].read() if s_ver.find("coreutils") > 0: args = "-S 30%" else: args = "" os.system("LC_ALL=C sort %s -T . -o %s.sorted %s" % (args, dump_src, dump_src)) self.log.info("Sorting dst table: %s" % tbl) os.system("LC_ALL=C sort %s -T . -o %s.sorted %s" % (args, dump_dst, dump_dst)) self.dump_compare(tbl, dump_src + ".sorted", dump_dst + ".sorted") os.unlink(dump_src) os.unlink(dump_dst) os.unlink(dump_src + ".sorted") os.unlink(dump_dst + ".sorted") def gen_copy_tbl(self, tbl, src_curs, dst_curs): self.pkey_list = get_pkey_list(src_curs, tbl) dst_pkey = get_pkey_list(dst_curs, tbl) if dst_pkey != self.pkey_list: self.log.error('pkeys do not match') sys.exit(1) src_cols = get_column_list(src_curs, tbl) dst_cols = get_column_list(dst_curs, tbl) field_list = [] for f in self.pkey_list: field_list.append(f) for f in src_cols: if f in self.pkey_list: continue if f in dst_cols: field_list.append(f) self.common_fields = field_list fqlist = [skytools.quote_ident(col) for col in field_list] tbl_expr = "%s (%s)" % (skytools.quote_fqident(tbl), ",".join(fqlist)) self.log.debug("using copy expr: %s" % tbl_expr) return tbl_expr def dump_table(self, tbl, copy_tbl, curs, fn): f = open(fn, "w", 64*1024) curs.copy_to(f, copy_tbl) size = f.tell() f.close() self.log.info('Got %d bytes' % size) def get_row(self, ln): if not ln: return None t = ln[:-1].split('\t') row = {} for i in range(len(self.common_fields)): row[self.common_fields[i]] = t[i] return row def dump_compare(self, tbl, src_fn, dst_fn): self.log.info("Comparing dumps: %s" % tbl) self.cnt_insert = 0 self.cnt_update = 0 self.cnt_delete = 0 self.total_src = 0 self.total_dst = 0 f1 = open(src_fn, "r", 64*1024) f2 = open(dst_fn, "r", 64*1024) src_ln = f1.readline() dst_ln = f2.readline() if src_ln: self.total_src += 1 if dst_ln: self.total_dst += 1 fix = "fix.%s.sql" % tbl if os.path.isfile(fix): os.unlink(fix) while src_ln or dst_ln: keep_src = keep_dst = 0 if src_ln != dst_ln: src_row = self.get_row(src_ln) dst_row = self.get_row(dst_ln) cmp = self.cmp_keys(src_row, dst_row) if cmp > 0: # src > dst self.got_missed_delete(tbl, dst_row) keep_src = 1 elif cmp < 0: # src < dst self.got_missed_insert(tbl, src_row) keep_dst = 1 else: if self.cmp_data(src_row, dst_row) != 0: self.got_missed_update(tbl, src_row, dst_row) if not keep_src: src_ln = f1.readline() if src_ln: self.total_src += 1 if not keep_dst: dst_ln = f2.readline() if dst_ln: self.total_dst += 1 self.log.info("finished %s: src: %d rows, dst: %d rows,"\ " missed: %d inserts, %d updates, %d deletes" % ( tbl, self.total_src, self.total_dst, self.cnt_insert, self.cnt_update, self.cnt_delete)) def got_missed_insert(self, tbl, src_row): self.cnt_insert += 1 fld_list = self.common_fields fq_list = [] val_list = [] for f in fld_list: fq_list.append(skytools.quote_ident(f)) v = unescape(src_row[f]) val_list.append(skytools.quote_literal(v)) q = "insert into %s (%s) values (%s);" % ( tbl, ", ".join(fq_list), ", ".join(val_list)) self.show_fix(tbl, q, 'insert') def got_missed_update(self, tbl, src_row, dst_row): self.cnt_update += 1 fld_list = self.common_fields set_list = [] whe_list = [] for f in self.pkey_list: self.addcmp(whe_list, skytools.quote_ident(f), unescape(src_row[f])) for f in fld_list: v1 = src_row[f] v2 = dst_row[f] if self.cmp_value(v1, v2) == 0: continue self.addeq(set_list, skytools.quote_ident(f), unescape(v1)) self.addcmp(whe_list, skytools.quote_ident(f), unescape(v2)) q = "update only %s set %s where %s;" % ( tbl, ", ".join(set_list), " and ".join(whe_list)) self.show_fix(tbl, q, 'update') def got_missed_delete(self, tbl, dst_row): self.cnt_delete += 1 whe_list = [] for f in self.pkey_list: self.addcmp(whe_list, skytools.quote_ident(f), unescape(dst_row[f])) q = "delete from only %s where %s;" % (skytools.quote_fqident(tbl), " and ".join(whe_list)) self.show_fix(tbl, q, 'delete') def show_fix(self, tbl, q, desc): #self.log.warning("missed %s: %s" % (desc, q)) fn = "fix.%s.sql" % tbl open(fn, "a").write("%s\n" % q) def addeq(self, list, f, v): vq = skytools.quote_literal(v) s = "%s = %s" % (f, vq) list.append(s) def addcmp(self, list, f, v): if v is None: s = "%s is null" % f else: vq = skytools.quote_literal(v) s = "%s = %s" % (f, vq) list.append(s) def cmp_data(self, src_row, dst_row): for k in self.common_fields: v1 = src_row[k] v2 = dst_row[k] if self.cmp_value(v1, v2) != 0: return -1 return 0 def cmp_value(self, v1, v2): if v1 == v2: return 0 # try to work around tz vs. notz z1 = len(v1) z2 = len(v2) if z1 == z2 + 3 and z2 >= 19 and v1[z2] == '+': v1 = v1[:-3] if v1 == v2: return 0 elif z1 + 3 == z2 and z1 >= 19 and v2[z1] == '+': v2 = v2[:-3] if v1 == v2: return 0 return -1 def cmp_keys(self, src_row, dst_row): """Compare primary keys of the rows. Returns 1 if src > dst, -1 if src < dst and 0 if src == dst""" # None means table is done. tag it larger than any existing row. if src_row is None: if dst_row is None: return 0 return 1 elif dst_row is None: return -1 for k in self.pkey_list: v1 = src_row[k] v2 = dst_row[k] if v1 < v2: return -1 elif v1 > v2: return 1 return 0 skytools-2.1.13/python/londiste/setup.py0000644000175000017500000007361511670174255017371 0ustar markomarko#! /usr/bin/env python """Londiste setup and sanity checker. """ import sys, os, skytools from installer import * # support set() on 2.3 if 'set' not in __builtins__: from sets import Set as set __all__ = ['ProviderSetup', 'SubscriberSetup'] def find_column_types(curs, table): table_oid = skytools.get_table_oid(curs, table) if table_oid == None: return None key_sql = """ SELECT k.attname FROM pg_index i, pg_attribute k WHERE i.indrelid = %d AND k.attrelid = i.indexrelid AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped """ % table_oid # find columns q = """ SELECT a.attname as name, CASE WHEN k.attname IS NOT NULL THEN 'k' ELSE 'v' END AS type FROM pg_attribute a LEFT JOIN (%s) k ON (k.attname = a.attname) WHERE a.attrelid = %d AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum """ % (key_sql, table_oid) curs.execute(q) rows = curs.dictfetchall() return rows def make_type_string(col_rows): res = map(lambda x: x['type'], col_rows) return "".join(res) def convertGlobs(s): return s.replace('?', '.').replace('*', '.*') def glob2regex(gpat): plist = [convertGlobs(s) for s in gpat.split('.')] return '^%s$' % '[.]'.join(plist) class CommonSetup(skytools.DBScript): def __init__(self, args): skytools.DBScript.__init__(self, 'londiste', args) self.set_single_loop(1) self.pidfile = self.pidfile + ".setup" self.pgq_queue_name = self.cf.get("pgq_queue_name") self.consumer_id = self.cf.get("pgq_consumer_id", self.job_name) self.fake = self.cf.getint('fake', 0) self.lock_timeout = self.cf.getfloat('lock_timeout', 10) if len(self.args) < 3: self.log.error("need subcommand") sys.exit(1) def set_lock_timeout(self, curs): ms = int(1000 * self.lock_timeout) if ms > 0: q = "SET LOCAL statement_timeout = %d" % ms self.log.debug(q) curs.execute(q) def run(self): self.admin() def fetch_provider_table_list(self, curs, pattern='*'): q = """select table_name, trigger_name from londiste.provider_get_table_list(%s) where table_name ~ %s""" curs.execute(q, [self.pgq_queue_name, glob2regex(pattern)]) return curs.dictfetchall() def get_provider_table_list(self, pattern='*'): src_db = self.get_database('provider_db') src_curs = src_db.cursor() list = self.fetch_provider_table_list(src_curs, pattern) src_db.commit() res = [] for row in list: res.append(row['table_name']) return res def get_provider_seqs(self): src_db = self.get_database('provider_db') src_curs = src_db.cursor() q = """SELECT * from londiste.provider_get_seq_list(%s)""" src_curs.execute(q, [self.pgq_queue_name]) src_db.commit() res = [] for row in src_curs.fetchall(): res.append(row[0]) return res def get_all_seqs(self, curs): q = """SELECT n.nspname || '.' || c.relname from pg_class c, pg_namespace n where n.oid = c.relnamespace and c.relkind = 'S' and n.nspname not in ('pgq', 'londiste', 'pgq_node') and n.nspname !~ '^pg_temp_.*' order by 1""" curs.execute(q) res = [] for row in curs.fetchall(): res.append(row[0]) return res def check_provider_queue(self): src_db = self.get_database('provider_db') src_curs = src_db.cursor() q = "select count(1) from pgq.get_queue_info(%s)" src_curs.execute(q, [self.pgq_queue_name]) ok = src_curs.fetchone()[0] src_db.commit() if not ok: self.log.error('Event queue does not exist yet') sys.exit(1) def fetch_subscriber_tables(self, curs, pattern = '*'): q = "select * from londiste.subscriber_get_table_list(%s) where table_name ~ %s" curs.execute(q, [self.pgq_queue_name, glob2regex(pattern)]) return curs.dictfetchall() def get_subscriber_table_list(self, pattern = '*'): dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() list = self.fetch_subscriber_tables(dst_curs, pattern) dst_db.commit() res = [] for row in list: res.append(row['table_name']) return res def init_optparse(self, parser=None): p = skytools.DBScript.init_optparse(self, parser) p.add_option("--expect-sync", action="store_true", dest="expect_sync", help = "no copy needed", default=False) p.add_option("--skip-truncate", action="store_true", dest="skip_truncate", help = "dont delete old data", default=False) p.add_option("--force", action="store_true", help="force", default=False) p.add_option("--all", action="store_true", help="include all tables", default=False) return p # # Provider commands # class ProviderSetup(CommonSetup): def admin(self): cmd = self.args[2] if cmd == "tables": self.provider_show_tables() elif cmd == "add": self.provider_add_tables(self.args[3:]) elif cmd == "remove": self.provider_remove_tables(self.args[3:]) elif cmd == "add-seq": self.provider_add_seq_list(self.args[3:]) elif cmd == "remove-seq": self.provider_remove_seq_list(self.args[3:]) elif cmd == "install": self.provider_install() elif cmd == "seqs": self.provider_list_seqs() else: self.log.error('bad subcommand') sys.exit(1) def provider_list_seqs(self): list = self.get_provider_seqs() for seq in list: print seq def provider_get_all_seqs(self): src_db = self.get_database('provider_db') src_curs = src_db.cursor() list = self.get_all_seqs(src_curs) src_db.commit() return list def provider_add_seq_list(self, seq_list): if not seq_list and self.options.all: seq_list = self.provider_get_all_seqs() cur_list = self.get_provider_seqs() gotnew = False for seq in seq_list: seq = skytools.fq_name(seq) if seq in cur_list: self.log.info('Seq %s already subscribed' % seq) continue gotnew = True self.provider_add_seq(seq) #if gotnew: # self.provider_notify_change() def provider_remove_seq_list(self, seq_list): if not seq_list and self.options.all: seq_list = self.get_provider_seqs() for seq in seq_list: self.provider_remove_seq(seq) self.provider_notify_change() def provider_install(self): src_db = self.get_database('provider_db') src_curs = src_db.cursor() install_provider(src_curs, self.log) # create event queue q = "select pgq.create_queue(%s)" self.exec_provider(q, [self.pgq_queue_name]) def find_missing_provider_tables(self, pattern='*'): src_db = self.get_database('provider_db') src_curs = src_db.cursor() q = """select schemaname || '.' || tablename as full_name from pg_tables where schemaname not in ('pgq', 'londiste', 'pg_catalog', 'information_schema') and schemaname !~ 'pg_.*' and (schemaname || '.' || tablename) ~ %s except select table_name from londiste.provider_get_table_list(%s)""" src_curs.execute(q, [glob2regex(pattern), self.pgq_queue_name]) rows = src_curs.fetchall() src_db.commit() list = [] for row in rows: list.append(row[0]) return list def provider_add_tables(self, table_list): self.check_provider_queue() if self.options.all and not table_list: table_list = ['*.*'] cur_list = self.get_provider_table_list() for tbl in table_list: tbls = self.find_missing_provider_tables(skytools.fq_name(tbl)) for tbl in tbls: if tbl not in cur_list: self.log.info('Adding %s' % tbl) self.provider_add_table(tbl) else: self.log.info("Table %s already added" % tbl) #self.provider_notify_change() def provider_remove_tables(self, table_list): self.check_provider_queue() cur_list = self.get_provider_table_list() if not table_list and self.options.all: table_list = cur_list for tbl in table_list: tbls = self.get_provider_table_list(skytools.fq_name(tbl)) for tbl in tbls: if tbl not in cur_list: self.log.info('%s already removed' % tbl) else: self.log.info("Removing %s" % tbl) self.provider_remove_table(tbl) self.provider_notify_change() def provider_add_table(self, tbl): src_db = self.get_database('provider_db') src_curs = src_db.cursor() pg_vers = src_curs.connection.server_version q = "select londiste.provider_add_table(%s, %s)" self.exec_provider(q, [self.pgq_queue_name, tbl]) # detect dangerous triggers if pg_vers >= 90100: q = """ select tg.trigger_name from londiste.provider_table tbl, information_schema.triggers tg where tbl.queue_name = %s and tbl.table_name = %s and tg.event_object_schema = %s and tg.event_object_table = %s and tg.action_timing = 'AFTER' and tg.trigger_name != tbl.trigger_name and tg.trigger_name < tbl.trigger_name and substring(tg.trigger_name from 1 for 10) != '_londiste_' and substring(tg.trigger_name from char_length(tg.trigger_name) - 6) != '_logger' """ else: q = """ select tg.trigger_name from londiste.provider_table tbl, information_schema.triggers tg where tbl.queue_name = %s and tbl.table_name = %s and tg.event_object_schema = %s and tg.event_object_table = %s and tg.condition_timing = 'AFTER' and tg.trigger_name != tbl.trigger_name and tg.trigger_name < tbl.trigger_name and substring(tg.trigger_name from 1 for 10) != '_londiste_' and substring(tg.trigger_name from char_length(tg.trigger_name) - 6) != '_logger' """ sname, tname = skytools.fq_name_parts(tbl) src_curs.execute(q, [self.pgq_queue_name, tbl, sname, tname]) for r in src_curs.fetchall(): self.log.warning("Table %s has AFTER trigger '%s' which runs before Londiste trigger. "\ "If it modifies data, then events will appear in queue in wrong order." % ( tbl, r[0])) src_db.commit() def provider_remove_table(self, tbl): q = "select londiste.provider_remove_table(%s, %s)" self.exec_provider(q, [self.pgq_queue_name, tbl]) def provider_show_tables(self): self.check_provider_queue() list = self.get_provider_table_list() for tbl in list: print tbl def provider_notify_change(self): q = "select londiste.provider_notify_change(%s)" self.exec_provider(q, [self.pgq_queue_name]) def provider_add_seq(self, seq): seq = skytools.fq_name(seq) q = "select londiste.provider_add_seq(%s, %s)" self.exec_provider(q, [self.pgq_queue_name, seq]) def provider_remove_seq(self, seq): seq = skytools.fq_name(seq) q = "select londiste.provider_remove_seq(%s, %s)" self.exec_provider(q, [self.pgq_queue_name, seq]) def exec_provider(self, sql, args): src_db = self.get_database('provider_db') src_curs = src_db.cursor() self.set_lock_timeout(src_curs) src_curs.execute(sql, args) if self.fake: src_db.rollback() else: src_db.commit() # # Subscriber commands # class SubscriberSetup(CommonSetup): def admin(self): cmd = self.args[2] if cmd == "tables": self.subscriber_show_tables() elif cmd == "missing": self.subscriber_missing_tables() elif cmd == "add": self.subscriber_add_tables(self.args[3:]) elif cmd == "remove": self.subscriber_remove_tables(self.args[3:]) elif cmd == "resync": self.subscriber_resync_tables(self.args[3:]) elif cmd == "register": self.subscriber_register() elif cmd == "unregister": self.subscriber_unregister() elif cmd == "install": self.subscriber_install() elif cmd == "check": self.check_tables(self.get_provider_table_list()) elif cmd in ["fkeys", "triggers"]: self.collect_meta(self.get_provider_table_list(), cmd, self.args[3:]) elif cmd == "seqs": self.subscriber_list_seqs() elif cmd == "add-seq": self.subscriber_add_seq(self.args[3:]) elif cmd == "remove-seq": self.subscriber_remove_seq(self.args[3:]) elif cmd == "restore-triggers": self.restore_triggers(self.args[3], self.args[4:]) else: self.log.error('bad subcommand: ' + cmd) sys.exit(1) def collect_meta(self, table_list, meta, args): """Display fkey/trigger info.""" if args == []: args = ['pending', 'active'] field_map = {'triggers': ['table_name', 'trigger_name', 'trigger_def'], 'fkeys': ['from_table', 'to_table', 'fkey_name', 'fkey_def']} query_map = {'pending': "select %s from londiste.subscriber_get_table_pending_%s(%%s)", 'active' : "select %s from londiste.find_table_%s(%%s)"} table_list = self.clean_subscriber_tables(table_list) if len(table_list) == 0: self.log.info("No tables, no fkeys") return dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() for which in args: union_list = [] fields = field_map[meta] q = query_map[which] % (",".join(fields), meta) for tbl in table_list: union_list.append(q % skytools.quote_literal(tbl)) # use union as fkey may appear in duplicate sql = " union ".join(union_list) + " order by 1" desc = "%s %s" % (which, meta) self.display_table(desc, dst_curs, fields, sql) dst_db.commit() def display_table(self, desc, curs, fields, sql, args = []): """Display multirow query as a table.""" curs.execute(sql, args) rows = curs.dictfetchall() if len(rows) == 0: return 0 widths = [15] * len(fields) for row in rows: for i, k in enumerate(fields): widths[i] = widths[i] > len(row[k]) and widths[i] or len(row[k]) widths = [w + 2 for w in widths] fmt = '%%-%ds' * (len(widths) - 1) + '%%s' fmt = fmt % tuple(widths[:-1]) print desc print fmt % tuple(fields) print fmt % tuple(['-'*15] * len(fields)) for row in rows: print fmt % tuple([row[k] for k in fields]) print '\n' return 1 def clean_subscriber_tables(self, table_list): """Returns fully-quelifies table list of tables that are registered on subscriber. """ subscriber_tables = self.get_subscriber_table_list() if not table_list and self.options.all: table_list = subscriber_tables else: newlist = [] for tbl in table_list: tbl = skytools.fq_name(tbl) if tbl in subscriber_tables: newlist.append(tbl) else: #self.log.warning("table %s not subscribed" % tbl) pass table_list = newlist return table_list def check_tables(self, table_list): src_db = self.get_database('provider_db') src_curs = src_db.cursor() dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() failed = 0 for tbl in table_list: self.log.info('Checking %s' % tbl) if not skytools.exists_table(src_curs, tbl): self.log.error('Table %s missing from provider side' % tbl) failed += 1 elif not skytools.exists_table(dst_curs, tbl): self.log.error('Table %s missing from subscriber side' % tbl) failed += 1 else: failed += self.check_table_columns(src_curs, dst_curs, tbl) src_db.commit() dst_db.commit() return failed def restore_triggers(self, tbl, triggers=None): tbl = skytools.fq_name(tbl) if tbl not in self.get_subscriber_table_list(): self.log.error("Table %s is not in the subscriber queue." % tbl) sys.exit(1) dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() if not triggers: q = "select count(1) from londiste.subscriber_get_table_pending_triggers(%s)" dst_curs.execute(q, [tbl]) if not dst_curs.fetchone()[0]: self.log.info("No pending triggers found for %s." % tbl) else: q = "select londiste.subscriber_restore_all_table_triggers(%s)" dst_curs.execute(q, [tbl]) else: for trigger in triggers: q = "select count(1) from londiste.find_table_triggers(%s) where trigger_name=%s" dst_curs.execute(q, [tbl, trigger]) if dst_curs.fetchone()[0]: self.log.info("Trigger %s on %s is already active." % (trigger, tbl)) continue q = "select count(1) from londiste.subscriber_get_table_pending_triggers(%s) where trigger_name=%s" dst_curs.execute(q, [tbl, trigger]) if not dst_curs.fetchone()[0]: self.log.info("Trigger %s not found on %s" % (trigger, tbl)) continue q = "select londiste.subscriber_restore_table_trigger(%s, %s)" dst_curs.execute(q, [tbl, trigger]) dst_db.commit() def check_table_columns(self, src_curs, dst_curs, tbl): src_colrows = find_column_types(src_curs, tbl) dst_colrows = find_column_types(dst_curs, tbl) src_cols = make_type_string(src_colrows) dst_cols = make_type_string(dst_colrows) if src_cols.find('k') < 0: self.log.error('provider table %s has no primary key (%s)' % ( tbl, src_cols)) return 1 if dst_cols.find('k') < 0: self.log.error('subscriber table %s has no primary key (%s)' % ( tbl, dst_cols)) return 1 if src_cols != dst_cols: self.log.warning('table %s structure is not same (%s/%s)'\ ', trying to continue' % (tbl, src_cols, dst_cols)) err = 0 for row in src_colrows: found = 0 for row2 in dst_colrows: if row2['name'] == row['name']: found = 1 break if not found: err = 1 self.log.error('%s: column %s on provider not on subscriber' % (tbl, row['name'])) elif row['type'] != row2['type']: err = 1 self.log.error('%s: pk different on column %s' % (tbl, row['name'])) return err def subscriber_install(self): dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() install_subscriber(dst_curs, self.log) if self.fake: self.log.debug('rollback') dst_db.rollback() else: self.log.debug('commit') dst_db.commit() def subscriber_register(self): src_db = self.get_database('provider_db') src_curs = src_db.cursor() src_curs.execute("select pgq.register_consumer(%s, %s)", [self.pgq_queue_name, self.consumer_id]) src_db.commit() def subscriber_unregister(self): q = "select londiste.subscriber_set_table_state(%s, %s, NULL, NULL)" dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() tbl_rows = self.fetch_subscriber_tables(dst_curs) for row in tbl_rows: dst_curs.execute(q, [self.pgq_queue_name, row['table_name']]) dst_db.commit() src_db = self.get_database('provider_db') src_curs = src_db.cursor() src_curs.execute("select pgq.unregister_consumer(%s, %s)", [self.pgq_queue_name, self.consumer_id]) src_db.commit() def subscriber_show_tables(self): """print out subscriber table list, with state and snapshot""" dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() list = self.fetch_subscriber_tables(dst_curs) dst_db.commit() format = "%-30s %20s" print format % ("Table", "State") for tbl in list: print format % (tbl['table_name'], tbl['merge_state'] or '-') def subscriber_missing_tables(self): provider_tables = self.get_provider_table_list() subscriber_tables = self.get_subscriber_table_list() for tbl in provider_tables: if tbl not in subscriber_tables: print "Table: %s" % tbl provider_seqs = self.get_provider_seqs() subscriber_seqs = self.get_subscriber_seq_list() for seq in provider_seqs: if seq not in subscriber_seqs: print "Sequence: %s" % seq def find_missing_subscriber_tables(self, pattern='*'): src_db = self.get_database('subscriber_db') src_curs = src_db.cursor() q = """select schemaname || '.' || tablename as full_name from pg_tables where schemaname not in ('pgq', 'londiste', 'pg_catalog', 'information_schema') and schemaname !~ 'pg_.*' and schemaname || '.' || tablename ~ %s except select table_name from londiste.provider_get_table_list(%s)""" src_curs.execute(q, [glob2regex(pattern), self.pgq_queue_name]) rows = src_curs.fetchall() src_db.commit() list = [] for row in rows: list.append(row[0]) return list def subscriber_add_tables(self, table_list): provider_tables = self.get_provider_table_list() subscriber_tables = self.get_subscriber_table_list() if not table_list and self.options.all: table_list = ['*.*'] for tbl in provider_tables: if tbl not in subscriber_tables: table_list.append(tbl) tbls = [] for tbl in table_list: more = self.find_missing_subscriber_tables(skytools.fq_name(tbl)) if more == []: self.log.info("No tables found that match %s" % tbl) tbls.extend(more) tbls = list(set(tbls)) err = 0 table_list = [] for tbl in tbls: if tbl not in provider_tables: err = 1 self.log.error("Table %s not attached to queue" % tbl) if not self.options.force: continue table_list.append(tbl) if err: if self.options.force: self.log.warning('--force used, ignoring errors') err = self.check_tables(table_list) if err: if self.options.force: self.log.warning('--force used, ignoring errors') else: sys.exit(1) dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() for tbl in table_list: if tbl in subscriber_tables: self.log.info("Table %s already added" % tbl) else: self.log.info("Adding %s" % tbl) self.subscriber_add_one_table(dst_curs, tbl) dst_db.commit() def subscriber_remove_tables(self, table_list): subscriber_tables = self.get_subscriber_table_list() if not table_list and self.options.all: table_list = ['*.*'] for tbl in table_list: tbls = self.get_subscriber_table_list(skytools.fq_name(tbl)) for tbl in tbls: if tbl in subscriber_tables: self.log.info("Removing: %s" % tbl) self.subscriber_remove_one_table(tbl) else: self.log.info("Table %s already removed" % tbl) def subscriber_resync_tables(self, table_list): dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() list = self.fetch_subscriber_tables(dst_curs) if not table_list and self.options.all: table_list = self.get_subscriber_table_list() for tbl in table_list: tbl = skytools.fq_name(tbl) tbl_row = None for row in list: if row['table_name'] == tbl: tbl_row = row break if not tbl_row: self.log.warning("Table %s not found" % tbl) elif tbl_row['merge_state'] != 'ok': self.log.warning("Table %s is not in stable state" % tbl) else: self.log.info("Resyncing %s" % tbl) q = "select londiste.subscriber_set_table_state(%s, %s, NULL, NULL)" dst_curs.execute(q, [self.pgq_queue_name, tbl]) dst_db.commit() def subscriber_add_one_table(self, dst_curs, tbl): q_add = "select londiste.subscriber_add_table(%s, %s)" q_triggers = "select londiste.subscriber_drop_all_table_triggers(%s)" if self.options.expect_sync and self.options.skip_truncate: self.log.error("Too many options: --expect-sync and --skip-truncate") sys.exit(1) dst_curs.execute(q_add, [self.pgq_queue_name, tbl]) if self.options.expect_sync: q = "select londiste.subscriber_set_table_state(%s, %s, null, 'ok')" dst_curs.execute(q, [self.pgq_queue_name, tbl]) return dst_curs.execute(q_triggers, [tbl]) if self.options.skip_truncate: q = "select londiste.subscriber_set_skip_truncate(%s, %s, true)" dst_curs.execute(q, [self.pgq_queue_name, tbl]) def subscriber_remove_one_table(self, tbl): q_remove = "select londiste.subscriber_remove_table(%s, %s)" q_triggers = "select londiste.subscriber_restore_all_table_triggers(%s)" dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() dst_curs.execute(q_remove, [self.pgq_queue_name, tbl]) dst_curs.execute(q_triggers, [tbl]) dst_db.commit() def get_subscriber_seq_list(self): dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() q = "SELECT * from londiste.subscriber_get_seq_list(%s)" dst_curs.execute(q, [self.pgq_queue_name]) list = dst_curs.fetchall() dst_db.commit() res = [] for row in list: res.append(row[0]) return res def subscriber_list_seqs(self): list = self.get_subscriber_seq_list() for seq in list: print seq def subscriber_add_seq(self, seq_list): src_db = self.get_database('provider_db') src_curs = src_db.cursor() dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() prov_list = self.get_provider_seqs() full_list = self.get_all_seqs(dst_curs) cur_list = self.get_subscriber_seq_list() if not seq_list and self.options.all: seq_list = prov_list for seq in seq_list: seq = skytools.fq_name(seq) if seq not in prov_list: self.log.error('Seq %s does not exist on provider side' % seq) continue if seq not in full_list: self.log.error('Seq %s does not exist on subscriber side' % seq) continue if seq in cur_list: self.log.info('Seq %s already subscribed' % seq) continue self.log.info('Adding sequence: %s' % seq) q = "select londiste.subscriber_add_seq(%s, %s)" dst_curs.execute(q, [self.pgq_queue_name, seq]) dst_db.commit() def subscriber_remove_seq(self, seq_list): dst_db = self.get_database('subscriber_db') dst_curs = dst_db.cursor() cur_list = self.get_subscriber_seq_list() if not seq_list and self.options.all: seq_list = cur_list for seq in seq_list: seq = skytools.fq_name(seq) if seq not in cur_list: self.log.warning('Seq %s not subscribed') else: self.log.info('Removing sequence: %s' % seq) q = "select londiste.subscriber_remove_seq(%s, %s)" dst_curs.execute(q, [self.pgq_queue_name, seq]) dst_db.commit() skytools-2.1.13/python/londiste/table_copy.py0000644000175000017500000001027011670441266020335 0ustar markomarko#! /usr/bin/env python """Do a full table copy. For internal usage. """ import sys, os, skytools from skytools.dbstruct import * from playback import * __all__ = ['CopyTable'] class CopyTable(Replicator): def __init__(self, args, copy_thread = 1): Replicator.__init__(self, args) if copy_thread: self.pidfile += ".copy" self.consumer_id += "_copy" self.copy_thread = 1 def do_copy(self, tbl_stat): src_db = self.get_database('provider_db') dst_db = self.get_database('subscriber_db') # it should not matter to pgq src_db.commit() dst_db.commit() # we need to get the COPY snapshot later src_db.set_isolation_level(skytools.I_REPEATABLE_READ) src_db.commit() self.sync_database_encodings(src_db, dst_db) # initial sync copy src_curs = src_db.cursor() dst_curs = dst_db.cursor() self.log.info("Starting full copy of %s" % tbl_stat.name) # just in case, drop all fkeys (in case "replay" was skipped) # !! this may commit, so must be done before anything else !! self.drop_fkeys(dst_db, tbl_stat.name) # just in case, drop all triggers (in case "subscriber add" was skipped) q_triggers = "select londiste.subscriber_drop_all_table_triggers(%s)" dst_curs.execute(q_triggers, [tbl_stat.name]) # find dst struct src_struct = TableStruct(src_curs, tbl_stat.name) dst_struct = TableStruct(dst_curs, tbl_stat.name) # check if columns match dlist = dst_struct.get_column_list() for c in src_struct.get_column_list(): if c not in dlist: raise Exception('Column %s does not exist on dest side' % c) # drop unnecessary stuff objs = T_CONSTRAINT | T_INDEX | T_RULE | T_PARENT dst_struct.drop(dst_curs, objs, log = self.log) # do truncate & copy self.real_copy(src_curs, dst_curs, tbl_stat) # get snapshot src_curs.execute("select txid_current_snapshot()") snapshot = src_curs.fetchone()[0] src_db.commit() # restore READ COMMITTED behaviour src_db.set_isolation_level(skytools.I_READ_COMMITTED) src_db.commit() # create previously dropped objects dst_struct.create(dst_curs, objs, log = self.log) dst_db.commit() # set state if self.copy_thread: tbl_stat.change_state(TABLE_CATCHING_UP) else: tbl_stat.change_state(TABLE_OK) tbl_stat.change_snapshot(snapshot) self.save_table_state(dst_curs) dst_db.commit() self.log.debug("%s: ANALYZE" % tbl_stat.name) dst_curs.execute("analyze " + skytools.quote_fqident(tbl_stat.name)) dst_db.commit() # if copy done, request immidiate tick from pgqadm, # to make state juggling faster. on mostly idle db-s # each step may take tickers idle_timeout secs, which is pain. q = "select pgq.force_tick(%s)" src_curs.execute(q, [self.pgq_queue_name]) src_db.commit() def real_copy(self, srccurs, dstcurs, tbl_stat): "Main copy logic." tablename = tbl_stat.name # drop data if tbl_stat.skip_truncate: self.log.info("%s: skipping truncate" % tablename) else: self.log.info("%s: truncating" % tablename) # truncate behaviour changed in 8.4 dstcurs.execute("show server_version_num") pgver = int(dstcurs.fetchone()[0]) if pgver >= 80400: dstcurs.execute("truncate only " + skytools.quote_fqident(tablename)) else: dstcurs.execute("truncate " + skytools.quote_fqident(tablename)) # do copy self.log.info("%s: start copy" % tablename) col_list = skytools.get_table_columns(srccurs, tablename) stats = skytools.full_copy(tablename, srccurs, dstcurs, col_list) if stats: self.log.info("%s: copy finished: %d bytes, %d rows" % ( tablename, stats[0], stats[1])) if __name__ == '__main__': script = CopyTable(sys.argv[1:]) script.start() skytools-2.1.13/python/londiste/__init__.py0000644000175000017500000000143211670174255017754 0ustar markomarko """Replication on top of PgQ.""" import londiste.playback import londiste.compare import londiste.file_read import londiste.file_write import londiste.setup import londiste.table_copy import londiste.installer import londiste.repair from londiste.playback import * from londiste.compare import * from londiste.file_read import * from londiste.file_write import * from londiste.setup import * from londiste.table_copy import * from londiste.installer import * from londiste.repair import * __all__ = ( londiste.playback.__all__ + londiste.compare.__all__ + londiste.file_read.__all__ + londiste.file_write.__all__ + londiste.setup.__all__ + londiste.table_copy.__all__ + londiste.installer.__all__ + londiste.repair.__all__ ) skytools-2.1.13/python/londiste/compare.py0000644000175000017500000000221311670174255017641 0ustar markomarko#! /usr/bin/env python """Compares tables in replication set. Currently just does count(1) on both sides. """ import sys, os, time, skytools __all__ = ['Comparator'] from syncer import Syncer class Comparator(Syncer): def process_sync(self, tbl, src_db, dst_db): """Actual comparision.""" src_curs = src_db.cursor() dst_curs = dst_db.cursor() self.log.info('Counting %s' % tbl) q = "select count(1) from only _TABLE_" q = self.cf.get('compare_sql', q) q = q.replace('_TABLE_', skytools.quote_fqident(tbl)) self.log.debug("srcdb: " + q) src_curs.execute(q) src_row = src_curs.fetchone() src_str = ", ".join(map(str, src_row)) self.log.info("srcdb: res = %s" % src_str) self.log.debug("dstdb: " + q) dst_curs.execute(q) dst_row = dst_curs.fetchone() dst_str = ", ".join(map(str, dst_row)) self.log.info("dstdb: res = %s" % dst_str) if src_str != dst_str: self.log.warning("%s: Results do not match!" % tbl) if __name__ == '__main__': script = Comparator(sys.argv[1:]) script.start() skytools-2.1.13/python/londiste/syncer.py0000644000175000017500000001520711670441156017522 0ustar markomarko """Catch moment when tables are in sync on master and slave. """ import sys, time, skytools class Syncer(skytools.DBScript): """Walks tables in primary key order and checks if data matches.""" def __init__(self, args): skytools.DBScript.__init__(self, 'londiste', args) self.set_single_loop(1) self.pgq_queue_name = self.cf.get("pgq_queue_name") self.pgq_consumer_id = self.cf.get('pgq_consumer_id', self.job_name) self.lock_timeout = self.cf.getfloat('lock_timeout', 10) if self.pidfile: self.pidfile += ".repair" def set_lock_timeout(self, curs): ms = int(1000 * self.lock_timeout) if ms > 0: q = "SET LOCAL statement_timeout = %d" % ms self.log.debug(q) curs.execute(q) def init_optparse(self, p=None): p = skytools.DBScript.init_optparse(self, p) p.add_option("--force", action="store_true", help="ignore lag") return p def check_consumer(self, setup_curs): # before locking anything check if consumer is working ok q = "select extract(epoch from ticker_lag) from pgq.get_queue_info(%s)" setup_curs.execute(q, [self.pgq_queue_name]) ticker_lag = setup_curs.fetchone()[0] q = "select extract(epoch from lag)"\ " from pgq.get_consumer_info(%s, %s)" setup_curs.execute(q, [self.pgq_queue_name, self.pgq_consumer_id]) res = setup_curs.fetchall() if len(res) == 0: self.log.error('No such consumer') sys.exit(1) consumer_lag = res[0][0] if consumer_lag > ticker_lag + 10 and not self.options.force: self.log.error('Consumer lagging too much, cannot proceed') sys.exit(1) def get_subscriber_table_state(self, dst_db): dst_curs = dst_db.cursor() q = "select * from londiste.subscriber_get_table_list(%s)" dst_curs.execute(q, [self.pgq_queue_name]) res = dst_curs.dictfetchall() dst_db.commit() return res def work(self): src_loc = self.cf.get('provider_db') lock_db = self.get_database('provider_db', cache='lock_db') setup_db = self.get_database('provider_db', cache='setup_db', autocommit = 1) src_db = self.get_database('provider_db', isolation_level = skytools.I_REPEATABLE_READ) dst_db = self.get_database('subscriber_db', isolation_level = skytools.I_REPEATABLE_READ) setup_curs = setup_db.cursor() self.check_consumer(setup_curs) state_list = self.get_subscriber_table_state(dst_db) state_map = {} full_list = [] for ts in state_list: name = ts['table_name'] full_list.append(name) state_map[name] = ts if len(self.args) > 2: tlist = self.args[2:] else: tlist = full_list for tbl in tlist: tbl = skytools.fq_name(tbl) if not tbl in state_map: self.log.warning('Table not subscribed: %s' % tbl) continue st = state_map[tbl] if st['merge_state'] != 'ok': self.log.info('Table %s not synced yet, no point' % tbl) continue self.check_table(tbl, lock_db, src_db, dst_db, setup_curs) lock_db.commit() src_db.commit() dst_db.commit() def force_tick(self, setup_curs): q = "select pgq.force_tick(%s)" setup_curs.execute(q, [self.pgq_queue_name]) res = setup_curs.fetchone() cur_pos = res[0] start = time.time() while 1: time.sleep(0.5) setup_curs.execute(q, [self.pgq_queue_name]) res = setup_curs.fetchone() if res[0] != cur_pos: # new pos return res[0] # dont loop more than 10 secs dur = time.time() - start if dur > 10 and not self.options.force: raise Exception("Ticker seems dead") def check_table(self, tbl, lock_db, src_db, dst_db, setup_curs): """Get transaction to same state, then process.""" lock_curs = lock_db.cursor() src_curs = src_db.cursor() dst_curs = dst_db.cursor() if not skytools.exists_table(src_curs, tbl): self.log.warning("Table %s does not exist on provider side" % tbl) return if not skytools.exists_table(dst_curs, tbl): self.log.warning("Table %s does not exist on subscriber side" % tbl) return # lock table in separate connection self.log.info('Locking %s' % tbl) lock_db.commit() self.set_lock_timeout(lock_curs) lock_time = time.time() lock_curs.execute("LOCK TABLE %s IN SHARE MODE" % skytools.quote_fqident(tbl)) # now wait until consumer has updated target table until locking self.log.info('Syncing %s' % tbl) # consumer must get futher than this tick tick_id = self.force_tick(setup_curs) # try to force second tick also self.force_tick(setup_curs) # take server time setup_curs.execute("select to_char(now(), 'YYYY-MM-DD HH24:MI:SS.MS')") tpos = setup_curs.fetchone()[0] # now wait while 1: time.sleep(0.5) q = "select now() - lag > timestamp %s, now(), lag"\ " from pgq.get_consumer_info(%s, %s)" setup_curs.execute(q, [tpos, self.pgq_queue_name, self.pgq_consumer_id]) res = setup_curs.fetchall() if len(res) == 0: raise Exception('No such consumer') row = res[0] self.log.debug("tpos=%s now=%s lag=%s ok=%s" % (tpos, row[1], row[2], row[0])) if row[0]: break # limit lock time if time.time() > lock_time + self.lock_timeout and not self.options.force: self.log.error('Consumer lagging too much, exiting') lock_db.rollback() sys.exit(1) # take snapshot on provider side src_db.commit() src_curs.execute("SELECT 1") # take snapshot on subscriber side dst_db.commit() dst_curs.execute("SELECT 1") # release lock lock_db.commit() # do work self.process_sync(tbl, src_db, dst_db) # done src_db.commit() dst_db.commit() def process_sync(self, tbl, src_db, dst_db): """It gets 2 connections in state where tbl should be in same state. """ raise Exception('process_sync not implemented') skytools-2.1.13/python/londiste/file_read.py0000644000175000017500000000233011670174255020125 0ustar markomarko """Reads events from file instead of db queue.""" import sys, os, re, skytools from playback import * from table_copy import * __all__ = ['FileRead'] file_regex = r"^tick_0*([0-9]+)\.sql$" file_rc = re.compile(file_regex) class FileRead(CopyTable): """Reads events from file instead of db queue. Incomplete implementation. """ def __init__(self, args, log = None): CopyTable.__init__(self, args, log, copy_thread = 0) def launch_copy(self, tbl): # copy immidiately self.do_copy(t) def work(self): last_batch = self.get_last_batch(curs) list = self.get_file_list() def get_list(self): """Return list of (first_batch, full_filename) pairs.""" src_dir = self.cf.get('file_src') list = os.listdir(src_dir) list.sort() res = [] for fn in list: m = file_rc.match(fn) if not m: self.log.debug("Ignoring file: %s" % fn) continue full = os.path.join(src_dir, fn) batch_id = int(m.group(1)) res.append((batch_id, full)) return res if __name__ == '__main__': script = Replicator(sys.argv[1:]) script.start() skytools-2.1.13/python/londiste/playback.py0000644000175000017500000005773011670174255020017 0ustar markomarko#! /usr/bin/env python """Basic replication core.""" import sys, os, time import skytools, pgq __all__ = ['Replicator', 'TableState', 'TABLE_MISSING', 'TABLE_IN_COPY', 'TABLE_CATCHING_UP', 'TABLE_WANNA_SYNC', 'TABLE_DO_SYNC', 'TABLE_OK'] # state # owner - who is allowed to change TABLE_MISSING = 0 # main TABLE_IN_COPY = 1 # copy TABLE_CATCHING_UP = 2 # copy TABLE_WANNA_SYNC = 3 # main TABLE_DO_SYNC = 4 # copy TABLE_OK = 5 # setup SYNC_OK = 0 # continue with batch SYNC_LOOP = 1 # sleep, try again SYNC_EXIT = 2 # nothing to do, exit skript class Counter(object): """Counts table statuses.""" missing = 0 copy = 0 catching_up = 0 wanna_sync = 0 do_sync = 0 ok = 0 def __init__(self, tables): """Counts and sanity checks.""" for t in tables: if t.state == TABLE_MISSING: self.missing += 1 elif t.state == TABLE_IN_COPY: self.copy += 1 elif t.state == TABLE_CATCHING_UP: self.catching_up += 1 elif t.state == TABLE_WANNA_SYNC: self.wanna_sync += 1 elif t.state == TABLE_DO_SYNC: self.do_sync += 1 elif t.state == TABLE_OK: self.ok += 1 # only one table is allowed to have in-progress copy if self.in_progress_copy() > 1: raise Exception('Bad table state') def in_progress_copy(self): """ return how many tables currently having in-progress copy """ return self.copy + self.catching_up + self.wanna_sync + self.do_sync def get_running_copy_state(self): """ return TABLE_STATE of current running COPY table """ if self.in_progress_copy() == 0: return None if self.copy: return TABLE_IN_COPY elif self.catching_up: return TABLE_CATCHING_UP elif self.wanna_sync: return TABLE_WANNA_SYNC elif self.do_sync: return TABLE_DO_SYNC class TableState(object): """Keeps state about one table.""" def __init__(self, name, log): self.name = name self.log = log self.forget() self.changed = 0 self.skip_truncate = False def forget(self): self.state = TABLE_MISSING self.last_snapshot_tick = None self.str_snapshot = None self.from_snapshot = None self.sync_tick_id = None self.ok_batch_count = 0 self.last_tick = 0 self.skip_truncate = False self.changed = 1 def change_snapshot(self, str_snapshot, tag_changed = 1): if self.str_snapshot == str_snapshot: return self.log.debug("%s: change_snapshot to %s" % (self.name, str_snapshot)) self.str_snapshot = str_snapshot if str_snapshot: self.from_snapshot = skytools.Snapshot(str_snapshot) else: self.from_snapshot = None if tag_changed: self.ok_batch_count = 0 self.last_tick = None self.changed = 1 def change_state(self, state, tick_id = None): if self.state == state and self.sync_tick_id == tick_id: return self.state = state self.sync_tick_id = tick_id self.changed = 1 self.log.debug("%s: change_state to %s" % (self.name, self.render_state())) def render_state(self): """Make a string to be stored in db.""" if self.state == TABLE_MISSING: return None elif self.state == TABLE_IN_COPY: return 'in-copy' elif self.state == TABLE_CATCHING_UP: return 'catching-up' elif self.state == TABLE_WANNA_SYNC: return 'wanna-sync:%d' % self.sync_tick_id elif self.state == TABLE_DO_SYNC: return 'do-sync:%d' % self.sync_tick_id elif self.state == TABLE_OK: return 'ok' def parse_state(self, merge_state): """Read state from string.""" state = -1 if merge_state == None: state = TABLE_MISSING elif merge_state == "in-copy": state = TABLE_IN_COPY elif merge_state == "catching-up": state = TABLE_CATCHING_UP elif merge_state == "ok": state = TABLE_OK elif merge_state == "?": state = TABLE_OK else: tmp = merge_state.split(':') if len(tmp) == 2: self.sync_tick_id = int(tmp[1]) if tmp[0] == 'wanna-sync': state = TABLE_WANNA_SYNC elif tmp[0] == 'do-sync': state = TABLE_DO_SYNC if state < 0: raise Exception("Bad table state: %s" % merge_state) return state def loaded_state(self, merge_state, str_snapshot, skip_truncate): self.log.debug("loaded_state: %s: %s / %s" % ( self.name, merge_state, str_snapshot)) self.change_snapshot(str_snapshot, 0) self.state = self.parse_state(merge_state) self.changed = 0 self.skip_truncate = skip_truncate if merge_state == "?": self.changed = 1 def interesting(self, ev, tick_id, copy_thread): """Check if table wants this event.""" if copy_thread: if self.state not in (TABLE_CATCHING_UP, TABLE_DO_SYNC): return False else: if self.state != TABLE_OK: return False # if no snapshot tracking, then accept always if not self.from_snapshot: return True # uninteresting? if self.from_snapshot.contains(ev.txid): return False # after couple interesting batches there no need to check snapshot # as there can be only one partially interesting batch if tick_id != self.last_tick: self.last_tick = tick_id self.ok_batch_count += 1 # disable batch tracking if self.ok_batch_count > 3: self.change_snapshot(None) return True def gc_snapshot(self, copy_thread, prev_tick, cur_tick, no_lag): """Remove attached snapshot if possible. If the event processing is in current moment. the snapshot is not needed beyond next batch. The logic is needed for mostly unchanging tables, where the .ok_batch_count check in .interesting() method can take a lot of time. """ # check if gc is needed if self.str_snapshot is None: return # check if allowed to modify if copy_thread: if self.state != TABLE_CATCHING_UP: return else: if self.state != TABLE_OK: return False # aquire last tick if not self.last_snapshot_tick: if no_lag: self.last_snapshot_tick = cur_tick return # reset snapshot if not needed anymore if self.last_snapshot_tick < prev_tick: self.change_snapshot(None) class SeqCache(object): def __init__(self): self.seq_list = [] self.fq_seq_list = [] self.val_cache = {} def set_seq_list(self, seq_list): self.seq_list = seq_list self.fq_seq_list = [skytools.quote_fqident(s) for s in seq_list] new_cache = {} for seq in seq_list: val = self.val_cache.get(seq) if val: new_cache[seq] = val self.val_cache = new_cache def resync(self, src_curs, dst_curs): if len(self.seq_list) == 0: return dat = ".last_value, ".join(self.fq_seq_list) dat += ".last_value" q = "select %s from %s" % (dat, ",".join(self.fq_seq_list)) src_curs.execute(q) row = src_curs.fetchone() for i in range(len(self.seq_list)): seq = self.seq_list[i] fqseq = self.fq_seq_list[i] cur = row[i] old = self.val_cache.get(seq) if old != cur: q = "select setval(%s, %s)" dst_curs.execute(q, [fqseq, cur]) self.val_cache[seq] = cur class Replicator(pgq.SerialConsumer): """Replication core.""" sql_command = { 'I': "insert into %s %s;", 'U': "update only %s set %s;", 'D': "delete from only %s where %s;", } # batch info cur_tick = 0 prev_tick = 0 def __init__(self, args): pgq.SerialConsumer.__init__(self, 'londiste', 'provider_db', 'subscriber_db', args) # where get/set_last_tick() function reside for SerialConsumer(). # default is pgq_ext, but lets keep londiste code under one schema self.dst_schema = "londiste" self.table_list = [] self.table_map = {} self.copy_thread = 0 self.maint_time = 0 self.checked_copy = False self.seq_cache = SeqCache() self.maint_delay = self.cf.getint('maint_delay', 600) self.mirror_queue = self.cf.get('mirror_queue', '') def process_remote_batch(self, src_db, batch_id, ev_list, dst_db): "All work for a batch. Entry point from SerialConsumer." # this part can play freely with transactions dst_curs = dst_db.cursor() self.cur_tick = self.cur_batch_info['tick_id'] self.prev_tick = self.cur_batch_info['prev_tick_id'] self.load_table_state(dst_curs) self.sync_tables(dst_db) self.copy_snapshot_cleanup(dst_db) # only main thread is allowed to restore fkeys if not self.copy_thread: self.restore_fkeys(dst_db) # now the actual event processing happens. # they must be done all in one tx in dst side # and the transaction must be kept open so that # the SerialConsumer can save last tick and commit. self.sync_database_encodings(src_db, dst_db) self.handle_seqs(dst_curs) self.handle_events(dst_curs, ev_list) self.save_table_state(dst_curs) def handle_seqs(self, dst_curs): if self.copy_thread: return q = "select * from londiste.subscriber_get_seq_list(%s)" dst_curs.execute(q, [self.pgq_queue_name]) seq_list = [] for row in dst_curs.fetchall(): seq_list.append(row[0]) self.seq_cache.set_seq_list(seq_list) src_curs = self.get_database('provider_db').cursor() self.seq_cache.resync(src_curs, dst_curs) def sync_tables(self, dst_db): """Table sync loop. Calls appropriate handles, which is expected to return one of SYNC_* constants.""" self.log.debug('Sync tables') while 1: cnt = Counter(self.table_list) if self.copy_thread: res = self.sync_from_copy_thread(cnt, dst_db) else: res = self.sync_from_main_thread(cnt, dst_db) if res == SYNC_EXIT: self.log.debug('Sync tables: exit') self.detach() sys.exit(0) elif res == SYNC_OK: return elif res != SYNC_LOOP: raise Exception('Program error') self.log.debug('Sync tables: sleeping') time.sleep(3) dst_db.commit() self.load_table_state(dst_db.cursor()) dst_db.commit() def sync_from_main_thread(self, cnt, dst_db): "Main thread sync logic." if not self.checked_copy: self.relaunch_copy(cnt) self.checked_copy = True # # decide what to do - order is important # if cnt.do_sync: # wait for copy thread to catch up return SYNC_LOOP elif cnt.wanna_sync: # copy thread wants sync, if not behind, do it t = self.get_table_by_state(TABLE_WANNA_SYNC) if self.cur_tick >= t.sync_tick_id: self.change_table_state(dst_db, t, TABLE_DO_SYNC, self.cur_tick) return SYNC_LOOP else: return SYNC_OK elif cnt.catching_up: # active copy, dont worry return SYNC_OK elif cnt.copy: # active copy, dont worry return SYNC_OK elif cnt.missing: # seems there is no active copy thread, launch new t = self.get_table_by_state(TABLE_MISSING) # drop all foreign keys to and from this table self.drop_fkeys(dst_db, t.name) # change state after fkeys are dropped thus allowing # failure inbetween self.change_table_state(dst_db, t, TABLE_IN_COPY) # the copy _may_ happen immidiately self.launch_copy(t) # there cannot be interesting events in current batch # but maybe there's several tables, lets do them in one go return SYNC_LOOP else: # seems everything is in sync return SYNC_OK def sync_from_copy_thread(self, cnt, dst_db): "Copy thread sync logic." # # decide what to do - order is important # if cnt.do_sync: # main thread is waiting, catch up, then handle over t = self.get_table_by_state(TABLE_DO_SYNC) if self.cur_tick == t.sync_tick_id: self.change_table_state(dst_db, t, TABLE_OK) return SYNC_EXIT elif self.cur_tick < t.sync_tick_id: return SYNC_OK else: self.log.error("copy_sync: cur_tick=%d sync_tick=%d" % ( self.cur_tick, t.sync_tick_id)) raise Exception('Invalid table state') elif cnt.wanna_sync: # wait for main thread to react return SYNC_LOOP elif cnt.catching_up: # is there more work? if self.work_state: return SYNC_OK # seems we have catched up t = self.get_table_by_state(TABLE_CATCHING_UP) self.change_table_state(dst_db, t, TABLE_WANNA_SYNC, self.cur_tick) return SYNC_LOOP elif cnt.copy: # table is not copied yet, do it t = self.get_table_by_state(TABLE_IN_COPY) self.do_copy(t) # forget previous value self.work_state = 1 return SYNC_LOOP else: # nothing to do return SYNC_EXIT def handle_events(self, dst_curs, ev_list): "Actual event processing happens here." ignored_events = 0 self.sql_list = [] mirror_list = [] for ev in ev_list: if not self.interesting(ev): ignored_events += 1 ev.tag_done() continue if ev.type in ('I', 'U', 'D'): self.handle_data_event(ev, dst_curs) else: self.handle_system_event(ev, dst_curs) if self.mirror_queue: mirror_list.append(ev) # finalize table changes self.flush_sql(dst_curs) self.stat_add('ignored', ignored_events) # put events into mirror queue if requested if self.mirror_queue: self.fill_mirror_queue(mirror_list, dst_curs) def handle_data_event(self, ev, dst_curs): # buffer SQL statements, then send them together fqname = skytools.quote_fqident(ev.extra1) fmt = self.sql_command[ev.type] sql = fmt % (fqname, ev.data) self.sql_list.append(sql) if len(self.sql_list) > 200: self.flush_sql(dst_curs) ev.tag_done() def flush_sql(self, dst_curs): # send all buffered statements at once if len(self.sql_list) == 0: return buf = "\n".join(self.sql_list) self.sql_list = [] dst_curs.execute(buf) def interesting(self, ev): if ev.type not in ('I', 'U', 'D'): return 1 t = self.get_table_by_name(ev.extra1) if t: return t.interesting(ev, self.cur_tick, self.copy_thread) else: return 0 def handle_system_event(self, ev, dst_curs): "System event." if ev.type == "T": self.log.info("got new table event: "+ev.data) # check tables to be dropped name_list = [] for name in ev.data.split(','): name_list.append(name.strip()) del_list = [] for tbl in self.table_list: if tbl.name in name_list: continue del_list.append(tbl) # separate loop to avoid changing while iterating for tbl in del_list: self.log.info("Removing table %s from set" % tbl.name) self.remove_table(tbl, dst_curs) ev.tag_done() else: self.log.warning("Unknows op %s" % ev.type) ev.tag_failed("Unknown operation") def remove_table(self, tbl, dst_curs): del self.table_map[tbl.name] self.table_list.remove(tbl) q = "select londiste.subscriber_remove_table(%s, %s)" dst_curs.execute(q, [self.pgq_queue_name, tbl.name]) def load_table_state(self, curs): """Load table state from database. Todo: if all tables are OK, there is no need to load state on every batch. """ q = "select table_name, snapshot, merge_state, skip_truncate"\ " from londiste.subscriber_get_table_list(%s)" curs.execute(q, [self.pgq_queue_name]) new_list = [] new_map = {} for row in curs.dictfetchall(): t = self.get_table_by_name(row['table_name']) if not t: t = TableState(row['table_name'], self.log) t.loaded_state(row['merge_state'], row['snapshot'], row['skip_truncate']) new_list.append(t) new_map[t.name] = t self.table_list = new_list self.table_map = new_map def save_table_state(self, curs): """Store changed table state in database.""" got_changes = 0 for t in self.table_list: if not t.changed: continue merge_state = t.render_state() self.log.info("storing state of %s: copy:%d new_state:%s" % ( t.name, self.copy_thread, merge_state)) q = "select londiste.subscriber_set_table_state(%s, %s, %s, %s)" curs.execute(q, [self.pgq_queue_name, t.name, t.str_snapshot, merge_state]) t.changed = 0 got_changes = 1 def change_table_state(self, dst_db, tbl, state, tick_id = None): tbl.change_state(state, tick_id) self.save_table_state(dst_db.cursor()) dst_db.commit() self.log.info("Table %s status changed to '%s'" % ( tbl.name, tbl.render_state())) def get_table_by_state(self, state): "get first table with specific state" for t in self.table_list: if t.state == state: return t raise Exception('No table was found with state: %d' % state) def get_table_by_name(self, name): if name.find('.') < 0: name = "public.%s" % name if name in self.table_map: return self.table_map[name] return None def fill_mirror_queue(self, ev_list, dst_curs): # insert events rows = [] fields = ['ev_type', 'ev_data', 'ev_extra1'] for ev in mirror_list: rows.append((ev.type, ev.data, ev.extra1)) pgq.bulk_insert_events(dst_curs, rows, fields, self.mirror_queue) # create tick q = "select pgq.ticker(%s, %s)" dst_curs.execute(q, [self.mirror_queue, self.cur_tick]) def launch_copy(self, tbl_stat): self.log.info("Launching copy process") script = sys.argv[0] conf = self.cf.filename if self.options.verbose: cmd = "%s -d -v %s copy" else: cmd = "%s -d %s copy" cmd = cmd % (script, conf) # wait until existing copy finishes copy_pidfile = self.pidfile + ".copy" while skytools.signal_pidfile(copy_pidfile, 0): self.log.info("Waiting for existing copy to exit") time.sleep(2) self.log.debug("Launch args: "+repr(cmd)) res = os.system(cmd) self.log.debug("Launch result: "+repr(res)) def relaunch_copy(self, cnt): """ check if a copy was killed before completion """ # We decide to force to run a COPY if: # - a table is in "copy" state # - copy pidfile either does not exists or matches no running pid self.log.debug("Sync(main) in_progress_copy = %d" % (cnt.in_progress_copy())) if cnt.in_progress_copy() == 0: return copy_pidfile = self.pidfile + ".copy" if skytools.signal_pidfile(copy_pidfile, 0): # copy is running return self.log.info("Table have in-progress-copy but no process") if os.path.isfile(copy_pidfile): self.log.debug("removing stale copy pid file %s" \ % copy_pidfile) os.remove(copy_pidfile) state = cnt.get_running_copy_state() if state: t = self.get_table_by_state(state) self.log.debug("launch copy for %s in state %s" % (str(t), str(state))) self.launch_copy(t) else: self.log.error("Can't find copy-in-progress table " +\ "state to re-launch stale copy") def sync_database_encodings(self, src_db, dst_db): """Make sure client_encoding is same on both side.""" try: # psycopg2 if src_db.encoding != dst_db.encoding: dst_db.set_client_encoding(src_db.encoding) except AttributeError: # psycopg1 src_curs = src_db.cursor() dst_curs = dst_db.cursor() src_curs.execute("show client_encoding") src_enc = src_curs.fetchone()[0] dst_curs.execute("show client_encoding") dst_enc = dst_curs.fetchone()[0] if src_enc != dst_enc: dst_curs.execute("set client_encoding = %s", [src_enc]) def copy_snapshot_cleanup(self, dst_db): """Remove unnecassary snapshot info from tables.""" no_lag = not self.work_state changes = False for t in self.table_list: t.gc_snapshot(self.copy_thread, self.prev_tick, self.cur_tick, no_lag) if t.changed: changes = True if changes: self.save_table_state(dst_db.cursor()) dst_db.commit() def restore_fkeys(self, dst_db): """Restore fkeys that have both tables on sync.""" dst_curs = dst_db.cursor() # restore fkeys -- one at a time q = "select * from londiste.subscriber_get_queue_valid_pending_fkeys(%s)" dst_curs.execute(q, [self.pgq_queue_name]) list = dst_curs.dictfetchall() for row in list: self.log.info('Creating fkey: %(fkey_name)s (%(from_table)s --> %(to_table)s)' % row) q2 = "select londiste.subscriber_restore_table_fkey(%(from_table)s, %(fkey_name)s)" dst_curs.execute(q2, row) dst_db.commit() def drop_fkeys(self, dst_db, table_name): # drop all foreign keys to and from this table # they need to be dropped one at a time to avoid deadlocks with user code dst_curs = dst_db.cursor() q = "select * from londiste.find_table_fkeys(%s)" dst_curs.execute(q, [table_name]) list = dst_curs.dictfetchall() for row in list: self.log.info('Dropping fkey: %s' % row['fkey_name']) q2 = "select londiste.subscriber_drop_table_fkey(%(from_table)s, %(fkey_name)s)" dst_curs.execute(q2, row) dst_db.commit() if __name__ == '__main__': script = Replicator(sys.argv[1:]) script.start() skytools-2.1.13/python/londiste/installer.py0000644000175000017500000000145711670174255020221 0ustar markomarko """Functions to install londiste and its depentencies into database.""" import os, skytools __all__ = ['install_provider', 'install_subscriber'] provider_object_list = [ skytools.DBLanguage("plpgsql"), skytools.DBFunction('txid_current_snapshot', 0, sql_file = "txid.sql"), skytools.DBSchema('pgq', sql_file = "pgq.sql"), skytools.DBSchema('londiste', sql_file = "londiste.sql") ] subscriber_object_list = [ skytools.DBLanguage("plpgsql"), skytools.DBSchema('londiste', sql_file = "londiste.sql") ] def install_provider(curs, log): """Installs needed code into provider db.""" skytools.db_install(curs, provider_object_list, log) def install_subscriber(curs, log): """Installs needed code into subscriber db.""" skytools.db_install(curs, subscriber_object_list, log) skytools-2.1.13/python/conf/0000755000175000017500000000000011727601174014745 5ustar markomarkoskytools-2.1.13/python/conf/wal-slave.ini0000644000175000017500000000175011670174255017346 0ustar markomarko[wal-slave] job_name = servername_walmgr_slave logfile = ~/log/wal-slave.log use_skylog = 1 slave_data = /var/lib/postgresql/8.3/main slave_bin = /usr/lib/postgresql/8.3/bin slave_stop_cmd = /etc/init.d/postgresql-8.3 stop slave_start_cmd = /etc/init.d/postgresql-8.3 start slave_config_dir = /etc/postgresql/8.3/main # alternative pg_xlog directory for slave, symlinked to pg_xlog on restore #slave_pg_xlog = /vol2/pg_xlog slave = /var/lib/postgresql/walshipping completed_wals = %(slave)s/logs.complete partial_wals = %(slave)s/logs.partial full_backup = %(slave)s/data.master config_backup = %(slave)s/config.backup backup_datadir = yes keep_backups = 0 archive_command = # primary database connect string for hot standby -- enabling # this will cause the slave to be started in hot standby mode. #primary_conninfo = host=master port=5432 user=postgres skytools-2.1.13/python/conf/londiste.ini0000644000175000017500000000110111670174255017262 0ustar markomarko [londiste] # should be unique job_name = test_to_subcriber # source queue location provider_db = dbname=provider port=6000 host=127.0.0.1 # target database - it's preferable to run "londiste replay" # on same machine and use unix-socket or localhost to connect subscriber_db = dbname=subscriber port=6000 host=127.0.0.1 # source queue name pgq_queue_name = londiste.replika logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid # how often to poll event from provider #loop_delay = 1 # max locking time on provider (in seconds, float) #lock_timeout = 10.0 skytools-2.1.13/python/conf/wal-master.ini0000644000175000017500000000234511670174255017530 0ustar markomarko[wal-master] job_name = servername_walmgr_master logfile = /var/lib/postgresql/log/wal-master.log pidfile = /var/lib/postgresql/pid/wal-master.pid use_skylog = 1 master_db = dbname=template1 master_data = /var/lib/postgresql/8.3/main master_config = /etc/postgresql/8.3/main/postgresql.conf # set this only if you can afford database restarts during setup and stop. #master_restart_cmd = /etc/init.d/postgresql-8.3 restart slave_config = /var/lib/postgresql/conf/wal-slave.ini slave = slave:/var/lib/postgresql/walshipping completed_wals = %(slave)s/logs.complete partial_wals = %(slave)s/logs.partial full_backup = %(slave)s/data.master config_backup = %(slave)s/config.backup # syncdaemon update frequency loop_delay = 10.0 # use record based shipping available since 8.2 use_xlog_functions = 0 # pass -z to rsync, useful on low bandwidth links compression = 0 # keep symlinks for pg_xlog and pg_log keep_symlinks = 1 # tell walmgr to set wal_level to hot_standby during setup #hot_standby = 1 # periodic sync #command_interval = 600 #periodic_command = /var/lib/postgresql/walshipping/periodic.sh skytools-2.1.13/python/conf/pgqadm.ini0000644000175000017500000000047411670174255016726 0ustar markomarko [pgqadm] # should be globally unique job_name = pgqadm_somedb db = dbname=provider port=6000 host=127.0.0.1 # how often to run maintenance [minutes] maint_delay_min = 5 # how often to check for activity [secs] loop_delay = 0.1 logfile = ~/log/%(job_name)s.log pidfile = ~/pid/%(job_name)s.pid use_skylog = 0 skytools-2.1.13/python/conf/skylog.ini0000644000175000017500000000276511670174255016772 0ustar markomarko; notes: ; - 'args' is mandatory in [handler_*] sections ; - in lists there must not be spaces ; ; top-level config ; ; list of all loggers [loggers] keys=root ; root logger sees everything. there can be per-job configs by ; specifing loggers with job_name of the script ; list of all handlers [handlers] ;; seems logger module immidiately initalized all handlers, ;; whether they are actually used or not. so better ;; keep this list in sync with actual handler list ;keys=stderr,logdb,logsrv,logfile keys=stderr ; list of all formatters [formatters] keys=short,long,none ; ; map specific loggers to specifig handlers ; [logger_root] level=DEBUG ;handlers=stderr,logdb,logsrv,logfile handlers=stderr ; ; configure formatters ; [formatter_short] format=%(asctime)s %(levelname)s %(message)s datefmt=%H:%M [formatter_long] format=%(asctime)s %(process)s %(levelname)s %(message)s [formatter_none] format=%(message)s ; ; configure handlers ; ; file. args: stream [handler_stderr] class=StreamHandler args=(sys.stderr,) formatter=short ; log into db. args: conn_string [handler_logdb] class=skylog.LogDBHandler args=("host=127.0.0.1 port=5432 user=logger dbname=logdb",) formatter=none level=INFO ; JSON messages over UDP. args: host, port [handler_logsrv] class=skylog.UdpLogServerHandler args=('127.0.0.1', 6666) formatter=none ; rotating logfile. args: filename, maxsize, maxcount [handler_logfile] class=skylog.EasyRotatingFileHandler args=('~/log/%(job_name)s.log', 100*1024*1024, 3) formatter=long skytools-2.1.13/python/skytools/0000755000175000017500000000000011727601174015707 5ustar markomarkoskytools-2.1.13/python/skytools/gzlog.py0000644000175000017500000000141411670174255017405 0ustar markomarko """Atomic append of gzipped data. The point is - if several gzip streams are concated, they are read back as one whose stream. """ import gzip from cStringIO import StringIO __all__ = ['gzip_append'] # # gzip storage # def gzip_append(filename, data, level = 6): """Append a block of data to file with safety checks.""" # compress data buf = StringIO() g = gzip.GzipFile(fileobj = buf, compresslevel = level, mode = "w") g.write(data) g.close() zdata = buf.getvalue() # append, safely f = open(filename, "a+", 0) f.seek(0, 2) pos = f.tell() try: f.write(zdata) f.close() except Exception, ex: # rollback on error f.seek(pos, 0) f.truncate() f.close() raise ex skytools-2.1.13/python/skytools/config.py0000644000175000017500000001121011670174255017523 0ustar markomarko """Nicer config class.""" import sys, os, ConfigParser, socket __all__ = ['Config'] class Config(object): """Bit improved ConfigParser. Additional features: - Remembers section. - Acceps defaults in get() functions. - List value support. """ def __init__(self, main_section, filename, sane_config = 1, user_defs = {}): """Initialize Config and read from file. @param sane_config: chooses between ConfigParser/SafeConfigParser. """ defs = { 'job_name': main_section, 'service_name': main_section, 'host_name': socket.gethostname(), } defs.update(user_defs) self.main_section = main_section self.filename = filename self.sane_config = sane_config if sane_config: self.cf = ConfigParser.SafeConfigParser(defs) else: self.cf = ConfigParser.ConfigParser(defs) if filename is None: self.cf.add_section(main_section) return if not os.path.isfile(filename): raise Exception('Config file not found: '+filename) self.cf.read(filename) if not self.cf.has_section(main_section): raise Exception("Wrong config file, no section '%s'"%main_section) def reload(self): """Re-reads config file.""" if self.filename: self.cf.read(self.filename) def get(self, key, default=None): """Reads string value, if not set then default.""" try: return self.cf.get(self.main_section, key) except ConfigParser.NoOptionError, det: if default == None: raise Exception("Config value not set: " + key) return default def getint(self, key, default=None): """Reads int value, if not set then default.""" try: return self.cf.getint(self.main_section, key) except ConfigParser.NoOptionError, det: if default == None: raise Exception("Config value not set: " + key) return default def getboolean(self, key, default=None): """Reads boolean value, if not set then default.""" try: return self.cf.getboolean(self.main_section, key) except ConfigParser.NoOptionError, det: if default == None: raise Exception("Config value not set: " + key) return default def getfloat(self, key, default=None): """Reads float value, if not set then default.""" try: return self.cf.getfloat(self.main_section, key) except ConfigParser.NoOptionError, det: if default == None: raise Exception("Config value not set: " + key) return default def getlist(self, key, default=None): """Reads comma-separated list from key.""" try: s = self.cf.get(self.main_section, key).strip() res = [] if not s: return res for v in s.split(","): res.append(v.strip()) return res except ConfigParser.NoOptionError, det: if default == None: raise Exception("Config value not set: " + key) return default def getfile(self, key, default=None): """Reads filename from config. In addition to reading string value, expands ~ to user directory. """ fn = self.get(key, default) if fn == "" or fn == "-": return fn # simulate that the cwd is script location #path = os.path.dirname(sys.argv[0]) # seems bad idea, cwd should be cwd fn = os.path.expanduser(fn) return fn def get_wildcard(self, key, values=[], default=None): """Reads a wildcard property from conf and returns its string value, if not set then default.""" orig_key = key keys = [key] for wild in values: key = key.replace('*', wild, 1) keys.append(key) keys.reverse() for key in keys: try: return self.cf.get(self.main_section, key) except ConfigParser.NoOptionError, det: pass if default == None: raise Exception("Config value not set: " + orig_key) return default def sections(self): """Returns list of sections in config file, excluding DEFAULT.""" return self.cf.sections() def clone(self, main_section): """Return new Config() instance with new main section on same config file.""" return Config(main_section, self.filename, self.sane_config) skytools-2.1.13/python/skytools/skylog.py0000644000175000017500000001350511670174255017577 0ustar markomarko"""Our log handlers for Python's logging package. """ import sys, os, time, socket import logging, logging.handlers from skytools.psycopgwrapper import connect_database from skytools.quoting import quote_json _service_name = 'unknown_svc' def set_service_name(service_name): global _service_name _service_name = service_name # configurable file logger class EasyRotatingFileHandler(logging.handlers.RotatingFileHandler): """Easier setup for RotatingFileHandler.""" def __init__(self, filename, maxBytes = 10*1024*1024, backupCount = 3): """Args same as for RotatingFileHandler, but in filename '~' is expanded.""" fn = os.path.expanduser(filename) logging.handlers.RotatingFileHandler.__init__(self, fn, maxBytes=maxBytes, backupCount=backupCount) # send JSON message over UDP class UdpLogServerHandler(logging.handlers.DatagramHandler): """Sends log records over UDP to logserver in JSON format.""" # map logging levels to logserver levels _level_map = { logging.DEBUG : 'DEBUG', logging.INFO : 'INFO', logging.WARNING : 'WARN', logging.ERROR : 'ERROR', logging.CRITICAL: 'FATAL', } # JSON message template _log_template = '{\n\t'\ '"logger": "skytools.UdpLogServer",\n\t'\ '"timestamp": %.0f,\n\t'\ '"level": "%s",\n\t'\ '"thread": null,\n\t'\ '"message": %s,\n\t'\ '"properties": {"application":"%s", "apptype": "%s", "type": "sys", "hostname":"%s", "hostaddr": "%s"}\n'\ '}\n' # cut longer msgs MAXMSG = 1024 def makePickle(self, record): """Create message in JSON format.""" # get & cut msg msg = self.format(record) if len(msg) > self.MAXMSG: msg = msg[:self.MAXMSG] txt_level = self._level_map.get(record.levelno, "ERROR") hostname = socket.gethostname() try: hostaddr = socket.gethostbyname(hostname) except: hostaddr = "0.0.0.0" jobname = record.name svcname = _service_name pkt = self._log_template % (time.time()*1000, txt_level, quote_json(msg), jobname, svcname, hostname, hostaddr) return pkt def send(self, s): """Disable socket caching.""" sock = self.makeSocket() sock.sendto(s, (self.host, self.port)) sock.close() class LogDBHandler(logging.handlers.SocketHandler): """Sends log records into PostgreSQL server. Additionally, does some statistics aggregating, to avoid overloading log server. It subclasses SocketHandler to get throtthling for failed connections. """ # map codes to string _level_map = { logging.DEBUG : 'DEBUG', logging.INFO : 'INFO', logging.WARNING : 'WARNING', logging.ERROR : 'ERROR', logging.CRITICAL: 'FATAL', } def __init__(self, connect_string): """ Initializes the handler with a specific connection string. """ logging.handlers.SocketHandler.__init__(self, None, None) self.closeOnError = 1 self.connect_string = connect_string self.stat_cache = {} self.stat_flush_period = 60 # send first stat line immidiately self.last_stat_flush = 0 def createSocket(self): try: logging.handlers.SocketHandler.createSocket(self) except: self.sock = self.makeSocket() def makeSocket(self): """Create server connection. In this case its not socket but database connection.""" db = connect_database(self.connect_string) db.set_isolation_level(0) # autocommit return db def emit(self, record): """Process log record.""" # we do not want log debug messages if record.levelno < logging.INFO: return try: self.process_rec(record) except (SystemExit, KeyboardInterrupt): raise except: self.handleError(record) def process_rec(self, record): """Aggregate stats if needed, and send to logdb.""" # render msg msg = self.format(record) # dont want to send stats too ofter if record.levelno == logging.INFO and msg and msg[0] == "{": self.aggregate_stats(msg) if time.time() - self.last_stat_flush >= self.stat_flush_period: self.flush_stats(record.name) return if record.levelno < logging.INFO: self.flush_stats(record.name) # dont send more than one line ln = msg.find('\n') if ln > 0: msg = msg[:ln] txt_level = self._level_map.get(record.levelno, "ERROR") self.send_to_logdb(record.name, txt_level, msg) def aggregate_stats(self, msg): """Sum stats together, to lessen load on logdb.""" msg = msg[1:-1] for rec in msg.split(", "): k, v = rec.split(": ") agg = self.stat_cache.get(k, 0) if v.find('.') >= 0: agg += float(v) else: agg += int(v) self.stat_cache[k] = agg def flush_stats(self, service): """Send awuired stats to logdb.""" res = [] for k, v in self.stat_cache.items(): res.append("%s: %s" % (k, str(v))) if len(res) > 0: logmsg = "{%s}" % ", ".join(res) self.send_to_logdb(service, "INFO", logmsg) self.stat_cache = {} self.last_stat_flush = time.time() def send_to_logdb(self, service, type, msg): """Actual sending is done here.""" if self.sock is None: self.createSocket() if self.sock: logcur = self.sock.cursor() query = "select * from log.add(%s, %s, %s)" logcur.execute(query, [type, service, msg]) skytools-2.1.13/python/skytools/dbstruct.py0000644000175000017500000003454011670174255020123 0ustar markomarko"""Find table structure and allow CREATE/DROP elements from it. """ import sys, re from skytools.sqltools import fq_name_parts, get_table_oid from skytools.quoting import quote_ident, quote_fqident __all__ = ['TableStruct', 'T_TABLE', 'T_CONSTRAINT', 'T_INDEX', 'T_TRIGGER', 'T_RULE', 'T_GRANT', 'T_OWNER', 'T_PARENT', 'T_PKEY', 'T_ALL'] T_TABLE = 1 << 0 T_CONSTRAINT = 1 << 1 T_INDEX = 1 << 2 T_TRIGGER = 1 << 3 T_RULE = 1 << 4 T_GRANT = 1 << 5 T_OWNER = 1 << 6 T_PARENT = 1 << 7 T_PKEY = 1 << 20 # special, one of constraints T_ALL = ( T_TABLE | T_CONSTRAINT | T_INDEX | T_TRIGGER | T_RULE | T_GRANT | T_OWNER ) # # Utility functions # def find_new_name(curs, name): """Create new object name for case the old exists. Needed when creating a new table besides old one. """ # cut off previous numbers m = re.search('_[0-9]+$', name) if m: name = name[:m.start()] # now loop for i in range(1, 1000): tname = "%s_%d" % (name, i) q = "select count(1) from pg_class where relname = %s" curs.execute(q, [tname]) if curs.fetchone()[0] == 0: return tname # failed raise Exception('find_new_name failed') def rx_replace(rx, sql, new_part): """Find a regex match and replace that part with new_part.""" m = re.search(rx, sql, re.I) if not m: raise Exception('rx_replace failed') p1 = sql[:m.start()] p2 = sql[m.end():] return p1 + new_part + p2 # # Schema objects # class TElem(object): """Keeps info about one metadata object.""" SQL = "" type = 0 def get_create_sql(self, curs): """Return SQL statement for creating or None if not supported.""" return None def get_drop_sql(self, curs): """Return SQL statement for dropping or None of not supported.""" return None def get_load_sql(cls, pg_vers): """Return SQL statement for finding objects.""" return cls.SQL get_load_sql = classmethod(get_load_sql) class TConstraint(TElem): """Info about constraint.""" type = T_CONSTRAINT SQL = """ SELECT c.conname as name, pg_get_constraintdef(c.oid) as def, c.contype, i.indisclustered as is_clustered FROM pg_constraint c LEFT JOIN pg_index i ON c.conrelid = i.indrelid AND c.conname = (SELECT r.relname FROM pg_class r WHERE r.oid = i.indexrelid) WHERE c.conrelid = %(oid)s AND c.contype != 'f' """ def __init__(self, table_name, row): self.table_name = table_name self.name = row['name'] self.defn = row['def'] self.contype = row['contype'] self.is_clustered = row['is_clustered'] # tag pkeys if self.contype == 'p': self.type += T_PKEY def get_create_sql(self, curs, new_table_name=None): # no ONLY here as table with childs (only case that matters) # cannot have contraints that childs do not have fmt = "ALTER TABLE %s ADD CONSTRAINT %s %s;" if new_table_name: name = self.name if self.contype in ('p', 'u'): name = find_new_name(curs, self.name) qtbl = quote_fqident(new_table_name) qname = quote_ident(name) else: qtbl = quote_fqident(self.table_name) qname = quote_ident(self.name) sql = fmt % (qtbl, qname, self.defn) if self.is_clustered: sql +=' ALTER TABLE ONLY %s CLUSTER ON %s;' % (qtbl, qname) return sql def get_drop_sql(self, curs): fmt = "ALTER TABLE ONLY %s DROP CONSTRAINT %s;" sql = fmt % (quote_fqident(self.table_name), quote_ident(self.name)) return sql class TIndex(TElem): """Info about index.""" type = T_INDEX SQL = """ SELECT n.nspname || '.' || c.relname as name, pg_get_indexdef(i.indexrelid) as defn, c.relname as local_name, i.indisclustered as is_clustered FROM pg_index i, pg_class c, pg_namespace n WHERE c.oid = i.indexrelid AND i.indrelid = %(oid)s AND n.oid = c.relnamespace AND NOT EXISTS (select objid from pg_depend where classid = %(pg_class_oid)s and objid = c.oid and deptype = 'i') """ def __init__(self, table_name, row): self.name = row['name'] self.defn = row['defn'] + ';' self.is_clustered = row['is_clustered'] self.table_name = table_name self.local_name = row['local_name'] def get_create_sql(self, curs, new_table_name = None): if new_table_name: # fixme: seems broken iname = find_new_name(curs, self.name) tname = new_table_name pnew = "INDEX %s ON %s " % (quote_ident(iname), quote_fqident(tname)) rx = r"\bINDEX[ ][a-z0-9._]+[ ]ON[ ][a-z0-9._]+[ ]" sql = rx_replace(rx, self.defn, pnew) else: sql = self.defn iname = self.local_name tname = self.table_name if self.is_clustered: sql += ' ALTER TABLE ONLY %s CLUSTER ON %s;' % ( quote_fqident(tname), quote_ident(iname)) return sql def get_drop_sql(self, curs): return 'DROP INDEX %s;' % quote_fqident(self.name) class TRule(TElem): """Info about rule.""" type = T_RULE SQL = """ SELECT rulename as name, pg_get_ruledef(oid) as def FROM pg_rewrite WHERE ev_class = %(oid)s AND rulename <> '_RETURN'::name """ def __init__(self, table_name, row, new_name = None): self.table_name = table_name self.name = row['name'] self.defn = row['def'] def get_create_sql(self, curs, new_table_name = None): if not new_table_name: return self.defn # fixme: broken rx = r"\bTO[ ][a-z0-9._]+[ ]DO[ ]" pnew = "TO %s DO " % new_table_name return rx_replace(rx, self.defn, pnew) def get_drop_sql(self, curs): return 'DROP RULE %s ON %s' % (quote_ident(self.name), quote_fqident(self.table_name)) class TTrigger(TElem): """Info about trigger.""" type = T_TRIGGER def get_load_sql(cls, pg_vers): """Return SQL statement for finding objects.""" sql = "SELECT tgname as name, pg_get_triggerdef(oid) as def "\ " FROM pg_trigger "\ " WHERE tgrelid = %(oid)s AND " if pg_vers >= 90000: sql += "NOT tgisinternal" else: sql += "NOT tgisconstraint" return sql get_load_sql = classmethod(get_load_sql) def __init__(self, table_name, row): self.table_name = table_name self.name = row['name'] self.defn = row['def'] + ';' def get_create_sql(self, curs, new_table_name = None): if not new_table_name: return self.defn # fixme: broken rx = r"\bON[ ][a-z0-9._]+[ ]" pnew = "ON %s " % new_table_name return rx_replace(rx, self.defn, pnew) def get_drop_sql(self, curs): return 'DROP TRIGGER %s ON %s' % (quote_ident(self.name), quote_fqident(self.table_name)) class TParent(TElem): """Info about trigger.""" type = T_PARENT SQL = """ SELECT n.nspname||'.'||c.relname AS name FROM pg_inherits i JOIN pg_class c ON i.inhparent = c.oid JOIN pg_namespace n ON c.relnamespace = n.oid WHERE i.inhrelid = %(oid)s """ def __init__(self, table_name, row): self.name = table_name self.parent_name = row['name'] def get_create_sql(self, curs, new_table_name = None): return 'ALTER TABLE ONLY %s INHERIT %s' % (quote_fqident(self.name), quote_fqident(self.parent_name)) def get_drop_sql(self, curs): return 'ALTER TABLE ONLY %s NO INHERIT %s' % (quote_fqident(self.name), quote_fqident(self.parent_name)) class TOwner(TElem): """Info about table owner.""" type = T_OWNER SQL = """ SELECT pg_get_userbyid(relowner) as owner FROM pg_class WHERE oid = %(oid)s """ def __init__(self, table_name, row, new_name = None): self.table_name = table_name self.name = 'Owner' self.owner = row['owner'] def get_create_sql(self, curs, new_name = None): if not new_name: new_name = self.table_name return 'ALTER TABLE %s OWNER TO %s;' % (quote_fqident(new_name), quote_ident(self.owner)) class TGrant(TElem): """Info about permissions.""" type = T_GRANT SQL = "SELECT relacl FROM pg_class where oid = %(oid)s" acl_map = { 'r': 'SELECT', 'w': 'UPDATE', 'a': 'INSERT', 'd': 'DELETE', 'R': 'RULE', 'x': 'REFERENCES', 't': 'TRIGGER', 'X': 'EXECUTE', 'U': 'USAGE', 'C': 'CREATE', 'T': 'TEMPORARY' } def acl_to_grants(self, acl): if acl == "arwdRxt": # ALL for tables return "ALL" return ", ".join([ self.acl_map[c] for c in acl ]) def parse_relacl(self, relacl): if relacl is None: return [] if len(relacl) > 0 and relacl[0] == '{' and relacl[-1] == '}': relacl = relacl[1:-1] list = [] for f in relacl.split(','): user, tmp = f.strip('"').split('=') acl, who = tmp.split('/') list.append((user, acl, who)) return list def __init__(self, table_name, row, new_name = None): self.name = table_name self.acl_list = self.parse_relacl(row['relacl']) def get_create_sql(self, curs, new_name = None): if not new_name: new_name = self.name list = [] for user, acl, who in self.acl_list: astr = self.acl_to_grants(acl) sql = "GRANT %s ON %s TO %s;" % (astr, quote_fqident(new_name), quote_ident(user)) list.append(sql) return "\n".join(list) def get_drop_sql(self, curs): list = [] for user, acl, who in self.acl_list: sql = "REVOKE ALL FROM %s ON %s;" % (quote_ident(user), quote_fqident(self.name)) list.append(sql) return "\n".join(list) class TColumn(TElem): """Info about table column.""" SQL = """ select a.attname as name, a.attname || ' ' || format_type(a.atttypid, a.atttypmod) || case when a.attnotnull then ' not null' else '' end || case when a.atthasdef then ' default ' || d.adsrc else '' end as def from pg_attribute a left join pg_attrdef d on (d.adrelid = a.attrelid and d.adnum = a.attnum) where a.attrelid = %(oid)s and not a.attisdropped and a.attnum > 0 order by a.attnum; """ def __init__(self, table_name, row): self.name = row['name'] self.column_def = row['def'] class TTable(TElem): """Info about table only (columns).""" type = T_TABLE def __init__(self, table_name, col_list): self.name = table_name self.col_list = col_list def get_create_sql(self, curs, new_name = None): if not new_name: new_name = self.name sql = "create table %s (" % quote_fqident(new_name) sep = "\n\t" for c in self.col_list: sql += sep + c.column_def sep = ",\n\t" sql += "\n);" return sql def get_drop_sql(self, curs): return "DROP TABLE %s;" % quote_fqident(self.name) # # Main table object, loads all the others # class TableStruct(object): """Collects and manages all info about table. Allow to issue CREATE/DROP statements about any group of elements. """ def __init__(self, curs, table_name): """Initializes class by loading info about table_name from database.""" self.table_name = table_name # fill args schema, name = fq_name_parts(table_name) args = { 'schema': schema, 'table': name, 'oid': get_table_oid(curs, table_name), 'pg_class_oid': get_table_oid(curs, 'pg_catalog.pg_class'), } # load table struct self.col_list = self._load_elem(curs, args, TColumn) self.object_list = [ TTable(table_name, self.col_list) ] # load additional objects to_load = [TConstraint, TIndex, TTrigger, TRule, TGrant, TOwner, TParent] for eclass in to_load: self.object_list += self._load_elem(curs, args, eclass) def _load_elem(self, curs, args, eclass): list = [] sql = eclass.get_load_sql(curs.connection.server_version) curs.execute(sql % args) for row in curs.dictfetchall(): list.append(eclass(self.table_name, row)) return list def create(self, curs, objs, new_table_name = None, log = None): """Issues CREATE statements for requested set of objects. If new_table_name is giver, creates table under that name and also tries to rename all indexes/constraints that conflict with existing table. """ for o in self.object_list: if o.type & objs: sql = o.get_create_sql(curs, new_table_name) if not sql: continue if log: log.info('Creating %s' % o.name) log.debug(sql) curs.execute(sql) def drop(self, curs, objs, log = None): """Issues DROP statements for requested set of objects.""" # make sure the creating & dropping happen in reverse order olist = self.object_list[:] olist.reverse() for o in olist: if o.type & objs: sql = o.get_drop_sql(curs) if not sql: continue if log: log.info('Dropping %s' % o.name) log.debug(sql) curs.execute(sql) def get_column_list(self): """Returns list of column names the table has.""" res = [] for c in self.col_list: res.append(c.name) return res def test(): from skytools import connect_database db = connect_database("dbname=fooz") curs = db.cursor() s = TableStruct(curs, "public.data1") s.drop(curs, T_ALL) s.create(curs, T_ALL) s.create(curs, T_ALL, "data1_new") s.create(curs, T_PKEY) if __name__ == '__main__': test() skytools-2.1.13/python/skytools/sqltools.py0000644000175000017500000002733211670174255020152 0ustar markomarko """Database tools.""" import os from cStringIO import StringIO from skytools.quoting import quote_copy, quote_literal, quote_ident, quote_fqident import skytools.installer_config __all__ = [ "fq_name_parts", "fq_name", "get_table_oid", "get_table_pkeys", "get_table_columns", "exists_schema", "exists_table", "exists_type", "exists_function", "exists_language", "Snapshot", "magic_insert", "CopyPipe", "full_copy", "DBObject", "DBSchema", "DBTable", "DBFunction", "DBLanguage", "db_install", "installer_find_file", "installer_apply_file", ] # # Fully qualified table name # def fq_name_parts(tbl): "Return fully qualified name parts." tmp = tbl.split('.', 1) if len(tmp) == 1: return ('public', tbl) elif len(tmp) == 2: return tmp else: raise Exception('Syntax error in table name:'+tbl) def fq_name(tbl): "Return fully qualified name." return '.'.join(fq_name_parts(tbl)) # # info about table # def get_table_oid(curs, table_name): schema, name = fq_name_parts(table_name) q = """select c.oid from pg_namespace n, pg_class c where c.relnamespace = n.oid and n.nspname = %s and c.relname = %s""" curs.execute(q, [schema, name]) res = curs.fetchall() if len(res) == 0: raise Exception('Table not found: '+table_name) return res[0][0] def get_table_pkeys(curs, tbl): oid = get_table_oid(curs, tbl) q = "SELECT k.attname FROM pg_index i, pg_attribute k"\ " WHERE i.indrelid = %s AND k.attrelid = i.indexrelid"\ " AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped"\ " ORDER BY k.attnum" curs.execute(q, [oid]) return map(lambda x: x[0], curs.fetchall()) def get_table_columns(curs, tbl): oid = get_table_oid(curs, tbl) q = "SELECT k.attname FROM pg_attribute k"\ " WHERE k.attrelid = %s"\ " AND k.attnum > 0 AND NOT k.attisdropped"\ " ORDER BY k.attnum" curs.execute(q, [oid]) return map(lambda x: x[0], curs.fetchall()) # # exist checks # def exists_schema(curs, schema): q = "select count(1) from pg_namespace where nspname = %s" curs.execute(q, [schema]) res = curs.fetchone() return res[0] def exists_table(curs, table_name): schema, name = fq_name_parts(table_name) q = """select count(1) from pg_namespace n, pg_class c where c.relnamespace = n.oid and c.relkind = 'r' and n.nspname = %s and c.relname = %s""" curs.execute(q, [schema, name]) res = curs.fetchone() return res[0] def exists_type(curs, type_name): schema, name = fq_name_parts(type_name) q = """select count(1) from pg_namespace n, pg_type t where t.typnamespace = n.oid and n.nspname = %s and t.typname = %s""" curs.execute(q, [schema, name]) res = curs.fetchone() return res[0] def exists_function(curs, function_name, nargs): # this does not check arg types, so may match several functions schema, name = fq_name_parts(function_name) q = """select count(1) from pg_namespace n, pg_proc p where p.pronamespace = n.oid and p.pronargs = %s and n.nspname = %s and p.proname = %s""" curs.execute(q, [nargs, schema, name]) res = curs.fetchone() # if unqualified function, check builtin functions too if not res[0] and function_name.find('.') < 0: name = "pg_catalog." + function_name return exists_function(curs, name, nargs) return res[0] def exists_language(curs, lang_name): q = """select count(1) from pg_language where lanname = %s""" curs.execute(q, [lang_name]) res = curs.fetchone() return res[0] # # Support for PostgreSQL snapshot # class Snapshot(object): "Represents a PostgreSQL snapshot." def __init__(self, str): "Create snapshot from string." self.sn_str = str tmp = str.split(':') if len(tmp) != 3: raise Exception('Unknown format for snapshot') self.xmin = int(tmp[0]) self.xmax = int(tmp[1]) self.txid_list = [] if tmp[2] != "": for s in tmp[2].split(','): self.txid_list.append(int(s)) def contains(self, txid): "Is txid visible in snapshot." txid = int(txid) if txid < self.xmin: return True if txid >= self.xmax: return False if txid in self.txid_list: return False return True # # Copy helpers # def _gen_dict_copy(tbl, row, fields, qfields): tmp = [] for f in fields: v = row.get(f) tmp.append(quote_copy(v)) return "\t".join(tmp) def _gen_dict_insert(tbl, row, fields, qfields): tmp = [] for f in fields: v = row.get(f) tmp.append(quote_literal(v)) fmt = "insert into %s (%s) values (%s);" return fmt % (tbl, ",".join(qfields), ",".join(tmp)) def _gen_list_copy(tbl, row, fields, qfields): tmp = [] for i in range(len(fields)): v = row[i] tmp.append(quote_copy(v)) return "\t".join(tmp) def _gen_list_insert(tbl, row, fields, qfields): tmp = [] for i in range(len(fields)): v = row[i] tmp.append(quote_literal(v)) fmt = "insert into %s (%s) values (%s);" return fmt % (tbl, ",".join(qfields), ",".join(tmp)) def magic_insert(curs, tablename, data, fields = None, use_insert = 0): """Copy/insert a list of dict/list data to database. If curs == None, then the copy or insert statements are returned as string. For list of dict the field list is optional, as its possible to guess them from dict keys. """ if len(data) == 0: return # decide how to process if hasattr(data[0], 'keys'): if fields == None: fields = data[0].keys() if use_insert: row_func = _gen_dict_insert else: row_func = _gen_dict_copy else: if fields == None: raise Exception("Non-dict data needs field list") if use_insert: row_func = _gen_list_insert else: row_func = _gen_list_copy qfields = [quote_ident(f) for f in fields] qtablename = quote_fqident(tablename) # init processing buf = StringIO() if curs == None and use_insert == 0: fmt = "COPY %s (%s) FROM STDIN;\n" buf.write(fmt % (qtablename, ",".join(qfields))) # process data for row in data: buf.write(row_func(qtablename, row, fields, qfields)) buf.write("\n") # if user needs only string, return it if curs == None: if use_insert == 0: buf.write("\\.\n") return buf.getvalue() # do the actual copy/inserts if use_insert: curs.execute(buf.getvalue()) else: buf.seek(0) hdr = "%s (%s)" % (qtablename, ",".join(qfields)) curs.copy_from(buf, hdr) # # Full COPY of table from one db to another # class CopyPipe(object): "Splits one big COPY to chunks." def __init__(self, dstcurs, tablename = None, limit = 512*1024, cancel_func=None, sql_from = None): self.tablename = tablename self.sql_from = sql_from self.dstcurs = dstcurs self.buf = StringIO() self.limit = limit self.cancel_func = None self.total_rows = 0 self.total_bytes = 0 def write(self, data): "New data from psycopg" self.total_bytes += len(data) self.total_rows += data.count("\n") if self.buf.tell() >= self.limit: pos = data.find('\n') if pos >= 0: # split at newline p1 = data[:pos + 1] p2 = data[pos + 1:] self.buf.write(p1) self.flush() data = p2 self.buf.write(data) def flush(self): "Send data out." if self.cancel_func: self.cancel_func() if self.buf.tell() <= 0: return self.buf.seek(0) if self.sql_from: self.dstcurs.copy_expert(self.sql_from, self.buf) else: self.dstcurs.copy_from(self.buf, self.tablename) self.buf.seek(0) self.buf.truncate() def full_copy(tablename, src_curs, dst_curs, column_list = []): """COPY table from one db to another.""" qtable = quote_fqident(tablename) if column_list: qfields = [quote_ident(f) for f in column_list] hdr = "%s (%s)" % (qtable, ",".join(qfields)) else: hdr = qtable if hasattr(src_curs, 'copy_expert'): sql_to = "COPY %s TO stdout" % hdr sql_from = "COPY %s FROM stdin" % hdr buf = CopyPipe(dst_curs, sql_from = sql_from) src_curs.copy_expert(sql_to, buf) else: buf = CopyPipe(dst_curs, hdr) src_curs.copy_to(buf, hdr) buf.flush() return (buf.total_bytes, buf.total_rows) # # SQL installer # class DBObject(object): """Base class for installable DB objects.""" name = None sql = None sql_file = None def __init__(self, name, sql = None, sql_file = None): self.name = name self.sql = sql self.sql_file = sql_file def create(self, curs, log = None): if log: log.info('Installing %s' % self.name) if self.sql: sql = self.sql elif self.sql_file: fn = self.find_file() if log: log.info(" Reading from %s" % fn) sql = open(fn, "r").read() else: raise Exception('object not defined') curs.execute(sql) def find_file(self): full_fn = None if self.sql_file[0] == "/": full_fn = self.sql_file else: dir_list = skytools.installer_config.sql_locations for dir in dir_list: fn = os.path.join(dir, self.sql_file) if os.path.isfile(fn): full_fn = fn break if not full_fn: raise Exception('File not found: '+self.sql_file) return full_fn class DBSchema(DBObject): """Handles db schema.""" def exists(self, curs): return exists_schema(curs, self.name) class DBTable(DBObject): """Handles db table.""" def exists(self, curs): return exists_table(curs, self.name) class DBFunction(DBObject): """Handles db function.""" def __init__(self, name, nargs, sql = None, sql_file = None): DBObject.__init__(self, name, sql, sql_file) self.nargs = nargs def exists(self, curs): return exists_function(curs, self.name, self.nargs) class DBLanguage(DBObject): """Handles db language.""" def __init__(self, name): DBObject.__init__(self, name, sql = "create language %s" % name) def exists(self, curs): return exists_language(curs, self.name) def db_install(curs, list, log = None): """Installs list of objects into db.""" for obj in list: if not obj.exists(curs): obj.create(curs, log) else: if log: log.info('%s is installed' % obj.name) def installer_find_file(filename): full_fn = None if filename[0] == "/": if os.path.isfile(filename): full_fn = filename else: dir_list = ["."] + skytools.installer_config.sql_locations for dir in dir_list: fn = os.path.join(dir, filename) if os.path.isfile(fn): full_fn = fn break if not full_fn: raise Exception('File not found: '+filename) return full_fn def installer_apply_file(db, filename, log): fn = installer_find_file(filename) sql = open(fn, "r").read() if log: log.info("applying %s" % fn) curs = db.cursor() for stmt in skytools.parse_statements(sql): log.debug(repr(stmt)) curs.execute(stmt) skytools-2.1.13/python/skytools/parsing.py0000644000175000017500000001713311670174255017733 0ustar markomarko """Various parsers for Postgres-specific data formats.""" import re from skytools.quoting import unescape, unquote_literal, unquote_ident __all__ = ["parse_pgarray", "parse_logtriga_sql", "parse_tabbed_table", "parse_statements"] _rc_listelem = re.compile(r'( [^,"}]+ | ["] ( [^"\\]+ | [\\]. )* ["] )', re.X) # _parse_pgarray def parse_pgarray(array): """ Parse Postgres array and return list of items inside it Used to deserialize data recived from service layer parameters """ if not array or array[0] != "{": raise Exception("bad array format: must start with {") res = [] pos = 1 while 1: m = _rc_listelem.search(array, pos) if not m: break pos2 = m.end() item = array[pos:pos2] if len(item) > 0 and item[0] == '"': item = item[1:-1] item = unescape(item) res.append(item) pos = pos2 + 1 if array[pos2] == "}": break elif array[pos2] != ",": raise Exception("bad array format: expected ,} got " + array[pos2]) return res # # parse logtriga partial sql # class _logtriga_parser: def tokenizer(self, sql): for typ, tok in sql_tokenizer(sql, ignore_whitespace = True): yield tok def parse_insert(self, tk, fields, values): # (col1, col2) values ('data', null) if tk.next() != "(": raise Exception("syntax error") while 1: fields.append(tk.next()) t = tk.next() if t == ")": break elif t != ",": raise Exception("syntax error") if tk.next().lower() != "values": raise Exception("syntax error, expected VALUES") if tk.next() != "(": raise Exception("syntax error, expected (") while 1: values.append(tk.next()) t = tk.next() if t == ")": break if t == ",": continue raise Exception("expected , or ) got "+t) t = tk.next() raise Exception("expected EOF, got " + repr(t)) def parse_update(self, tk, fields, values): # col1 = 'data1', col2 = null where pk1 = 'pk1' and pk2 = 'pk2' while 1: fields.append(tk.next()) if tk.next() != "=": raise Exception("syntax error") values.append(tk.next()) t = tk.next() if t == ",": continue elif t.lower() == "where": break else: raise Exception("syntax error, expected WHERE or , got "+repr(t)) while 1: fields.append(tk.next()) if tk.next() != "=": raise Exception("syntax error") values.append(tk.next()) t = tk.next() if t.lower() != "and": raise Exception("syntax error, expected AND got "+repr(t)) def parse_delete(self, tk, fields, values): # pk1 = 'pk1' and pk2 = 'pk2' while 1: fields.append(tk.next()) if tk.next() != "=": raise Exception("syntax error") values.append(tk.next()) t = tk.next() if t.lower() != "and": raise Exception("syntax error, expected AND, got "+repr(t)) def parse_sql(self, op, sql): tk = self.tokenizer(sql) fields = [] values = [] try: if op == "I": self.parse_insert(tk, fields, values) elif op == "U": self.parse_update(tk, fields, values) elif op == "D": self.parse_delete(tk, fields, values) raise Exception("syntax error") except StopIteration: # last sanity check if len(fields) == 0 or len(fields) != len(values): raise Exception("syntax error, fields do not match values") fields = [unquote_ident(f) for f in fields] values = [unquote_literal(f) for f in values] return dict(zip(fields, values)) def parse_logtriga_sql(op, sql): """Parse partial SQL used by logtriga() back to data values. Parser has following limitations: - Expects standard_quoted_strings = off - Does not support dollar quoting. - Does not support complex expressions anywhere. (hashtext(col1) = hashtext(val1)) - WHERE expression must not contain IS (NOT) NULL - Does not support updateing pk value. Returns dict of col->data pairs. """ return _logtriga_parser().parse_sql(op, sql) def parse_tabbed_table(txt): """Parse a tab-separated table into list of dicts. Expect first row to be column names. Very primitive. """ txt = txt.replace("\r\n", "\n") fields = None data = [] for ln in txt.split("\n"): if not ln: continue if not fields: fields = ln.split("\t") continue cols = ln.split("\t") if len(cols) != len(fields): continue row = dict(zip(fields, cols)) data.append(row) return data _extstr = r""" ['] (?: [^'\\]+ | \\. | [']['] )* ['] """ _stdstr = r""" ['] (?: [^']+ | [']['] )* ['] """ _base_sql = r""" (?P [a-z][a-z0-9_$]* | ["] (?: [^"]+ | ["]["] )* ["] ) | (?P (?P [$] (?: [_a-z][_a-z0-9]*)? [$] ) .*? (?P=dname) ) | (?P [0-9][0-9.e]* ) | (?P [$] [0-9]+ ) | (?P [%][(] [a-z0-9_]+ [)][s] | [%][%] ) | (?P [{] [^}]+ [}] | [{][{] | [}] [}] ) | (?P (?: \s+ | [/][*] .*? [*][/] | [-][-][^\n]* )+ ) | (?P . )""" _std_sql = r"""(?: (?P [E] %s | %s ) | %s )""" % (_extstr, _stdstr, _base_sql) _ext_sql = r"""(?: (?P [E]? %s ) | %s )""" % (_extstr, _base_sql) _std_sql_rc = _ext_sql_rc = None def sql_tokenizer(sql, standard_quoting = False, ignore_whitespace = False): """Parser SQL to tokens. Iterator, returns (toktype, tokstr) tuples. """ global _std_sql_rc, _ext_sql_rc if not _std_sql_rc: _std_sql_rc = re.compile(_std_sql, re.X | re.I | re.S) _ext_sql_rc = re.compile(_ext_sql, re.X | re.I | re.S) if standard_quoting: rc = _std_sql_rc else: rc = _ext_sql_rc pos = 0 while 1: m = rc.match(sql, pos) if not m: break pos = m.end() typ = m.lastgroup if not ignore_whitespace or typ != "ws": yield (m.lastgroup, m.group()) _copy_from_stdin_re = "copy.*from\s+stdin" _copy_from_stdin_rc = None def parse_statements(sql, standard_quoting = False): """Parse multi-statement string into separate statements. Returns list of statements. """ global _copy_from_stdin_rc if not _copy_from_stdin_rc: _copy_from_stdin_rc = re.compile(_copy_from_stdin_re, re.X | re.I) tokens = [] pcount = 0 # '(' level for typ, t in sql_tokenizer(sql, standard_quoting = standard_quoting): # skip whitespace and comments before statement if len(tokens) == 0 and typ == "ws": continue # keep the rest tokens.append(t) if t == "(": pcount += 1 elif t == ")": pcount -= 1 elif t == ";" and pcount == 0: sql = "".join(tokens) if _copy_from_stdin_rc.match(sql): raise Exception("copy from stdin not supported") yield ("".join(tokens)) tokens = [] if len(tokens) > 0: yield ("".join(tokens)) if pcount != 0: raise Exception("syntax error - unbalanced parenthesis") skytools-2.1.13/python/skytools/__init__.py0000644000175000017500000000156211670174255020026 0ustar markomarko """Tools for Python database scripts.""" import skytools.quoting import skytools.config import skytools.psycopgwrapper import skytools.sqltools import skytools.gzlog import skytools.scripting import skytools.parsing import skytools.dbstruct from skytools.psycopgwrapper import * from skytools.config import * from skytools.dbstruct import * from skytools.gzlog import * from skytools.scripting import * from skytools.sqltools import * from skytools.quoting import * from skytools.parsing import * __all__ = (skytools.psycopgwrapper.__all__ + skytools.config.__all__ + skytools.dbstruct.__all__ + skytools.gzlog.__all__ + skytools.scripting.__all__ + skytools.sqltools.__all__ + skytools.quoting.__all__ + skytools.parsing.__all__) import skytools.installer_config __version__ = skytools.installer_config.package_version skytools-2.1.13/python/skytools/psycopgwrapper.py0000644000175000017500000000730211670440763021352 0ustar markomarko """Wrapper around psycopg1/2. Preferred is psycopg2, fallback to psycopg1. Interface provided is psycopg1: - dict* methods. - new columns can be assigned to row. """ import sys __all__ = ['I_AUTOCOMMIT', 'I_READ_COMMITTED', 'I_REPEATABLE_READ', 'I_SERIALIZABLE'] try: ##from psycopg2.psycopg1 import connect as _pgconnect # psycopg2.psycopg1.cursor is too backwards compatible, # to the point of avoiding optimized access. # only backwards compat thing we need is dict* methods import psycopg2.extensions, psycopg2.extras from psycopg2.extensions import QuotedString I_AUTOCOMMIT = psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT I_READ_COMMITTED = psycopg2.extensions.ISOLATION_LEVEL_READ_COMMITTED I_REPEATABLE_READ = psycopg2.extensions.ISOLATION_LEVEL_REPEATABLE_READ I_SERIALIZABLE = psycopg2.extensions.ISOLATION_LEVEL_SERIALIZABLE class _CompatRow(psycopg2.extras.DictRow): """Make DictRow more dict-like.""" def __setitem__(self, k, v): """Allow adding new key-value pairs. Such operation adds new field to global _index. But that is OK, as .description is unchanged, and access to such fields before setting them should raise exception anyway. """ if type(k) != int: if k not in self._index: self._index[k] = len(self._index) k = self._index[k] while k >= len(self): self.append(None) return list.__setitem__(self, k, v) def __contains__(self, k): """Returns if such row has such column.""" return k in self._index def copy(self): """Return regular dict.""" return dict(self.items()) def iterkeys(self): return self._index.iterkeys() def itervalues(self): return list.__iter__(self) class _CompatCursor(psycopg2.extras.DictCursor): """Regular psycopg2 DictCursor with dict* methods.""" def __init__(self, *args, **kwargs): psycopg2.extras.DictCursor.__init__(self, *args, **kwargs) self.row_factory = _CompatRow dictfetchone = psycopg2.extras.DictCursor.fetchone dictfetchall = psycopg2.extras.DictCursor.fetchall dictfetchmany = psycopg2.extras.DictCursor.fetchmany class _CompatConnection(psycopg2.extensions.connection): """Connection object that uses _CompatCursor.""" def cursor(self): return psycopg2.extensions.connection.cursor(self, cursor_factory = _CompatCursor) def _pgconnect(cstr): """Create a psycopg2 connection.""" return _CompatConnection(cstr) except ImportError: # use psycopg 1 I_AUTOCOMMIT = 0 I_READ_COMMITTED = 1 I_REPEATABLE_READ = 2 I_SERIALIZABLE = 2 try: from psycopg import connect as _pgconnect from psycopg import QuotedString except ImportError: print "Please install psycopg2 module" sys.exit(1) def connect_database(connstr): """Create a db connection with connect_timeout option. Default connect_timeout is 15, to change put it directly into dsn. """ # allow override if connstr.find("connect_timeout") < 0: connstr += " connect_timeout=15" # create connection db = _pgconnect(connstr) # fill .server_version on older psycopg if not hasattr(db, 'server_version'): iso = db.isolation_level db.set_isolation_level(0) curs = db.cursor() curs.execute('show server_version_num') db.server_version = int(curs.fetchone()[0]) db.set_isolation_level(iso) return db skytools-2.1.13/python/skytools/installer_config.py.in0000644000175000017500000000011511670174255022207 0ustar markomarko sql_locations = [ "@SQLDIR@", ] package_version = "@PACKAGE_VERSION@" skytools-2.1.13/python/skytools/scripting.py0000644000175000017500000004104011670440400020250 0ustar markomarko """Useful functions and classes for database scripts.""" import sys, os, signal, optparse, traceback, time, errno import logging, logging.handlers, logging.config from skytools.config import * from skytools.psycopgwrapper import connect_database from skytools.psycopgwrapper import I_AUTOCOMMIT, I_READ_COMMITTED, I_SERIALIZABLE import skytools.skylog __all__ = ['DBScript', 'signal_pidfile'] #__all__ += ['daemonize', 'run_single_process'] # # utils # def signal_pidfile(pidfile, sig): """Send a signal to process whose ID is located in pidfile. Read only first line of pidfile to support multiline pidfiles like postmaster.pid. Returns True is successful, False if pidfile does not exist or process itself is dead. Any other errors will passed as exceptions. """ try: pid = int(open(pidfile, 'r').readline()) os.kill(pid, sig) return True except IOError, ex: if ex.errno != errno.ENOENT: raise except OSError, ex: if ex.errno != errno.ESRCH: raise except ValueError, ex: raise ValueError('Corrupt pidfile: %s' % pidfile) return False # # daemon mode # def daemonize(): """Turn the process into daemon. Goes background and disables all i/o. """ # launch new process, kill parent pid = os.fork() if pid != 0: os._exit(0) # start new session os.setsid() # stop i/o fd = os.open("/dev/null", os.O_RDWR) os.dup2(fd, 0) os.dup2(fd, 1) os.dup2(fd, 2) if fd > 2: os.close(fd) # # Pidfile locking+cleanup & daemonization combined # def run_single_process(runnable, daemon, pidfile): """Run runnable class, possibly daemonized, locked on pidfile.""" # check if another process is running if pidfile and os.path.isfile(pidfile): if signal_pidfile(pidfile, 0): print "Pidfile exists, another process running?" sys.exit(1) else: print "Ignoring stale pidfile" # daemonize if needed if daemon: daemonize() # clean only own pidfile own_pidfile = False try: if pidfile: f = open(pidfile, 'w') own_pidfile = True f.write(str(os.getpid())) f.close() runnable.run() finally: if own_pidfile: try: os.remove(pidfile) except: pass # # logging setup # _log_config_done = 0 _log_init_done = {} def _init_log(job_name, service_name, cf, log_level): """Logging setup happens here.""" global _log_init_done, _log_config_done got_skylog = 0 use_skylog = cf.getint("use_skylog", 0) # load logging config if needed if use_skylog and not _log_config_done: # python logging.config braindamage: # cannot specify external classess without such hack logging.skylog = skytools.skylog skytools.skylog.set_service_name(service_name) # load general config list = ['skylog.ini', '~/.skylog.ini', '/etc/skylog.ini'] for fn in list: fn = os.path.expanduser(fn) if os.path.isfile(fn): defs = {'job_name': job_name, 'service_name': service_name} logging.config.fileConfig(fn, defs) got_skylog = 1 break _log_config_done = 1 if not got_skylog: sys.stderr.write("skylog.ini not found!\n") sys.exit(1) # avoid duplicate logging init for job_name log = logging.getLogger(job_name) if job_name in _log_init_done: return log _log_init_done[job_name] = 1 # compatibility: specify ini file in script config logfile = cf.getfile("logfile", "") if logfile: fmt = logging.Formatter('%(asctime)s %(process)s %(levelname)s %(message)s') size = cf.getint('log_size', 10*1024*1024) num = cf.getint('log_count', 3) hdlr = logging.handlers.RotatingFileHandler( logfile, 'a', size, num) hdlr.setFormatter(fmt) log.addHandler(hdlr) # if skylog.ini is disabled or not available, log at least to stderr if not got_skylog: hdlr = logging.StreamHandler() fmt = logging.Formatter('%(asctime)s %(process)s %(levelname)s %(message)s') hdlr.setFormatter(fmt) log.addHandler(hdlr) log.setLevel(log_level) return log #: how old connections need to be closed DEF_CONN_AGE = 20*60 # 20 min class DBCachedConn(object): """Cache a db connection.""" def __init__(self, name, loc, max_age = DEF_CONN_AGE): self.name = name self.loc = loc self.conn = None self.conn_time = 0 self.max_age = max_age self.autocommit = -1 self.isolation_level = -1 def get_connection(self, autocommit = 0, isolation_level = -1): # autocommit overrider isolation_level if autocommit: isolation_level = I_AUTOCOMMIT # default isolation_level is READ COMMITTED if isolation_level < 0: isolation_level = I_READ_COMMITTED # new conn? if not self.conn: self.isolation_level = isolation_level self.conn = connect_database(self.loc) self.conn.set_isolation_level(isolation_level) self.conn_time = time.time() else: if self.isolation_level != isolation_level: raise Exception("Conflict in isolation_level") # done return self.conn def refresh(self): if not self.conn: return #for row in self.conn.notifies(): # if row[0].lower() == "reload": # self.reset() # return if not self.max_age: return if time.time() - self.conn_time >= self.max_age: self.reset() def reset(self): if not self.conn: return # drop reference conn = self.conn self.conn = None if self.isolation_level == I_AUTOCOMMIT: return # rollback & close try: conn.rollback() except: pass try: conn.close() except: pass class DBScript(object): """Base class for database scripts. Handles logging, daemonizing, config, errors. """ service_name = None job_name = None cf = None log = None def __init__(self, service_name, args): """Script setup. User class should override work() and optionally __init__(), startup(), reload(), reset() and init_optparse(). NB: in case of daemon, the __init__() and startup()/work() will be run in different processes. So nothing fancy should be done in __init__(). @param service_name: unique name for script. It will be also default job_name, if not specified in config. @param args: cmdline args (sys.argv[1:]), but can be overrided """ self.service_name = service_name self.db_cache = {} self.go_daemon = 0 self.do_single_loop = 0 self.looping = 1 self.need_reload = 1 self.stat_dict = {} self.log_level = logging.INFO self.work_state = 1 # parse command line parser = self.init_optparse() self.options, self.args = parser.parse_args(args) # check args if self.options.version: print 'Skytools version', skytools.__version__ sys.exit(0) if self.options.daemon: self.go_daemon = 1 if self.options.quiet: self.log_level = logging.WARNING if self.options.verbose: self.log_level = logging.DEBUG if len(self.args) < 1: print "need config file" sys.exit(1) # read config file self.cf = self.load_config() self.reload() # init logging self.log = _init_log(self.job_name, self.service_name, self.cf, self.log_level) # send signal, if needed if self.options.cmd == "kill": self.send_signal(signal.SIGTERM) elif self.options.cmd == "stop": self.send_signal(signal.SIGINT) elif self.options.cmd == "reload": self.send_signal(signal.SIGHUP) def load_config(self): conf_file = self.args[0] return Config(self.service_name, conf_file) def init_optparse(self, parser = None): """Initialize a OptionParser() instance that will be used to parse command line arguments. Note that it can be overrided both directions - either DBScript will initialize a instance and passes to user code or user can initialize and then pass to DBScript.init_optparse(). @param parser: optional OptionParser() instance, where DBScript should attachs its own arguments. @return: initialized OptionParser() instance. """ if parser: p = parser else: p = optparse.OptionParser() p.set_usage("%prog [options] INI") # generic options p.add_option("-q", "--quiet", action="store_true", help = "make program silent") p.add_option("-v", "--verbose", action="store_true", help = "make program verbose") p.add_option("-d", "--daemon", action="store_true", help = "go background") p.add_option("-V", "--version", action="store_true", help = "print version info and exit") # control options g = optparse.OptionGroup(p, 'control running process') g.add_option("-r", "--reload", action="store_const", const="reload", dest="cmd", help = "reload config (send SIGHUP)") g.add_option("-s", "--stop", action="store_const", const="stop", dest="cmd", help = "stop program safely (send SIGINT)") g.add_option("-k", "--kill", action="store_const", const="kill", dest="cmd", help = "kill program immidiately (send SIGTERM)") p.add_option_group(g) return p def send_signal(self, sig): if not self.pidfile: self.log.warning("No pidfile in config, nothing todo") elif os.path.isfile(self.pidfile): alive = signal_pidfile(self.pidfile, sig) if not alive: self.log.warning("pidfile exist, but process not running") else: self.log.warning("No pidfile, process not running") sys.exit(0) def set_single_loop(self, do_single_loop): """Changes whether the script will loop or not.""" self.do_single_loop = do_single_loop def start(self): """This will launch main processing thread.""" if self.go_daemon: if not self.pidfile: self.log.error("Daemon needs pidfile") sys.exit(1) run_single_process(self, self.go_daemon, self.pidfile) def stop(self): """Safely stops processing loop.""" self.looping = 0 def reload(self): "Reload config." self.cf.reload() self.job_name = self.cf.get("job_name", self.service_name) self.pidfile = self.cf.getfile("pidfile", '') self.loop_delay = self.cf.getfloat("loop_delay", 1.0) def hook_sighup(self, sig, frame): "Internal SIGHUP handler. Minimal code here." self.need_reload = 1 def hook_sigint(self, sig, frame): "Internal SIGINT handler. Minimal code here." self.stop() def stat_add(self, key, value): self.stat_put(key, value) def stat_put(self, key, value): """Sets a stat value.""" self.stat_dict[key] = value def stat_increase(self, key, increase = 1): """Increases a stat value.""" if key in self.stat_dict: self.stat_dict[key] += increase else: self.stat_dict[key] = increase def send_stats(self): "Send statistics to log." res = [] for k, v in self.stat_dict.items(): res.append("%s: %s" % (k, str(v))) if len(res) == 0: return logmsg = "{%s}" % ", ".join(res) self.log.info(logmsg) self.stat_dict = {} def get_database(self, dbname, autocommit = 0, isolation_level = -1, cache = None, connstr = None): """Load cached database connection. User must not store it permanently somewhere, as all connections will be invalidated on reset. """ max_age = self.cf.getint('connection_lifetime', DEF_CONN_AGE) if not cache: cache = dbname if cache in self.db_cache: dbc = self.db_cache[cache] else: if not connstr: connstr = self.cf.get(dbname) dbc = DBCachedConn(cache, connstr, max_age) self.db_cache[cache] = dbc return dbc.get_connection(autocommit, isolation_level) def close_database(self, dbname): """Explicitly close a cached connection. Next call to get_database() will reconnect. """ if dbname in self.db_cache: dbc = self.db_cache[dbname] dbc.reset() def reset(self): "Something bad happened, reset all connections." for dbc in self.db_cache.values(): dbc.reset() self.db_cache = {} def run(self): "Thread main loop." # run startup, safely try: self.startup() except KeyboardInterrupt, det: raise except SystemExit, det: raise except Exception, det: exc, msg, tb = sys.exc_info() self.log.fatal("Job %s crashed: %s: '%s' (%s: %s)" % ( self.job_name, str(exc), str(msg).rstrip(), str(tb), repr(traceback.format_tb(tb)))) del tb self.reset() sys.exit(1) while self.looping: # reload config, if needed if self.need_reload: self.reload() self.need_reload = 0 # do some work work = self.run_once() # send stats that was added self.send_stats() # reconnect if needed for dbc in self.db_cache.values(): dbc.refresh() # exit if needed if self.do_single_loop: self.log.debug("Only single loop requested, exiting") break # remember work state self.work_state = work # should sleep? if not work: try: time.sleep(self.loop_delay) except Exception, d: self.log.debug("sleep failed: "+str(d)) sys.exit(0) def run_once(self): "Run users work function, safely." try: return self.work() except SystemExit, d: self.send_stats() self.log.info("got SystemExit(%s), exiting" % str(d)) self.reset() raise except KeyboardInterrupt: self.send_stats() self.log.info("got KeyboardInterrupt, exiting") self.reset() sys.exit(1) except MemoryError: exc, msg, tb = sys.exc_info() self.log.fatal("Job %s out of memory: %s: '%s' (%s: %s)" % ( self.job_name, str(exc), str(msg).rstrip(), str(tb), repr(traceback.format_tb(tb)))) del tb sys.exit(1) except Exception: self.send_stats() exc, msg, tb = sys.exc_info() self.log.fatal("Job %s crashed: %s: '%s' (%s: %s)" % ( self.job_name, str(exc), str(msg).rstrip(), str(tb), repr(traceback.format_tb(tb)))) del tb self.reset() if self.looping and not self.do_single_loop: time.sleep(20) return 1 def work(self): """Here should user's processing happen. Return value is taken as boolean - if true, the next loop starts immidiately. If false, DBScript sleeps for a loop_delay. """ raise Exception("Nothing implemented?") def startup(self): """Will be called just before entering main loop. In case of daemon, if will be called in same process as work(), unlike __init__(). """ # set signals signal.signal(signal.SIGHUP, self.hook_sighup) signal.signal(signal.SIGINT, self.hook_sigint) skytools-2.1.13/python/skytools/_pyquoting.py0000644000175000017500000001073211670174255020464 0ustar markomarko# _pyquoting.py """Various helpers for string quoting/unquoting. Here is pure Python that should match C code in _cquoting. """ import urllib, re __all__ = [ "quote_literal", "quote_copy", "quote_bytea_raw", "db_urlencode", "db_urldecode", "unescape", "unquote_literal", ] # # SQL quoting # def quote_literal(s): """Quote a literal value for SQL. If string contains '\\', extended E'' quoting is used, otherwise standard quoting. Input value of None results in string "null" without quotes. Python implementation. """ if s == None: return "null" s = str(s).replace("'", "''") s2 = s.replace("\\", "\\\\") if len(s) != len(s2): return "E'" + s2 + "'" return "'" + s2 + "'" def quote_copy(s): """Quoting for copy command. None is converted to \\N. Python implementation. """ if s == None: return "\\N" s = str(s) s = s.replace("\\", "\\\\") s = s.replace("\t", "\\t") s = s.replace("\n", "\\n") s = s.replace("\r", "\\r") return s _bytea_map = None def quote_bytea_raw(s): """Quoting for bytea parser. Returns None as None. Python implementation. """ global _bytea_map if s == None: return None if 1 and _bytea_map is None: _bytea_map = {} for i in xrange(256): c = chr(i) if i < 0x20 or i >= 0x7F: _bytea_map[c] = "\\%03o" % i elif c == "\\": _bytea_map[c] = r"\\" else: _bytea_map[c] = c return "".join([_bytea_map[c] for c in s]) # # Database specific urlencode and urldecode. # def db_urlencode(dict): """Database specific urlencode. Encode None as key without '='. That means that in "foo&bar=", foo is NULL and bar is empty string. Python implementation. """ elem_list = [] for k, v in dict.items(): if v is None: elem = urllib.quote_plus(str(k)) else: elem = urllib.quote_plus(str(k)) + '=' + urllib.quote_plus(str(v)) elem_list.append(elem) return '&'.join(elem_list) def db_urldecode(qs): """Database specific urldecode. Decode key without '=' as None. This also does not support one key several times. Python implementation. """ res = {} for elem in qs.split('&'): if not elem: continue pair = elem.split('=', 1) name = urllib.unquote_plus(pair[0]) # keep only one instance around name = intern(str(name)) if len(pair) == 1: res[name] = None else: res[name] = urllib.unquote_plus(pair[1]) return res # # Remove C-like backslash escapes # _esc_re = r"\\([0-7]{1,3}|.)" _esc_rc = re.compile(_esc_re) _esc_map = { 't': '\t', 'n': '\n', 'r': '\r', 'a': '\a', 'b': '\b', "'": "'", '"': '"', '\\': '\\', } def _sub_unescape_c(m): v = m.group(1) if (len(v) == 1) and (v < '0' or v > '7'): try: return _esc_map[v] except KeyError: return v else: return chr(int(v, 8)) def unescape(val): """Removes C-style escapes from string. Python implementation. """ return _esc_rc.sub(_sub_unescape_c, val) _esql_re = r"''|\\([0-7]{1,3}|.)" _esql_rc = re.compile(_esc_re) def _sub_unescape_sqlext(m): if m.group() == "''": return "'" v = m.group(1) if (len(v) == 1) and (v < '0' or v > '7'): try: return _esc_map[v] except KeyError: return v return chr(int(v, 8)) def unquote_literal(val, stdstr = False): """Unquotes SQL string. E'..' -> extended quoting. '..' -> standard or extended quoting null -> None other -> returned as-is """ if val[0] == "'" and val[-1] == "'": if stdstr: return val[1:-1].replace("''", "'") else: return _esql_rc.sub(_sub_unescape_sqlext, val[1:-1]) elif len(val) > 2 and val[0] in ('E', 'e') and val[1] == "'" and val[-1] == "'": return _esql_rc.sub(_sub_unescape_sqlext, val[2:-1]) elif len(val) >= 2 and val[0] == '$' and val[-1] == '$': p1 = val.find('$', 1) p2 = val.rfind('$', 1, -1) if p1 > 0 and p2 > p1: t1 = val[:p1+1] t2 = val[p2:] if t1 == t2: return val[len(t1):-len(t1)] raise Exception("Bad dollar-quoted string") elif val.lower() == "null": return None return val skytools-2.1.13/python/skytools/quoting.py0000644000175000017500000000617411670174255017761 0ustar markomarko# quoting.py """Various helpers for string quoting/unquoting.""" import re __all__ = [ # _pyqoting / _cquoting "quote_literal", "quote_copy", "quote_bytea_raw", "db_urlencode", "db_urldecode", "unescape", "unquote_literal", # local "quote_bytea_literal", "quote_bytea_copy", "quote_statement", "quote_ident", "quote_fqident", "quote_json", "unescape_copy", "unquote_ident", ] try: from skytools._cquoting import * except ImportError: from skytools._pyquoting import * # # SQL quoting # def quote_bytea_literal(s): """Quote bytea for regular SQL.""" return quote_literal(quote_bytea_raw(s)) def quote_bytea_copy(s): """Quote bytea for COPY.""" return quote_copy(quote_bytea_raw(s)) def quote_statement(sql, dict): """Quote whole statement. Data values are taken from dict. """ xdict = {} for k, v in dict.items(): xdict[k] = quote_literal(v) return sql % xdict # reserved keywords _ident_kwmap = { "all":1, "analyse":1, "analyze":1, "and":1, "any":1, "array":1, "as":1, "asc":1, "asymmetric":1, "both":1, "case":1, "cast":1, "check":1, "collate":1, "column":1, "constraint":1, "create":1, "current_date":1, "current_role":1, "current_time":1, "current_timestamp":1, "current_user":1, "default":1, "deferrable":1, "desc":1, "distinct":1, "do":1, "else":1, "end":1, "except":1, "false":1, "for":1, "foreign":1, "from":1, "grant":1, "group":1, "having":1, "in":1, "initially":1, "intersect":1, "into":1, "leading":1, "limit":1, "localtime":1, "localtimestamp":1, "new":1, "not":1, "null":1, "off":1, "offset":1, "old":1, "on":1, "only":1, "or":1, "order":1, "placing":1, "primary":1, "references":1, "returning":1, "select":1, "session_user":1, "some":1, "symmetric":1, "table":1, "then":1, "to":1, "trailing":1, "true":1, "union":1, "unique":1, "user":1, "using":1, "when":1, "where":1, } _ident_bad = re.compile(r"[^a-z0-9_]|^[0-9]") def quote_ident(s): """Quote SQL identifier. If is checked against weird symbols and keywords. """ if _ident_bad.search(s) or s in _ident_kwmap: s = '"%s"' % s.replace('"', '""') return s def quote_fqident(s): """Quote fully qualified SQL identifier. The '.' is taken as namespace separator and all parts are quoted separately """ return '.'.join(map(quote_ident, s.split('.', 1))) # # quoting for JSON strings # _jsre = re.compile(r'[\x00-\x1F\\/"]') _jsmap = { "\b": "\\b", "\f": "\\f", "\n": "\\n", "\r": "\\r", "\t": "\\t", "\\": "\\\\", '"': '\\"', "/": "\\/", # to avoid html attacks } def _json_quote_char(m): c = m.group(0) try: return _jsmap[c] except KeyError: return r"\u%04x" % ord(c) def quote_json(s): """JSON style quoting.""" if s is None: return "null" return '"%s"' % _jsre.sub(_json_quote_char, s) def unescape_copy(val): """Removes C-style escapes, also converts "\N" to None.""" if val == r"\N": return None return unescape(val) def unquote_ident(val): """Unquotes possibly quoted SQL identifier.""" if val[0] == '"' and val[-1] == '"': return val[1:-1].replace('""', '"') return val skytools-2.1.13/python/pgqadm.py0000755000175000017500000001305111670174255015650 0ustar markomarko#! /usr/bin/env python """PgQ ticker and maintenance. """ import sys import skytools from pgq.ticker import SmartTicker from pgq.status import PGQStatus #from pgq.admin import PGQAdmin """TODO: pgqadm ini check """ command_usage = """ %prog [options] INI CMD [subcmd args] commands: ticker start ticking & maintenance process status show overview of queue health install install code into db create QNAME create queue drop QNAME drop queue register QNAME CONS install code into db unregister QNAME CONS install code into db config QNAME [VAR=VAL] show or change queue config """ config_allowed_list = { 'queue_ticker_max_count': 'int', 'queue_ticker_max_lag': 'interval', 'queue_ticker_idle_period': 'interval', 'queue_rotation_period': 'interval', } class PGQAdmin(skytools.DBScript): def __init__(self, args): skytools.DBScript.__init__(self, 'pgqadm', args) self.set_single_loop(1) if len(self.args) < 2: print "need command" sys.exit(1) int_cmds = { 'create': self.create_queue, 'drop': self.drop_queue, 'register': self.register, 'unregister': self.unregister, 'install': self.installer, 'config': self.change_config, } cmd = self.args[1] if cmd == "ticker": script = SmartTicker(args) elif cmd == "status": script = PGQStatus(args) elif cmd in int_cmds: script = None self.work = int_cmds[cmd] else: print "unknown command" sys.exit(1) if self.pidfile: self.pidfile += ".admin" self.run_script = script def start(self): if self.run_script: self.run_script.start() else: skytools.DBScript.start(self) def init_optparse(self, parser=None): p = skytools.DBScript.init_optparse(self, parser) p.set_usage(command_usage.strip()) return p def installer(self): objs = [ skytools.DBLanguage("plpgsql"), skytools.DBFunction("txid_current_snapshot", 0, sql_file="txid.sql"), skytools.DBSchema("pgq", sql_file="pgq.sql"), ] db = self.get_database('db') curs = db.cursor() skytools.db_install(curs, objs, self.log) db.commit() def create_queue(self): qname = self.args[2] self.log.info('Creating queue: %s' % qname) self.exec_sql("select pgq.create_queue(%s)", [qname]) def drop_queue(self): qname = self.args[2] self.log.info('Dropping queue: %s' % qname) self.exec_sql("select pgq.drop_queue(%s)", [qname]) def register(self): qname = self.args[2] cons = self.args[3] self.log.info('Registering consumer %s on queue %s' % (cons, qname)) self.exec_sql("select pgq.register_consumer(%s, %s)", [qname, cons]) def unregister(self): qname = self.args[2] cons = self.args[3] self.log.info('Unregistering consumer %s from queue %s' % (cons, qname)) self.exec_sql("select pgq.unregister_consumer(%s, %s)", [qname, cons]) def change_config(self): if len(self.args) < 3: list = self.get_queue_list() for qname in list: self.show_config(qname) return qname = self.args[2] if len(self.args) == 3: self.show_config(qname) return alist = [] for el in self.args[3:]: k, v = el.split('=') if k not in config_allowed_list: qk = "queue_" + k if qk not in config_allowed_list: raise Exception('unknown config var: '+k) k = qk expr = "%s=%s" % (k, skytools.quote_literal(v)) alist.append(expr) self.log.info('Change queue %s config to: %s' % (qname, ", ".join(alist))) sql = "update pgq.queue set %s where queue_name = %s" % ( ", ".join(alist), skytools.quote_literal(qname)) self.exec_sql(sql, []) def exec_sql(self, q, args): self.log.debug(q) db = self.get_database('db') curs = db.cursor() curs.execute(q, args) db.commit() def show_config(self, qname): fields = [] for f, kind in config_allowed_list.items(): if kind == 'interval': sql = "extract('epoch' from %s)::text as %s" % (f, f) fields.append(sql) else: fields.append(f) klist = ", ".join(fields) q = "select " + klist + " from pgq.queue where queue_name = %s" db = self.get_database('db') curs = db.cursor() curs.execute(q, [qname]) res = curs.dictfetchone() db.commit() if res is None: print "no such queue:", qname return print qname for k in config_allowed_list: n = k if k[:6] == "queue_": n = k[6:] print " %s\t=%7s" % (n, res[k]) def get_queue_list(self): db = self.get_database('db') curs = db.cursor() curs.execute("select queue_name from pgq.queue order by 1") rows = curs.fetchall() db.commit() list = [] for r in rows: list.append(r[0]) return list if __name__ == '__main__': script = PGQAdmin(sys.argv[1:]) script.start() skytools-2.1.13/python/modules/0000755000175000017500000000000011727601174015470 5ustar markomarkoskytools-2.1.13/python/modules/cquoting.c0000644000175000017500000004146111670174255017475 0ustar markomarko/* * Fast quoting functions for Python. */ #define PY_SSIZE_T_CLEAN #include #if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN) typedef int Py_ssize_t; #define PY_SSIZE_T_MAX INT_MAX #define PY_SSIZE_T_MIN INT_MIN #endif typedef enum { false = 0, true = 1 } bool; /* * Common buffer management. */ struct Buf { unsigned char *ptr; unsigned long pos; unsigned long alloc; }; static unsigned char *buf_init(struct Buf *buf, unsigned init_size) { if (init_size < 256) init_size = 256; buf->ptr = PyMem_Malloc(init_size); if (buf->ptr) { buf->pos = 0; buf->alloc = init_size; } return buf->ptr; } /* return new pos */ static unsigned char *buf_enlarge(struct Buf *buf, unsigned need_room) { unsigned alloc = buf->alloc; unsigned need_size = buf->pos + need_room; unsigned char *ptr; /* no alloc needed */ if (need_size < alloc) return buf->ptr + buf->pos; if (alloc <= need_size / 2) alloc = need_size; else alloc = alloc * 2; ptr = PyMem_Realloc(buf->ptr, alloc); if (!ptr) return NULL; buf->ptr = ptr; buf->alloc = alloc; return buf->ptr + buf->pos; } static void buf_free(struct Buf *buf) { PyMem_Free(buf->ptr); buf->ptr = NULL; buf->pos = buf->alloc = 0; } static inline unsigned char *buf_get_target_for(struct Buf *buf, unsigned len) { if (buf->pos + len <= buf->alloc) return buf->ptr + buf->pos; else return buf_enlarge(buf, len); } static inline void buf_set_target(struct Buf *buf, unsigned char *newpos) { assert(buf->ptr + buf->pos <= newpos); assert(buf->ptr + buf->alloc >= newpos); buf->pos = newpos - buf->ptr; } static inline int buf_put(struct Buf *buf, unsigned char c) { if (buf->pos < buf->alloc) { buf->ptr[buf->pos++] = c; return 1; } else if (buf_enlarge(buf, 1)) { buf->ptr[buf->pos++] = c; return 1; } return 0; } static PyObject *buf_pystr(struct Buf *buf, unsigned start_pos, unsigned char *newpos) { PyObject *res; if (newpos) buf_set_target(buf, newpos); res = PyString_FromStringAndSize((char *)buf->ptr + start_pos, buf->pos - start_pos); buf_free(buf); return res; } /* * Get string data */ static Py_ssize_t get_buffer(PyObject *obj, unsigned char **buf_p, PyObject **tmp_obj_p) { PyBufferProcs *bfp; PyObject *str = NULL; Py_ssize_t res; /* check for None */ if (obj == Py_None) { PyErr_Format(PyExc_TypeError, "None is not allowed here"); return -1; } /* is string or unicode ? */ if (PyString_Check(obj) || PyUnicode_Check(obj)) { if (PyString_AsStringAndSize(obj, (char**)buf_p, &res) < 0) return -1; return res; } /* try to get buffer */ bfp = obj->ob_type->tp_as_buffer; if (bfp && bfp->bf_getsegcount && bfp->bf_getreadbuffer) { if (bfp->bf_getsegcount(obj, NULL) == 1) return bfp->bf_getreadbuffer(obj, 0, (void**)buf_p); } /* * Not a string-like object, run str() or it. */ /* are we in recursion? */ if (tmp_obj_p == NULL) { PyErr_Format(PyExc_TypeError, "Cannot convert to string - get_buffer() recusively failed"); return -1; } /* do str() then */ str = PyObject_Str(obj); res = -1; if (str != NULL) { res = get_buffer(str, buf_p, NULL); if (res >= 0) { *tmp_obj_p = str; } else { Py_CLEAR(str); } } return res; } /* * Common argument parsing. */ typedef PyObject *(*quote_fn)(unsigned char *src, Py_ssize_t src_len); static PyObject *common_quote(PyObject *args, quote_fn qfunc) { unsigned char *src = NULL; Py_ssize_t src_len = 0; PyObject *arg, *res, *strtmp = NULL; if (!PyArg_ParseTuple(args, "O", &arg)) return NULL; if (arg != Py_None) { src_len = get_buffer(arg, &src, &strtmp); if (src_len < 0) return NULL; } res = qfunc(src, src_len); Py_CLEAR(strtmp); return res; } /* * Simple quoting functions. */ static const char doc_quote_literal[] = "Quote a literal value for SQL.\n" "\n" "If string contains '\\', it is quoted and result is prefixed with E.\n" "Input value of None results in string \"null\" without quotes.\n" "\n" "C implementation.\n"; static PyObject *quote_literal_body(unsigned char *src, Py_ssize_t src_len) { struct Buf buf; unsigned char *esc, *dst, *src_end = src + src_len; unsigned int start_ofs = 1; if (src == NULL) return PyString_FromString("null"); esc = dst = buf_init(&buf, src_len * 2 + 2 + 1); if (!dst) return NULL; *dst++ = ' '; *dst++ = '\''; while (src < src_end) { if (*src == '\\') { *dst++ = '\\'; start_ofs = 0; } else if (*src == '\'') { *dst++ = '\''; } *dst++ = *src++; } *dst++ = '\''; if (start_ofs == 0) *esc = 'E'; return buf_pystr(&buf, start_ofs, dst); } static PyObject *quote_literal(PyObject *self, PyObject *args) { return common_quote(args, quote_literal_body); } /* COPY field */ static const char doc_quote_copy[] = "Quoting for COPY data. None is converted to \\N.\n\n" "C implementation."; static PyObject *quote_copy_body(unsigned char *src, Py_ssize_t src_len) { unsigned char *dst, *src_end = src + src_len; struct Buf buf; if (src == NULL) return PyString_FromString("\\N"); dst = buf_init(&buf, src_len * 2); if (!dst) return NULL; while (src < src_end) { switch (*src) { case '\t': *dst++ = '\\'; *dst++ = 't'; src++; break; case '\n': *dst++ = '\\'; *dst++ = 'n'; src++; break; case '\r': *dst++ = '\\'; *dst++ = 'r'; src++; break; case '\\': *dst++ = '\\'; *dst++ = '\\'; src++; break; default: *dst++ = *src++; break; } } return buf_pystr(&buf, 0, dst); } static PyObject *quote_copy(PyObject *self, PyObject *args) { return common_quote(args, quote_copy_body); } /* raw bytea for byteain() */ static const char doc_quote_bytea_raw[] = "Quoting for bytea parser. Returns None as None.\n" "\n" "C implementation."; static PyObject *quote_bytea_raw_body(unsigned char *src, Py_ssize_t src_len) { unsigned char *dst, *src_end = src + src_len; struct Buf buf; if (src == NULL) { Py_INCREF(Py_None); return Py_None; } dst = buf_init(&buf, src_len * 4); if (!dst) return NULL; while (src < src_end) { if (*src < 0x20 || *src >= 0x7F) { *dst++ = '\\'; *dst++ = '0' + (*src >> 6); *dst++ = '0' + ((*src >> 3) & 7); *dst++ = '0' + (*src & 7); src++; } else { if (*src == '\\') *dst++ = '\\'; *dst++ = *src++; } } return buf_pystr(&buf, 0, dst); } static PyObject *quote_bytea_raw(PyObject *self, PyObject *args) { return common_quote(args, quote_bytea_raw_body); } /* SQL unquote */ static const char doc_unquote_literal[] = "Unquote SQL value.\n\n" "E'..' -> extended quoting.\n" "'..' -> standard or extended quoting\n" "null -> None\n" "other -> returned as-is\n\n" "C implementation.\n"; static PyObject *do_sql_ext(unsigned char *src, Py_ssize_t src_len) { unsigned char *dst, *src_end = src + src_len; struct Buf buf; dst = buf_init(&buf, src_len); if (!dst) return NULL; while (src < src_end) { if (*src == '\'') { src++; if (src < src_end && *src == '\'') { *dst++ = *src++; continue; } goto failed; } if (*src != '\\') { *dst++ = *src++; continue; } if (++src >= src_end) goto failed; switch (*src) { case 't': *dst++ = '\t'; src++; break; case 'n': *dst++ = '\n'; src++; break; case 'r': *dst++ = '\r'; src++; break; case 'a': *dst++ = '\a'; src++; break; case 'b': *dst++ = '\b'; src++; break; default: if (*src >= '0' && *src <= '7') { unsigned char c = *src++ - '0'; if (src < src_end && *src >= '0' && *src <= '7') { c = (c << 3) | ((*src++) - '0'); if (src < src_end && *src >= '0' && *src <= '7') c = (c << 3) | ((*src++) - '0'); } *dst++ = c; } else { *dst++ = *src++; } } } return buf_pystr(&buf, 0, dst); failed: PyErr_Format(PyExc_ValueError, "Broken exteded SQL string"); return NULL; } static PyObject *do_sql_std(unsigned char *src, Py_ssize_t src_len) { unsigned char *dst, *src_end = src + src_len; struct Buf buf; dst = buf_init(&buf, src_len); if (!dst) return NULL; while (src < src_end) { if (*src != '\'') { *dst++ = *src++; continue; } src++; if (src >= src_end || *src != '\'') goto failed; *dst++ = *src++; } return buf_pystr(&buf, 0, dst); failed: PyErr_Format(PyExc_ValueError, "Broken standard SQL string"); return NULL; } static PyObject *do_dolq(unsigned char *src, Py_ssize_t src_len) { /* src_len >= 2, '$' in start and end */ unsigned char *src_end = src + src_len; unsigned char *p1 = src + 1, *p2 = src_end - 2; while (p1 < src_end && *p1 != '$') p1++; while (p2 > src && *p2 != '$') p2--; if (p2 <= p1) goto failed; p1++; /* position after '$' */ if ((p1 - src) != (src_end - p2)) goto failed; if (memcmp(src, p2, p1 - src) != 0) goto failed; return PyString_FromStringAndSize((char *)p1, p2 - p1); failed: PyErr_Format(PyExc_ValueError, "Broken dollar-quoted string"); return NULL; } static PyObject *unquote_literal(PyObject *self, PyObject *args) { unsigned char *src = NULL; Py_ssize_t src_len = 0; int stdstr = 0; PyObject *value = NULL; if (!PyArg_ParseTuple(args, "O|i", &value, &stdstr)) return NULL; if (PyString_AsStringAndSize(value, (char **)&src, &src_len) < 0) return NULL; if (src_len == 4 && strcasecmp((char *)src, "null") == 0) { Py_INCREF(Py_None); return Py_None; } if (src_len >= 2 && src[0] == '$' && src[src_len - 1] == '$') return do_dolq(src, src_len); if (src_len < 2 || src[src_len - 1] != '\'') goto badstr; if (src[0] == '\'') { src++; src_len -= 2; return stdstr ? do_sql_std(src, src_len) : do_sql_ext(src, src_len); } else if (src_len > 2 && (src[0] | 0x20) == 'e' && src[1] == '\'') { src += 2; src_len -= 3; return do_sql_ext(src, src_len); } badstr: Py_INCREF(value); return value; } /* C unescape */ static const char doc_unescape[] = "Unescape C-style escaped string.\n\n" "C implementation."; static PyObject *unescape_body(unsigned char *src, Py_ssize_t src_len) { unsigned char *dst, *src_end = src + src_len; struct Buf buf; if (src == NULL) { PyErr_Format(PyExc_TypeError, "None not allowed"); return NULL; } dst = buf_init(&buf, src_len); if (!dst) return NULL; while (src < src_end) { if (*src != '\\') { *dst++ = *src++; continue; } if (++src >= src_end) goto failed; switch (*src) { case 't': *dst++ = '\t'; src++; break; case 'n': *dst++ = '\n'; src++; break; case 'r': *dst++ = '\r'; src++; break; case 'a': *dst++ = '\a'; src++; break; case 'b': *dst++ = '\b'; src++; break; default: if (*src >= '0' && *src <= '7') { unsigned char c = *src++ - '0'; if (src < src_end && *src >= '0' && *src <= '7') { c = (c << 3) | ((*src++) - '0'); if (src < src_end && *src >= '0' && *src <= '7') c = (c << 3) | ((*src++) - '0'); } *dst++ = c; } else { *dst++ = *src++; } } } return buf_pystr(&buf, 0, dst); failed: PyErr_Format(PyExc_ValueError, "Broken string - \\ at the end"); return NULL; } static PyObject *unescape(PyObject *self, PyObject *args) { return common_quote(args, unescape_body); } /* * urlencode of dict */ static bool urlenc(struct Buf *buf, PyObject *obj) { Py_ssize_t len; unsigned char *src, *dst; PyObject *strtmp = NULL; static const unsigned char hextbl[] = "0123456789abcdef"; bool ok = false; len = get_buffer(obj, &src, &strtmp); if (len < 0) goto failed; dst = buf_get_target_for(buf, len * 3); if (!dst) goto failed; while (len--) { if ((*src >= 'a' && *src <= 'z') || (*src >= 'A' && *src <= 'Z') || (*src >= '0' && *src <= '9') || (*src == '.' || *src == '_' || *src == '-')) { *dst++ = *src++; } else if (*src == ' ') { *dst++ = '+'; src++; } else { *dst++ = '%'; *dst++ = hextbl[*src >> 4]; *dst++ = hextbl[*src & 0xF]; src++; } } buf_set_target(buf, dst); ok = true; failed: Py_CLEAR(strtmp); return ok; } /* urlencode key+val pair. val can be None */ static bool urlenc_keyval(struct Buf *buf, PyObject *key, PyObject *value, bool needAmp) { if (needAmp && !buf_put(buf, '&')) return false; if (!urlenc(buf, key)) return false; if (value != Py_None) { if (!buf_put(buf, '=')) return false; if (!urlenc(buf, value)) return false; } return true; } /* encode native dict using PyDict_Next */ static PyObject *encode_dict(PyObject *data) { PyObject *key, *value; Py_ssize_t pos = 0; bool needAmp = false; struct Buf buf; if (!buf_init(&buf, 1024)) return NULL; while (PyDict_Next(data, &pos, &key, &value)) { if (!urlenc_keyval(&buf, key, value, needAmp)) goto failed; needAmp = true; } return buf_pystr(&buf, 0, NULL); failed: buf_free(&buf); return NULL; } /* encode custom object using .iteritems() */ static PyObject *encode_dictlike(PyObject *data) { PyObject *key = NULL, *value = NULL, *tup, *iter; struct Buf buf; bool needAmp = false; if (!buf_init(&buf, 1024)) return NULL; iter = PyObject_CallMethod(data, "iteritems", NULL); if (iter == NULL) { buf_free(&buf); return NULL; } while ((tup = PyIter_Next(iter))) { key = PySequence_GetItem(tup, 0); value = key ? PySequence_GetItem(tup, 1) : NULL; Py_CLEAR(tup); if (!key || !value) goto failed; if (!urlenc_keyval(&buf, key, value, needAmp)) goto failed; needAmp = true; Py_CLEAR(key); Py_CLEAR(value); } /* allow error from iterator */ if (PyErr_Occurred()) goto failed; Py_CLEAR(iter); return buf_pystr(&buf, 0, NULL); failed: buf_free(&buf); Py_CLEAR(iter); Py_CLEAR(key); Py_CLEAR(value); return NULL; } static const char doc_db_urlencode[] = "Urlencode for database records.\n" "If a value is None the key is output without '='.\n" "\n" "C implementation."; static PyObject *db_urlencode(PyObject *self, PyObject *args) { PyObject *data; if (!PyArg_ParseTuple(args, "O", &data)) return NULL; if (PyDict_Check(data)) { return encode_dict(data); } else { return encode_dictlike(data); } } /* * urldecode to dict */ static inline int gethex(unsigned char c) { if (c >= '0' && c <= '9') return c - '0'; c |= 0x20; if (c >= 'a' && c <= 'f') return c - 'a' + 10; return -1; } static PyObject *get_elem(unsigned char *buf, unsigned char **src_p, unsigned char *src_end) { int c1, c2; unsigned char *src = *src_p; unsigned char *dst = buf; while (src < src_end) { switch (*src) { case '%': if (++src + 2 > src_end) goto hex_incomplete; if ((c1 = gethex(*src++)) < 0) goto hex_invalid; if ((c2 = gethex(*src++)) < 0) goto hex_invalid; *dst++ = (c1 << 4) | c2; break; case '+': *dst++ = ' '; src++; break; case '&': case '=': goto gotit; default: *dst++ = *src++; } } gotit: *src_p = src; return PyString_FromStringAndSize((char *)buf, dst - buf); hex_incomplete: PyErr_Format(PyExc_ValueError, "Incomplete hex code"); return NULL; hex_invalid: PyErr_Format(PyExc_ValueError, "Invalid hex code"); return NULL; } static const char doc_db_urldecode[] = "Urldecode from string to dict.\n" "NULL are detected by missing '='.\n" "Duplicate keys are ignored - only latest is kept.\n" "\n" "C implementation."; static PyObject *db_urldecode(PyObject *self, PyObject *args) { unsigned char *src, *src_end; Py_ssize_t src_len; PyObject *dict = NULL, *key = NULL, *value = NULL; struct Buf buf; if (!PyArg_ParseTuple(args, "t#", &src, &src_len)) return NULL; if (!buf_init(&buf, src_len)) return NULL; dict = PyDict_New(); if (!dict) { buf_free(&buf); return NULL; } src_end = src + src_len; while (src < src_end) { if (*src == '&') { src++; continue; } key = get_elem(buf.ptr, &src, src_end); if (!key) goto failed; if (src < src_end && *src == '=') { src++; value = get_elem(buf.ptr, &src, src_end); if (value == NULL) goto failed; } else { Py_INCREF(Py_None); value = Py_None; } /* lessen memory usage by intering */ PyString_InternInPlace(&key); if (PyDict_SetItem(dict, key, value) < 0) goto failed; Py_CLEAR(key); Py_CLEAR(value); } buf_free(&buf); return dict; failed: buf_free(&buf); Py_CLEAR(key); Py_CLEAR(value); Py_CLEAR(dict); return NULL; } /* * Module initialization */ static PyMethodDef cquoting_methods[] = { { "quote_literal", quote_literal, METH_VARARGS, doc_quote_literal }, { "quote_copy", quote_copy, METH_VARARGS, doc_quote_copy }, { "quote_bytea_raw", quote_bytea_raw, METH_VARARGS, doc_quote_bytea_raw }, { "unescape", unescape, METH_VARARGS, doc_unescape }, { "db_urlencode", db_urlencode, METH_VARARGS, doc_db_urlencode }, { "db_urldecode", db_urldecode, METH_VARARGS, doc_db_urldecode }, { "unquote_literal", unquote_literal, METH_VARARGS, doc_unquote_literal }, { NULL } }; PyMODINIT_FUNC init_cquoting(void) { PyObject *module; module = Py_InitModule("_cquoting", cquoting_methods); PyModule_AddStringConstant(module, "__doc__", "fast quoting for skytools"); } skytools-2.1.13/python/pgq/0000755000175000017500000000000011727601174014607 5ustar markomarkoskytools-2.1.13/python/pgq/event.py0000644000175000017500000000271711670174255016313 0ustar markomarko """PgQ event container. """ __all__ = ['EV_RETRY', 'EV_DONE', 'EV_FAILED', 'Event'] # Event status codes EV_RETRY = 0 EV_DONE = 1 EV_FAILED = 2 _fldmap = { 'ev_id': 'ev_id', 'ev_txid': 'ev_txid', 'ev_time': 'ev_time', 'ev_type': 'ev_type', 'ev_data': 'ev_data', 'ev_retry': 'ev_retry', 'ev_extra1': 'ev_extra1', 'ev_extra2': 'ev_extra2', 'ev_extra3': 'ev_extra3', 'ev_extra4': 'ev_extra4', 'id': 'ev_id', 'txid': 'ev_txid', 'time': 'ev_time', 'type': 'ev_type', 'data': 'ev_data', 'retry': 'ev_retry', 'extra1': 'ev_extra1', 'extra2': 'ev_extra2', 'extra3': 'ev_extra3', 'extra4': 'ev_extra4', } class Event(object): """Event data for consumers. Consumer is supposed to tag them after processing. If not, events will stay in retry queue. """ def __init__(self, queue_name, row): self._event_row = row self.status = EV_RETRY self.retry_time = 60 self.fail_reason = "Buggy consumer" self.queue_name = queue_name def __getattr__(self, key): return self._event_row[_fldmap[key]] def tag_done(self): self.status = EV_DONE def tag_retry(self, retry_time = 60): self.status = EV_RETRY self.retry_time = retry_time def tag_failed(self, reason): self.status = EV_FAILED self.fail_reason = reason skytools-2.1.13/python/pgq/producer.py0000644000175000017500000000215111670174255017005 0ustar markomarko """PgQ producer helpers for Python. """ import skytools __all__ = ['bulk_insert_events', 'insert_event'] _fldmap = { 'id': 'ev_id', 'time': 'ev_time', 'type': 'ev_type', 'data': 'ev_data', 'extra1': 'ev_extra1', 'extra2': 'ev_extra2', 'extra3': 'ev_extra3', 'extra4': 'ev_extra4', 'ev_id': 'ev_id', 'ev_time': 'ev_time', 'ev_type': 'ev_type', 'ev_data': 'ev_data', 'ev_extra1': 'ev_extra1', 'ev_extra2': 'ev_extra2', 'ev_extra3': 'ev_extra3', 'ev_extra4': 'ev_extra4', } def bulk_insert_events(curs, rows, fields, queue_name): q = "select pgq.current_event_table(%s)" curs.execute(q, [queue_name]) tbl = curs.fetchone()[0] db_fields = map(_fldmap.get, fields) skytools.magic_insert(curs, tbl, rows, db_fields) def insert_event(curs, queue, ev_type, ev_data, extra1=None, extra2=None, extra3=None, extra4=None): q = "select pgq.insert_event(%s, %s, %s, %s, %s, %s, %s)" curs.execute(q, [queue, ev_type, ev_data, extra1, extra2, extra3, extra4]) return curs.fetchone()[0] skytools-2.1.13/python/pgq/consumer.py0000644000175000017500000003635411670174255017031 0ustar markomarko """PgQ consumer framework for Python. API problems(?): - process_event() and process_batch() should have db as argument. - should ev.tag*() update db immidiately? """ import sys, time, skytools from pgq.event import * __all__ = ['Consumer', 'RemoteConsumer', 'SerialConsumer'] class _WalkerEvent(Event): """Redirects status flags to BatchWalker. That way event data can gc-d immidiately and tag_done() events dont need to be remembered. """ def __init__(self, walker, queue, row): Event.__init__(self, queue, row) self._walker = walker def tag_done(self): self._walker.tag_event_done(self) def tag_retry(self, retry_time = 60): self._walker.tag_event_retry(self, retry_time) def tag_failed(self, reason): self._walker.tag_failed(self, reason) class _BatchWalker(object): """Lazy iterator over batch events. Events are loaded using cursor. It will be given as ev_list to process_batch(). It allows: - one for loop over events - len() after that """ def __init__(self, curs, batch_id, queue_name, fetch_size = 300): self.queue_name = queue_name self.fetch_size = fetch_size self.sql_cursor = "batch_walker" self.curs = curs self.length = 0 self.status_map = {} curs.execute("select pgq.batch_event_sql(%s)", [batch_id]) self.batch_sql = curs.fetchone()[0] self.fetch_status = 0 # 0-not started, 1-in-progress, 2-done def __iter__(self): if self.fetch_status: raise Exception("BatchWalker: double fetch? (%d)" % self.fetch_status) self.fetch_status = 1 q = "declare %s no scroll cursor for %s" % (self.sql_cursor, self.batch_sql) self.curs.execute(q) q = "fetch %d from batch_walker" % self.fetch_size while 1: self.curs.execute(q) rows = self.curs.dictfetchall() if not len(rows): break self.length += len(rows) for row in rows: ev = _WalkerEvent(self, self.queue_name, row) ev.tag_retry() yield ev self.curs.execute("close %s" % self.sql_cursor) self.fetch_status = 2 def __len__(self): if self.fetch_status != 2: raise Exception("BatchWalker: len() for incomplete result. (%d)" % self.fetch_status) return self.length def tag_event_done(self, event): del self.status_map[event.id] def tag_event_retry(self, event, retry_time): self.status_map[event.id] = (EV_RETRY, retry_time) def tag_event_failed(self, event, reason): self.status_map[event.id] = (EV_FAILED, reason) def iter_status(self): for res in self.status_map.iteritems(): yield res class Consumer(skytools.DBScript): """Consumer base class. """ def __init__(self, service_name, db_name, args): """Initialize new consumer. @param service_name: service_name for DBScript @param db_name: name of database for get_database() @param args: cmdline args for DBScript """ skytools.DBScript.__init__(self, service_name, args) self.db_name = db_name self.reg_list = [] self.consumer_id = self.cf.get("pgq_consumer_id", self.job_name) self.pgq_queue_name = self.cf.get("pgq_queue_name") self.pgq_lazy_fetch = self.cf.getint("pgq_lazy_fetch", 0) def attach(self): """Attach consumer to interesting queues.""" res = self.register_consumer(self.pgq_queue_name) return res def detach(self): """Detach consumer from all queues.""" tmp = self.reg_list[:] for q in tmp: self.unregister_consumer(q) def process_event(self, db, event): """Process one event. Should be overrided by user code. Event should be tagged as done, retry or failed. If not, it will be tagged as for retry. """ raise Exception("needs to be implemented") def process_batch(self, db, batch_id, event_list): """Process all events in batch. By default calls process_event for each. Can be overrided by user code. Events should be tagged as done, retry or failed. If not, they will be tagged as for retry. """ for ev in event_list: self.process_event(db, ev) def work(self): """Do the work loop, once (internal).""" if len(self.reg_list) == 0: self.log.debug("Attaching") self.attach() db = self.get_database(self.db_name) curs = db.cursor() data_avail = 0 for queue in self.reg_list: self.stat_start() # acquire batch batch_id = self._load_next_batch(curs, queue) db.commit() if batch_id == None: continue data_avail = 1 # load events list = self._load_batch_events(curs, batch_id, queue) db.commit() # process events self._launch_process_batch(db, batch_id, list) # done self._finish_batch(curs, batch_id, list) db.commit() self.stat_end(len(list)) # if false, script sleeps return data_avail def register_consumer(self, queue_name): db = self.get_database(self.db_name) cx = db.cursor() cx.execute("select pgq.register_consumer(%s, %s)", [queue_name, self.consumer_id]) res = cx.fetchone()[0] db.commit() self.reg_list.append(queue_name) return res def unregister_consumer(self, queue_name): db = self.get_database(self.db_name) cx = db.cursor() cx.execute("select pgq.unregister_consumer(%s, %s)", [queue_name, self.consumer_id]) db.commit() self.reg_list.remove(queue_name) def _launch_process_batch(self, db, batch_id, list): self.process_batch(db, batch_id, list) def _load_batch_events_old(self, curs, batch_id, queue_name): """Fetch all events for this batch.""" # load events sql = "select * from pgq.get_batch_events(%d)" % batch_id curs.execute(sql) rows = curs.dictfetchall() # map them to python objects list = [] for r in rows: ev = Event(queue_name, r) list.append(ev) return list def _load_batch_events(self, curs, batch_id, queue_name): """Fetch all events for this batch.""" if self.pgq_lazy_fetch: return _BatchWalker(curs, batch_id, queue_name, self.pgq_lazy_fetch) else: return self._load_batch_events_old(curs, batch_id, queue_name) def _load_next_batch(self, curs, queue_name): """Allocate next batch. (internal)""" q = "select pgq.next_batch(%s, %s)" curs.execute(q, [queue_name, self.consumer_id]) return curs.fetchone()[0] def _finish_batch(self, curs, batch_id, list): """Tag events and notify that the batch is done.""" retry = failed = 0 if self.pgq_lazy_fetch: for ev_id, stat in list.iter_status(): if stat[0] == EV_RETRY: self._tag_retry(curs, batch_id, ev_id, stat[1]) retry += 1 elif stat[0] == EV_FAILED: self._tag_failed(curs, batch_id, ev_id, stat[1]) failed += 1 else: for ev in list: if ev.status == EV_FAILED: self._tag_failed(curs, batch_id, ev.id, ev.fail_reason) failed += 1 elif ev.status == EV_RETRY: self._tag_retry(curs, batch_id, ev.id, ev.retry_time) retry += 1 # report weird events if retry: self.stat_add('retry-events', retry) if failed: self.stat_add('failed-events', failed) curs.execute("select pgq.finish_batch(%s)", [batch_id]) def _tag_failed(self, curs, batch_id, ev_id, fail_reason): """Tag event as failed. (internal)""" curs.execute("select pgq.event_failed(%s, %s, %s)", [batch_id, ev_id, fail_reason]) def _tag_retry(self, cx, batch_id, ev_id, retry_time): """Tag event for retry. (internal)""" cx.execute("select pgq.event_retry(%s, %s, %s)", [batch_id, ev_id, retry_time]) def get_batch_info(self, batch_id): """Get info about batch. @return: Return value is a dict of: - queue_name: queue name - consumer_name: consumers name - batch_start: batch start time - batch_end: batch end time - tick_id: end tick id - prev_tick_id: start tick id - lag: how far is batch_end from current moment. """ db = self.get_database(self.db_name) cx = db.cursor() q = "select queue_name, consumer_name, batch_start, batch_end,"\ " prev_tick_id, tick_id, lag"\ " from pgq.get_batch_info(%s)" cx.execute(q, [batch_id]) row = cx.dictfetchone() db.commit() return row def stat_start(self): self.stat_batch_start = time.time() def stat_end(self, count): t = time.time() self.stat_add('count', count) self.stat_add('duration', t - self.stat_batch_start) class RemoteConsumer(Consumer): """Helper for doing event processing in another database. Requires that whole batch is processed in one TX. """ def __init__(self, service_name, db_name, remote_db, args): Consumer.__init__(self, service_name, db_name, args) self.remote_db = remote_db def process_batch(self, db, batch_id, event_list): """Process all events in batch. By default calls process_event for each. """ dst_db = self.get_database(self.remote_db) curs = dst_db.cursor() if self.is_last_batch(curs, batch_id): for ev in event_list: ev.tag_done() return self.process_remote_batch(db, batch_id, event_list, dst_db) self.set_last_batch(curs, batch_id) dst_db.commit() def is_last_batch(self, dst_curs, batch_id): """Helper function to keep track of last successful batch in external database. """ q = "select pgq_ext.is_batch_done(%s, %s)" dst_curs.execute(q, [ self.consumer_id, batch_id ]) return dst_curs.fetchone()[0] def set_last_batch(self, dst_curs, batch_id): """Helper function to set last successful batch in external database. """ q = "select pgq_ext.set_batch_done(%s, %s)" dst_curs.execute(q, [ self.consumer_id, batch_id ]) def process_remote_batch(self, db, batch_id, event_list, dst_db): raise Exception('process_remote_batch not implemented') class SerialConsumer(Consumer): """Consumer that applies batches sequentially in second database. Requirements: - Whole batch in one TX. - Must not use retry queue. Features: - Can detect if several batches are already applied to dest db. - If some ticks are lost. allows to seek back on queue. Whether it succeeds, depends on pgq configuration. """ def __init__(self, service_name, db_name, remote_db, args): Consumer.__init__(self, service_name, db_name, args) self.remote_db = remote_db self.dst_schema = "pgq_ext" self.cur_batch_info = None def startup(self): if self.options.rewind: self.rewind() sys.exit(0) if self.options.reset: self.dst_reset() sys.exit(0) return Consumer.startup(self) def init_optparse(self, parser = None): p = Consumer.init_optparse(self, parser) p.add_option("--rewind", action = "store_true", help = "change queue position according to destination") p.add_option("--reset", action = "store_true", help = "reset queue pos on destination side") return p def process_batch(self, db, batch_id, event_list): """Process all events in batch. """ dst_db = self.get_database(self.remote_db) curs = dst_db.cursor() self.cur_batch_info = self.get_batch_info(batch_id) # check if done if self.is_batch_done(curs): for ev in event_list: ev.tag_done() return # actual work self.process_remote_batch(db, batch_id, event_list, dst_db) # finish work self.set_batch_done(curs) dst_db.commit() def is_batch_done(self, dst_curs): """Helper function to keep track of last successful batch in external database. """ cur_tick = self.cur_batch_info['tick_id'] prev_tick = self.cur_batch_info['prev_tick_id'] dst_tick = self.get_last_tick(dst_curs) if not dst_tick: # seems this consumer has not run yet against dst_db return False if prev_tick == dst_tick: # on track return False if cur_tick == dst_tick: # current batch is already applied, skip it return True # anything else means problems raise Exception('Lost position: batch %d..%d, dst has %d' % ( prev_tick, cur_tick, dst_tick)) def set_batch_done(self, dst_curs): """Helper function to set last successful batch in external database. """ tick_id = self.cur_batch_info['tick_id'] self.set_last_tick(dst_curs, tick_id) def attach(self): new = Consumer.attach(self) if new: self.dst_reset() def detach(self): """If detaching, also clean completed tick table on dest.""" Consumer.detach(self) self.dst_reset() def process_remote_batch(self, db, batch_id, event_list, dst_db): raise Exception('process_remote_batch not implemented') def rewind(self): self.log.info("Rewinding queue") src_db = self.get_database(self.db_name) dst_db = self.get_database(self.remote_db) src_curs = src_db.cursor() dst_curs = dst_db.cursor() dst_tick = self.get_last_tick(dst_curs) if dst_tick: q = "select pgq.register_consumer_at(%s, %s, %s)" src_curs.execute(q, [self.pgq_queue_name, self.consumer_id, dst_tick]) else: self.log.warning('No tick found on dst side') dst_db.commit() src_db.commit() def dst_reset(self): self.log.info("Resetting queue tracking on dst side") dst_db = self.get_database(self.remote_db) dst_curs = dst_db.cursor() self.set_last_tick(dst_curs, None) dst_db.commit() def get_last_tick(self, dst_curs): q = "select %s.get_last_tick(%%s)" % self.dst_schema dst_curs.execute(q, [self.consumer_id]) res = dst_curs.fetchone() return res[0] def set_last_tick(self, dst_curs, tick_id): q = "select %s.set_last_tick(%%s, %%s)" % self.dst_schema dst_curs.execute(q, [ self.consumer_id, tick_id ]) skytools-2.1.13/python/pgq/ticker.py0000644000175000017500000001172011670174255016445 0ustar markomarko"""PgQ ticker. It will also launch maintenance job. """ import sys, os, time, threading import skytools from maint import MaintenanceJob __all__ = ['SmartTicker'] def is_txid_sane(curs): curs.execute("select txid_current()") txid = curs.fetchone()[0] # on 8.2 theres no such table if not skytools.exists_table(curs, 'txid.epoch'): return 1 curs.execute("select epoch, last_value from txid.epoch") epoch, last_val = curs.fetchone() stored_val = (epoch << 32) | last_val if stored_val <= txid: return 1 else: return 0 class QueueStatus(object): def __init__(self, name): self.queue_name = name self.seq_name = None self.idle_period = 60 self.max_lag = 3 self.max_count = 200 self.last_tick_time = 0 self.last_count = 0 self.quiet_count = 0 def set_data(self, row): self.seq_name = row['queue_event_seq'] self.idle_period = row['queue_ticker_idle_period'] self.max_lag = row['queue_ticker_max_lag'] self.max_count = row['queue_ticker_max_count'] def need_tick(self, cur_count, cur_time): # check if tick is needed need_tick = 0 lag = cur_time - self.last_tick_time if cur_count == self.last_count: # totally idle database # don't go immidiately to big delays, as seq grows before commit if self.quiet_count < 5: if lag >= self.max_lag: need_tick = 1 self.quiet_count += 1 else: if lag >= self.idle_period: need_tick = 1 else: self.quiet_count = 0 # somewhat loaded machine if cur_count - self.last_count >= self.max_count: need_tick = 1 elif lag >= self.max_lag: need_tick = 1 if need_tick: self.last_tick_time = cur_time self.last_count = cur_count return need_tick class SmartTicker(skytools.DBScript): last_tick_event = 0 last_tick_time = 0 quiet_count = 0 tick_count = 0 maint_thread = None def __init__(self, args): skytools.DBScript.__init__(self, 'pgqadm', args) self.ticker_log_time = 0 self.ticker_log_delay = 5*60 self.queue_map = {} self.refresh_time = 0 def reload(self): skytools.DBScript.reload(self) self.ticker_log_delay = self.cf.getfloat("ticker_log_delay", 5*60) def startup(self): if self.maint_thread: return db = self.get_database("db", autocommit = 1) cx = db.cursor() ok = is_txid_sane(cx) if not ok: self.log.error('txid in bad state') sys.exit(1) self.maint_thread = MaintenanceJob(self, [self.cf.filename]) t = threading.Thread(name = 'maint_thread', target = self.maint_thread.run) t.setDaemon(1) t.start() def refresh_queues(self, cx): q = "select queue_name, queue_event_seq,"\ " extract('epoch' from queue_ticker_idle_period) as queue_ticker_idle_period,"\ " extract('epoch' from queue_ticker_max_lag) as queue_ticker_max_lag,"\ " queue_ticker_max_count"\ " from pgq.queue"\ " where not queue_external_ticker" cx.execute(q) new_map = {} data_list = [] for row in cx.dictfetchall(): queue_name = row['queue_name'] try: que = self.queue_map[queue_name] except KeyError, x: que = QueueStatus(queue_name) que.set_data(row) new_map[queue_name] = que p1 = "'%s', (select last_value from %s)" % (queue_name, que.seq_name) data_list.append(p1) self.queue_map = new_map self.seq_query = "select %s" % ( ", ".join(data_list)) if len(data_list) == 0: self.seq_query = None self.refresh_time = time.time() def work(self): db = self.get_database("db", autocommit = 1) cx = db.cursor() queue_refresh = self.cf.getint('queue_refresh_period', 30) cur_time = time.time() if cur_time >= self.refresh_time + queue_refresh: self.refresh_queues(cx) if not self.seq_query: return # now check seqs cx.execute(self.seq_query) res = cx.fetchone() pos = 0 while pos < len(res): id = res[pos] val = res[pos + 1] pos += 2 que = self.queue_map[id] if que.need_tick(val, cur_time): cx.execute("select pgq.ticker(%s)", [que.queue_name]) self.tick_count += 1 if cur_time > self.ticker_log_time + self.ticker_log_delay: self.ticker_log_time = cur_time self.stat_add('ticks', self.tick_count) self.tick_count = 0 skytools-2.1.13/python/pgq/__init__.py0000644000175000017500000000040511670174255016721 0ustar markomarko"""PgQ framework for Python.""" import pgq.event import pgq.consumer import pgq.producer from pgq.event import * from pgq.consumer import * from pgq.producer import * __all__ = ( pgq.event.__all__ + pgq.consumer.__all__ + pgq.producer.__all__ ) skytools-2.1.13/python/pgq/maint.py0000644000175000017500000000617611670174255016305 0ustar markomarko"""PgQ maintenance functions.""" import skytools, time def get_pgq_api_version(curs): q = "select count(1) from pg_proc p, pg_namespace n"\ " where n.oid = p.pronamespace and n.nspname='pgq'"\ " and p.proname='version';" curs.execute(q) if not curs.fetchone()[0]: return '1.0.0' curs.execute("select pgq.version()") return curs.fetchone()[0] def version_ge(curs, want_ver): """Check is db version of pgq is greater than want_ver.""" db_ver = get_pgq_api_version(curs) want_tuple = map(int, want_ver.split('.')) db_tuple = map(int, db_ver.split('.')) if db_tuple[0] != want_tuple[0]: raise Exception('Wrong major version') if db_tuple[1] >= want_tuple[1]: return 1 return 0 class MaintenanceJob(skytools.DBScript): """Periodic maintenance.""" def __init__(self, ticker, args): skytools.DBScript.__init__(self, 'pgqadm', args) self.ticker = ticker self.last_time = 0 # start immidiately self.last_ticks = 0 self.clean_ticks = 1 self.maint_delay = 5*60 def startup(self): # disable regular DBScript startup() pass def reload(self): skytools.DBScript.reload(self) # force loop_delay self.loop_delay = 5 # compat var self.maint_delay = 60 * self.cf.getfloat('maint_delay_min', -1) if self.maint_delay < 0: self.maint_delay = self.cf.getfloat('maint_delay', 5*60) self.maint_delay = self.cf.getfloat('maint_delay', self.maint_delay) def work(self): t = time.time() if self.last_time + self.maint_delay > t: return self.do_maintenance() self.last_time = t duration = time.time() - t self.stat_add('maint_duration', duration) def do_maintenance(self): """Helper function for running maintenance.""" db = self.get_database('db', autocommit=1) cx = db.cursor() if skytools.exists_function(cx, "pgq.maint_rotate_tables_step1", 1): # rotate each queue in own TX q = "select queue_name from pgq.get_queue_info()" cx.execute(q) for row in cx.fetchall(): cx.execute("select pgq.maint_rotate_tables_step1(%s)", [row[0]]) res = cx.fetchone()[0] if res: self.log.info('Rotating %s' % row[0]) else: cx.execute("select pgq.maint_rotate_tables_step1();") # finish rotation cx.execute("select pgq.maint_rotate_tables_step2();") # move retry events to main queue in small blocks rcount = 0 while 1: cx.execute('select pgq.maint_retry_events();') res = cx.fetchone()[0] rcount += res if res == 0: break if rcount: self.log.info('Got %d events for retry' % rcount) # vacuum tables that are needed cx.execute('set maintenance_work_mem = 32768') cx.execute('select * from pgq.maint_tables_to_vacuum()') for row in cx.fetchall(): cx.execute('vacuum %s;' % row[0]) skytools-2.1.13/python/pgq/status.py0000644000175000017500000000577111670174255016520 0ustar markomarko """Status display. """ import sys, os, skytools def ival(data, _as = None): "Format interval for output" if not _as: _as = data.split('.')[-1] numfmt = 'FM9999999' expr = "coalesce(to_char(extract(epoch from %s), '%s') || 's', 'NULL') as %s" return expr % (data, numfmt, _as) class PGQStatus(skytools.DBScript): def __init__(self, args, check = 0): skytools.DBScript.__init__(self, 'pgqadm', args) self.show_status() sys.exit(0) def show_status(self): db = self.get_database("db", autocommit=1) cx = db.cursor() cx.execute("show server_version") pgver = cx.fetchone()[0] cx.execute("select pgq.version()") qver = cx.fetchone()[0] print "Postgres version: %s PgQ version: %s" % (pgver, qver) q = """select f.queue_name, f.queue_ntables, %s, %s, %s, %s, q.queue_ticker_max_count from pgq.get_queue_info() f, pgq.queue q where q.queue_name = f.queue_name""" % ( ival('f.queue_rotation_period'), ival('f.ticker_lag'), ival('q.queue_ticker_max_lag'), ival('q.queue_ticker_idle_period'), ) cx.execute(q) event_rows = cx.dictfetchall() q = """select queue_name, consumer_name, %s, %s from pgq.get_consumer_info()""" % ( ival('lag'), ival('last_seen'), ) cx.execute(q) consumer_rows = cx.dictfetchall() print "\n%-45s %9s %13s %6s" % ('Event queue', 'Rotation', 'Ticker', 'TLag') print '-' * 78 for ev_row in event_rows: tck = "%s/%s/%s" % (ev_row['queue_ticker_max_count'], ev_row['queue_ticker_max_lag'], ev_row['queue_ticker_idle_period']) rot = "%s/%s" % (ev_row['queue_ntables'], ev_row['queue_rotation_period']) print "%-45s %9s %13s %6s" % ( ev_row['queue_name'], rot, tck, ev_row['ticker_lag'], ) print '-' * 78 print "\n%-56s %9s %9s" % ( 'Consumer', 'Lag', 'LastSeen') print '-' * 78 for ev_row in event_rows: cons = self.pick_consumers(ev_row, consumer_rows) self.show_queue(ev_row, cons) print '-' * 78 db.commit() def show_consumer(self, cons): print " %-54s %9s %9s" % ( cons['consumer_name'], cons['lag'], cons['last_seen']) def show_queue(self, ev_row, consumer_rows): print "%(queue_name)s:" % ev_row for cons in consumer_rows: self.show_consumer(cons) def pick_consumers(self, ev_row, consumer_rows): res = [] for con in consumer_rows: if con['queue_name'] != ev_row['queue_name']: continue res.append(con) return res skytools-2.1.13/config.mak.in0000644000175000017500000000072311670174255015047 0ustar markomarko prefix = @prefix@ datarootdir = @datarootdir@ mandir = @mandir@ override PYTHON = @PYTHON@ override PG_CONFIG = @PG_CONFIG@ # additional CPPFLAGS to pgxs modules PG_CPPFLAGS = $(filter -DHAVE%, @DEFS@) SQLDIR = $(prefix)/share/skytools PGXS = $(shell $(PG_CONFIG) --pgxs) DESTDIR = / ASCIIDOC = @ASCIIDOC@ XMLTO = @XMLTO@ PACKAGE_NAME = @PACKAGE_NAME@ PACKAGE_TARNAME = @PACKAGE_TARNAME@ PACKAGE_VERSION = @PACKAGE_VERSION@ PACKAGE_STRING = @PACKAGE_STRING@ skytools-2.1.13/PKG-INFO0000644000175000017500000000034511727601174013576 0ustar markomarkoMetadata-Version: 1.0 Name: skytools Version: 2.1.13 Summary: UNKNOWN Home-page: http://pgfoundry.org/projects/skytools/ Author: Marko Kreen Author-email: marko.kreen@skype.net License: BSD Description: UNKNOWN Platform: UNKNOWN skytools-2.1.13/upgrade/0000755000175000017500000000000011727601174014126 5ustar markomarkoskytools-2.1.13/upgrade/src/0000755000175000017500000000000011727601174014715 5ustar markomarkoskytools-2.1.13/upgrade/src/v2.1.7_londiste.sql0000644000175000017500000000051111670174255020171 0ustar markomarkobegin; \i ../sql/londiste/functions/londiste.provider_create_trigger.sql \i ../sql/londiste/functions/londiste.provider_refresh_trigger.sql \i ../sql/londiste/functions/londiste.provider_remove_table.sql \i ../sql/londiste/functions/londiste.subscriber_trigger_funcs.sql \i ../sql/londiste/functions/londiste.version.sql end; skytools-2.1.13/upgrade/src/v2.1.6_londiste.sql0000644000175000017500000000010111670174255020163 0ustar markomarkobegin; \i ../sql/londiste/functions/londiste.version.sql end; skytools-2.1.13/upgrade/src/v2.1.5_pgq_core.sql0000644000175000017500000000124611670174255020153 0ustar markomarkobegin; alter table pgq.subscription add constraint subscription_ukey unique (sub_queue, sub_consumer); create index rq_retry_owner_idx on pgq.retry_queue (ev_owner, ev_id); \i ../sql/pgq/functions/pgq.current_event_table.sql \i ../sql/pgq/functions/pgq.event_failed.sql \i ../sql/pgq/functions/pgq.event_retry.sql \i ../sql/pgq/functions/pgq.force_tick.sql \i ../sql/pgq/functions/pgq.grant_perms.sql \i ../sql/pgq/functions/pgq.insert_event.sql \i ../sql/pgq/functions/pgq.maint_tables_to_vacuum.sql \i ../sql/pgq/functions/pgq.next_batch.sql \i ../sql/pgq/functions/pgq.register_consumer.sql \i ../sql/pgq/functions/pgq.version.sql \i ../sql/pgq/structure/grants.sql end; skytools-2.1.13/upgrade/src/v2.1.6_pgq_ext.sql0000644000175000017500000000006711670174255020024 0ustar markomarkobegin; \i ../sql/pgq_ext/functions/version.sql end; skytools-2.1.13/upgrade/src/v2.1.8_pgq_core.sql0000644000175000017500000000015311670174255020152 0ustar markomarkobegin; \i ../sql/pgq/functions/pgq.maint_rotate_tables.sql \i ../sql/pgq/functions/pgq.version.sql end; skytools-2.1.13/upgrade/src/v2.1.7_pgq_core.sql0000644000175000017500000000015211670174255020150 0ustar markomarkobegin; \i ../sql/pgq/functions/pgq.maint_retry_events.sql \i ../sql/pgq/functions/pgq.version.sql end; skytools-2.1.13/upgrade/src/v2.1.5_londiste.sql0000644000175000017500000000262411670174255020176 0ustar markomarkobegin; create table londiste.subscriber_pending_fkeys( from_table text not null, to_table text not null, fkey_name text not null, fkey_def text not null, primary key (from_table, fkey_name) ); create table londiste.subscriber_pending_triggers ( table_name text not null, trigger_name text not null, trigger_def text not null, primary key (table_name, trigger_name) ); -- drop function londiste.denytrigger(); \i ../sql/londiste/functions/londiste.find_table_fkeys.sql \i ../sql/londiste/functions/londiste.find_table_triggers.sql \i ../sql/londiste/functions/londiste.find_column_types.sql \i ../sql/londiste/functions/londiste.subscriber_fkeys_funcs.sql \i ../sql/londiste/functions/londiste.subscriber_trigger_funcs.sql \i ../sql/londiste/functions/londiste.quote_fqname.sql \i ../sql/londiste/functions/londiste.find_table_oid.sql \i ../sql/londiste/functions/londiste.get_last_tick.sql \i ../sql/londiste/functions/londiste.provider_add_table.sql \i ../sql/londiste/functions/londiste.provider_create_trigger.sql \i ../sql/londiste/functions/londiste.provider_notify_change.sql \i ../sql/londiste/functions/londiste.provider_remove_table.sql \i ../sql/londiste/functions/londiste.set_last_tick.sql \i ../sql/londiste/functions/londiste.subscriber_remove_table.sql \i ../sql/londiste/structure/grants.sql end; skytools-2.1.13/upgrade/src/v2.1.5_pgq_ext.sql0000644000175000017500000000007211670174255020017 0ustar markomarkobegin; \i ../sql/pgq_ext/functions/track_tick.sql end; skytools-2.1.13/upgrade/final/0000755000175000017500000000000011727601174015217 5ustar markomarkoskytools-2.1.13/upgrade/final/v2.1.7_londiste.sql0000644000175000017500000001330411670174255020477 0ustar markomarko begin; create or replace function londiste.provider_create_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; begin select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; execute 'create trigger ' || quote_ident(tgname) || ' after insert or update or delete on ' || londiste.quote_fqname(i_table_name) || ' for each row execute procedure pgq.logtriga(' || quote_literal(i_queue_name) || ', ' || quote_literal(i_col_types) || ', ' || quote_literal(i_table_name) || ')'; return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_refresh_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare t_name text; tbl_oid oid; begin select trigger_name into t_name from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; tbl_oid := londiste.find_table_oid(i_table_name); perform 1 from pg_trigger where tgrelid = tbl_oid and tgname = t_name; if found then execute 'drop trigger ' || quote_ident(t_name) || ' on ' || londiste.quote_fqname(i_table_name); end if; perform londiste.provider_create_trigger(i_queue_name, i_table_name, i_col_types); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_refresh_trigger( i_queue_name text, i_table_name text ) returns integer strict as $$ begin return londiste.provider_refresh_trigger(i_queue_name, i_table_name, londiste.find_column_types(i_table_name)); end; $$ language plpgsql security definer; create or replace function londiste.provider_remove_table( i_queue_name text, i_table_name text ) returns integer as $$ declare tgname text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'no such table registered'; end if; begin execute 'drop trigger ' || quote_ident(tgname) || ' on ' || londiste.quote_fqname(i_table_name); exception when undefined_table then raise notice 'table % does not exist', i_table_name; when undefined_object then raise notice 'trigger % does not exist on table %', tgname, i_table_name; end; delete from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; return 1; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_get_table_pending_triggers(i_table_name text) returns setof londiste.subscriber_pending_triggers as $$ declare trigger record; begin for trigger in select * from londiste.subscriber_pending_triggers where table_name = i_table_name loop return next trigger; end loop; return; end; $$ language plpgsql strict stable; create or replace function londiste.subscriber_drop_table_trigger(i_table_name text, i_trigger_name text) returns integer as $$ declare trig_def record; begin select * into trig_def from londiste.find_table_triggers(i_table_name) where trigger_name = i_trigger_name; if FOUND is not true then return 0; end if; insert into londiste.subscriber_pending_triggers(table_name, trigger_name, trigger_def) values (i_table_name, i_trigger_name, trig_def.trigger_def); execute 'drop trigger ' || quote_ident(i_trigger_name) || ' on ' || londiste.quote_fqname(i_table_name); return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_drop_all_table_triggers(i_table_name text) returns integer as $$ declare trigger record; begin for trigger in select trigger_name as name from londiste.find_table_triggers(i_table_name) loop perform londiste.subscriber_drop_table_trigger(i_table_name, trigger.name); end loop; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_table_trigger(i_table_name text, i_trigger_name text) returns integer as $$ declare trig_def text; begin select trigger_def into trig_def from londiste.subscriber_pending_triggers where (table_name, trigger_name) = (i_table_name, i_trigger_name); if not found then return 0; end if; delete from londiste.subscriber_pending_triggers where table_name = i_table_name and trigger_name = i_trigger_name; execute trig_def; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_all_table_triggers(i_table_name text) returns integer as $$ declare trigger record; begin for trigger in select trigger_name as name from londiste.subscriber_get_table_pending_triggers(i_table_name) loop perform londiste.subscriber_restore_table_trigger(i_table_name, trigger.name); end loop; return 1; end; $$ language plpgsql; create or replace function londiste.version() returns text as $$ begin return '2.1.7'; end; $$ language plpgsql; end; skytools-2.1.13/upgrade/final/v2.1.6_londiste.sql0000644000175000017500000000021211670174255020470 0ustar markomarko begin; create or replace function londiste.version() returns text as $$ begin return '2.1.6'; end; $$ language plpgsql; end; skytools-2.1.13/upgrade/final/v2.1.5_pgq_core.sql0000644000175000017500000004210411670174255020453 0ustar markomarko begin; alter table pgq.subscription add constraint subscription_ukey unique (sub_queue, sub_consumer); create index rq_retry_owner_idx on pgq.retry_queue (ev_owner, ev_id); create or replace function pgq.current_event_table(x_queue_name text) returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.current_event_table(1) -- -- Return active event table for particular queue. -- Event can be added to it without going via functions, -- e.g. by COPY. -- -- Note: -- The result is valid only during current transaction. -- -- Permissions: -- Actual insertion requires superuser access. -- -- Parameters: -- x_queue_name - Queue name. -- ---------------------------------------------------------------------- declare res text; begin select queue_data_pfx || '_' || queue_cur_table into res from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'Event queue not found'; end if; return res; end; $$ language plpgsql; -- no perms needed create or replace function pgq.event_failed( x_batch_id bigint, x_event_id bigint, x_reason text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_failed(3) -- -- Copies the event to failed queue so it can be looked at later. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - Event id -- x_reason - Text to associate with event. -- -- Returns: -- 0 if event was already in queue, 1 otherwise. -- ---------------------------------------------------------------------- begin insert into pgq.failed_queue (ev_failed_reason, ev_failed_time, ev_id, ev_time, ev_txid, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) select x_reason, now(), ev_id, ev_time, NULL, sub_id, coalesce(ev_retry, 0), ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.get_batch_events(x_batch_id), pgq.subscription where sub_batch = x_batch_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; -- dont worry if the event is already in queue exception when unique_violation then return 0; end; $$ language plpgsql security definer; create or replace function pgq.event_retry( x_batch_id bigint, x_event_id bigint, x_retry_time timestamptz) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry(3) -- -- Put the event into retry queue, to be processed again later. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - event id -- x_retry_time - Time when the event should be put back into queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- begin insert into pgq.retry_queue (ev_retry_after, ev_id, ev_time, ev_txid, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) select x_retry_time, ev_id, ev_time, NULL, sub_id, coalesce(ev_retry, 0) + 1, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.get_batch_events(x_batch_id), pgq.subscription where sub_batch = x_batch_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; -- dont worry if the event is already in queue exception when unique_violation then return 0; end; $$ language plpgsql security definer; create or replace function pgq.event_retry( x_batch_id bigint, x_event_id bigint, x_retry_seconds integer) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry(3) -- -- Put the event into retry queue, to be processed later again. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - event id -- x_retry_seconds - Time when the event should be put back into queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare new_retry timestamptz; begin new_retry := current_timestamp + ((x_retry_seconds || ' seconds')::interval); return pgq.event_retry(x_batch_id, x_event_id, new_retry); end; $$ language plpgsql security definer; create or replace function pgq.force_tick(i_queue_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.force_tick(2) -- -- Simulate lots of events happening to force ticker to tick. -- -- Should be called in loop, with some delay until last tick -- changes or too much time is passed. -- -- Such function is needed because paraller calls of pgq.ticker() are -- dangerous, and cannot be protected with locks as snapshot -- is taken before locking. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- Currently last tick id. -- ---------------------------------------------------------------------- declare q record; t record; begin -- bump seq and get queue id select queue_id, setval(queue_event_seq, nextval(queue_event_seq) + queue_ticker_max_count * 2) as tmp into q from pgq.queue where queue_name = i_queue_name and not queue_external_ticker; if not found then raise exception 'queue not found or ticks not allowed'; end if; -- return last tick id select tick_id into t from pgq.tick where tick_queue = q.queue_id order by tick_queue desc, tick_id desc limit 1; return t.tick_id; end; $$ language plpgsql security definer; create or replace function pgq.grant_perms(x_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.grant_perms(1) -- -- Make event tables readable by public. -- -- Parameters: -- x_queue_name - Name of the queue. -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare q record; i integer; tbl_perms text; seq_perms text; begin select * from pgq.queue into q where queue_name = x_queue_name; if not found then raise exception 'Queue not found'; end if; if true then -- safe, all access must go via functions seq_perms := 'select'; tbl_perms := 'select'; else -- allow ordinery users to directly insert -- to event tables. dangerous. seq_perms := 'select, update'; tbl_perms := 'select, insert'; end if; -- tick seq, normal users don't need to modify it execute 'grant ' || seq_perms || ' on ' || q.queue_tick_seq || ' to public'; -- event seq execute 'grant ' || seq_perms || ' on ' || q.queue_event_seq || ' to public'; -- parent table for events execute 'grant select on ' || q.queue_data_pfx || ' to public'; -- real event tables for i in 0 .. q.queue_ntables - 1 loop execute 'grant ' || tbl_perms || ' on ' || q.queue_data_pfx || '_' || i || ' to public'; end loop; return 1; end; $$ language plpgsql security definer; create or replace function pgq.insert_event(queue_name text, ev_type text, ev_data text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.insert_event(3) -- -- Insert a event into queue. -- -- Parameters: -- queue_name - Name of the queue -- ev_type - User-specified type for the event -- ev_data - User data for the event -- -- Returns: -- Event ID -- ---------------------------------------------------------------------- begin return pgq.insert_event(queue_name, ev_type, ev_data, null, null, null, null); end; $$ language plpgsql security definer; create or replace function pgq.insert_event( queue_name text, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.insert_event(7) -- -- Insert a event into queue with all the extra fields. -- -- Parameters: -- queue_name - Name of the queue -- ev_type - User-specified type for the event -- ev_data - User data for the event -- ev_extra1 - Extra data field for the event -- ev_extra2 - Extra data field for the event -- ev_extra3 - Extra data field for the event -- ev_extra4 - Extra data field for the event -- -- Returns: -- Event ID -- ---------------------------------------------------------------------- begin return pgq.insert_event_raw(queue_name, null, now(), null, null, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4); end; $$ language plpgsql security definer; create or replace function pgq.maint_tables_to_vacuum() returns setof text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_tables_to_vacuum(0) -- -- Returns list of tablenames that need frequent vacuuming. -- -- The goal is to avoid hardcoding them into maintenance process. -- -- Returns: -- List of table names. -- ---------------------------------------------------------------------- declare row record; begin return next 'pgq.subscription'; return next 'pgq.consumer'; return next 'pgq.queue'; return next 'pgq.tick'; return next 'pgq.retry_queue'; -- include also txid, pgq_ext and londiste tables if they exist for row in select n.nspname as scm, t.relname as tbl from pg_class t, pg_namespace n where n.oid = t.relnamespace and n.nspname = 'txid' and t.relname = 'epoch' union all select n.nspname as scm, t.relname as tbl from pg_class t, pg_namespace n where n.oid = t.relnamespace and n.nspname = 'londiste' and t.relname = 'completed' union all select n.nspname as scm, t.relname as tbl from pg_class t, pg_namespace n where n.oid = t.relnamespace and n.nspname = 'pgq_ext' and t.relname in ('completed_tick', 'completed_batch', 'completed_event', 'partial_batch') loop return next row.scm || '.' || row.tbl; end loop; return; end; $$ language plpgsql; create or replace function pgq.next_batch(x_queue_name text, x_consumer_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.next_batch(2) -- -- Makes next block of events active. -- -- If it returns NULL, there is no events available in queue. -- Consumer should sleep a bith then. -- -- Parameters: -- x_queue_name - Name of the queue -- x_consumer_name - Name of the consumer -- -- Returns: -- Batch ID or NULL if there are no more events available. -- ---------------------------------------------------------------------- declare next_tick bigint; batch_id bigint; errmsg text; sub record; begin select sub_queue, sub_consumer, sub_id, sub_last_tick, sub_batch into sub from pgq.queue q, pgq.consumer c, pgq.subscription s where q.queue_name = x_queue_name and c.co_name = x_consumer_name and s.sub_queue = q.queue_id and s.sub_consumer = c.co_id; if not found then errmsg := 'Not subscriber to queue: ' || coalesce(x_queue_name, 'NULL') || '/' || coalesce(x_consumer_name, 'NULL'); raise exception '%', errmsg; end if; -- has already active batch if sub.sub_batch is not null then return sub.sub_batch; end if; -- find next tick select tick_id into next_tick from pgq.tick where tick_id > sub.sub_last_tick and tick_queue = sub.sub_queue order by tick_queue asc, tick_id asc limit 1; if not found then -- nothing to do return null; end if; -- get next batch batch_id := nextval('pgq.batch_id_seq'); update pgq.subscription set sub_batch = batch_id, sub_next_tick = next_tick, sub_active = now() where sub_queue = sub.sub_queue and sub_consumer = sub.sub_consumer; return batch_id; end; $$ language plpgsql security definer; create or replace function pgq.register_consumer( x_queue_name text, x_consumer_id text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.register_consumer(2) -- -- Subscribe consumer on a queue. -- -- From this moment forward, consumer will see all events in the queue. -- -- Parameters: -- x_queue_name - Name of queue -- x_consumer_name - Name of consumer -- -- Returns: -- 0 - if already registered -- 1 - if new registration -- ---------------------------------------------------------------------- begin return pgq.register_consumer(x_queue_name, x_consumer_id, NULL); end; $$ language plpgsql security definer; create or replace function pgq.register_consumer( x_queue_name text, x_consumer_name text, x_tick_pos bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.register_consumer(3) -- -- Extended registration, allows to specify tick_id. -- -- Note: -- For usage in special situations. -- -- Parameters: -- x_queue_name - Name of a queue -- x_consumer_name - Name of consumer -- x_tick_pos - Tick ID -- -- Returns: -- 0/1 whether consumer has already registered. -- ---------------------------------------------------------------------- declare tmp text; last_tick bigint; x_queue_id integer; x_consumer_id integer; queue integer; sub record; begin select queue_id into x_queue_id from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'Event queue not created yet'; end if; -- get consumer and create if new select co_id into x_consumer_id from pgq.consumer where co_name = x_consumer_name; if not found then insert into pgq.consumer (co_name) values (x_consumer_name); x_consumer_id := currval('pgq.consumer_co_id_seq'); end if; -- if particular tick was requested, check if it exists if x_tick_pos is not null then perform 1 from pgq.tick where tick_queue = x_queue_id and tick_id = x_tick_pos; if not found then raise exception 'cannot reposition, tick not found: %', x_tick_pos; end if; end if; -- check if already registered select sub_last_tick, sub_batch into sub from pgq.subscription where sub_consumer = x_consumer_id and sub_queue = x_queue_id; if found then if x_tick_pos is not null then if sub.sub_batch is not null then raise exception 'reposition while active not allowed'; end if; -- update tick pos if requested update pgq.subscription set sub_last_tick = x_tick_pos where sub_consumer = x_consumer_id and sub_queue = x_queue_id; end if; -- already registered return 0; end if; -- new registration if x_tick_pos is null then -- start from current tick select tick_id into last_tick from pgq.tick where tick_queue = x_queue_id order by tick_queue desc, tick_id desc limit 1; if not found then raise exception 'No ticks for this queue. Please run ticker on database.'; end if; else last_tick := x_tick_pos; end if; -- register insert into pgq.subscription (sub_queue, sub_consumer, sub_last_tick) values (x_queue_id, x_consumer_id, last_tick); return 1; end; $$ language plpgsql security definer; create or replace function pgq.version() returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.version(0) -- -- Returns verison string for pgq. ATM its SkyTools version -- that is only bumped when PGQ database code changes. -- ---------------------------------------------------------------------- begin return '2.1.5'; end; $$ language plpgsql; grant usage on schema pgq to public; grant select on table pgq.consumer to public; grant select on table pgq.queue to public; grant select on table pgq.tick to public; grant select on table pgq.queue to public; grant select on table pgq.subscription to public; grant select on table pgq.event_template to public; grant select on table pgq.retry_queue to public; grant select on table pgq.failed_queue to public; end; skytools-2.1.13/upgrade/final/v2.1.6_pgq_ext.sql0000644000175000017500000000021111670174255020315 0ustar markomarko begin; create or replace function pgq_ext.version() returns text as $$ begin return '2.1.6'; end; $$ language plpgsql; end; skytools-2.1.13/upgrade/final/v2.1.8_pgq_core.sql0000644000175000017500000001002011670174255020446 0ustar markomarko begin; create or replace function pgq.maint_rotate_tables_step1(i_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_rotate_tables_step1(1) -- -- Rotate tables for one queue. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- 1 if rotation happened, otherwise 0. -- ---------------------------------------------------------------------- declare badcnt integer; cf record; nr integer; tbl text; lowest_tick_id int8; lowest_xmin int8; begin -- check if needed and load record select * from pgq.queue into cf where queue_name = i_queue_name and queue_rotation_period is not null and queue_switch_step2 is not null and queue_switch_time + queue_rotation_period < current_timestamp for update; if not found then return 0; end if; -- find lowest tick for that queue select min(sub_last_tick) into lowest_tick_id from pgq.subscription where sub_queue = cf.queue_id; -- if some consumer exists if lowest_tick_id is not null then -- is the slowest one still on previous table? select txid_snapshot_xmin(tick_snapshot) into lowest_xmin from pgq.tick where tick_queue = cf.queue_id and tick_id = lowest_tick_id; if lowest_xmin <= cf.queue_switch_step2 then return 0; -- skip rotation then end if; end if; -- nobody on previous table, we can rotate -- calc next table number and name nr := cf.queue_cur_table + 1; if nr = cf.queue_ntables then nr := 0; end if; tbl := cf.queue_data_pfx || '_' || nr; -- there may be long lock on the table from pg_dump, -- detect it and skip rotate then begin execute 'lock table ' || tbl || ' nowait'; execute 'truncate ' || tbl; exception when lock_not_available then -- cannot truncate, skipping rotate return 0; end; -- remember the moment update pgq.queue set queue_cur_table = nr, queue_switch_time = current_timestamp, queue_switch_step1 = txid_current(), queue_switch_step2 = NULL where queue_id = cf.queue_id; -- Clean ticks by using step2 txid from previous rotation. -- That should keep all ticks for all batches that are completely -- in old table. This keeps them for longer than needed, but: -- 1. we want the pgq.tick table to be big, to avoid Postgres -- accitentally switching to seqscans on that. -- 2. that way we guarantee to consumers that they an be moved -- back on the queue at least for one rotation_period. -- (may help in disaster recovery) delete from pgq.tick where tick_queue = cf.queue_id and txid_snapshot_xmin(tick_snapshot) < cf.queue_switch_step2; return 1; end; $$ language plpgsql; -- need admin access create or replace function pgq.maint_rotate_tables_step2() returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_rotate_tables_step2(0) -- -- Stores the txid when the rotation was visible. It should be -- called in separate transaction than pgq.maint_rotate_tables_step1() -- ---------------------------------------------------------------------- begin update pgq.queue set queue_switch_step2 = txid_current() where queue_switch_step2 is null; return 1; end; $$ language plpgsql; -- need admin access create or replace function pgq.version() returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.version(0) -- -- Returns verison string for pgq. ATM its SkyTools version -- that is only bumped when PGQ database code changes. -- ---------------------------------------------------------------------- begin return '2.1.8'; end; $$ language plpgsql; end; skytools-2.1.13/upgrade/final/v2.1.7_pgq_core.sql0000644000175000017500000000345011670174255020456 0ustar markomarko begin; create or replace function pgq.maint_retry_events() returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_retry_events(0) -- -- Moves retry events back to main queue. -- -- It moves small amount at a time. It should be called -- until it returns 0 -- -- Returns: -- Number of events processed. -- ---------------------------------------------------------------------- declare cnt integer; rec record; begin cnt := 0; for rec in select queue_name, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.retry_queue, pgq.queue, pgq.subscription where ev_retry_after <= current_timestamp and sub_id = ev_owner and queue_id = sub_queue order by ev_retry_after limit 10 loop cnt := cnt + 1; perform pgq.insert_event_raw(rec.queue_name, rec.ev_id, rec.ev_time, rec.ev_owner, rec.ev_retry, rec.ev_type, rec.ev_data, rec.ev_extra1, rec.ev_extra2, rec.ev_extra3, rec.ev_extra4); delete from pgq.retry_queue where ev_owner = rec.ev_owner and ev_id = rec.ev_id; end loop; return cnt; end; $$ language plpgsql; -- need admin access create or replace function pgq.version() returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.version(0) -- -- Returns verison string for pgq. ATM its SkyTools version -- that is only bumped when PGQ database code changes. -- ---------------------------------------------------------------------- begin return '2.1.7'; end; $$ language plpgsql; end; skytools-2.1.13/upgrade/final/v2.1.5_londiste.sql0000644000175000017500000003620211670174255020477 0ustar markomarko begin; create table londiste.subscriber_pending_fkeys( from_table text not null, to_table text not null, fkey_name text not null, fkey_def text not null, primary key (from_table, fkey_name) ); create table londiste.subscriber_pending_triggers ( table_name text not null, trigger_name text not null, trigger_def text not null, primary key (table_name, trigger_name) ); -- drop function londiste.denytrigger(); create or replace function londiste.find_table_fkeys(i_table_name text) returns setof londiste.subscriber_pending_fkeys as $$ declare fkey record; tbl_oid oid; begin select londiste.find_table_oid(i_table_name) into tbl_oid; for fkey in select n1.nspname || '.' || t1.relname as from_table, n2.nspname || '.' || t2.relname as to_table, conname::text as fkey_name, 'alter table only ' || quote_ident(n1.nspname) || '.' || quote_ident(t1.relname) || ' add constraint ' || quote_ident(conname::text) || ' ' || pg_get_constraintdef(c.oid) as fkey_def from pg_constraint c, pg_namespace n1, pg_class t1, pg_namespace n2, pg_class t2 where c.contype = 'f' and (c.conrelid = tbl_oid or c.confrelid = tbl_oid) and t1.oid = c.conrelid and n1.oid = t1.relnamespace and t2.oid = c.confrelid and n2.oid = t2.relnamespace order by 1,2,3 loop return next fkey; end loop; return; end; $$ language plpgsql strict stable; create or replace function londiste.find_table_triggers(i_table_name text) returns setof londiste.subscriber_pending_triggers as $$ declare tg record; begin for tg in select n.nspname || '.' || c.relname as table_name, t.tgname::text as name, pg_get_triggerdef(t.oid) as def from pg_trigger t, pg_class c, pg_namespace n where n.oid = c.relnamespace and c.oid = t.tgrelid and t.tgrelid = londiste.find_table_oid(i_table_name) and not t.tgisconstraint loop return next tg; end loop; return; end; $$ language plpgsql strict stable; create or replace function londiste.find_column_types(tbl text) returns text as $$ declare res text; col record; tbl_oid oid; begin tbl_oid := londiste.find_table_oid(tbl); res := ''; for col in SELECT CASE WHEN k.attname IS NOT NULL THEN 'k' ELSE 'v' END AS type FROM pg_attribute a LEFT JOIN ( SELECT k.attname FROM pg_index i, pg_attribute k WHERE i.indrelid = tbl_oid AND k.attrelid = i.indexrelid AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped ) k ON (k.attname = a.attname) WHERE a.attrelid = tbl_oid AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum loop res := res || col.type; end loop; return res; end; $$ language plpgsql strict stable; create or replace function londiste.subscriber_get_table_pending_fkeys(i_table_name text) returns setof londiste.subscriber_pending_fkeys as $$ declare fkeys record; begin for fkeys in select * from londiste.subscriber_pending_fkeys where from_table=i_table_name or to_table=i_table_name order by 1,2,3 loop return next fkeys; end loop; return; end; $$ language plpgsql; create or replace function londiste.subscriber_get_queue_valid_pending_fkeys(i_queue_name text) returns setof londiste.subscriber_pending_fkeys as $$ declare fkeys record; begin for fkeys in select pf.* from londiste.subscriber_pending_fkeys pf left join londiste.subscriber_table st_from on (st_from.table_name = pf.from_table) left join londiste.subscriber_table st_to on (st_to.table_name = pf.to_table) where (st_from.table_name is null or (st_from.merge_state = 'ok' and st_from.snapshot is null)) and (st_to.table_name is null or (st_to.merge_state = 'ok' and st_to.snapshot is null)) and (coalesce(st_from.queue_name = i_queue_name, false) or coalesce(st_to.queue_name = i_queue_name, false)) order by 1, 2, 3 loop return next fkeys; end loop; return; end; $$ language plpgsql; create or replace function londiste.subscriber_drop_table_fkey(i_from_table text, i_fkey_name text) returns integer as $$ declare fkey record; begin select * into fkey from londiste.find_table_fkeys(i_from_table) where fkey_name = i_fkey_name and from_table = i_from_table; if not found then return 0; end if; insert into londiste.subscriber_pending_fkeys values (fkey.from_table, fkey.to_table, i_fkey_name, fkey.fkey_def); execute 'alter table only ' || londiste.quote_fqname(fkey.from_table) || ' drop constraint ' || quote_ident(i_fkey_name); return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_table_fkey(i_from_table text, i_fkey_name text) returns integer as $$ declare fkey record; begin select * into fkey from londiste.subscriber_pending_fkeys where fkey_name = i_fkey_name and from_table = i_from_table; if not found then return 0; end if; delete from londiste.subscriber_pending_fkeys where fkey_name = fkey.fkey_name; execute fkey.fkey_def; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_get_table_pending_triggers(i_table_name text) returns setof londiste.subscriber_pending_triggers as $$ declare trigger record; begin for trigger in select * from londiste.subscriber_pending_triggers where table_name = i_table_name loop return next trigger; end loop; return; end; $$ language plpgsql strict stable; create or replace function londiste.subscriber_drop_table_trigger(i_table_name text, i_trigger_name text) returns integer as $$ declare trig_def record; begin select * into trig_def from londiste.find_table_triggers(i_table_name) where trigger_name = i_trigger_name; if FOUND is not true then return 0; end if; insert into londiste.subscriber_pending_triggers(table_name, trigger_name, trigger_def) values (i_table_name, i_trigger_name, trig_def.trigger_def); execute 'drop trigger ' || i_trigger_name || ' on ' || i_table_name; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_drop_all_table_triggers(i_table_name text) returns integer as $$ declare trigger record; begin for trigger in select trigger_name as name from londiste.find_table_triggers(i_table_name) loop perform londiste.subscriber_drop_table_trigger(i_table_name, trigger.name); end loop; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_table_trigger(i_table_name text, i_trigger_name text) returns integer as $$ declare trig_def text; begin select trigger_def into trig_def from londiste.subscriber_pending_triggers where (table_name, trigger_name) = (i_table_name, i_trigger_name); if not found then return 0; end if; delete from londiste.subscriber_pending_triggers where table_name = i_table_name and trigger_name = i_trigger_name; execute trig_def; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_all_table_triggers(i_table_name text) returns integer as $$ declare trigger record; begin for trigger in select trigger_name as name from londiste.subscriber_get_table_pending_triggers(i_table_name) loop perform londiste.subscriber_restore_table_trigger(i_table_name, trigger.name); end loop; return 1; end; $$ language plpgsql; create or replace function londiste.quote_fqname(i_name text) returns text as $$ declare res text; pos integer; s text; n text; begin pos := position('.' in i_name); if pos > 0 then s := substring(i_name for pos - 1); n := substring(i_name from pos + 1); else s := 'public'; n := i_name; end if; return quote_ident(s) || '.' || quote_ident(n); end; $$ language plpgsql strict immutable; create or replace function londiste.find_rel_oid(tbl text, kind text) returns oid as $$ declare res oid; pos integer; schema text; name text; begin pos := position('.' in tbl); if pos > 0 then schema := substring(tbl for pos - 1); name := substring(tbl from pos + 1); else schema := 'public'; name := tbl; end if; select c.oid into res from pg_namespace n, pg_class c where c.relnamespace = n.oid and c.relkind = kind and n.nspname = schema and c.relname = name; if not found then if kind = 'r' then raise exception 'table not found'; elsif kind = 'S' then raise exception 'seq not found'; else raise exception 'weird relkind'; end if; end if; return res; end; $$ language plpgsql strict stable; create or replace function londiste.find_table_oid(tbl text) returns oid as $$ begin return londiste.find_rel_oid(tbl, 'r'); end; $$ language plpgsql strict stable; create or replace function londiste.find_seq_oid(tbl text) returns oid as $$ begin return londiste.find_rel_oid(tbl, 'S'); end; $$ language plpgsql strict stable; create or replace function londiste.get_last_tick(i_consumer text) returns bigint as $$ declare res bigint; begin select last_tick_id into res from londiste.completed where consumer_id = i_consumer; return res; end; $$ language plpgsql security definer strict stable; create or replace function londiste.provider_add_table( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; sql text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; if position('k' in i_col_types) < 1 then raise exception 'need key column'; end if; if position('.' in i_table_name) < 1 then raise exception 'need fully-qualified table name'; end if; select queue_name into tgname from pgq.queue where queue_name = i_queue_name; if not found then raise exception 'no such event queue'; end if; tgname := i_queue_name || '_logger'; tgname := replace(lower(tgname), '.', '_'); insert into londiste.provider_table (queue_name, table_name, trigger_name) values (i_queue_name, i_table_name, tgname); perform londiste.provider_create_trigger( i_queue_name, i_table_name, i_col_types); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_add_table( i_queue_name text, i_table_name text ) returns integer as $$ begin return londiste.provider_add_table(i_queue_name, i_table_name, londiste.find_column_types(i_table_name)); end; $$ language plpgsql security definer; create or replace function londiste.provider_create_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; begin select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; execute 'create trigger ' || tgname || ' after insert or update or delete on ' || i_table_name || ' for each row execute procedure pgq.logtriga(' || quote_literal(i_queue_name) || ', ' || quote_literal(i_col_types) || ', ' || quote_literal(i_table_name) || ')'; return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_notify_change(i_queue_name text) returns integer as $$ declare res text; tbl record; begin res := ''; for tbl in select table_name from londiste.provider_table where queue_name = i_queue_name order by nr loop if res = '' then res := tbl.table_name; else res := res || ',' || tbl.table_name; end if; end loop; perform pgq.insert_event(i_queue_name, 'T', res); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_remove_table( i_queue_name text, i_table_name text ) returns integer as $$ declare tgname text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'no such table registered'; end if; begin execute 'drop trigger ' || tgname || ' on ' || i_table_name; exception when undefined_table then raise notice 'table % does not exist', i_table_name; when undefined_object then raise notice 'trigger % does not exist on table %', tgname, i_table_name; end; delete from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; return 1; end; $$ language plpgsql security definer; create or replace function londiste.set_last_tick( i_consumer text, i_tick_id bigint) returns integer as $$ begin if i_tick_id is null then delete from londiste.completed where consumer_id = i_consumer; else update londiste.completed set last_tick_id = i_tick_id where consumer_id = i_consumer; if not found then insert into londiste.completed (consumer_id, last_tick_id) values (i_consumer, i_tick_id); end if; end if; return 1; end; $$ language plpgsql security definer; create or replace function londiste.subscriber_remove_table( i_queue_name text, i_table text) returns integer as $$ declare link text; begin delete from londiste.subscriber_table where queue_name = i_queue_name and table_name = i_table; if not found then raise exception 'no such table'; end if; -- sync link link := londiste.link_dest(i_queue_name); if link is not null then delete from londiste.provider_table where queue_name = link and table_name = i_table; perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql security definer; grant usage on schema londiste to public; grant select on londiste.provider_table to public; grant select on londiste.completed to public; grant select on londiste.link to public; grant select on londiste.subscriber_table to public; end; skytools-2.1.13/upgrade/final/v2.1.5_pgq_ext.sql0000644000175000017500000000157411670174255020331 0ustar markomarko begin; create or replace function pgq_ext.get_last_tick(a_consumer text) returns int8 as $$ declare res int8; begin select last_tick_id into res from pgq_ext.completed_tick where consumer_id = a_consumer; return res; end; $$ language plpgsql security definer; create or replace function pgq_ext.set_last_tick(a_consumer text, a_tick_id bigint) returns integer as $$ begin if a_tick_id is null then delete from pgq_ext.completed_tick where consumer_id = a_consumer; else update pgq_ext.completed_tick set last_tick_id = a_tick_id where consumer_id = a_consumer; if not found then insert into pgq_ext.completed_tick (consumer_id, last_tick_id) values (a_consumer, a_tick_id); end if; end if; return 1; end; $$ language plpgsql security definer; end; skytools-2.1.13/upgrade/Makefile0000644000175000017500000000055011670174255015570 0ustar markomarko #SQLS = v2.1.5_londiste.sql v2.1.5_pgq_core.sql v2.1.5_pgq_ext.sql #SQLS = v2.1.6_londiste.sql v2.1.6_pgq_ext.sql #SQLS = v2.1.7_pgq_core.sql v2.1.7_londiste.sql SQLS = v2.1.8_pgq_core.sql SRCS = $(addprefix src/, $(SQLS)) DSTS = $(addprefix final/, $(SQLS)) CATSQL = python ../scripts/catsql.py all: $(DSTS) final/%.sql: src/%.sql $(CATSQL) $< > $@ skytools-2.1.13/sql/0000755000175000017500000000000011727601174013276 5ustar markomarkoskytools-2.1.13/sql/txid/0000755000175000017500000000000011727601174014246 5ustar markomarkoskytools-2.1.13/sql/txid/expected/0000755000175000017500000000000011727601174016047 5ustar markomarkoskytools-2.1.13/sql/txid/expected/txid.out0000644000175000017500000000420311670174255017551 0ustar markomarko-- init \set ECHO none -- i/o select '12:13:'::txid_snapshot; txid_snapshot --------------- 12:13: (1 row) select '12:13:1,2'::txid_snapshot; ERROR: illegal txid_snapshot input format -- errors select '31:12:'::txid_snapshot; ERROR: illegal txid_snapshot input format select '0:1:'::txid_snapshot; ERROR: illegal txid_snapshot input format select '12:13:0'::txid_snapshot; ERROR: illegal txid_snapshot input format select '12:13:2,1'::txid_snapshot; ERROR: illegal txid_snapshot input format create table snapshot_test ( nr integer, snap txid_snapshot ); insert into snapshot_test values (1, '12:13:'); insert into snapshot_test values (2, '12:20:13,15,18'); insert into snapshot_test values (3, '100001:100009:100005,100007,100008'); select snap from snapshot_test order by nr; snap ------------------------------------ 12:13: 12:20:13,15,18 100001:100009:100005,100007,100008 (3 rows) select txid_snapshot_xmin(snap), txid_snapshot_xmax(snap), txid_snapshot_xip(snap) from snapshot_test order by nr; txid_snapshot_xmin | txid_snapshot_xmax | txid_snapshot_xip --------------------+--------------------+------------------- 12 | 20 | 13 12 | 20 | 15 12 | 20 | 18 100001 | 100009 | 100005 100001 | 100009 | 100007 100001 | 100009 | 100008 (6 rows) select id, txid_visible_in_snapshot(id, snap) from snapshot_test, generate_series(11, 21) id where nr = 2; id | txid_visible_in_snapshot ----+-------------------------- 11 | t 12 | t 13 | f 14 | t 15 | f 16 | t 17 | t 18 | f 19 | t 20 | f 21 | f (11 rows) -- test current values also select txid_current() >= txid_snapshot_xmin(txid_current_snapshot()); ?column? ---------- t (1 row) -- select txid_current_txid() < txid_snapshot_xmax(txid_current_snapshot()); -- select txid_in_snapshot(txid_current_txid(), txid_current_snapshot()), -- txid_not_in_snapshot(txid_current_txid(), txid_current_snapshot()); skytools-2.1.13/sql/txid/txid.h0000644000175000017500000000242611670174255015375 0ustar markomarko#ifndef _TXID_H_ #define _TXID_H_ #define MAX_INT64 0x7FFFFFFFFFFFFFFFLL /* Use unsigned variant internally */ typedef uint64 txid; typedef struct { int32 __varsz; /* should not be touched directly */ uint32 nxip; txid xmin; txid xmax; txid xip[1]; } TxidSnapshot; #define TXID_SNAPSHOT_SIZE(nxip) (offsetof(TxidSnapshot, xip) + sizeof(txid) * nxip) typedef struct { uint64 last_value; uint64 epoch; } TxidEpoch; /* internal functions */ void txid_load_epoch(TxidEpoch *state, int try_write); txid txid_convert_xid(TransactionId xid, TxidEpoch *state); /* public functions */ Datum txid_current(PG_FUNCTION_ARGS); Datum txid_current_snapshot(PG_FUNCTION_ARGS); Datum txid_snapshot_in(PG_FUNCTION_ARGS); Datum txid_snapshot_out(PG_FUNCTION_ARGS); Datum txid_snapshot_recv(PG_FUNCTION_ARGS); Datum txid_snapshot_send(PG_FUNCTION_ARGS); Datum txid_snapshot_xmin(PG_FUNCTION_ARGS); Datum txid_snapshot_xmax(PG_FUNCTION_ARGS); Datum txid_snapshot_xip(PG_FUNCTION_ARGS); Datum txid_visible_in_snapshot(PG_FUNCTION_ARGS); Datum txid_snapshot_active(PG_FUNCTION_ARGS); Datum txid_in_snapshot(PG_FUNCTION_ARGS); Datum txid_not_in_snapshot(PG_FUNCTION_ARGS); #endif /* _TXID_H_ */ skytools-2.1.13/sql/txid/txid.c0000644000175000017500000002360211670174255015367 0ustar markomarko/*------------------------------------------------------------------------- * txid.c * * Safe handling of transaction ID's. * * Copyright (c) 2003-2004, PostgreSQL Global Development Group * Author: Jan Wieck, Afilias USA INC. * * 64-bit output: Marko Kreen, Skype Technologies *------------------------------------------------------------------------- */ #include "postgres.h" #include #include "access/xact.h" #include "funcapi.h" #include "lib/stringinfo.h" #include "libpq/pqformat.h" #include "txid.h" #ifdef INT64_IS_BUSTED #error txid needs working int64 #endif #ifdef PG_MODULE_MAGIC PG_MODULE_MAGIC; #endif #ifndef SET_VARSIZE #define SET_VARSIZE(x, len) VARATT_SIZEP(x) = len #endif /* txid will be signed int8 in database, so must limit to 63 bits */ #define MAX_TXID UINT64CONST(0x7FFFFFFFFFFFFFFF) /* * If defined, use bsearch() function for searching * txid's inside snapshots that have more than given values. */ #define USE_BSEARCH_FOR 100 /* * public functions */ PG_FUNCTION_INFO_V1(txid_current); PG_FUNCTION_INFO_V1(txid_snapshot_in); PG_FUNCTION_INFO_V1(txid_snapshot_out); PG_FUNCTION_INFO_V1(txid_snapshot_recv); PG_FUNCTION_INFO_V1(txid_snapshot_send); PG_FUNCTION_INFO_V1(txid_current_snapshot); PG_FUNCTION_INFO_V1(txid_snapshot_xmin); PG_FUNCTION_INFO_V1(txid_snapshot_xmax); /* new API in 8.3 */ PG_FUNCTION_INFO_V1(txid_visible_in_snapshot); PG_FUNCTION_INFO_V1(txid_snapshot_xip); /* old API */ PG_FUNCTION_INFO_V1(txid_in_snapshot); PG_FUNCTION_INFO_V1(txid_not_in_snapshot); PG_FUNCTION_INFO_V1(txid_snapshot_active); /* * utility functions */ static int _cmp_txid(const void *aa, const void *bb) { const uint64 *a = aa; const uint64 *b = bb; if (*a < *b) return -1; if (*a > *b) return 1; return 0; } static void sort_snapshot(TxidSnapshot *snap) { qsort(snap->xip, snap->nxip, sizeof(txid), _cmp_txid); } static StringInfo buf_init(txid xmin, txid xmax) { TxidSnapshot snap; StringInfo buf; snap.xmin = xmin; snap.xmax = xmax; snap.nxip = 0; buf = makeStringInfo(); appendBinaryStringInfo(buf, (char *)&snap, offsetof(TxidSnapshot, xip)); return buf; } static void buf_add_txid(StringInfo buf, txid xid) { TxidSnapshot *snap = (TxidSnapshot *)buf->data; snap->nxip++; appendBinaryStringInfo(buf, (char *)&xid, sizeof(xid)); } static TxidSnapshot * buf_finalize(StringInfo buf) { TxidSnapshot *snap = (TxidSnapshot *)buf->data; SET_VARSIZE(snap, buf->len); /* buf is not needed anymore */ buf->data = NULL; pfree(buf); return snap; } static TxidSnapshot * parse_snapshot(const char *str) { txid xmin; txid xmax; txid last_val = 0, val; char *endp; StringInfo buf; xmin = (txid) strtoull(str, &endp, 0); if (*endp != ':') goto bad_format; str = endp + 1; xmax = (txid) strtoull(str, &endp, 0); if (*endp != ':') goto bad_format; str = endp + 1; /* it should look sane */ if (xmin >= xmax || xmin == 0 || xmax > MAX_INT64) goto bad_format; /* allocate buffer */ buf = buf_init(xmin, xmax); /* loop over values */ while (*str != '\0') { /* read next value */ val = (txid) strtoull(str, &endp, 0); str = endp; /* require the input to be in order */ if (val < xmin || val <= last_val || val >= xmax) goto bad_format; buf_add_txid(buf, val); last_val = val; if (*str == ',') str++; else if (*str != '\0') goto bad_format; } return buf_finalize(buf); bad_format: elog(ERROR, "illegal txid_snapshot input format"); return NULL; } /* * Public functions */ /* * txid_current - Return the current transaction ID as txid */ Datum txid_current(PG_FUNCTION_ARGS) { txid val; TxidEpoch state; txid_load_epoch(&state, 0); val = txid_convert_xid(GetTopTransactionId(), &state); PG_RETURN_INT64(val); } /* * txid_current_snapshot - return current snapshot */ Datum txid_current_snapshot(PG_FUNCTION_ARGS) { TxidSnapshot *snap; unsigned num, i, size; TxidEpoch state; if (SerializableSnapshot == NULL) elog(ERROR, "get_current_snapshot: SerializableSnapshot == NULL"); txid_load_epoch(&state, 1); num = SerializableSnapshot->xcnt; size = offsetof(TxidSnapshot, xip) + sizeof(txid) * num; snap = palloc(size); SET_VARSIZE(snap, size); snap->xmin = txid_convert_xid(SerializableSnapshot->xmin, &state); snap->xmax = txid_convert_xid(SerializableSnapshot->xmax, &state); snap->nxip = num; for (i = 0; i < num; i++) snap->xip[i] = txid_convert_xid(SerializableSnapshot->xip[i], &state); /* we want them guaranteed ascending order */ sort_snapshot(snap); PG_RETURN_POINTER(snap); } /* * txid_snapshot_in - input function for type txid_snapshot */ Datum txid_snapshot_in(PG_FUNCTION_ARGS) { TxidSnapshot *snap; char *str = PG_GETARG_CSTRING(0); snap = parse_snapshot(str); PG_RETURN_POINTER(snap); } /* * txid_snapshot_out - output function for type txid_snapshot */ Datum txid_snapshot_out(PG_FUNCTION_ARGS) { TxidSnapshot *snap; StringInfoData str; int i; snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(0); initStringInfo(&str); appendStringInfo(&str, "%llu:", (unsigned long long)snap->xmin); appendStringInfo(&str, "%llu:", (unsigned long long)snap->xmax); for (i = 0; i < snap->nxip; i++) { appendStringInfo(&str, "%s%llu", ((i > 0) ? "," : ""), (unsigned long long)snap->xip[i]); } PG_FREE_IF_COPY(snap, 0); PG_RETURN_CSTRING(str.data); } /* * txid_snapshot_recv(internal) returns txid_snapshot * * binary input function for type txid_snapshot * * format: int4 nxip, int8 xmin, int8 xmax, int8 xip */ Datum txid_snapshot_recv(PG_FUNCTION_ARGS) { StringInfo buf = (StringInfo) PG_GETARG_POINTER(0); TxidSnapshot *snap; txid last = 0; int nxip; int i; int avail; int expect; txid xmin, xmax; /* * load nxip and check for nonsense. * * (nxip > avail) check is against int overflows in 'expect'. */ nxip = pq_getmsgint(buf, 4); avail = buf->len - buf->cursor; expect = 8 + 8 + nxip * 8; if (nxip < 0 || nxip > avail || expect > avail) goto bad_format; xmin = pq_getmsgint64(buf); xmax = pq_getmsgint64(buf); if (xmin == 0 || xmax == 0 || xmin > xmax || xmax > MAX_TXID) goto bad_format; snap = palloc(TXID_SNAPSHOT_SIZE(nxip)); snap->xmin = xmin; snap->xmax = xmax; snap->nxip = nxip; SET_VARSIZE(snap, TXID_SNAPSHOT_SIZE(nxip)); for (i = 0; i < nxip; i++) { txid cur = pq_getmsgint64(buf); if (cur <= last || cur < xmin || cur >= xmax) goto bad_format; snap->xip[i] = cur; last = cur; } PG_RETURN_POINTER(snap); bad_format: elog(ERROR, "invalid snapshot data"); return (Datum)NULL; } /* * txid_snapshot_send(txid_snapshot) returns bytea * * binary output function for type txid_snapshot * * format: int4 nxip, int8 xmin, int8 xmax, int8 xip */ Datum txid_snapshot_send(PG_FUNCTION_ARGS) { TxidSnapshot *snap = (TxidSnapshot *)PG_GETARG_VARLENA_P(0); StringInfoData buf; uint32 i; pq_begintypsend(&buf); pq_sendint(&buf, snap->nxip, 4); pq_sendint64(&buf, snap->xmin); pq_sendint64(&buf, snap->xmax); for (i = 0; i < snap->nxip; i++) pq_sendint64(&buf, snap->xip[i]); PG_RETURN_BYTEA_P(pq_endtypsend(&buf)); } static int _txid_in_snapshot(txid value, const TxidSnapshot *snap) { if (value < snap->xmin) return true; else if (value >= snap->xmax) return false; #ifdef USE_BSEARCH_FOR else if (snap->nxip >= USE_BSEARCH_FOR) { void *res; res = bsearch(&value, snap->xip, snap->nxip, sizeof(txid), _cmp_txid); return (res) ? false : true; } #endif else { int i; for (i = 0; i < snap->nxip; i++) { if (value == snap->xip[i]) return false; } return true; } } /* * txid_in_snapshot - is txid visible in snapshot ? */ Datum txid_in_snapshot(PG_FUNCTION_ARGS) { txid value = PG_GETARG_INT64(0); TxidSnapshot *snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(1); int res; res = _txid_in_snapshot(value, snap) ? true : false; PG_FREE_IF_COPY(snap, 1); PG_RETURN_BOOL(res); } /* * changed api */ Datum txid_visible_in_snapshot(PG_FUNCTION_ARGS) { txid value = PG_GETARG_INT64(0); TxidSnapshot *snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(1); int res; res = _txid_in_snapshot(value, snap) ? true : false; PG_FREE_IF_COPY(snap, 1); PG_RETURN_BOOL(res); } /* * txid_not_in_snapshot - is txid invisible in snapshot ? */ Datum txid_not_in_snapshot(PG_FUNCTION_ARGS) { txid value = PG_GETARG_INT64(0); TxidSnapshot *snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(1); int res; res = _txid_in_snapshot(value, snap) ? false : true; PG_FREE_IF_COPY(snap, 1); PG_RETURN_BOOL(res); } /* * txid_snapshot_xmin - return snapshot's xmin */ Datum txid_snapshot_xmin(PG_FUNCTION_ARGS) { TxidSnapshot *snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(0); txid res = snap->xmin; PG_FREE_IF_COPY(snap, 0); PG_RETURN_INT64(res); } /* * txid_snapshot_xmin - return snapshot's xmax */ Datum txid_snapshot_xmax(PG_FUNCTION_ARGS) { TxidSnapshot *snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(0); txid res = snap->xmax; PG_FREE_IF_COPY(snap, 0); PG_RETURN_INT64(res); } /* remember state between function calls */ struct snap_state { int pos; TxidSnapshot *snap; }; /* * txid_snapshot_active - returns uncommitted TXID's in snapshot. */ Datum txid_snapshot_xip(PG_FUNCTION_ARGS) { FuncCallContext *fctx; struct snap_state *state; if (SRF_IS_FIRSTCALL()) { TxidSnapshot *snap; int statelen; snap = (TxidSnapshot *) PG_GETARG_VARLENA_P(0); fctx = SRF_FIRSTCALL_INIT(); statelen = sizeof(*state) + VARSIZE(snap); state = MemoryContextAlloc(fctx->multi_call_memory_ctx, statelen); state->pos = 0; state->snap = (TxidSnapshot *)((char *)state + sizeof(*state)); memcpy(state->snap, snap, VARSIZE(snap)); fctx->user_fctx = state; PG_FREE_IF_COPY(snap, 0); } fctx = SRF_PERCALL_SETUP(); state = fctx->user_fctx; if (state->pos < state->snap->nxip) { Datum res = Int64GetDatum(state->snap->xip[state->pos]); state->pos++; SRF_RETURN_NEXT(fctx, res); } else { SRF_RETURN_DONE(fctx); } } /* old api */ Datum txid_snapshot_active(PG_FUNCTION_ARGS) { return txid_snapshot_xip(fcinfo); } skytools-2.1.13/sql/txid/Makefile0000644000175000017500000000221011670174255015703 0ustar markomarko include ../../config.mak PGVER := $(shell $(PG_CONFIG) --version | sed 's/PostgreSQL //') ifeq ($(PGVER),) $(error skytools not configured, cannot continue) else # postgres >= manages epoch itself, so skip epoch tables pg83 = $(shell test $(PGVER) "<" "8.3" && echo "false" || echo "true") pg82 = $(shell test $(PGVER) "<" "8.2" && echo "false" || echo "true") endif ifeq ($(pg83),true) # we have 8.3 with internal txid # create empty file txid.sql: echo > txid.sql EXTRA_CLEAN = txid.sql.in txid.sql else # 8.2 or 8.1 # # pg < 8.3 needs this module # MODULE_big = txid SRCS = txid.c epoch.c OBJS = $(SRCS:.c=.o) REGRESS = txid REGRESS_OPTS = --load-language=plpgsql DATA = uninstall_txid.sql DOCS = README.txid DATA_built = txid.sql EXTRA_CLEAN = txid.sql.in ifeq ($(pg82),true) # 8.2 tracks epoch internally TXID_SQL = txid.std.sql else # 8.1 needs epoch-tracking code TXID_SQL = txid.std.sql txid.schema.sql endif # ! 8.2 endif # ! 8.3 # PGXS build procedure include $(PGXS) # additional deps txid.o: txid.h epoch.o: txid.h txid.sql.in: $(TXID_SQL) cat $(TXID_SQL) > $@ test: install make installcheck || { less regression.diffs; exit 1; } skytools-2.1.13/sql/txid/epoch.c0000644000175000017500000001215311670174255015514 0ustar markomarko/*------------------------------------------------------------------------- * epoch.c * * Detect current epoch. *------------------------------------------------------------------------- */ #include "postgres.h" #include #include "access/transam.h" #include "executor/spi.h" #include "miscadmin.h" #include "catalog/pg_control.h" #include "access/xlog.h" #include "txid.h" /* * do a TransactionId -> txid conversion */ txid txid_convert_xid(TransactionId xid, TxidEpoch *state) { uint64 epoch; /* avoid issues with the the special meaning of 0 */ if (xid == InvalidTransactionId) return MAX_INT64; /* return special xid's as-is */ if (xid < FirstNormalTransactionId) return xid; /* xid can on both sides on wrap-around */ epoch = state->epoch; if (TransactionIdPrecedes(xid, state->last_value)) { if (xid > state->last_value) epoch--; } else if (TransactionIdFollows(xid, state->last_value)) { if (xid < state->last_value) epoch++; } return (epoch << 32) | xid; } #if PG_CONTROL_VERSION >= 820 /* * PostgreSQl 8.2 keeps track of epoch internally. */ void txid_load_epoch(TxidEpoch *state, int try_write) { TransactionId xid; uint32 epoch; GetNextXidAndEpoch(&xid, &epoch); state->epoch = epoch; state->last_value = xid; } #else /* * For older PostgreSQL keep epoch in table. */ /* * this caches the txid_epoch table. * The struct should be updated only together with the table. */ static TxidEpoch epoch_state = { 0, 0 }; /* * load values from txid_epoch table. */ static int load_epoch(void) { HeapTuple row; TupleDesc rdesc; bool isnull = false; Datum tmp; int res; uint64 db_epoch, db_value; res = SPI_connect(); if (res < 0) elog(ERROR, "cannot connect to SPI"); res = SPI_execute("select epoch, last_value from txid.epoch", true, 0); if (res != SPI_OK_SELECT) elog(ERROR, "load_epoch: select failed?"); if (SPI_processed != 1) elog(ERROR, "load_epoch: there must be exactly 1 row"); row = SPI_tuptable->vals[0]; rdesc = SPI_tuptable->tupdesc; tmp = SPI_getbinval(row, rdesc, 1, &isnull); if (isnull) elog(ERROR, "load_epoch: epoch is NULL"); db_epoch = DatumGetInt64(tmp); tmp = SPI_getbinval(row, rdesc, 2, &isnull); if (isnull) elog(ERROR, "load_epoch: last_value is NULL"); db_value = DatumGetInt64(tmp); SPI_finish(); /* * If the db has lesser values, then some updates were lost. * * Should that be special-cased? ATM just use db values. * Thus immidiate update. */ epoch_state.epoch = db_epoch; epoch_state.last_value = db_value; return 1; } /* * updates last_value and epoch, if needed */ static void save_epoch(void) { int res; char qbuf[200]; uint64 new_epoch, new_value; TransactionId xid = GetTopTransactionId(); TransactionId old_value; /* store old state */ MemoryContext oldcontext = CurrentMemoryContext; ResourceOwner oldowner = CurrentResourceOwner; /* * avoid changing internal values. */ new_value = xid; new_epoch = epoch_state.epoch; old_value = (TransactionId)epoch_state.last_value; if (xid < old_value) { if (TransactionIdFollows(xid, old_value)) new_epoch++; else return; } sprintf(qbuf, "update txid.epoch set epoch = %llu, last_value = %llu", (unsigned long long)new_epoch, (unsigned long long)new_value); /* * The update may fail in case of SERIALIZABLE transaction. * Try to catch the error and hide it. */ BeginInternalSubTransaction(NULL); PG_TRY(); { /* do the update */ res = SPI_connect(); if (res < 0) elog(ERROR, "cannot connect to SPI"); res = SPI_execute(qbuf, false, 0); SPI_finish(); ReleaseCurrentSubTransaction(); } PG_CATCH(); { /* we expect rollback to clean up inner SPI call */ RollbackAndReleaseCurrentSubTransaction(); FlushErrorState(); res = -1; /* remember failure */ } PG_END_TRY(); /* restore old state */ MemoryContextSwitchTo(oldcontext); CurrentResourceOwner = oldowner; if (res < 0) return; /* * Seems the update was successful, update internal state too. * * There is a chance that the TX will be rollbacked, but then * another backend will do the update, or this one at next * checkpoint. */ epoch_state.epoch = new_epoch; epoch_state.last_value = new_value; } static void check_epoch(int update_prio) { TransactionId xid = GetTopTransactionId(); TransactionId recheck, tx_next; int ok = 1; /* should not happen, but just in case */ if (xid == InvalidTransactionId) return; /* new backend */ if (epoch_state.last_value == 0) load_epoch(); /* try to avoid concurrent access */ if (update_prio) recheck = 50000 + 100 * (MyProcPid & 0x1FF); else recheck = 300000 + 1000 * (MyProcPid & 0x1FF); /* read table */ tx_next = (TransactionId)epoch_state.last_value + recheck; if (TransactionIdFollows(xid, tx_next)) ok = load_epoch(); /* * check if save is needed. last_value may be updated above. */ tx_next = (TransactionId)epoch_state.last_value + recheck; if (!ok || TransactionIdFollows(xid, tx_next)) save_epoch(); } void txid_load_epoch(TxidEpoch *state, int try_write) { check_epoch(try_write); state->epoch = epoch_state.epoch; state->last_value = epoch_state.last_value; } #endif skytools-2.1.13/sql/txid/txid.schema.sql0000644000175000017500000000210711670174255017200 0ustar markomarko-- ---------- -- txid.sql -- -- SQL script for loading the transaction ID compatible datatype -- -- Copyright (c) 2003-2004, PostgreSQL Global Development Group -- Author: Jan Wieck, Afilias USA INC. -- -- ---------- -- -- now the epoch storage -- CREATE SCHEMA txid; -- remember txid settings -- use bigint so we can do arithmetic with it create table txid.epoch ( epoch bigint, last_value bigint ); -- make sure there exist exactly one row insert into txid.epoch values (0, 1); -- then protect it create function txid.epoch_guard() returns trigger as $$ begin if TG_OP = 'UPDATE' then -- epoch: allow only small increase if NEW.epoch > OLD.epoch and NEW.epoch < (OLD.epoch + 3) then return NEW; end if; -- last_value: allow only increase if NEW.epoch = OLD.epoch and NEW.last_value > OLD.last_value then return NEW; end if; end if; raise exception 'bad operation on txid.epoch'; end; $$ language plpgsql; -- the trigger create trigger epoch_guard_trigger before insert or update or delete on txid.epoch for each row execute procedure txid.epoch_guard(); skytools-2.1.13/sql/txid/uninstall_txid.sql0000644000175000017500000000015611670174255020034 0ustar markomarko DROP DOMAIN txid; DROP TYPE txid_snapshot cascade; DROP SCHEMA txid CASCADE; DROP FUNCTION txid_current(); skytools-2.1.13/sql/txid/README.txid0000644000175000017500000000321111670174255016074 0ustar markomarko txid - 8 byte transaction ID's ============================== Based on xxid module from Slony-I. The goal is to make PostgreSQL internal transaction ID and snapshot data usable externally. They cannot be used directly as the internal 4-byte value wraps around and thus breaks indexing. This module extends the internal value with wraparound cound (epoch). It uses relaxed method for wraparound check. There is a table txid.epoch (epoch, last_value) which is used to check if the xid is in current, next or previous epoch. It requires only occasional read-write access - ca. after 100k - 500k transactions. Also it contains type 'txid_snapshot' and following functions: txid_current() returns int8 Current transaction ID txid_current_snapshot() returns txid_snapshot Current snapshot txid_snapshot_xmin( snap ) returns int8 Smallest TXID in snapshot. TXID's smaller than this are all visible in snapshot. txid_snapshot_xmax( snap ) returns int8 Largest TXID in snapshot. TXID's starting from this one are all invisible in snapshot. txid_snapshot_xip( snap ) setof int8 List of uncommitted TXID's in snapshot, that are invisible in snapshot. Values are between xmin and xmax. txid_visible_in_snapshot(id, snap) returns bool Is TXID visible in snapshot? Problems -------- - it breaks when there are more than 2G tx'es between calls. Fixed in 8.2 - functions that create new txid's should be 'security definers' thus better protecting txid_epoch table. - After loading database from backup you should do: UPDATE txid.epoch SET epoch = epoch + 1, last_value = (get_current_txid() & 4294967295); skytools-2.1.13/sql/txid/txid.std.sql0000644000175000017500000000534711670174255016543 0ustar markomarko-- ---------- -- txid.sql -- -- SQL script for loading the transaction ID compatible datatype -- -- Copyright (c) 2003-2004, PostgreSQL Global Development Group -- Author: Jan Wieck, Afilias USA INC. -- -- ---------- set client_min_messages = 'warning'; CREATE DOMAIN txid AS bigint CHECK (value > 0); -- -- A special transaction snapshot data type for faster visibility checks -- CREATE OR REPLACE FUNCTION txid_snapshot_in(cstring) RETURNS txid_snapshot AS 'MODULE_PATHNAME' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION txid_snapshot_out(txid_snapshot) RETURNS cstring AS 'MODULE_PATHNAME' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION txid_snapshot_recv(internal) RETURNS txid_snapshot AS 'MODULE_PATHNAME' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION txid_snapshot_send(txid_snapshot) RETURNS bytea AS 'MODULE_PATHNAME' LANGUAGE C IMMUTABLE STRICT; -- -- The data type itself -- CREATE TYPE txid_snapshot ( INPUT = txid_snapshot_in, OUTPUT = txid_snapshot_out, RECEIVE = txid_snapshot_recv, SEND = txid_snapshot_send, INTERNALLENGTH = variable, STORAGE = extended, ALIGNMENT = double ); --CREATE OR REPLACE FUNCTION get_current_txid() CREATE OR REPLACE FUNCTION txid_current() RETURNS bigint AS 'MODULE_PATHNAME', 'txid_current' LANGUAGE C STABLE SECURITY DEFINER; -- CREATE OR REPLACE FUNCTION get_current_snapshot() CREATE OR REPLACE FUNCTION txid_current_snapshot() RETURNS txid_snapshot AS 'MODULE_PATHNAME', 'txid_current_snapshot' LANGUAGE C STABLE SECURITY DEFINER; --CREATE OR REPLACE FUNCTION get_snapshot_xmin(txid_snapshot) CREATE OR REPLACE FUNCTION txid_snapshot_xmin(txid_snapshot) RETURNS bigint AS 'MODULE_PATHNAME', 'txid_snapshot_xmin' LANGUAGE C IMMUTABLE STRICT; -- CREATE OR REPLACE FUNCTION get_snapshot_xmax(txid_snapshot) CREATE OR REPLACE FUNCTION txid_snapshot_xmax(txid_snapshot) RETURNS bigint AS 'MODULE_PATHNAME', 'txid_snapshot_xmax' LANGUAGE C IMMUTABLE STRICT; -- CREATE OR REPLACE FUNCTION get_snapshot_active(txid_snapshot) CREATE OR REPLACE FUNCTION txid_snapshot_xip(txid_snapshot) RETURNS setof bigint AS 'MODULE_PATHNAME', 'txid_snapshot_xip' LANGUAGE C IMMUTABLE STRICT; -- -- Special comparision functions used by the remote worker -- for sync chunk selection -- CREATE OR REPLACE FUNCTION txid_visible_in_snapshot(bigint, txid_snapshot) RETURNS boolean AS 'MODULE_PATHNAME', 'txid_visible_in_snapshot' LANGUAGE C IMMUTABLE STRICT; /* CREATE OR REPLACE FUNCTION txid_in_snapshot(bigint, txid_snapshot) RETURNS boolean AS 'MODULE_PATHNAME', 'txid_in_snapshot' LANGUAGE C IMMUTABLE STRICT; CREATE OR REPLACE FUNCTION txid_not_in_snapshot(bigint, txid_snapshot) RETURNS boolean AS 'MODULE_PATHNAME', 'txid_not_in_snapshot' LANGUAGE C IMMUTABLE STRICT; */ skytools-2.1.13/sql/txid/sql/0000755000175000017500000000000011727601174015045 5ustar markomarkoskytools-2.1.13/sql/txid/sql/txid.sql0000644000175000017500000000212611670174255016541 0ustar markomarko-- init \set ECHO none \i txid.sql \set ECHO all -- i/o select '12:13:'::txid_snapshot; select '12:13:1,2'::txid_snapshot; -- errors select '31:12:'::txid_snapshot; select '0:1:'::txid_snapshot; select '12:13:0'::txid_snapshot; select '12:13:2,1'::txid_snapshot; create table snapshot_test ( nr integer, snap txid_snapshot ); insert into snapshot_test values (1, '12:13:'); insert into snapshot_test values (2, '12:20:13,15,18'); insert into snapshot_test values (3, '100001:100009:100005,100007,100008'); select snap from snapshot_test order by nr; select txid_snapshot_xmin(snap), txid_snapshot_xmax(snap), txid_snapshot_xip(snap) from snapshot_test order by nr; select id, txid_visible_in_snapshot(id, snap) from snapshot_test, generate_series(11, 21) id where nr = 2; -- test current values also select txid_current() >= txid_snapshot_xmin(txid_current_snapshot()); -- select txid_current_txid() < txid_snapshot_xmax(txid_current_snapshot()); -- select txid_in_snapshot(txid_current_txid(), txid_current_snapshot()), -- txid_not_in_snapshot(txid_current_txid(), txid_current_snapshot()); skytools-2.1.13/sql/londiste/0000755000175000017500000000000011727601174015117 5ustar markomarkoskytools-2.1.13/sql/londiste/functions/0000755000175000017500000000000011727601174017127 5ustar markomarkoskytools-2.1.13/sql/londiste/functions/londiste.provider_get_seq_list.sql0000644000175000017500000000057411670174255026074 0ustar markomarko create or replace function londiste.provider_get_seq_list(i_queue_name text) returns setof text as $$ declare rec record; begin for rec in select seq_name from londiste.provider_seq where queue_name = i_queue_name order by nr loop return next rec.seq_name; end loop; return; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_remove_table.sql0000644000175000017500000000124511670174255026363 0ustar markomarko create or replace function londiste.subscriber_remove_table( i_queue_name text, i_table text) returns integer as $$ declare link text; begin delete from londiste.subscriber_table where queue_name = i_queue_name and table_name = i_table; if not found then raise exception 'no such table'; end if; -- sync link link := londiste.link_dest(i_queue_name); if link is not null then delete from londiste.provider_table where queue_name = link and table_name = i_table; perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.link.sql0000644000175000017500000000640711670174255022436 0ustar markomarko create or replace function londiste.link_source(i_dst_name text) returns text as $$ declare res text; begin select source into res from londiste.link where dest = i_dst_name; return res; end; $$ language plpgsql security definer; create or replace function londiste.link_dest(i_source_name text) returns text as $$ declare res text; begin select dest into res from londiste.link where source = i_source_name; return res; end; $$ language plpgsql security definer; create or replace function londiste.cmp_list(list1 text, a_queue text, a_table text, a_field text) returns boolean as $$ declare sql text; tmp record; list2 text; begin sql := 'select ' || quote_ident(a_field) || ' as name from ' || londiste.quote_fqname(a_table) || ' where queue_name = ' || quote_literal(a_queue) || ' order by 1'; list2 := ''; for tmp in execute sql loop if list2 = '' then list2 := tmp.name; else list2 := list2 || ',' || tmp.name; end if; end loop; return list1 = list2; end; $$ language plpgsql security definer; create or replace function londiste.link(i_source_name text, i_dest_name text, prov_tick_id bigint, prov_tbl_list text, prov_seq_list text) returns text as $$ declare tmp text; list text; tick_seq text; external boolean; last_tick bigint; begin -- check if all matches if not londiste.cmp_list(prov_tbl_list, i_source_name, 'londiste.subscriber_table', 'table_name') then raise exception 'not all tables copied into subscriber'; end if; if not londiste.cmp_list(prov_seq_list, i_source_name, 'londiste.subscriber_seq', 'seq_name') then raise exception 'not all seqs copied into subscriber'; end if; if not londiste.cmp_list(prov_seq_list, i_dest_name, 'londiste.provider_table', 'table_name') then raise exception 'linked provider queue does not have all tables'; end if; if not londiste.cmp_list(prov_seq_list, i_dest_name, 'londiste.provider_seq', 'seq_name') then raise exception 'linked provider queue does not have all seqs'; end if; -- check pgq select queue_external_ticker, queue_tick_seq into external, tick_seq from pgq.queue where queue_name = i_dest_name; if not found then raise exception 'dest queue does not exist'; end if; if external then raise exception 'dest queue has already external_ticker turned on?'; end if; if nextval(tick_seq) >= prov_tick_id then raise exception 'dest queue ticks larger'; end if; update pgq.queue set queue_external_ticker = true where queue_name = i_dest_name; insert into londiste.link (source, dest) values (i_source_name, i_dest_name); return null; end; $$ language plpgsql security definer; create or replace function londiste.link_del(i_source_name text, i_dest_name text) returns text as $$ begin delete from londiste.link where source = i_source_name and dest = i_dest_name; if not found then raise exception 'no suck link'; end if; return null; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_get_table_list.sql0000644000175000017500000000160611670174255026701 0ustar markomarko create or replace function londiste.subscriber_get_table_list(i_queue_name text) returns setof londiste.ret_subscriber_table as $$ declare rec londiste.ret_subscriber_table%rowtype; begin for rec in select table_name, merge_state, snapshot, trigger_name, skip_truncate from londiste.subscriber_table where queue_name = i_queue_name order by nr loop return next rec; end loop; return; end; $$ language plpgsql security definer; -- compat create or replace function londiste.get_table_state(i_queue text) returns setof londiste.subscriber_table as $$ declare rec londiste.subscriber_table%rowtype; begin for rec in select * from londiste.subscriber_table where queue_name = i_queue order by nr loop return next rec; end loop; return; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_trigger_funcs.sql0000644000175000017500000000474211670174255026565 0ustar markomarko create or replace function londiste.subscriber_get_table_pending_triggers(i_table_name text) returns setof londiste.subscriber_pending_triggers as $$ declare trigger record; begin for trigger in select * from londiste.subscriber_pending_triggers where table_name = i_table_name loop return next trigger; end loop; return; end; $$ language plpgsql strict stable; create or replace function londiste.subscriber_drop_table_trigger(i_table_name text, i_trigger_name text) returns integer as $$ declare trig_def record; begin select * into trig_def from londiste.find_table_triggers(i_table_name) where trigger_name = i_trigger_name; if FOUND is not true then return 0; end if; insert into londiste.subscriber_pending_triggers(table_name, trigger_name, trigger_def) values (i_table_name, i_trigger_name, trig_def.trigger_def); execute 'drop trigger ' || quote_ident(i_trigger_name) || ' on ' || londiste.quote_fqname(i_table_name); return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_drop_all_table_triggers(i_table_name text) returns integer as $$ declare trigger record; begin for trigger in select trigger_name as name from londiste.find_table_triggers(i_table_name) loop perform londiste.subscriber_drop_table_trigger(i_table_name, trigger.name); end loop; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_table_trigger(i_table_name text, i_trigger_name text) returns integer as $$ declare trig_def text; begin select trigger_def into trig_def from londiste.subscriber_pending_triggers where (table_name, trigger_name) = (i_table_name, i_trigger_name); if not found then return 0; end if; delete from londiste.subscriber_pending_triggers where table_name = i_table_name and trigger_name = i_trigger_name; execute trig_def; return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_all_table_triggers(i_table_name text) returns integer as $$ declare trigger record; begin for trigger in select trigger_name as name from londiste.subscriber_get_table_pending_triggers(i_table_name) loop perform londiste.subscriber_restore_table_trigger(i_table_name, trigger.name); end loop; return 1; end; $$ language plpgsql; skytools-2.1.13/sql/londiste/functions/londiste.quote_fqname.sql0000644000175000017500000000073511670174255024163 0ustar markomarko create or replace function londiste.quote_fqname(i_name text) returns text as $$ declare res text; pos integer; s text; n text; begin pos := position('.' in i_name); if pos > 0 then s := substring(i_name for pos - 1); n := substring(i_name from pos + 1); else s := 'public'; n := i_name; end if; return quote_ident(s) || '.' || quote_ident(n); end; $$ language plpgsql strict immutable; skytools-2.1.13/sql/londiste/functions/londiste.provider_remove_seq.sql0000644000175000017500000000117311670174255025553 0ustar markomarko create or replace function londiste.provider_remove_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin -- check if linked queue link := londiste.link_source(i_queue_name); if link is not null then raise exception 'Linked queue, cannot modify'; end if; delete from londiste.provider_seq where queue_name = i_queue_name and seq_name = i_seq_name; if not found then raise exception 'seq not attached'; end if; perform londiste.provider_notify_change(i_queue_name); return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.find_table_oid.sql0000644000175000017500000000231111670174255024411 0ustar markomarkocreate or replace function londiste.find_rel_oid(tbl text, kind text) returns oid as $$ declare res oid; pos integer; schema text; name text; begin pos := position('.' in tbl); if pos > 0 then schema := substring(tbl for pos - 1); name := substring(tbl from pos + 1); else schema := 'public'; name := tbl; end if; select c.oid into res from pg_namespace n, pg_class c where c.relnamespace = n.oid and c.relkind = kind and n.nspname = schema and c.relname = name; if not found then if kind = 'r' then raise exception 'table not found'; elsif kind = 'S' then raise exception 'seq not found'; else raise exception 'weird relkind'; end if; end if; return res; end; $$ language plpgsql strict stable; create or replace function londiste.find_table_oid(tbl text) returns oid as $$ begin return londiste.find_rel_oid(tbl, 'r'); end; $$ language plpgsql strict stable; create or replace function londiste.find_seq_oid(tbl text) returns oid as $$ begin return londiste.find_rel_oid(tbl, 'S'); end; $$ language plpgsql strict stable; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_fkeys_funcs.sql0000644000175000017500000000477111670174255026245 0ustar markomarko create or replace function londiste.subscriber_get_table_pending_fkeys(i_table_name text) returns setof londiste.subscriber_pending_fkeys as $$ declare fkeys record; begin for fkeys in select * from londiste.subscriber_pending_fkeys where from_table=i_table_name or to_table=i_table_name order by 1,2,3 loop return next fkeys; end loop; return; end; $$ language plpgsql; create or replace function londiste.subscriber_get_queue_valid_pending_fkeys(i_queue_name text) returns setof londiste.subscriber_pending_fkeys as $$ declare fkeys record; begin for fkeys in select pf.* from londiste.subscriber_pending_fkeys pf join londiste.subscriber_table st_from on (st_from.table_name = pf.from_table and st_from.merge_state = 'ok' and st_from.snapshot is null) join londiste.subscriber_table st_to on (st_to.table_name = pf.to_table and st_to.merge_state = 'ok' and st_to.snapshot is null) -- change the AND to OR to allow fkeys between tables coming from different queues where (st_from.queue_name = i_queue_name and st_to.queue_name = i_queue_name) order by 1, 2, 3 loop return next fkeys; end loop; return; end; $$ language plpgsql; create or replace function londiste.subscriber_drop_table_fkey(i_from_table text, i_fkey_name text) returns integer as $$ declare fkey record; begin select * into fkey from londiste.find_table_fkeys(i_from_table) where fkey_name = i_fkey_name and from_table = i_from_table; if not found then return 0; end if; insert into londiste.subscriber_pending_fkeys values (fkey.from_table, fkey.to_table, i_fkey_name, fkey.fkey_def); execute 'alter table only ' || londiste.quote_fqname(fkey.from_table) || ' drop constraint ' || quote_ident(i_fkey_name); return 1; end; $$ language plpgsql; create or replace function londiste.subscriber_restore_table_fkey(i_from_table text, i_fkey_name text) returns integer as $$ declare fkey record; begin select * into fkey from londiste.subscriber_pending_fkeys where fkey_name = i_fkey_name and from_table = i_from_table; if not found then return 0; end if; delete from londiste.subscriber_pending_fkeys where fkey_name = fkey.fkey_name; execute fkey.fkey_def; return 1; end; $$ language plpgsql; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_set_skip_truncate.sql0000644000175000017500000000064611670174255027451 0ustar markomarko create or replace function londiste.subscriber_set_skip_truncate( i_queue text, i_table text, i_value bool) returns integer as $$ begin update londiste.subscriber_table set skip_truncate = i_value where queue_name = i_queue and table_name = i_table; if not found then raise exception 'table not found'; end if; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_add_table.sql0000644000175000017500000000053411670174255025616 0ustar markomarko create or replace function londiste.subscriber_add_table( i_queue_name text, i_table text) returns integer as $$ begin insert into londiste.subscriber_table (queue_name, table_name) values (i_queue_name, i_table); -- linked queue is updated, when the table is copied return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_get_seq_list.sql0000644000175000017500000000060011670174255026373 0ustar markomarko create or replace function londiste.subscriber_get_seq_list(i_queue_name text) returns setof text as $$ declare rec record; begin for rec in select seq_name from londiste.subscriber_seq where queue_name = i_queue_name order by nr loop return next rec.seq_name; end loop; return; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_add_seq.sql0000644000175000017500000000113511670174255025315 0ustar markomarko create or replace function londiste.subscriber_add_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin insert into londiste.subscriber_seq (queue_name, seq_name) values (i_queue_name, i_seq_name); -- update linked queue if needed link := londiste.link_dest(i_queue_name); if link is not null then insert into londiste.provider_seq (queue_name, seq_name) values (link, i_seq_name); perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.provider_add_table.sql0000644000175000017500000000261111670174255025303 0ustar markomarkocreate or replace function londiste.provider_add_table( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; sql text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; if position('k' in i_col_types) < 1 then raise exception 'need key column'; end if; if position('.' in i_table_name) < 1 then raise exception 'need fully-qualified table name'; end if; select queue_name into tgname from pgq.queue where queue_name = i_queue_name; if not found then raise exception 'no such event queue'; end if; tgname := i_queue_name || '_logger'; tgname := replace(lower(tgname), '.', '_'); insert into londiste.provider_table (queue_name, table_name, trigger_name) values (i_queue_name, i_table_name, tgname); perform londiste.provider_create_trigger( i_queue_name, i_table_name, i_col_types); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_add_table( i_queue_name text, i_table_name text ) returns integer as $$ begin return londiste.provider_add_table(i_queue_name, i_table_name, londiste.find_column_types(i_table_name)); end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.provider_remove_table.sql0000644000175000017500000000206311670174255026051 0ustar markomarko create or replace function londiste.provider_remove_table( i_queue_name text, i_table_name text ) returns integer as $$ declare tgname text; begin if londiste.link_source(i_queue_name) is not null then raise exception 'Linked queue, manipulation not allowed'; end if; select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'no such table registered'; end if; begin execute 'drop trigger ' || quote_ident(tgname) || ' on ' || londiste.quote_fqname(i_table_name); exception when undefined_table then raise notice 'table % does not exist', i_table_name; when undefined_object then raise notice 'trigger % does not exist on table %', tgname, i_table_name; end; delete from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.find_table_fkeys.sql0000644000175000017500000000205611670174255024765 0ustar markomarko create or replace function londiste.find_table_fkeys(i_table_name text) returns setof londiste.subscriber_pending_fkeys as $$ declare fkey record; tbl_oid oid; begin select londiste.find_table_oid(i_table_name) into tbl_oid; for fkey in select n1.nspname || '.' || t1.relname as from_table, n2.nspname || '.' || t2.relname as to_table, conname::text as fkey_name, 'alter table only ' || quote_ident(n1.nspname) || '.' || quote_ident(t1.relname) || ' add constraint ' || quote_ident(conname::text) || ' ' || pg_get_constraintdef(c.oid) as fkey_def from pg_constraint c, pg_namespace n1, pg_class t1, pg_namespace n2, pg_class t2 where c.contype = 'f' and (c.conrelid = tbl_oid or c.confrelid = tbl_oid) and t1.oid = c.conrelid and n1.oid = t1.relnamespace and t2.oid = c.confrelid and n2.oid = t2.relnamespace order by 1,2,3 loop return next fkey; end loop; return; end; $$ language plpgsql strict stable; skytools-2.1.13/sql/londiste/functions/londiste.provider_add_seq.sql0000644000175000017500000000120211670174255024777 0ustar markomarko create or replace function londiste.provider_add_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin -- check if linked queue link := londiste.link_source(i_queue_name); if link is not null then raise exception 'Linked queue, cannot modify'; end if; perform 1 from pg_class where oid = londiste.find_seq_oid(i_seq_name); if not found then raise exception 'seq not found'; end if; insert into londiste.provider_seq (queue_name, seq_name) values (i_queue_name, i_seq_name); return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.provider_create_trigger.sql0000644000175000017500000000151211670174255026371 0ustar markomarko create or replace function londiste.provider_create_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare tgname text; begin select trigger_name into tgname from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; execute 'create trigger ' || quote_ident(tgname) || ' after insert or update or delete on ' || londiste.quote_fqname(i_table_name) || ' for each row execute procedure pgq.logtriga(' || quote_literal(i_queue_name) || ', ' || quote_literal(i_col_types) || ', ' || quote_literal(i_table_name) || ')'; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.find_column_types.sql0000644000175000017500000000150611670174255025215 0ustar markomarkocreate or replace function londiste.find_column_types(tbl text) returns text as $$ declare res text; col record; tbl_oid oid; begin tbl_oid := londiste.find_table_oid(tbl); res := ''; for col in SELECT CASE WHEN k.attname IS NOT NULL THEN 'k' ELSE 'v' END AS type FROM pg_attribute a LEFT JOIN ( SELECT k.attname FROM pg_index i, pg_attribute k WHERE i.indrelid = tbl_oid AND k.attrelid = i.indexrelid AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped ) k ON (k.attname = a.attname) WHERE a.attrelid = tbl_oid AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum loop res := res || col.type; end loop; return res; end; $$ language plpgsql strict stable; skytools-2.1.13/sql/londiste/functions/londiste.provider_get_table_list.sql0000644000175000017500000000072011670174255026364 0ustar markomarko create or replace function londiste.provider_get_table_list(i_queue text) returns setof londiste.ret_provider_table_list as $$ declare rec londiste.ret_provider_table_list%rowtype; begin for rec in select table_name, trigger_name from londiste.provider_table where queue_name = i_queue order by nr loop return next rec; end loop; return; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.set_last_tick.sql0000644000175000017500000000111511670174255024320 0ustar markomarko create or replace function londiste.set_last_tick( i_consumer text, i_tick_id bigint) returns integer as $$ begin if i_tick_id is null then delete from londiste.completed where consumer_id = i_consumer; else update londiste.completed set last_tick_id = i_tick_id where consumer_id = i_consumer; if not found then insert into londiste.completed (consumer_id, last_tick_id) values (i_consumer, i_tick_id); end if; end if; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.provider_refresh_trigger.sql0000644000175000017500000000232311670174255026565 0ustar markomarko create or replace function londiste.provider_refresh_trigger( i_queue_name text, i_table_name text, i_col_types text ) returns integer strict as $$ declare t_name text; tbl_oid oid; begin select trigger_name into t_name from londiste.provider_table where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'table not found'; end if; tbl_oid := londiste.find_table_oid(i_table_name); perform 1 from pg_trigger where tgrelid = tbl_oid and tgname = t_name; if found then execute 'drop trigger ' || quote_ident(t_name) || ' on ' || londiste.quote_fqname(i_table_name); end if; perform londiste.provider_create_trigger(i_queue_name, i_table_name, i_col_types); return 1; end; $$ language plpgsql security definer; create or replace function londiste.provider_refresh_trigger( i_queue_name text, i_table_name text ) returns integer strict as $$ begin return londiste.provider_refresh_trigger(i_queue_name, i_table_name, londiste.find_column_types(i_table_name)); end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_set_table_state.sql0000644000175000017500000000361011670174255027057 0ustar markomarko create or replace function londiste.subscriber_set_table_state( i_queue_name text, i_table_name text, i_snapshot text, i_merge_state text) returns integer as $$ declare link text; ok integer; begin update londiste.subscriber_table set snapshot = i_snapshot, merge_state = i_merge_state, -- reset skip_snapshot when table is copied over skip_truncate = case when i_merge_state = 'ok' then null else skip_truncate end where queue_name = i_queue_name and table_name = i_table_name; if not found then raise exception 'no such table'; end if; -- sync link state also link := londiste.link_dest(i_queue_name); if link then select * from londiste.provider_table where queue_name = linkdst and table_name = i_table_name; if found then if i_merge_state is null or i_merge_state <> 'ok' then delete from londiste.provider_table where queue_name = link and table_name = i_table_name; perform londiste.notify_change(link); end if; else if i_merge_state = 'ok' then insert into londiste.provider_table (queue_name, table_name) values (link, i_table_name); perform londiste.notify_change(link); end if; end if; end if; return 1; end; $$ language plpgsql security definer; create or replace function londiste.set_table_state( i_queue_name text, i_table_name text, i_snapshot text, i_merge_state text) returns integer as $$ begin return londiste.subscriber_set_table_state(i_queue_name, i_table_name, i_snapshot, i_merge_state); end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.version.sql0000644000175000017500000000017011670174255023155 0ustar markomarko create or replace function londiste.version() returns text as $$ begin return '2.1.12'; end; $$ language plpgsql; skytools-2.1.13/sql/londiste/functions/londiste.provider_notify_change.sql0000644000175000017500000000111311670174255026215 0ustar markomarko create or replace function londiste.provider_notify_change(i_queue_name text) returns integer as $$ declare res text; tbl record; begin res := ''; for tbl in select table_name from londiste.provider_table where queue_name = i_queue_name order by nr loop if res = '' then res := tbl.table_name; else res := res || ',' || tbl.table_name; end if; end loop; perform pgq.insert_event(i_queue_name, 'T', res); return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/functions/londiste.find_table_triggers.sql0000644000175000017500000000233211670174255025467 0ustar markomarko create or replace function londiste.find_table_triggers(i_table_name text) returns setof londiste.subscriber_pending_triggers as $$ declare tg record; ver int4; begin select setting::int4 into ver from pg_settings where name = 'server_version_num'; if ver >= 90000 then for tg in select n.nspname || '.' || c.relname as table_name, t.tgname::text as name, pg_get_triggerdef(t.oid) as def from pg_trigger t, pg_class c, pg_namespace n where n.oid = c.relnamespace and c.oid = t.tgrelid and t.tgrelid = londiste.find_table_oid(i_table_name) and not t.tgisinternal loop return next tg; end loop; else for tg in select n.nspname || '.' || c.relname as table_name, t.tgname::text as name, pg_get_triggerdef(t.oid) as def from pg_trigger t, pg_class c, pg_namespace n where n.oid = c.relnamespace and c.oid = t.tgrelid and t.tgrelid = londiste.find_table_oid(i_table_name) and not t.tgisconstraint loop return next tg; end loop; end if; return; end; $$ language plpgsql strict stable; skytools-2.1.13/sql/londiste/functions/londiste.get_last_tick.sql0000644000175000017500000000044711670174255024313 0ustar markomarko create or replace function londiste.get_last_tick(i_consumer text) returns bigint as $$ declare res bigint; begin select last_tick_id into res from londiste.completed where consumer_id = i_consumer; return res; end; $$ language plpgsql security definer strict stable; skytools-2.1.13/sql/londiste/functions/londiste.subscriber_remove_seq.sql0000644000175000017500000000126611670174255026067 0ustar markomarko create or replace function londiste.subscriber_remove_seq( i_queue_name text, i_seq_name text) returns integer as $$ declare link text; begin delete from londiste.subscriber_seq where queue_name = i_queue_name and seq_name = i_seq_name; if not found then raise exception 'no such seq?'; end if; -- update linked queue if needed link := londiste.link_dest(i_queue_name); if link is not null then delete from londiste.provider_seq where queue_name = link and seq_name = i_seq_name; perform londiste.provider_notify_change(link); end if; return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/londiste/expected/0000755000175000017500000000000011727601174016720 5ustar markomarkoskytools-2.1.13/sql/londiste/expected/londiste_provider.out0000644000175000017500000000643611670174255023217 0ustar markomarkoset client_min_messages = 'warning'; \set VERBOSITY 'terse' -- -- tables -- create table testdata ( id serial primary key, data text ); create table testdata_nopk ( id serial, data text ); select londiste.provider_add_table('pqueue', 'public.testdata_nopk'); ERROR: need key column select londiste.provider_add_table('pqueue', 'public.testdata'); ERROR: no such event queue select pgq.create_queue('pqueue'); create_queue -------------- 1 (1 row) select londiste.provider_add_table('pqueue', 'public.testdata'); provider_add_table -------------------- 1 (1 row) select londiste.provider_add_table('pqueue', 'public.testdata'); ERROR: duplicate key value violates unique constraint "provider_table_pkey" select londiste.provider_refresh_trigger('pqueue', 'public.testdata'); provider_refresh_trigger -------------------------- 1 (1 row) select * from londiste.provider_get_table_list('pqueue'); table_name | trigger_name -----------------+--------------- public.testdata | pqueue_logger (1 row) select londiste.provider_remove_table('pqueue', 'public.nonexist'); ERROR: no such table registered select londiste.provider_remove_table('pqueue', 'public.testdata'); provider_remove_table ----------------------- 1 (1 row) select * from londiste.provider_get_table_list('pqueue'); table_name | trigger_name ------------+-------------- (0 rows) -- -- seqs -- select * from londiste.provider_get_seq_list('pqueue'); provider_get_seq_list ----------------------- (0 rows) select londiste.provider_add_seq('pqueue', 'public.no_seq'); ERROR: seq not found select londiste.provider_add_seq('pqueue', 'public.testdata_id_seq'); provider_add_seq ------------------ 0 (1 row) select londiste.provider_add_seq('pqueue', 'public.testdata_id_seq'); ERROR: duplicate key value violates unique constraint "provider_seq_pkey" select * from londiste.provider_get_seq_list('pqueue'); provider_get_seq_list ------------------------ public.testdata_id_seq (1 row) select londiste.provider_remove_seq('pqueue', 'public.testdata_id_seq'); provider_remove_seq --------------------- 0 (1 row) select londiste.provider_remove_seq('pqueue', 'public.testdata_id_seq'); ERROR: seq not attached select * from londiste.provider_get_seq_list('pqueue'); provider_get_seq_list ----------------------- (0 rows) -- -- linked queue -- select londiste.provider_add_table('pqueue', 'public.testdata'); provider_add_table -------------------- 1 (1 row) insert into londiste.link (source, dest) values ('mqueue', 'pqueue'); select londiste.provider_add_table('pqueue', 'public.testdata'); ERROR: Linked queue, manipulation not allowed select londiste.provider_remove_table('pqueue', 'public.testdata'); ERROR: Linked queue, manipulation not allowed select londiste.provider_add_seq('pqueue', 'public.testdata_id_seq'); ERROR: Linked queue, cannot modify select londiste.provider_remove_seq('pqueue', 'public.testdata_seq'); ERROR: Linked queue, cannot modify -- -- cleanup -- delete from londiste.link; drop table testdata; drop table testdata_nopk; delete from londiste.provider_seq; delete from londiste.provider_table; select pgq.drop_queue('pqueue'); drop_queue ------------ 1 (1 row) skytools-2.1.13/sql/londiste/expected/londiste_fkeys.out0000644000175000017500000002435511670174255022506 0ustar markomarkoset log_error_verbosity = 'terse'; create table ref_1 ( id int4 primary key, val text ); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "ref_1_pkey" for table "ref_1" create table ref_2 ( id int4 primary key, ref int4 not null references ref_1, val text ); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "ref_2_pkey" for table "ref_2" create table ref_3 ( id int4 primary key, ref2 int4 not null references ref_2, val text ); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "ref_3_pkey" for table "ref_3" select * from londiste.subscriber_add_table('refqueue', 'public.ref_1'); subscriber_add_table ---------------------- 0 (1 row) select * from londiste.subscriber_add_table('refqueue', 'public.ref_2'); subscriber_add_table ---------------------- 0 (1 row) select * from londiste.subscriber_add_table('refqueue', 'public.ref_3'); subscriber_add_table ---------------------- 0 (1 row) select * from londiste.find_table_fkeys('public.ref_1'); from_table | to_table | fkey_name | fkey_def --------------+--------------+----------------+---------------------------------------------------------------------------------------------------- public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) (1 row) select * from londiste.find_table_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (2 rows) select * from londiste.find_table_fkeys('public.ref_3'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (1 row) select * from londiste.subscriber_get_table_pending_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) -- drop fkeys select * from londiste.subscriber_drop_table_fkey('public.ref_2', 'ref_2_ref_fkey'); subscriber_drop_table_fkey ---------------------------- 1 (1 row) select * from londiste.find_table_fkeys('public.ref_1'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.find_table_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (1 row) select * from londiste.find_table_fkeys('public.ref_3'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (1 row) select * from londiste.subscriber_drop_table_fkey('public.ref_3', 'ref_3_ref2_fkey'); subscriber_drop_table_fkey ---------------------------- 1 (1 row) -- check if dropped select * from londiste.find_table_fkeys('public.ref_1'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.find_table_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.find_table_fkeys('public.ref_3'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) -- look state select * from londiste.subscriber_get_table_pending_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (2 rows) select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) -- toggle sync select * from londiste.subscriber_set_table_state('refqueue', 'public.ref_1', null, 'ok'); subscriber_set_table_state ---------------------------- 1 (1 row) select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.subscriber_set_table_state('refqueue', 'public.ref_2', null, 'ok'); subscriber_set_table_state ---------------------------- 1 (1 row) select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); from_table | to_table | fkey_name | fkey_def --------------+--------------+----------------+---------------------------------------------------------------------------------------------------- public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) (1 row) select * from londiste.subscriber_set_table_state('refqueue', 'public.ref_3', null, 'ok'); subscriber_set_table_state ---------------------------- 1 (1 row) select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (2 rows) -- restore select * from londiste.subscriber_restore_table_fkey('public.ref_2', 'ref_2_ref_fkey'); subscriber_restore_table_fkey ------------------------------- 1 (1 row) select * from londiste.subscriber_restore_table_fkey('public.ref_3', 'ref_3_ref2_fkey'); subscriber_restore_table_fkey ------------------------------- 1 (1 row) -- look state select * from londiste.subscriber_get_table_pending_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); from_table | to_table | fkey_name | fkey_def ------------+----------+-----------+---------- (0 rows) select * from londiste.find_table_fkeys('public.ref_1'); from_table | to_table | fkey_name | fkey_def --------------+--------------+----------------+---------------------------------------------------------------------------------------------------- public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) (1 row) select * from londiste.find_table_fkeys('public.ref_2'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_2 | public.ref_1 | ref_2_ref_fkey | alter table only public.ref_2 add constraint ref_2_ref_fkey FOREIGN KEY (ref) REFERENCES ref_1(id) public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (2 rows) select * from londiste.find_table_fkeys('public.ref_3'); from_table | to_table | fkey_name | fkey_def --------------+--------------+-----------------+------------------------------------------------------------------------------------------------------ public.ref_3 | public.ref_2 | ref_3_ref2_fkey | alter table only public.ref_3 add constraint ref_3_ref2_fkey FOREIGN KEY (ref2) REFERENCES ref_2(id) (1 row) skytools-2.1.13/sql/londiste/expected/londiste_denytrigger.out0000644000175000017500000000201111670174255023671 0ustar markomarkocreate table denytest ( val integer); insert into denytest values (1); create trigger xdeny after insert or update or delete on denytest for each row execute procedure londiste.deny_trigger(); insert into denytest values (2); ERROR: ('Changes no allowed on this table',) update denytest set val = 2; ERROR: ('Changes no allowed on this table',) delete from denytest; ERROR: ('Changes no allowed on this table',) select londiste.disable_deny_trigger(true); disable_deny_trigger ---------------------- t (1 row) update denytest set val = 2; select londiste.disable_deny_trigger(true); disable_deny_trigger ---------------------- t (1 row) update denytest set val = 2; select londiste.disable_deny_trigger(false); disable_deny_trigger ---------------------- f (1 row) update denytest set val = 2; ERROR: ('Changes no allowed on this table',) select londiste.disable_deny_trigger(false); disable_deny_trigger ---------------------- f (1 row) update denytest set val = 2; ERROR: ('Changes no allowed on this table',) skytools-2.1.13/sql/londiste/expected/londiste_install.out0000644000175000017500000000001611670174255023017 0ustar markomarko\set ECHO off skytools-2.1.13/sql/londiste/expected/londiste_subscriber.out0000644000175000017500000001764711670174255023536 0ustar markomarkoset client_min_messages = 'warning'; \set VERBOSITY 'terse' create table testdata ( id serial primary key, data text ); -- -- tables -- select londiste.subscriber_add_table('pqueue', 'public.testdata_nopk'); subscriber_add_table ---------------------- 0 (1 row) select londiste.subscriber_add_table('pqueue', 'public.testdata'); subscriber_add_table ---------------------- 0 (1 row) select pgq.create_queue('pqueue'); create_queue -------------- 1 (1 row) select londiste.subscriber_add_table('pqueue', 'public.testdata'); ERROR: duplicate key value violates unique constraint "subscriber_table_pkey" select londiste.subscriber_add_table('pqueue', 'public.testdata'); ERROR: duplicate key value violates unique constraint "subscriber_table_pkey" select * from londiste.subscriber_get_table_list('pqueue'); table_name | merge_state | snapshot | trigger_name | skip_truncate ----------------------+-------------+----------+--------------+--------------- public.testdata_nopk | | | | public.testdata | | | | (2 rows) select londiste.subscriber_remove_table('pqueue', 'public.nonexist'); ERROR: no such table select londiste.subscriber_remove_table('pqueue', 'public.testdata'); subscriber_remove_table ------------------------- 0 (1 row) select * from londiste.subscriber_get_table_list('pqueue'); table_name | merge_state | snapshot | trigger_name | skip_truncate ----------------------+-------------+----------+--------------+--------------- public.testdata_nopk | | | | (1 row) -- -- seqs -- select * from londiste.subscriber_get_seq_list('pqueue'); subscriber_get_seq_list ------------------------- (0 rows) select londiste.subscriber_add_seq('pqueue', 'public.no_seq'); subscriber_add_seq -------------------- 0 (1 row) select londiste.subscriber_add_seq('pqueue', 'public.testdata_id_seq'); subscriber_add_seq -------------------- 0 (1 row) select londiste.subscriber_add_seq('pqueue', 'public.testdata_id_seq'); ERROR: duplicate key value violates unique constraint "subscriber_seq_pkey" select * from londiste.subscriber_get_seq_list('pqueue'); subscriber_get_seq_list ------------------------- public.no_seq public.testdata_id_seq (2 rows) select londiste.subscriber_remove_seq('pqueue', 'public.testdata_id_seq'); subscriber_remove_seq ----------------------- 0 (1 row) select londiste.subscriber_remove_seq('pqueue', 'public.testdata_id_seq'); ERROR: no such seq? select * from londiste.subscriber_get_seq_list('pqueue'); subscriber_get_seq_list ------------------------- public.no_seq (1 row) -- -- linked queue -- select londiste.subscriber_add_table('pqueue', 'public.testdata'); subscriber_add_table ---------------------- 0 (1 row) insert into londiste.link (source, dest) values ('mqueue', 'pqueue'); select londiste.subscriber_add_table('pqueue', 'public.testdata'); ERROR: duplicate key value violates unique constraint "subscriber_table_pkey" select londiste.subscriber_remove_table('pqueue', 'public.testdata'); subscriber_remove_table ------------------------- 0 (1 row) select londiste.subscriber_add_seq('pqueue', 'public.testdata_id_seq'); subscriber_add_seq -------------------- 0 (1 row) select londiste.subscriber_remove_seq('pqueue', 'public.testdata_seq'); ERROR: no such seq? -- -- skip-truncate, set_table_state -- select londiste.subscriber_add_table('pqueue', 'public.skiptest'); subscriber_add_table ---------------------- 0 (1 row) select skip_truncate from londiste.subscriber_table where table_name = 'public.skiptest'; skip_truncate --------------- (1 row) select londiste.subscriber_set_skip_truncate('pqueue', 'public.skiptest', true); subscriber_set_skip_truncate ------------------------------ 1 (1 row) select skip_truncate from londiste.subscriber_table where table_name = 'public.skiptest'; skip_truncate --------------- t (1 row) select londiste.subscriber_set_table_state('pqueue', 'public.skiptest', 'snap1', 'in-copy'); subscriber_set_table_state ---------------------------- 1 (1 row) select skip_truncate, snapshot from londiste.subscriber_table where table_name = 'public.skiptest'; skip_truncate | snapshot ---------------+---------- t | snap1 (1 row) select londiste.subscriber_set_table_state('pqueue', 'public.skiptest', null, 'ok'); subscriber_set_table_state ---------------------------- 1 (1 row) select skip_truncate, snapshot from londiste.subscriber_table where table_name = 'public.skiptest'; skip_truncate | snapshot ---------------+---------- | (1 row) -- -- test tick tracking -- select londiste.get_last_tick('c'); get_last_tick --------------- (1 row) select londiste.set_last_tick('c', 1); set_last_tick --------------- 1 (1 row) select londiste.get_last_tick('c'); get_last_tick --------------- 1 (1 row) select londiste.set_last_tick('c', 2); set_last_tick --------------- 1 (1 row) select londiste.get_last_tick('c'); get_last_tick --------------- 2 (1 row) select londiste.set_last_tick('c', NULL); set_last_tick --------------- 1 (1 row) select londiste.get_last_tick('c'); get_last_tick --------------- (1 row) -- test triggers create table tgfk ( id int4 primary key, data text ); create table tgtest ( id int4 primary key, fk int4 references tgfk, data text ); create or replace function notg() returns trigger as $$ begin return null; end; $$ language plpgsql; create trigger tg_nop after insert on tgtest for each row execute procedure notg(); select * from londiste.find_table_triggers('tgtest'); table_name | trigger_name | trigger_def ---------------+--------------+------------------------------------------------------------------------------------ public.tgtest | tg_nop | CREATE TRIGGER tg_nop AFTER INSERT ON tgtest FOR EACH ROW EXECUTE PROCEDURE notg() (1 row) select * from londiste.subscriber_get_table_pending_triggers('tgtest'); table_name | trigger_name | trigger_def ------------+--------------+------------- (0 rows) select * from londiste.subscriber_drop_all_table_triggers('tgtest'); subscriber_drop_all_table_triggers ------------------------------------ 1 (1 row) select * from londiste.find_table_triggers('tgtest'); table_name | trigger_name | trigger_def ------------+--------------+------------- (0 rows) select * from londiste.subscriber_get_table_pending_triggers('tgtest'); table_name | trigger_name | trigger_def ------------+--------------+------------------------------------------------------------------------------------ tgtest | tg_nop | CREATE TRIGGER tg_nop AFTER INSERT ON tgtest FOR EACH ROW EXECUTE PROCEDURE notg() (1 row) select * from londiste.subscriber_restore_all_table_triggers('tgtest'); subscriber_restore_all_table_triggers --------------------------------------- 1 (1 row) select * from londiste.find_table_triggers('tgtest'); table_name | trigger_name | trigger_def ---------------+--------------+------------------------------------------------------------------------------------ public.tgtest | tg_nop | CREATE TRIGGER tg_nop AFTER INSERT ON tgtest FOR EACH ROW EXECUTE PROCEDURE notg() (1 row) select * from londiste.subscriber_get_table_pending_triggers('tgtest'); table_name | trigger_name | trigger_def ------------+--------------+------------- (0 rows) skytools-2.1.13/sql/londiste/structure/0000755000175000017500000000000011727601174017157 5ustar markomarkoskytools-2.1.13/sql/londiste/structure/grants.sql0000644000175000017500000000035311670174255021201 0ustar markomarko grant usage on schema londiste to public; grant select on londiste.provider_table to public; grant select on londiste.completed to public; grant select on londiste.link to public; grant select on londiste.subscriber_table to public; skytools-2.1.13/sql/londiste/structure/tables.sql0000644000175000017500000000335011670174255021155 0ustar markomarkoset default_with_oids = 'off'; create schema londiste; create table londiste.provider_table ( nr serial not null, queue_name text not null, table_name text not null, trigger_name text, primary key (queue_name, table_name) ); create table londiste.provider_seq ( nr serial not null, queue_name text not null, seq_name text not null, primary key (queue_name, seq_name) ); create table londiste.completed ( consumer_id text not null, last_tick_id bigint not null, primary key (consumer_id) ); create table londiste.link ( source text not null, dest text not null, primary key (source), unique (dest) ); create table londiste.subscriber_table ( nr serial not null, queue_name text not null, table_name text not null, snapshot text, merge_state text, trigger_name text, skip_truncate bool, primary key (queue_name, table_name) ); create table londiste.subscriber_seq ( nr serial not null, queue_name text not null, seq_name text not null, primary key (queue_name, seq_name) ); create table londiste.subscriber_pending_fkeys ( from_table text not null, to_table text not null, fkey_name text not null, fkey_def text not null, primary key (from_table, fkey_name) ); create table londiste.subscriber_pending_triggers ( table_name text not null, trigger_name text not null, trigger_def text not null, primary key (table_name, trigger_name) ); skytools-2.1.13/sql/londiste/structure/types.sql0000644000175000017500000000040111670174255021041 0ustar markomarko create type londiste.ret_provider_table_list as ( table_name text, trigger_name text ); create type londiste.ret_subscriber_table as ( table_name text, merge_state text, snapshot text, trigger_name text, skip_truncate bool ); skytools-2.1.13/sql/londiste/Makefile0000644000175000017500000000113111670174255016555 0ustar markomarko DATA_built = londiste.sql londiste.upgrade.sql DOCS = README.londiste FUNCS = $(wildcard functions/*.sql) SRCS = structure/tables.sql structure/grants.sql structure/types.sql $(FUNCS) REGRESS = londiste_install londiste_provider londiste_subscriber londiste_fkeys # londiste_denytrigger REGRESS_OPTS = --load-language=plpythonu --load-language=plpgsql include ../../config.mak include $(PGXS) londiste.sql: $(SRCS) cat $(SRCS) > $@ londiste.upgrade.sql: $(FUNCS) cat $(FUNCS) > $@ test: londiste.sql $(MAKE) installcheck || { less regression.diffs; exit 1; } ack: cp results/* expected/ skytools-2.1.13/sql/londiste/README.londiste0000644000175000017500000000073411670174255017625 0ustar markomarko londiste database backend -------------------------- Provider side: -------------- londiste.provider_table londiste.provider_seq Subscriber side --------------- table londiste.completed table londiste.subscriber_table table londiste.subscriber_seq Open issues ------------ - notify behaviour - should notify-s given to db for processing? - link init functions - switchover - are set_last_tick()/get_last_tick() functions needed anymore? - typecheck for add_table()? skytools-2.1.13/sql/londiste/sql/0000755000175000017500000000000011727601174015716 5ustar markomarkoskytools-2.1.13/sql/londiste/sql/londiste_fkeys.sql0000644000175000017500000000507511670174255021472 0ustar markomarko set log_error_verbosity = 'terse'; create table ref_1 ( id int4 primary key, val text ); create table ref_2 ( id int4 primary key, ref int4 not null references ref_1, val text ); create table ref_3 ( id int4 primary key, ref2 int4 not null references ref_2, val text ); select * from londiste.subscriber_add_table('refqueue', 'public.ref_1'); select * from londiste.subscriber_add_table('refqueue', 'public.ref_2'); select * from londiste.subscriber_add_table('refqueue', 'public.ref_3'); select * from londiste.find_table_fkeys('public.ref_1'); select * from londiste.find_table_fkeys('public.ref_2'); select * from londiste.find_table_fkeys('public.ref_3'); select * from londiste.subscriber_get_table_pending_fkeys('public.ref_2'); select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); -- drop fkeys select * from londiste.subscriber_drop_table_fkey('public.ref_2', 'ref_2_ref_fkey'); select * from londiste.find_table_fkeys('public.ref_1'); select * from londiste.find_table_fkeys('public.ref_2'); select * from londiste.find_table_fkeys('public.ref_3'); select * from londiste.subscriber_drop_table_fkey('public.ref_3', 'ref_3_ref2_fkey'); -- check if dropped select * from londiste.find_table_fkeys('public.ref_1'); select * from londiste.find_table_fkeys('public.ref_2'); select * from londiste.find_table_fkeys('public.ref_3'); -- look state select * from londiste.subscriber_get_table_pending_fkeys('public.ref_2'); select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); -- toggle sync select * from londiste.subscriber_set_table_state('refqueue', 'public.ref_1', null, 'ok'); select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); select * from londiste.subscriber_set_table_state('refqueue', 'public.ref_2', null, 'ok'); select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); select * from londiste.subscriber_set_table_state('refqueue', 'public.ref_3', null, 'ok'); select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); -- restore select * from londiste.subscriber_restore_table_fkey('public.ref_2', 'ref_2_ref_fkey'); select * from londiste.subscriber_restore_table_fkey('public.ref_3', 'ref_3_ref2_fkey'); -- look state select * from londiste.subscriber_get_table_pending_fkeys('public.ref_2'); select * from londiste.subscriber_get_queue_valid_pending_fkeys('refqueue'); select * from londiste.find_table_fkeys('public.ref_1'); select * from londiste.find_table_fkeys('public.ref_2'); select * from londiste.find_table_fkeys('public.ref_3'); skytools-2.1.13/sql/londiste/sql/londiste_subscriber.sql0000644000175000017500000000706211670174255022512 0ustar markomarko set client_min_messages = 'warning'; \set VERBOSITY 'terse' create table testdata ( id serial primary key, data text ); -- -- tables -- select londiste.subscriber_add_table('pqueue', 'public.testdata_nopk'); select londiste.subscriber_add_table('pqueue', 'public.testdata'); select pgq.create_queue('pqueue'); select londiste.subscriber_add_table('pqueue', 'public.testdata'); select londiste.subscriber_add_table('pqueue', 'public.testdata'); select * from londiste.subscriber_get_table_list('pqueue'); select londiste.subscriber_remove_table('pqueue', 'public.nonexist'); select londiste.subscriber_remove_table('pqueue', 'public.testdata'); select * from londiste.subscriber_get_table_list('pqueue'); -- -- seqs -- select * from londiste.subscriber_get_seq_list('pqueue'); select londiste.subscriber_add_seq('pqueue', 'public.no_seq'); select londiste.subscriber_add_seq('pqueue', 'public.testdata_id_seq'); select londiste.subscriber_add_seq('pqueue', 'public.testdata_id_seq'); select * from londiste.subscriber_get_seq_list('pqueue'); select londiste.subscriber_remove_seq('pqueue', 'public.testdata_id_seq'); select londiste.subscriber_remove_seq('pqueue', 'public.testdata_id_seq'); select * from londiste.subscriber_get_seq_list('pqueue'); -- -- linked queue -- select londiste.subscriber_add_table('pqueue', 'public.testdata'); insert into londiste.link (source, dest) values ('mqueue', 'pqueue'); select londiste.subscriber_add_table('pqueue', 'public.testdata'); select londiste.subscriber_remove_table('pqueue', 'public.testdata'); select londiste.subscriber_add_seq('pqueue', 'public.testdata_id_seq'); select londiste.subscriber_remove_seq('pqueue', 'public.testdata_seq'); -- -- skip-truncate, set_table_state -- select londiste.subscriber_add_table('pqueue', 'public.skiptest'); select skip_truncate from londiste.subscriber_table where table_name = 'public.skiptest'; select londiste.subscriber_set_skip_truncate('pqueue', 'public.skiptest', true); select skip_truncate from londiste.subscriber_table where table_name = 'public.skiptest'; select londiste.subscriber_set_table_state('pqueue', 'public.skiptest', 'snap1', 'in-copy'); select skip_truncate, snapshot from londiste.subscriber_table where table_name = 'public.skiptest'; select londiste.subscriber_set_table_state('pqueue', 'public.skiptest', null, 'ok'); select skip_truncate, snapshot from londiste.subscriber_table where table_name = 'public.skiptest'; -- -- test tick tracking -- select londiste.get_last_tick('c'); select londiste.set_last_tick('c', 1); select londiste.get_last_tick('c'); select londiste.set_last_tick('c', 2); select londiste.get_last_tick('c'); select londiste.set_last_tick('c', NULL); select londiste.get_last_tick('c'); -- test triggers create table tgfk ( id int4 primary key, data text ); create table tgtest ( id int4 primary key, fk int4 references tgfk, data text ); create or replace function notg() returns trigger as $$ begin return null; end; $$ language plpgsql; create trigger tg_nop after insert on tgtest for each row execute procedure notg(); select * from londiste.find_table_triggers('tgtest'); select * from londiste.subscriber_get_table_pending_triggers('tgtest'); select * from londiste.subscriber_drop_all_table_triggers('tgtest'); select * from londiste.find_table_triggers('tgtest'); select * from londiste.subscriber_get_table_pending_triggers('tgtest'); select * from londiste.subscriber_restore_all_table_triggers('tgtest'); select * from londiste.find_table_triggers('tgtest'); select * from londiste.subscriber_get_table_pending_triggers('tgtest'); skytools-2.1.13/sql/londiste/sql/londiste_install.sql0000644000175000017500000000022211670174255022004 0ustar markomarko\set ECHO off set log_error_verbosity = 'terse'; \i ../txid/txid.sql \i ../pgq/pgq.sql \i ../logtriga/logtriga.sql \i londiste.sql \set ECHO all skytools-2.1.13/sql/londiste/sql/londiste_denytrigger.sql0000644000175000017500000000107711670174255022672 0ustar markomarko create table denytest ( val integer); insert into denytest values (1); create trigger xdeny after insert or update or delete on denytest for each row execute procedure londiste.deny_trigger(); insert into denytest values (2); update denytest set val = 2; delete from denytest; select londiste.disable_deny_trigger(true); update denytest set val = 2; select londiste.disable_deny_trigger(true); update denytest set val = 2; select londiste.disable_deny_trigger(false); update denytest set val = 2; select londiste.disable_deny_trigger(false); update denytest set val = 2; skytools-2.1.13/sql/londiste/sql/londiste_provider.sql0000644000175000017500000000372011670174255022176 0ustar markomarko set client_min_messages = 'warning'; \set VERBOSITY 'terse' -- -- tables -- create table testdata ( id serial primary key, data text ); create table testdata_nopk ( id serial, data text ); select londiste.provider_add_table('pqueue', 'public.testdata_nopk'); select londiste.provider_add_table('pqueue', 'public.testdata'); select pgq.create_queue('pqueue'); select londiste.provider_add_table('pqueue', 'public.testdata'); select londiste.provider_add_table('pqueue', 'public.testdata'); select londiste.provider_refresh_trigger('pqueue', 'public.testdata'); select * from londiste.provider_get_table_list('pqueue'); select londiste.provider_remove_table('pqueue', 'public.nonexist'); select londiste.provider_remove_table('pqueue', 'public.testdata'); select * from londiste.provider_get_table_list('pqueue'); -- -- seqs -- select * from londiste.provider_get_seq_list('pqueue'); select londiste.provider_add_seq('pqueue', 'public.no_seq'); select londiste.provider_add_seq('pqueue', 'public.testdata_id_seq'); select londiste.provider_add_seq('pqueue', 'public.testdata_id_seq'); select * from londiste.provider_get_seq_list('pqueue'); select londiste.provider_remove_seq('pqueue', 'public.testdata_id_seq'); select londiste.provider_remove_seq('pqueue', 'public.testdata_id_seq'); select * from londiste.provider_get_seq_list('pqueue'); -- -- linked queue -- select londiste.provider_add_table('pqueue', 'public.testdata'); insert into londiste.link (source, dest) values ('mqueue', 'pqueue'); select londiste.provider_add_table('pqueue', 'public.testdata'); select londiste.provider_remove_table('pqueue', 'public.testdata'); select londiste.provider_add_seq('pqueue', 'public.testdata_id_seq'); select londiste.provider_remove_seq('pqueue', 'public.testdata_seq'); -- -- cleanup -- delete from londiste.link; drop table testdata; drop table testdata_nopk; delete from londiste.provider_seq; delete from londiste.provider_table; select pgq.drop_queue('pqueue'); skytools-2.1.13/sql/logtriga/0000755000175000017500000000000011727601174015106 5ustar markomarkoskytools-2.1.13/sql/logtriga/expected/0000755000175000017500000000000011727601174016707 5ustar markomarkoskytools-2.1.13/sql/logtriga/expected/logtriga.out0000644000175000017500000000611711670174255021257 0ustar markomarko-- init \set ECHO none create table rtest ( id integer primary key, dat text ); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "rtest_pkey" for table "rtest" create table clog ( id serial, op text, data text ); NOTICE: CREATE TABLE will create implicit sequence "clog_id_seq" for serial column "clog.id" create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure logtriga('kv', 'insert into clog (op, data) values ($1, $2)'); -- simple test insert into rtest values (1, 'value1'); update rtest set dat = 'value2'; delete from rtest; select * from clog; delete from clog; id | op | data ----+----+-------------------------------- 1 | I | (id,dat) values ('1','value1') 2 | U | dat='value2' where id='1' 3 | D | id='1' (3 rows) -- test new fields alter table rtest add column dat2 text; insert into rtest values (1, 'value1'); update rtest set dat = 'value2'; delete from rtest; select * from clog; delete from clog; id | op | data ----+----+-------------------------------- 4 | I | (id,dat) values ('1','value1') 5 | U | dat='value2' where id='1' 6 | D | id='1' (3 rows) -- test field rename alter table rtest alter column dat type integer using 0; insert into rtest values (1, '666', 'newdat'); update rtest set dat = 5; delete from rtest; select * from clog; delete from clog; id | op | data ----+----+----------------------------- 7 | I | (id,dat) values ('1','666') 8 | U | dat='5' where id='1' 9 | D | id='1' (3 rows) -- test field ignore drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure logtriga('kiv', 'insert into clog (op, data) values ($1, $2)'); insert into rtest values (1, '666', 'newdat'); update rtest set dat = 5, dat2 = 'newdat2'; update rtest set dat = 6; delete from rtest; select * from clog; delete from clog; id | op | data ----+----+--------------------------------- 10 | I | (id,dat2) values ('1','newdat') 11 | U | dat2='newdat2' where id='1' 12 | D | id='1' (3 rows) -- test wrong key drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure logtriga('vik', 'insert into clog (op, data) values ($1, $2)'); insert into rtest values (1, 0, 'non-null'); insert into rtest values (2, 0, NULL); update rtest set dat2 = 'non-null2' where id=1; update rtest set dat2 = NULL where id=1; update rtest set dat2 = 'new-nonnull' where id=2; ERROR: logtriga: Unexpected NULL key value delete from rtest where id=1; ERROR: logtriga: Unexpected NULL key value delete from rtest where id=2; ERROR: logtriga: Unexpected NULL key value select * from clog; delete from clog; id | op | data ----+----+---------------------------------------- 13 | I | (id,dat2) values ('1','non-null') 14 | I | (id,dat2) values ('2',null) 15 | U | dat2='non-null2' where dat2='non-null' 16 | U | dat2=NULL where dat2='non-null2' (4 rows) skytools-2.1.13/sql/logtriga/logtriga.sql.in0000644000175000017500000000033411670174255020046 0ustar markomarko -- usage: logtriga(flds, query) -- -- query should include 2 args: -- $1 - for op type I/U/D, -- $2 - for op data CREATE OR REPLACE FUNCTION logtriga() RETURNS trigger AS 'MODULE_PATHNAME', 'logtriga' LANGUAGE C; skytools-2.1.13/sql/logtriga/logtriga.c0000644000175000017500000002656611727577077017117 0ustar markomarko/* ---------------------------------------------------------------------- * logtriga.c * * Generic trigger for logging table changes. * Based on Slony-I log trigger. * Does not depend on event storage. * * Copyright (c) 2003-2006, PostgreSQL Global Development Group * Author: Jan Wieck, Afilias USA INC. * * Generalized by Marko Kreen. * ---------------------------------------------------------------------- */ #include "postgres.h" #include "executor/spi.h" #include "commands/trigger.h" #include "catalog/pg_operator.h" #include "catalog/pg_type.h" #include "utils/typcache.h" #include "utils/rel.h" #include "textbuf.h" PG_FUNCTION_INFO_V1(logtriga); Datum logtriga(PG_FUNCTION_ARGS); #ifdef PG_MODULE_MAGIC PG_MODULE_MAGIC; #endif /* * There may be several plans to be cached. * * FIXME: plans are kept in singe-linked list * so not very fast access. Probably they should be * handled more intelligently. */ typedef struct PlanCache PlanCache; struct PlanCache { PlanCache *next; char *query; void *plan; }; /* * Cache result allocations. */ typedef struct ArgCache { TBuf *op_type; TBuf *op_data; } ArgCache; static PlanCache *plan_cache = NULL; static ArgCache *arg_cache = NULL; /* * Cache helpers */ static void *get_plan(const char *query) { PlanCache *c; void *plan; Oid plan_types[2]; for (c = plan_cache; c; c = c->next) if (strcmp(query, c->query) == 0) return c->plan; /* * Plan not cached, prepare new plan then. */ plan_types[0] = TEXTOID; plan_types[1] = TEXTOID; plan = SPI_saveplan(SPI_prepare(query, 2, plan_types)); if (plan == NULL) elog(ERROR, "logtriga: SPI_prepare() failed"); /* create cache object */ c = malloc(sizeof(*c)); if (!c) elog(ERROR, "logtriga: no memory for plan cache"); c->plan = plan; c->query = strdup(query); /* insert at start */ c->next = plan_cache; plan_cache = c; return plan; } static ArgCache * get_arg_cache(void) { if (arg_cache == NULL) { ArgCache *a = malloc(sizeof(*a)); if (!a) elog(ERROR, "logtriga: no memory"); memset(a, 0, sizeof(*a)); a->op_type = tbuf_alloc(8); a->op_data = tbuf_alloc(8192); arg_cache = a; } return arg_cache; } static void append_key_eq(TBuf *tbuf, const char *col_ident, const char *col_value) { if (col_value == NULL) elog(ERROR, "logtriga: Unexpected NULL key value"); tbuf_encode_cstring(tbuf, col_ident, "quote_ident"); tbuf_append_char(tbuf, '='); tbuf_encode_cstring(tbuf, col_value, "quote_literal"); } static void append_normal_eq(TBuf *tbuf, const char *col_ident, const char *col_value) { tbuf_encode_cstring(tbuf, col_ident, "quote_ident"); tbuf_append_char(tbuf, '='); if (col_value != NULL) tbuf_encode_cstring(tbuf, col_value, "quote_literal"); else tbuf_append_cstring(tbuf, "NULL"); } static void process_insert(ArgCache *cs, TriggerData *tg, char *attkind) { HeapTuple new_row = tg->tg_trigtuple; TupleDesc tupdesc = tg->tg_relation->rd_att; int i; int need_comma = false; int attkind_idx; /* * INSERT * * op_type = 'I' op_data = ("non-NULL-col" [, ...]) values ('value' [, * ...]) */ tbuf_append_cstring(cs->op_type, "I"); /* * Specify all the columns */ tbuf_append_char(cs->op_data, '('); attkind_idx = -1; for (i = 0; i < tg->tg_relation->rd_att->natts; i++) { char *col_ident; /* Skip dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; /* Check if allowed by colstring */ attkind_idx++; if (attkind[attkind_idx] == '\0') break; if (attkind[attkind_idx] == 'i') continue; if (need_comma) tbuf_append_char(cs->op_data, ','); else need_comma = true; /* quote column name */ col_ident = SPI_fname(tupdesc, i + 1); tbuf_encode_cstring(cs->op_data, col_ident, "quote_ident"); } /* * Append the string ") values (" */ tbuf_append_cstring(cs->op_data, ") values ("); /* * Append the values */ need_comma = false; attkind_idx = -1; for (i = 0; i < tg->tg_relation->rd_att->natts; i++) { char *col_value; /* Skip dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; /* Check if allowed by colstring */ attkind_idx++; if (attkind[attkind_idx] == '\0') break; if (attkind[attkind_idx] == 'i') continue; if (need_comma) tbuf_append_char(cs->op_data, ','); else need_comma = true; /* quote column value */ col_value = SPI_getvalue(new_row, tupdesc, i + 1); if (col_value == NULL) tbuf_append_cstring(cs->op_data, "null"); else tbuf_encode_cstring(cs->op_data, col_value, "quote_literal"); } /* * Terminate and done */ tbuf_append_char(cs->op_data, ')'); } static int process_update(ArgCache *cs, TriggerData *tg, char *attkind) { HeapTuple old_row = tg->tg_trigtuple; HeapTuple new_row = tg->tg_newtuple; TupleDesc tupdesc = tg->tg_relation->rd_att; Datum old_value; Datum new_value; bool old_isnull; bool new_isnull; char *col_ident; char *col_value; int i; int need_comma = false; int need_and = false; int attkind_idx; int ignore_count = 0; /* * UPDATE * * op_type = 'U' op_data = "col_ident"='value' [, ...] where "pk_ident" = * 'value' [ and ...] */ tbuf_append_cstring(cs->op_type, "U"); attkind_idx = -1; for (i = 0; i < tg->tg_relation->rd_att->natts; i++) { /* * Ignore dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (attkind[attkind_idx] == '\0') break; old_value = SPI_getbinval(old_row, tupdesc, i + 1, &old_isnull); new_value = SPI_getbinval(new_row, tupdesc, i + 1, &new_isnull); /* * If old and new value are NULL, the column is unchanged */ if (old_isnull && new_isnull) continue; /* * If both are NOT NULL, we need to compare the values and skip * setting the column if equal */ if (!old_isnull && !new_isnull) { Oid opr_oid; FmgrInfo *opr_finfo_p; /* * Lookup the equal operators function call info using the * typecache if available */ TypeCacheEntry *type_cache; type_cache = lookup_type_cache(SPI_gettypeid(tupdesc, i + 1), TYPECACHE_EQ_OPR | TYPECACHE_EQ_OPR_FINFO); opr_oid = type_cache->eq_opr; if (opr_oid == ARRAY_EQ_OP) opr_oid = InvalidOid; else opr_finfo_p = &(type_cache->eq_opr_finfo); /* * If we have an equal operator, use that to do binary * comparision. Else get the string representation of both * attributes and do string comparision. */ if (OidIsValid(opr_oid)) { if (DatumGetBool(FunctionCall2(opr_finfo_p, old_value, new_value))) continue; } else { char *old_strval = SPI_getvalue(old_row, tupdesc, i + 1); char *new_strval = SPI_getvalue(new_row, tupdesc, i + 1); if (strcmp(old_strval, new_strval) == 0) continue; } } if (attkind[attkind_idx] == 'i') { /* this change should be ignored */ ignore_count++; continue; } if (need_comma) tbuf_append_char(cs->op_data, ','); else need_comma = true; col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(new_row, tupdesc, i + 1); append_normal_eq(cs->op_data, col_ident, col_value); } /* * It can happen that the only UPDATE an application does is to set a * column to the same value again. In that case, we'd end up here with * no columns in the SET clause yet. We add the first key column here * with it's old value to simulate the same for the replication * engine. */ if (!need_comma) { /* there was change in ignored columns, skip whole event */ if (ignore_count > 0) return 0; for (i = 0, attkind_idx = -1; i < tg->tg_relation->rd_att->natts; i++) { if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (attkind[attkind_idx] == 'k') break; } col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(old_row, tupdesc, i + 1); append_key_eq(cs->op_data, col_ident, col_value); } tbuf_append_cstring(cs->op_data, " where "); for (i = 0, attkind_idx = -1; i < tg->tg_relation->rd_att->natts; i++) { /* * Ignore dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (attkind[attkind_idx] == '\0') break; if (attkind[attkind_idx] != 'k') continue; col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(old_row, tupdesc, i + 1); if (need_and) tbuf_append_cstring(cs->op_data, " and "); else need_and = true; append_key_eq(cs->op_data, col_ident, col_value); } return 1; } static void process_delete(ArgCache *cs, TriggerData *tg, char *attkind) { HeapTuple old_row = tg->tg_trigtuple; TupleDesc tupdesc = tg->tg_relation->rd_att; char *col_ident; char *col_value; int i; int need_and = false; int attkind_idx; /* * DELETE * * op_type = 'D' op_data = "pk_ident"='value' [and ...] */ tbuf_append_cstring(cs->op_type, "D"); for (i = 0, attkind_idx = -1; i < tg->tg_relation->rd_att->natts; i++) { if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (attkind[attkind_idx] == '\0') break; if (attkind[attkind_idx] != 'k') continue; col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(old_row, tupdesc, i + 1); if (need_and) tbuf_append_cstring(cs->op_data, " and "); else need_and = true; append_key_eq(cs->op_data, col_ident, col_value); } } Datum logtriga(PG_FUNCTION_ARGS) { TriggerData *tg; Datum argv[2]; int rc; ArgCache *cs; TupleDesc tupdesc; int i; int attcnt; char *attkind; char *kpos; char *query; int need_event = 1; /* * Get the trigger call context */ if (!CALLED_AS_TRIGGER(fcinfo)) elog(ERROR, "logtriga not called as trigger"); tg = (TriggerData *) (fcinfo->context); tupdesc = tg->tg_relation->rd_att; /* * Check all logTrigger() calling conventions */ if (!TRIGGER_FIRED_AFTER(tg->tg_event)) elog(ERROR, "logtriga must be fired AFTER"); if (!TRIGGER_FIRED_FOR_ROW(tg->tg_event)) elog(ERROR, "logtriga must be fired FOR EACH ROW"); if (tg->tg_trigger->tgnargs != 2) elog(ERROR, "logtriga must be defined with 2 args"); /* * Connect to the SPI manager */ if ((rc = SPI_connect()) < 0) elog(ERROR, "logtriga: SPI_connect() failed"); cs = get_arg_cache(); tbuf_reset(cs->op_type); tbuf_reset(cs->op_data); /* * Get all the trigger arguments */ attkind = tg->tg_trigger->tgargs[0]; query = tg->tg_trigger->tgargs[1]; /* * Count number of active columns */ for (i = 0, attcnt = 0; i < tg->tg_relation->rd_att->natts; i++) { if (tupdesc->attrs[i]->attisdropped) continue; attcnt++; } /* * Make sure all 'k' columns exist and there is at least one of them. */ kpos = strrchr(attkind, 'k'); if (kpos == NULL) elog(ERROR, "logtriga: need at least one key column"); if (kpos - attkind >= attcnt) elog(ERROR, "logtriga: key column does not exist"); /* * Determine cmdtype and op_data depending on the command type */ if (TRIGGER_FIRED_BY_INSERT(tg->tg_event)) process_insert(cs, tg, attkind); else if (TRIGGER_FIRED_BY_UPDATE(tg->tg_event)) need_event = process_update(cs, tg, attkind); else if (TRIGGER_FIRED_BY_DELETE(tg->tg_event)) process_delete(cs, tg, attkind); else elog(ERROR, "logtriga fired for unhandled event"); /* * Construct the parameter array and insert the log row. */ if (need_event) { argv[0] = PointerGetDatum(tbuf_look_text(cs->op_type)); argv[1] = PointerGetDatum(tbuf_look_text(cs->op_data)); SPI_execp(get_plan(query), argv, NULL, 0); } SPI_finish(); return PointerGetDatum(NULL); } skytools-2.1.13/sql/logtriga/Makefile0000644000175000017500000000035411670174255016552 0ustar markomarko include ../../config.mak MODULE_big = logtriga SRCS = logtriga.c textbuf.c OBJS = $(SRCS:.c=.o) DATA_built = logtriga.sql REGRESS = logtriga include $(PGXS) test: install make installcheck || { less regression.diffs; exit 1; } skytools-2.1.13/sql/logtriga/textbuf.h0000644000175000017500000000114111670174255016737 0ustar markomarkostruct TBuf; typedef struct TBuf TBuf; TBuf *tbuf_alloc(int start_size); void tbuf_free(TBuf *tbuf); int tbuf_get_size(TBuf *tbuf); void tbuf_reset(TBuf *tbuf); const text *tbuf_look_text(TBuf *tbuf); const char *tbuf_look_cstring(TBuf *tbuf); void tbuf_append_cstring(TBuf *tbuf, const char *str); void tbuf_append_text(TBuf *tbuf, const text *str); void tbuf_append_char(TBuf *tbuf, char chr); text *tbuf_steal_text(TBuf *tbuf); void tbuf_encode_cstring(TBuf *tbuf, const char *str, const char *encoding); void tbuf_encode_data(TBuf *tbuf, const uint8 *data, int len, const char *encoding); skytools-2.1.13/sql/logtriga/README.logtriga0000644000175000017500000000245411670174255017604 0ustar markomarko logtriga - generic table changes logger ======================================= logtriga provides generic table changes logging trigger. It prepares partial SQL statement about a change and gives it to user query. Usage ----- CREATE TRIGGER foo_log AFTER INSERT OR UPDATE OR DELETE ON foo_tbl FOR EACH ROW EXECUTE PROCEDURE logtriga(column_types, query); Where column_types is a string where each charater defines type of that column. Known types: * k - one of primary key columns for table. * v - data column * i - uninteresting column, to be ignored. Trigger function prepares 2 string arguments for query and executes it. * $1 - Operation type: I/U/D. * $2 - Partial SQL for event playback. * INSERT INTO FOO_TBL (field, list) values (val1, val2) * UPDATE FOO_TBL SET field1 = val1, field2 = val2 where key1 = kval1 * DELETE FROM FOO_TBL WHERE key1 = keyval1 The upper-case part is left out. Example ------- Following query emulates Slony-I behaviour: insert into SL_SCHEMA.sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values (CLUSTER_IDENT, SL_SCHEMA.getCurrentXid(), TABLE_OID, nextval('SL_SCHEMA.sl_action_seq'), $1, $2) The upper-case strings should be replaced with actual values on trigger creation. skytools-2.1.13/sql/logtriga/textbuf.c0000644000175000017500000001673111670174255016745 0ustar markomarko #include #include "funcapi.h" #include "mb/pg_wchar.h" #include "parser/keywords.h" #if 1 #define talloc(len) malloc(len) #define trealloc(p, len) realloc(p, len) #define tfree(p) free(p) #else #define talloc(len) palloc(len) #define trealloc(p, len) repalloc(p, len) #define tfree(p) pfree(p) #endif #include "textbuf.h" #ifndef SET_VARSIZE #define SET_VARSIZE(x, len) VARATT_SIZEP(x) = (len) #endif struct TBuf { text *data; int size; }; static void request_avail(TBuf *tbuf, int len) { int newlen = tbuf->size; int need = VARSIZE(tbuf->data) + len; if (need < newlen) return; while (need > newlen) newlen *= 2; tbuf->data = trealloc(tbuf->data, newlen); tbuf->size = newlen; } static inline char *get_endp(TBuf *tbuf) { char *p = VARDATA(tbuf->data); int len = VARSIZE(tbuf->data) - VARHDRSZ; return p + len; } static inline void inc_used(TBuf *tbuf, int len) { SET_VARSIZE(tbuf->data, VARSIZE(tbuf->data) + len); } static void tbuf_init(TBuf *tbuf, int start_size) { if (start_size < VARHDRSZ) start_size = VARHDRSZ; tbuf->data = talloc(start_size); tbuf->size = start_size; SET_VARSIZE(tbuf->data, VARHDRSZ); } TBuf *tbuf_alloc(int start_size) { TBuf *res; res = talloc(sizeof(TBuf)); tbuf_init(res, start_size); return res; } void tbuf_free(TBuf *tbuf) { if (tbuf->data) tfree(tbuf->data); tfree(tbuf); } int tbuf_get_size(TBuf *tbuf) { return VARSIZE(tbuf->data) - VARHDRSZ; } void tbuf_reset(TBuf *tbuf) { SET_VARSIZE(tbuf->data, VARHDRSZ); } const text *tbuf_look_text(TBuf *tbuf) { return tbuf->data; } const char *tbuf_look_cstring(TBuf *tbuf) { char *p; request_avail(tbuf, 1); p = get_endp(tbuf); *p = 0; return VARDATA(tbuf->data); } void tbuf_append_cstring(TBuf *tbuf, const char *str) { int len = strlen(str); request_avail(tbuf, len); memcpy(get_endp(tbuf), str, len); inc_used(tbuf, len); } void tbuf_append_text(TBuf *tbuf, const text *str) { int len = VARSIZE(str) - VARHDRSZ; request_avail(tbuf, len); memcpy(get_endp(tbuf), VARDATA(str), len); inc_used(tbuf, len); } void tbuf_append_char(TBuf *tbuf, char chr) { char *p; request_avail(tbuf, 1); p = get_endp(tbuf); *p = chr; inc_used(tbuf, 1); } text *tbuf_steal_text(TBuf *tbuf) { text *data = tbuf->data; tbuf->data = NULL; return data; } static const char b64tbl[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; static int b64encode(char *dst, const uint8 *src, int srclen) { char *p = dst; const uint8 *s = src, *end = src + srclen; int pos = 2; uint32 buf = 0; while (s < end) { buf |= (unsigned char) *s << (pos << 3); pos--; s++; /* write it out */ if (pos < 0) { *p++ = b64tbl[ (buf >> 18) & 0x3f ]; *p++ = b64tbl[ (buf >> 12) & 0x3f ]; *p++ = b64tbl[ (buf >> 6) & 0x3f ]; *p++ = b64tbl[ buf & 0x3f ]; pos = 2; buf = 0; } } if (pos != 2) { *p++ = b64tbl[ (buf >> 18) & 0x3f ]; *p++ = b64tbl[ (buf >> 12) & 0x3f ]; *p++ = (pos == 0) ? b64tbl[ (buf >> 6) & 0x3f ] : '='; *p++ = '='; } return p - dst; } static const char hextbl[] = "0123456789abcdef"; static int urlencode(char *dst, const uint8 *src, int srclen) { const uint8 *end = src + srclen; char *p = dst; while (src < end) { if (*src == '=') *p++ = '+'; else if ((*src >= '0' && *src <= '9') || (*src >= 'A' && *src <= 'Z') || (*src >= 'a' && *src <= 'z')) *p++ = *src; else { *p++ = '%'; *p++ = hextbl[*src >> 4]; *p++ = hextbl[*src & 15]; } } return p - dst; } static int quote_literal(char *dst, const uint8 *src, int srclen) { const uint8 *cp1; char *cp2; int wl; cp1 = src; cp2 = dst; *cp2++ = '\''; while (srclen > 0) { if ((wl = pg_mblen((const char *)cp1)) != 1) { srclen -= wl; while (wl-- > 0) *cp2++ = *cp1++; continue; } if (*cp1 == '\'') *cp2++ = '\''; if (*cp1 == '\\') *cp2++ = '\\'; *cp2++ = *cp1++; srclen--; } *cp2++ = '\''; return cp2 - dst; } /* * slon_quote_identifier - Quote an identifier only if needed * * When quotes are needed, we palloc the required space; slightly * space-wasteful but well worth it for notational simplicity. * * Version: pgsql/src/backend/utils/adt/ruleutils.c,v 1.188 2005/01/13 17:19:10 */ static int quote_ident(char *dst, const uint8 *src, int srclen) { /* * Can avoid quoting if ident starts with a lowercase letter or * underscore and contains only lowercase letters, digits, and * underscores, *and* is not any SQL keyword. Otherwise, supply * quotes. */ int nquotes = 0; bool safe; const char *ptr; char *optr; char ident[NAMEDATALEN + 1]; /* expect idents be not bigger than NAMEDATALEN */ if (srclen > NAMEDATALEN) srclen = NAMEDATALEN; memcpy(ident, src, srclen); ident[srclen] = 0; /* * would like to use macros here, but they might yield * unwanted locale-specific results... */ safe = ((ident[0] >= 'a' && ident[0] <= 'z') || ident[0] == '_'); for (ptr = ident; *ptr; ptr++) { char ch = *ptr; if ((ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9') || (ch == '_')) continue; /* okay */ safe = false; if (ch == '"') nquotes++; } if (safe) { /* * Check for keyword. This test is overly strong, since many of * the "keywords" known to the parser are usable as column names, * but the parser doesn't provide any easy way to test for whether * an identifier is safe or not... so be safe not sorry. * * Note: ScanKeywordLookup() does case-insensitive comparison, but * that's fine, since we already know we have all-lower-case. */ #if defined(PG_VERSION_NUM) && PG_VERSION_NUM >= 90000 if (ScanKeywordLookup(ident, ScanKeywords, NumScanKeywords) != NULL) #else if (ScanKeywordLookup(ident) != NULL) #endif safe = false; } optr = dst; if (!safe) *optr++ = '"'; for (ptr = ident; *ptr; ptr++) { char ch = *ptr; if (ch == '"') *optr++ = '"'; *optr++ = ch; } if (!safe) *optr++ = '"'; return optr - dst; } void tbuf_encode_cstring(TBuf *tbuf, const char *str, const char *encoding) { if (str == NULL) elog(ERROR, "tbuf_encode_cstring: NULL"); tbuf_encode_data(tbuf, (const uint8 *)str, strlen(str), encoding); } void tbuf_encode_data(TBuf *tbuf, const uint8 *data, int len, const char *encoding) { int dlen = 0; char *dst; if (strcmp(encoding, "url") == 0) { request_avail(tbuf, len*3); dst = get_endp(tbuf); dlen = urlencode(dst, data, len); } else if (strcmp(encoding, "base64") == 0) { request_avail(tbuf, (len + 2) * 4 / 3); dst = get_endp(tbuf); dlen = b64encode(dst, data, len); } else if (strcmp(encoding, "quote_literal") == 0) { request_avail(tbuf, len * 2 + 2); dst = get_endp(tbuf); dlen = quote_literal(dst, data, len); } else if (strcmp(encoding, "quote_ident") == 0) { request_avail(tbuf, len * 2 + 2); dst = get_endp(tbuf); dlen = quote_ident(dst, data, len); } else elog(ERROR, "bad encoding"); inc_used(tbuf, dlen); } skytools-2.1.13/sql/logtriga/sql/0000755000175000017500000000000011727601174015705 5ustar markomarkoskytools-2.1.13/sql/logtriga/sql/logtriga.sql0000644000175000017500000000344211670174255020243 0ustar markomarko-- init \set ECHO none \i logtriga.sql \set ECHO all create table rtest ( id integer primary key, dat text ); create table clog ( id serial, op text, data text ); create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure logtriga('kv', 'insert into clog (op, data) values ($1, $2)'); -- simple test insert into rtest values (1, 'value1'); update rtest set dat = 'value2'; delete from rtest; select * from clog; delete from clog; -- test new fields alter table rtest add column dat2 text; insert into rtest values (1, 'value1'); update rtest set dat = 'value2'; delete from rtest; select * from clog; delete from clog; -- test field rename alter table rtest alter column dat type integer using 0; insert into rtest values (1, '666', 'newdat'); update rtest set dat = 5; delete from rtest; select * from clog; delete from clog; -- test field ignore drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure logtriga('kiv', 'insert into clog (op, data) values ($1, $2)'); insert into rtest values (1, '666', 'newdat'); update rtest set dat = 5, dat2 = 'newdat2'; update rtest set dat = 6; delete from rtest; select * from clog; delete from clog; -- test wrong key drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure logtriga('vik', 'insert into clog (op, data) values ($1, $2)'); insert into rtest values (1, 0, 'non-null'); insert into rtest values (2, 0, NULL); update rtest set dat2 = 'non-null2' where id=1; update rtest set dat2 = NULL where id=1; update rtest set dat2 = 'new-nonnull' where id=2; delete from rtest where id=1; delete from rtest where id=2; select * from clog; delete from clog; skytools-2.1.13/sql/Makefile0000644000175000017500000000031411670174255014736 0ustar markomarko include ../config.mak SUBDIRS = logtriga londiste pgq pgq_ext txid all install clean distclean installcheck: for dir in $(SUBDIRS); do \ $(MAKE) -C $$dir $@ DESTDIR=$(DESTDIR) || exit $?; \ done skytools-2.1.13/sql/pgq/0000755000175000017500000000000011727601174014065 5ustar markomarkoskytools-2.1.13/sql/pgq/functions/0000755000175000017500000000000011727601174016075 5ustar markomarkoskytools-2.1.13/sql/pgq/functions/pgq.register_consumer.sql0000644000175000017500000000717711670174255023161 0ustar markomarkocreate or replace function pgq.register_consumer( x_queue_name text, x_consumer_id text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.register_consumer(2) -- -- Subscribe consumer on a queue. -- -- From this moment forward, consumer will see all events in the queue. -- -- Parameters: -- x_queue_name - Name of queue -- x_consumer_name - Name of consumer -- -- Returns: -- 0 - if already registered -- 1 - if new registration -- ---------------------------------------------------------------------- begin return pgq.register_consumer(x_queue_name, x_consumer_id, NULL); end; $$ language plpgsql security definer; create or replace function pgq.register_consumer( x_queue_name text, x_consumer_name text, x_tick_pos bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.register_consumer(3) -- -- Extended registration, allows to specify tick_id. -- -- Note: -- For usage in special situations. -- -- Parameters: -- x_queue_name - Name of a queue -- x_consumer_name - Name of consumer -- x_tick_pos - Tick ID -- -- Returns: -- 0/1 whether consumer has already registered. -- ---------------------------------------------------------------------- declare tmp text; last_tick bigint; x_queue_id integer; x_consumer_id integer; queue integer; sub record; begin select queue_id into x_queue_id from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'Event queue not created yet'; end if; -- get consumer and create if new select co_id into x_consumer_id from pgq.consumer where co_name = x_consumer_name; if not found then insert into pgq.consumer (co_name) values (x_consumer_name); x_consumer_id := currval('pgq.consumer_co_id_seq'); end if; -- if particular tick was requested, check if it exists if x_tick_pos is not null then perform 1 from pgq.tick where tick_queue = x_queue_id and tick_id = x_tick_pos; if not found then raise exception 'cannot reposition, tick not found: %', x_tick_pos; end if; end if; -- check if already registered select sub_last_tick, sub_batch into sub from pgq.subscription where sub_consumer = x_consumer_id and sub_queue = x_queue_id; if found then if x_tick_pos is not null then if sub.sub_batch is not null then raise exception 'reposition while active not allowed'; end if; -- update tick pos if requested update pgq.subscription set sub_last_tick = x_tick_pos where sub_consumer = x_consumer_id and sub_queue = x_queue_id; end if; -- already registered return 0; end if; -- new registration if x_tick_pos is null then -- start from current tick select tick_id into last_tick from pgq.tick where tick_queue = x_queue_id order by tick_queue desc, tick_id desc limit 1; if not found then raise exception 'No ticks for this queue. Please run ticker on database.'; end if; else last_tick := x_tick_pos; end if; -- register insert into pgq.subscription (sub_queue, sub_consumer, sub_last_tick) values (x_queue_id, x_consumer_id, last_tick); return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.maint_rotate_tables.sql0000644000175000017500000000713111670174255023430 0ustar markomarkocreate or replace function pgq.maint_rotate_tables_step1(i_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_rotate_tables_step1(1) -- -- Rotate tables for one queue. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- 1 if rotation happened, otherwise 0. -- ---------------------------------------------------------------------- declare badcnt integer; cf record; nr integer; tbl text; lowest_tick_id int8; lowest_xmin int8; begin -- check if needed and load record select * from pgq.queue into cf where queue_name = i_queue_name and queue_rotation_period is not null and queue_switch_step2 is not null and queue_switch_time + queue_rotation_period < current_timestamp for update; if not found then return 0; end if; -- find lowest tick for that queue select min(sub_last_tick) into lowest_tick_id from pgq.subscription where sub_queue = cf.queue_id; -- if some consumer exists if lowest_tick_id is not null then -- is the slowest one still on previous table? select txid_snapshot_xmin(tick_snapshot) into lowest_xmin from pgq.tick where tick_queue = cf.queue_id and tick_id = lowest_tick_id; if lowest_xmin <= cf.queue_switch_step2 then return 0; -- skip rotation then end if; end if; -- nobody on previous table, we can rotate -- calc next table number and name nr := cf.queue_cur_table + 1; if nr = cf.queue_ntables then nr := 0; end if; tbl := cf.queue_data_pfx || '_' || nr; -- there may be long lock on the table from pg_dump, -- detect it and skip rotate then begin execute 'lock table ' || tbl || ' nowait'; execute 'truncate ' || tbl; exception when lock_not_available then -- cannot truncate, skipping rotate return 0; end; -- remember the moment update pgq.queue set queue_cur_table = nr, queue_switch_time = current_timestamp, queue_switch_step1 = txid_current(), queue_switch_step2 = NULL where queue_id = cf.queue_id; -- Clean ticks by using step2 txid from previous rotation. -- That should keep all ticks for all batches that are completely -- in old table. This keeps them for longer than needed, but: -- 1. we want the pgq.tick table to be big, to avoid Postgres -- accitentally switching to seqscans on that. -- 2. that way we guarantee to consumers that they an be moved -- back on the queue at least for one rotation_period. -- (may help in disaster recovery) delete from pgq.tick where tick_queue = cf.queue_id and txid_snapshot_xmin(tick_snapshot) < cf.queue_switch_step2; return 1; end; $$ language plpgsql; -- need admin access create or replace function pgq.maint_rotate_tables_step2() returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_rotate_tables_step2(0) -- -- Stores the txid when the rotation was visible. It should be -- called in separate transaction than pgq.maint_rotate_tables_step1() -- ---------------------------------------------------------------------- begin update pgq.queue set queue_switch_step2 = txid_current() where queue_switch_step2 is null; return 1; end; $$ language plpgsql; -- need admin access skytools-2.1.13/sql/pgq/functions/pgq.event_retry_raw.sql0000644000175000017500000000366411670174255022636 0ustar markomarkocreate or replace function pgq.event_retry_raw( x_queue text, x_consumer text, x_retry_after timestamptz, x_ev_id bigint, x_ev_time timestamptz, x_ev_retry integer, x_ev_type text, x_ev_data text, x_ev_extra1 text, x_ev_extra2 text, x_ev_extra3 text, x_ev_extra4 text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry_raw(12) -- -- Allows full control over what goes to retry queue. -- -- Parameters: -- x_queue - name of the queue -- x_consumer - name of the consumer -- x_retry_after - when the event should be processed again -- x_ev_id - event id -- x_ev_time - creation time -- x_ev_retry - retry count -- x_ev_type - user data -- x_ev_data - user data -- x_ev_extra1 - user data -- x_ev_extra2 - user data -- x_ev_extra3 - user data -- x_ev_extra4 - user data -- -- Returns: -- Event ID. -- ---------------------------------------------------------------------- declare q record; id bigint; begin select sub_id, queue_event_seq into q from pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue and co_name = x_consumer and sub_consumer = co_id and sub_queue = queue_id; if not found then raise exception 'consumer not registered'; end if; id := x_ev_id; if id is null then id := nextval(q.queue_event_seq); end if; insert into pgq.retry_queue (ev_retry_after, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) values (x_retry_after, x_ev_id, x_ev_time, q.sub_id, x_ev_retry, x_ev_type, x_ev_data, x_ev_extra1, x_ev_extra2, x_ev_extra3, x_ev_extra4); return id; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.ticker.sql0000644000175000017500000000461211670174255020672 0ustar markomarkocreate or replace function pgq.ticker(i_queue_name text, i_tick_id bigint) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.ticker(2) -- -- Insert a tick with a particular tick_id. -- -- For external tickers. -- -- Parameters: -- i_queue_name - Name of the queue -- i_tick_id - Id of new tick. -- -- Returns: -- Tick id. -- ---------------------------------------------------------------------- begin insert into pgq.tick (tick_queue, tick_id) select queue_id, i_tick_id from pgq.queue where queue_name = i_queue_name and queue_external_ticker; if not found then raise exception 'queue not found'; end if; return i_tick_id; end; $$ language plpgsql security definer; -- unsure about access create or replace function pgq.ticker(i_queue_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.ticker(1) -- -- Insert a tick with a tick_id from sequence. -- -- For pgqadm usage. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- Tick id. -- ---------------------------------------------------------------------- declare res bigint; ext boolean; seq text; q record; begin select queue_id, queue_tick_seq, queue_external_ticker into q from pgq.queue where queue_name = i_queue_name; if not found then raise exception 'no such queue'; end if; if q.queue_external_ticker then raise exception 'This queue has external tick source.'; end if; insert into pgq.tick (tick_queue, tick_id) values (q.queue_id, nextval(q.queue_tick_seq)); res = currval(q.queue_tick_seq); return res; end; $$ language plpgsql security definer; -- unsure about access create or replace function pgq.ticker() returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.ticker(0) -- -- Creates ticks for all queues which dont have external ticker. -- -- Returns: -- Number of queues that were processed. -- ---------------------------------------------------------------------- declare res bigint; begin select count(pgq.ticker(queue_name)) into res from pgq.queue where not queue_external_ticker; return res; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.create_queue.sql0000644000175000017500000000413311670174255022056 0ustar markomarkocreate or replace function pgq.create_queue(i_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.create_queue(1) -- -- Creates new queue with given name. -- -- Returns: -- 0 - queue already exists -- 1 - queue created -- ---------------------------------------------------------------------- declare tblpfx text; tblname text; idxpfx text; idxname text; sql text; id integer; tick_seq text; ev_seq text; n_tables integer; begin if i_queue_name is null then raise exception 'Invalid NULL value'; end if; -- check if exists perform 1 from pgq.queue where queue_name = i_queue_name; if found then return 0; end if; -- insert event id := nextval('pgq.queue_queue_id_seq'); tblpfx := 'pgq.event_' || id; idxpfx := 'event_' || id; tick_seq := 'pgq.event_' || id || '_tick_seq'; ev_seq := 'pgq.event_' || id || '_id_seq'; insert into pgq.queue (queue_id, queue_name, queue_data_pfx, queue_event_seq, queue_tick_seq) values (id, i_queue_name, tblpfx, ev_seq, tick_seq); select queue_ntables into n_tables from pgq.queue where queue_id = id; -- create seqs execute 'CREATE SEQUENCE ' || tick_seq; execute 'CREATE SEQUENCE ' || ev_seq; -- create data tables execute 'CREATE TABLE ' || tblpfx || ' () ' || ' INHERITS (pgq.event_template)'; for i in 0 .. (n_tables - 1) loop tblname := tblpfx || '_' || i; idxname := idxpfx || '_' || i; execute 'CREATE TABLE ' || tblname || ' () ' || ' INHERITS (' || tblpfx || ')'; execute 'ALTER TABLE ' || tblname || ' ALTER COLUMN ev_id ' || ' SET DEFAULT nextval(' || quote_literal(ev_seq) || ')'; execute 'create index ' || idxname || '_txid_idx on ' || tblname || ' (ev_txid)'; end loop; perform pgq.grant_perms(i_queue_name); perform pgq.ticker(i_queue_name); return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.failed_queue.sql0000644000175000017500000001317711670174255022047 0ustar markomarko create or replace function pgq.failed_event_list( x_queue_name text, x_consumer_name text) returns setof pgq.failed_queue as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_list(2) -- -- Get list of all failed events for one consumer. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- -- Returns: -- List of failed events. -- ---------------------------------------------------------------------- declare rec pgq.failed_queue%rowtype; begin for rec in select fq.* from pgq.failed_queue fq, pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id and ev_owner = sub_id order by ev_id loop return next rec; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_list( x_queue_name text, x_consumer_name text, x_count integer, x_offset integer) returns setof pgq.failed_queue as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_list(4) -- -- Get list of failed events, from offset and specific count. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- x_count - Max amount of events to fetch -- x_offset - From this offset -- -- Returns: -- List of failed events. -- ---------------------------------------------------------------------- declare rec pgq.failed_queue%rowtype; begin for rec in select fq.* from pgq.failed_queue fq, pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id and ev_owner = sub_id order by ev_id limit x_count offset x_offset loop return next rec; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_count( x_queue_name text, x_consumer_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_count(2) -- -- Get size of failed event queue. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- -- Returns: -- Number of failed events in failed event queue. -- ---------------------------------------------------------------------- declare ret integer; begin select count(1) into ret from pgq.failed_queue, pgq.consumer, pgq.queue, pgq.subscription where queue_name = x_queue_name and co_name = x_consumer_name and sub_queue = queue_id and sub_consumer = co_id and ev_owner = sub_id; return ret; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_delete( x_queue_name text, x_consumer_name text, x_event_id bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_delete(3) -- -- Delete specific event from failed event queue. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- x_event_id - Event ID -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare x_sub_id integer; begin select sub_id into x_sub_id from pgq.subscription, pgq.consumer, pgq.queue where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id; if not found then raise exception 'no such queue/consumer'; end if; delete from pgq.failed_queue where ev_owner = x_sub_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; end; $$ language plpgsql security definer; create or replace function pgq.failed_event_retry( x_queue_name text, x_consumer_name text, x_event_id bigint) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.failed_event_retry(3) -- -- Insert specific event from failed queue to main queue. -- -- Parameters: -- x_queue_name - Queue name -- x_consumer_name - Consumer name -- x_event_id - Event ID -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare ret bigint; x_sub_id integer; begin select sub_id into x_sub_id from pgq.subscription, pgq.consumer, pgq.queue where queue_name = x_queue_name and co_name = x_consumer_name and sub_consumer = co_id and sub_queue = queue_id; if not found then raise exception 'no such queue/consumer'; end if; select pgq.insert_event_raw(x_queue_name, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) into ret from pgq.failed_queue, pgq.consumer, pgq.queue where ev_owner = x_sub_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; perform pgq.failed_event_delete(x_queue_name, x_consumer_name, x_event_id); return ret; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.batch_event_tables.sql0000644000175000017500000000414011670174255023221 0ustar markomarkocreate or replace function pgq.batch_event_tables(x_batch_id bigint) returns setof text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.batch_event_tables(1) -- -- Returns set of table names where this batch events may reside. -- -- Parameters: -- x_batch_id - ID of a active batch. -- ---------------------------------------------------------------------- declare nr integer; tbl text; use_prev integer; use_next integer; batch record; begin select txid_snapshot_xmin(last.tick_snapshot) as tx_min, -- absolute minimum txid_snapshot_xmax(cur.tick_snapshot) as tx_max, -- absolute maximum q.queue_data_pfx, q.queue_ntables, q.queue_cur_table, q.queue_switch_step1, q.queue_switch_step2 into batch from pgq.tick last, pgq.tick cur, pgq.subscription s, pgq.queue q where cur.tick_id = s.sub_next_tick and cur.tick_queue = s.sub_queue and last.tick_id = s.sub_last_tick and last.tick_queue = s.sub_queue and s.sub_batch = x_batch_id and q.queue_id = s.sub_queue; if not found then raise exception 'Cannot find data for batch %', x_batch_id; end if; -- if its definitely not in one or other, look into both if batch.tx_max < batch.queue_switch_step1 then use_prev := 1; use_next := 0; elsif batch.queue_switch_step2 is not null and (batch.tx_min > batch.queue_switch_step2) then use_prev := 0; use_next := 1; else use_prev := 1; use_next := 1; end if; if use_prev then nr := batch.queue_cur_table - 1; if nr < 0 then nr := batch.queue_ntables - 1; end if; tbl := batch.queue_data_pfx || '_' || nr; return next tbl; end if; if use_next then tbl := batch.queue_data_pfx || '_' || batch.queue_cur_table; return next tbl; end if; return; end; $$ language plpgsql; -- no perms needed skytools-2.1.13/sql/pgq/functions/pgq.unregister_consumer.sql0000644000175000017500000000226211670174255023512 0ustar markomarko create or replace function pgq.unregister_consumer( x_queue_name text, x_consumer_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.unregister_consumer(2) -- -- Unsubscriber consumer from the queue. Also consumer's failed -- and retry events are deleted. -- -- Parameters: -- x_queue_name - Name of the queue -- x_consumer_name - Name of the consumer -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare x_sub_id integer; begin select sub_id into x_sub_id from pgq.subscription, pgq.consumer, pgq.queue where sub_queue = queue_id and sub_consumer = co_id and queue_name = x_queue_name and co_name = x_consumer_name; if not found then raise exception 'consumer not registered on queue'; end if; delete from pgq.retry_queue where ev_owner = x_sub_id; delete from pgq.failed_queue where ev_owner = x_sub_id; delete from pgq.subscription where sub_id = x_sub_id; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.finish_batch.sql0000644000175000017500000000156011670174255022031 0ustar markomarko create or replace function pgq.finish_batch( x_batch_id bigint) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.finish_batch(1) -- -- Closes a batch. No more operations can be done with events -- of this batch. -- -- Parameters: -- x_batch_id - id of batch. -- -- Returns: -- If batch 1 if batch was found, 0 otherwise. -- ---------------------------------------------------------------------- begin update pgq.subscription set sub_active = now(), sub_last_tick = sub_next_tick, sub_next_tick = null, sub_batch = null where sub_batch = x_batch_id; if not found then raise warning 'finish_batch: batch % not found', x_batch_id; return 0; end if; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.maint_tables_to_vacuum.sql0000644000175000017500000000277411670174255024144 0ustar markomarkocreate or replace function pgq.maint_tables_to_vacuum() returns setof text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_tables_to_vacuum(0) -- -- Returns list of tablenames that need frequent vacuuming. -- -- The goal is to avoid hardcoding them into maintenance process. -- -- Returns: -- List of table names. -- ---------------------------------------------------------------------- declare row record; begin return next 'pgq.subscription'; return next 'pgq.consumer'; return next 'pgq.queue'; return next 'pgq.tick'; return next 'pgq.retry_queue'; -- include also txid, pgq_ext and londiste tables if they exist for row in select n.nspname as scm, t.relname as tbl from pg_class t, pg_namespace n where n.oid = t.relnamespace and n.nspname = 'txid' and t.relname = 'epoch' union all select n.nspname as scm, t.relname as tbl from pg_class t, pg_namespace n where n.oid = t.relnamespace and n.nspname = 'londiste' and t.relname = 'completed' union all select n.nspname as scm, t.relname as tbl from pg_class t, pg_namespace n where n.oid = t.relnamespace and n.nspname = 'pgq_ext' and t.relname in ('completed_tick', 'completed_batch', 'completed_event', 'partial_batch') loop return next row.scm || '.' || row.tbl; end loop; return; end; $$ language plpgsql; skytools-2.1.13/sql/pgq/functions/pgq.version.sql0000644000175000017500000000064211670174255021075 0ustar markomarkocreate or replace function pgq.version() returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.version(0) -- -- Returns verison string for pgq. ATM its SkyTools version -- that is only bumped when PGQ database code changes. -- ---------------------------------------------------------------------- begin return '2.1.8'; end; $$ language plpgsql; skytools-2.1.13/sql/pgq/functions/pgq.get_batch_info.sql0000644000175000017500000000215311670174255022342 0ustar markomarko create or replace function pgq.get_batch_info(x_batch_id bigint) returns pgq.ret_batch_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_batch_info(1) -- -- Returns detailed info about a batch. -- -- Parameters: -- x_batch_id - id of a active batch. -- -- Returns: -- Info -- ---------------------------------------------------------------------- declare ret pgq.ret_batch_info%rowtype; begin select queue_name, co_name, prev.tick_time as batch_start, cur.tick_time as batch_end, sub_last_tick, sub_next_tick, current_timestamp - cur.tick_time as lag into ret from pgq.subscription, pgq.tick cur, pgq.tick prev, pgq.queue, pgq.consumer where sub_batch = x_batch_id and prev.tick_id = sub_last_tick and prev.tick_queue = sub_queue and cur.tick_id = sub_next_tick and cur.tick_queue = sub_queue and queue_id = sub_queue and co_id = sub_consumer; return ret; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.force_tick.sql0000644000175000017500000000255311670174255021523 0ustar markomarko create or replace function pgq.force_tick(i_queue_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.force_tick(2) -- -- Simulate lots of events happening to force ticker to tick. -- -- Should be called in loop, with some delay until last tick -- changes or too much time is passed. -- -- Such function is needed because paraller calls of pgq.ticker() are -- dangerous, and cannot be protected with locks as snapshot -- is taken before locking. -- -- Parameters: -- i_queue_name - Name of the queue -- -- Returns: -- Currently last tick id. -- ---------------------------------------------------------------------- declare q record; t record; begin -- bump seq and get queue id select queue_id, setval(queue_event_seq, nextval(queue_event_seq) + queue_ticker_max_count * 2) as tmp into q from pgq.queue where queue_name = i_queue_name and not queue_external_ticker; if not found then raise exception 'queue not found or ticks not allowed'; end if; -- return last tick id select tick_id into t from pgq.tick where tick_queue = q.queue_id order by tick_queue desc, tick_id desc limit 1; return t.tick_id; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.batch_event_sql.sql0000644000175000017500000001217711670174255022557 0ustar markomarkocreate or replace function pgq.batch_event_sql(x_batch_id bigint) returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.batch_event_sql(1) -- Creates SELECT statement that fetches events for this batch. -- -- Parameters: -- x_batch_id - ID of a active batch. -- -- Returns: -- SQL statement. -- ---------------------------------------------------------------------- -- ---------------------------------------------------------------------- -- Algorithm description: -- Given 2 snapshots, sn1 and sn2 with sn1 having xmin1, xmax1 -- and sn2 having xmin2, xmax2 create expression that filters -- right txid's from event table. -- -- Simplest solution would be -- > WHERE ev_txid >= xmin1 AND ev_txid <= xmax2 -- > AND NOT txid_visible_in_snapshot(ev_txid, sn1) -- > AND txid_visible_in_snapshot(ev_txid, sn2) -- -- The simple solution has a problem with long transactions (xmin1 very low). -- All the batches that happen when the long tx is active will need -- to scan all events in that range. Here is 2 optimizations used: -- -- 1) Use [xmax1..xmax2] for range scan. That limits the range to -- txids that actually happened between two snapshots. For txids -- in the range [xmin1..xmax1] look which ones were actually -- committed between snapshots and search for them using exact -- values using IN (..) list. -- -- 2) As most TX are short, there could be lot of them that were -- just below xmax1, but were committed before xmax2. So look -- if there are ID's near xmax1 and lower the range to include -- them, thus decresing size of IN (..) list. -- ---------------------------------------------------------------------- declare rec record; sql text; tbl text; arr text; part text; select_fields text; retry_expr text; batch record; begin select s.sub_last_tick, s.sub_next_tick, s.sub_id, s.sub_queue, txid_snapshot_xmax(last.tick_snapshot) as tx_start, txid_snapshot_xmax(cur.tick_snapshot) as tx_end, last.tick_snapshot as last_snapshot, cur.tick_snapshot as cur_snapshot into batch from pgq.subscription s, pgq.tick last, pgq.tick cur where s.sub_batch = x_batch_id and last.tick_queue = s.sub_queue and last.tick_id = s.sub_last_tick and cur.tick_queue = s.sub_queue and cur.tick_id = s.sub_next_tick; if not found then raise exception 'batch not found'; end if; -- load older transactions arr := ''; for rec in -- active tx-es in prev_snapshot that were committed in cur_snapshot select id1 from txid_snapshot_xip(batch.last_snapshot) id1 left join txid_snapshot_xip(batch.cur_snapshot) id2 on (id1 = id2) where id2 is null order by 1 desc loop -- try to avoid big IN expression, so try to include nearby -- tx'es into range if batch.tx_start - 100 <= rec.id1 then batch.tx_start := rec.id1; else if arr = '' then arr := rec.id1; else arr := arr || ',' || rec.id1; end if; end if; end loop; -- must match pgq.event_template select_fields := 'select ev_id, ev_time, ev_txid, ev_retry, ev_type,' || ' ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4'; retry_expr := ' and (ev_owner is null or ev_owner = ' || batch.sub_id || ')'; -- now generate query that goes over all potential tables sql := ''; for rec in select xtbl from pgq.batch_event_tables(x_batch_id) xtbl loop tbl := rec.xtbl; -- this gets newer queries that definitely are not in prev_snapshot part := select_fields || ' from pgq.tick cur, pgq.tick last, ' || tbl || ' ev ' || ' where cur.tick_id = ' || batch.sub_next_tick || ' and cur.tick_queue = ' || batch.sub_queue || ' and last.tick_id = ' || batch.sub_last_tick || ' and last.tick_queue = ' || batch.sub_queue || ' and ev.ev_txid >= ' || batch.tx_start || ' and ev.ev_txid <= ' || batch.tx_end || ' and txid_visible_in_snapshot(ev.ev_txid, cur.tick_snapshot)' || ' and not txid_visible_in_snapshot(ev.ev_txid, last.tick_snapshot)' || retry_expr; -- now include older tx-es, that were ongoing -- at the time of prev_snapshot if arr <> '' then part := part || ' union all ' || select_fields || ' from ' || tbl || ' ev ' || ' where ev.ev_txid in (' || arr || ')' || retry_expr; end if; if sql = '' then sql := part; else sql := sql || ' union all ' || part; end if; end loop; if sql = '' then raise exception 'could not construct sql for batch %', x_batch_id; end if; return sql || ' order by 1'; end; $$ language plpgsql; -- no perms needed skytools-2.1.13/sql/pgq/functions/pgq.get_consumer_info.sql0000644000175000017500000000716211670174255023121 0ustar markomarko ------------------------------------------------------------------------- create or replace function pgq.get_consumer_info() returns setof pgq.ret_consumer_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_consumer_info(0) -- -- Returns info about all consumers on all queues. -- -- Returns: -- See pgq.get_consumer_info(2) -- ---------------------------------------------------------------------- declare ret pgq.ret_consumer_info%rowtype; i record; begin for i in select queue_name from pgq.queue order by 1 loop for ret in select * from pgq.get_consumer_info(i.queue_name) loop return next ret; end loop; end loop; return; end; $$ language plpgsql security definer; ------------------------------------------------------------------------- create or replace function pgq.get_consumer_info(x_queue_name text) returns setof pgq.ret_consumer_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_consumer_info(1) -- -- Returns info about consumers on one particular queue. -- -- Parameters: -- x_queue_name - Queue name -- -- Returns: -- See pgq.get_consumer_info(2) -- ---------------------------------------------------------------------- declare ret pgq.ret_consumer_info%rowtype; tmp record; begin for tmp in select queue_name, co_name from pgq.queue, pgq.consumer, pgq.subscription where queue_id = sub_queue and co_id = sub_consumer and queue_name = x_queue_name order by 1, 2 loop for ret in select * from pgq.get_consumer_info(tmp.queue_name, tmp.co_name) loop return next ret; end loop; end loop; return; end; $$ language plpgsql security definer; ------------------------------------------------------------------------ create or replace function pgq.get_consumer_info( x_queue_name text, x_consumer_name text) returns setof pgq.ret_consumer_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_consumer_info(2) -- -- Get info about particular consumer on particular queue. -- -- Parameters: -- x_queue_name - name of a queue. -- x_consumer_name - name of a consumer -- -- Returns: -- queue_name - Queue name -- consumer_name - Consumer name -- lag - How old are events the consumer is processing -- last_seen - When the consumer seen by pgq -- last_tick - Tick ID of last processed tick -- current_batch - Current batch ID, if one is active or NULL -- next_tick - If batch is active, then its final tick. -- ---------------------------------------------------------------------- declare ret pgq.ret_consumer_info%rowtype; begin for ret in select queue_name, co_name, current_timestamp - tick_time as lag, current_timestamp - sub_active as last_seen, sub_last_tick as last_tick, sub_batch as current_batch, sub_next_tick as next_tick from pgq.subscription, pgq.tick, pgq.queue, pgq.consumer where tick_id = sub_last_tick and queue_id = sub_queue and tick_queue = sub_queue and co_id = sub_consumer and queue_name = x_queue_name and co_name = x_consumer_name order by 1,2 loop return next ret; end loop; return; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.event_failed.sql0000644000175000017500000000250711670174255022037 0ustar markomarkocreate or replace function pgq.event_failed( x_batch_id bigint, x_event_id bigint, x_reason text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_failed(3) -- -- Copies the event to failed queue so it can be looked at later. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - Event id -- x_reason - Text to associate with event. -- -- Returns: -- 0 if event was already in queue, 1 otherwise. -- ---------------------------------------------------------------------- begin insert into pgq.failed_queue (ev_failed_reason, ev_failed_time, ev_id, ev_time, ev_txid, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) select x_reason, now(), ev_id, ev_time, NULL, sub_id, coalesce(ev_retry, 0), ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.get_batch_events(x_batch_id), pgq.subscription where sub_batch = x_batch_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; -- dont worry if the event is already in queue exception when unique_violation then return 0; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.get_queue_info.sql0000644000175000017500000000315011670174255022403 0ustar markomarkocreate or replace function pgq.get_queue_info() returns setof pgq.ret_queue_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_queue_info(0) -- -- Get info about all queues. -- -- Returns: -- List of pgq.ret_queue_info records. -- ---------------------------------------------------------------------- declare q record; ret pgq.ret_queue_info%rowtype; begin for q in select queue_name from pgq.queue order by 1 loop select * into ret from pgq.get_queue_info(q.queue_name); return next ret; end loop; return; end; $$ language plpgsql security definer; create or replace function pgq.get_queue_info(qname text) returns pgq.ret_queue_info as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_queue_info(1) -- -- Get info about particular queue. -- -- Returns: -- One pgq.ret_queue_info record. -- ---------------------------------------------------------------------- declare ret pgq.ret_queue_info%rowtype; begin select queue_name, queue_ntables, queue_cur_table, queue_rotation_period, queue_switch_time, queue_external_ticker, queue_ticker_max_count, queue_ticker_max_lag, queue_ticker_idle_period, (select current_timestamp - tick_time from pgq.tick where tick_queue = queue_id order by tick_queue desc, tick_id desc limit 1 ) as ticker_lag into ret from pgq.queue where queue_name = qname; return ret; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.maint_retry_events.sql0000644000175000017500000000277411670174255023341 0ustar markomarkocreate or replace function pgq.maint_retry_events() returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.maint_retry_events(0) -- -- Moves retry events back to main queue. -- -- It moves small amount at a time. It should be called -- until it returns 0 -- -- Returns: -- Number of events processed. -- ---------------------------------------------------------------------- declare cnt integer; rec record; begin cnt := 0; -- allow only single event mover at a time, without affecting inserts lock table pgq.retry_queue in share update exclusive mode; for rec in select queue_name, ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.retry_queue, pgq.queue, pgq.subscription where ev_retry_after <= current_timestamp and sub_id = ev_owner and queue_id = sub_queue order by ev_retry_after limit 10 loop cnt := cnt + 1; perform pgq.insert_event_raw(rec.queue_name, rec.ev_id, rec.ev_time, rec.ev_owner, rec.ev_retry, rec.ev_type, rec.ev_data, rec.ev_extra1, rec.ev_extra2, rec.ev_extra3, rec.ev_extra4); delete from pgq.retry_queue where ev_owner = rec.ev_owner and ev_id = rec.ev_id; end loop; return cnt; end; $$ language plpgsql; -- need admin access skytools-2.1.13/sql/pgq/functions/pgq.grant_perms.sql0000644000175000017500000000311211670174255021724 0ustar markomarkocreate or replace function pgq.grant_perms(x_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.grant_perms(1) -- -- Make event tables readable by public. -- -- Parameters: -- x_queue_name - Name of the queue. -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare q record; i integer; tbl_perms text; seq_perms text; begin select * from pgq.queue into q where queue_name = x_queue_name; if not found then raise exception 'Queue not found'; end if; if true then -- safe, all access must go via functions seq_perms := 'select'; tbl_perms := 'select'; else -- allow ordinery users to directly insert -- to event tables. dangerous. seq_perms := 'select, update'; tbl_perms := 'select, insert'; end if; -- tick seq, normal users don't need to modify it execute 'grant ' || seq_perms || ' on ' || q.queue_tick_seq || ' to public'; -- event seq execute 'grant ' || seq_perms || ' on ' || q.queue_event_seq || ' to public'; -- parent table for events execute 'grant select on ' || q.queue_data_pfx || ' to public'; -- real event tables for i in 0 .. q.queue_ntables - 1 loop execute 'grant ' || tbl_perms || ' on ' || q.queue_data_pfx || '_' || i || ' to public'; end loop; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.drop_queue.sql0000644000175000017500000000301011670174255021550 0ustar markomarkocreate or replace function pgq.drop_queue(x_queue_name text) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.drop_queue(1) -- -- Drop queue and all associated tables. -- No consumers must be listening on the queue. -- -- ---------------------------------------------------------------------- declare tblname text; q record; num integer; begin -- check ares if x_queue_name is null then raise exception 'Invalid NULL value'; end if; -- check if exists select * into q from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'No such event queue'; end if; -- check if no consumers select count(*) into num from pgq.subscription where sub_queue = q.queue_id; if num > 0 then raise exception 'cannot drop queue, consumers still attached'; end if; -- drop data tables for i in 0 .. (q.queue_ntables - 1) loop tblname := q.queue_data_pfx || '_' || i; execute 'DROP TABLE ' || tblname; end loop; execute 'DROP TABLE ' || q.queue_data_pfx; -- delete ticks delete from pgq.tick where tick_queue = q.queue_id; -- drop seqs -- FIXME: any checks needed here? execute 'DROP SEQUENCE ' || q.queue_tick_seq; execute 'DROP SEQUENCE ' || q.queue_event_seq; -- delete event delete from pgq.queue where queue_name = x_queue_name; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.get_batch_events.sql0000644000175000017500000000124711670174255022716 0ustar markomarkocreate or replace function pgq.get_batch_events(x_batch_id bigint) returns setof pgq.ret_batch_event as $$ -- ---------------------------------------------------------------------- -- Function: pgq.get_batch_events(1) -- -- Get all events in batch. -- -- Parameters: -- x_batch_id - ID of active batch. -- -- Returns: -- List of events. -- ---------------------------------------------------------------------- declare rec pgq.ret_batch_event%rowtype; sql text; begin sql := pgq.batch_event_sql(x_batch_id); for rec in execute sql loop return next rec; end loop; return; end; $$ language plpgsql; -- no perms needed skytools-2.1.13/sql/pgq/functions/pgq.event_retry.sql0000644000175000017500000000414611670174255021761 0ustar markomarkocreate or replace function pgq.event_retry( x_batch_id bigint, x_event_id bigint, x_retry_time timestamptz) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry(3) -- -- Put the event into retry queue, to be processed again later. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - event id -- x_retry_time - Time when the event should be put back into queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- begin insert into pgq.retry_queue (ev_retry_after, ev_id, ev_time, ev_txid, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4) select x_retry_time, ev_id, ev_time, NULL, sub_id, coalesce(ev_retry, 0) + 1, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4 from pgq.get_batch_events(x_batch_id), pgq.subscription where sub_batch = x_batch_id and ev_id = x_event_id; if not found then raise exception 'event not found'; end if; return 1; -- dont worry if the event is already in queue exception when unique_violation then return 0; end; $$ language plpgsql security definer; create or replace function pgq.event_retry( x_batch_id bigint, x_event_id bigint, x_retry_seconds integer) returns integer as $$ -- ---------------------------------------------------------------------- -- Function: pgq.event_retry(3) -- -- Put the event into retry queue, to be processed later again. -- -- Parameters: -- x_batch_id - ID of active batch. -- x_event_id - event id -- x_retry_seconds - Time when the event should be put back into queue -- -- Returns: -- nothing -- ---------------------------------------------------------------------- declare new_retry timestamptz; begin new_retry := current_timestamp + ((x_retry_seconds || ' seconds')::interval); return pgq.event_retry(x_batch_id, x_event_id, new_retry); end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.next_batch.sql0000644000175000017500000000402211670174255021523 0ustar markomarkocreate or replace function pgq.next_batch(x_queue_name text, x_consumer_name text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.next_batch(2) -- -- Makes next block of events active. -- -- If it returns NULL, there is no events available in queue. -- Consumer should sleep a bith then. -- -- Parameters: -- x_queue_name - Name of the queue -- x_consumer_name - Name of the consumer -- -- Returns: -- Batch ID or NULL if there are no more events available. -- ---------------------------------------------------------------------- declare next_tick bigint; batch_id bigint; errmsg text; sub record; begin select sub_queue, sub_consumer, sub_id, sub_last_tick, sub_batch into sub from pgq.queue q, pgq.consumer c, pgq.subscription s where q.queue_name = x_queue_name and c.co_name = x_consumer_name and s.sub_queue = q.queue_id and s.sub_consumer = c.co_id; if not found then errmsg := 'Not subscriber to queue: ' || coalesce(x_queue_name, 'NULL') || '/' || coalesce(x_consumer_name, 'NULL'); raise exception '%', errmsg; end if; -- has already active batch if sub.sub_batch is not null then return sub.sub_batch; end if; -- find next tick select tick_id into next_tick from pgq.tick where tick_id > sub.sub_last_tick and tick_queue = sub.sub_queue order by tick_queue asc, tick_id asc limit 1; if not found then -- nothing to do return null; end if; -- get next batch batch_id := nextval('pgq.batch_id_seq'); update pgq.subscription set sub_batch = batch_id, sub_next_tick = next_tick, sub_active = now() where sub_queue = sub.sub_queue and sub_consumer = sub.sub_consumer; return batch_id; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/functions/pgq.current_event_table.sql0000644000175000017500000000161611670174255023444 0ustar markomarkocreate or replace function pgq.current_event_table(x_queue_name text) returns text as $$ -- ---------------------------------------------------------------------- -- Function: pgq.current_event_table(1) -- -- Return active event table for particular queue. -- Event can be added to it without going via functions, -- e.g. by COPY. -- -- Note: -- The result is valid only during current transaction. -- -- Permissions: -- Actual insertion requires superuser access. -- -- Parameters: -- x_queue_name - Queue name. -- ---------------------------------------------------------------------- declare res text; begin select queue_data_pfx || '_' || queue_cur_table into res from pgq.queue where queue_name = x_queue_name; if not found then raise exception 'Event queue not found'; end if; return res; end; $$ language plpgsql; -- no perms needed skytools-2.1.13/sql/pgq/functions/pgq.insert_event.sql0000644000175000017500000000327211670174255022117 0ustar markomarkocreate or replace function pgq.insert_event(queue_name text, ev_type text, ev_data text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.insert_event(3) -- -- Insert a event into queue. -- -- Parameters: -- queue_name - Name of the queue -- ev_type - User-specified type for the event -- ev_data - User data for the event -- -- Returns: -- Event ID -- ---------------------------------------------------------------------- begin return pgq.insert_event(queue_name, ev_type, ev_data, null, null, null, null); end; $$ language plpgsql security definer; create or replace function pgq.insert_event( queue_name text, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text) returns bigint as $$ -- ---------------------------------------------------------------------- -- Function: pgq.insert_event(7) -- -- Insert a event into queue with all the extra fields. -- -- Parameters: -- queue_name - Name of the queue -- ev_type - User-specified type for the event -- ev_data - User data for the event -- ev_extra1 - Extra data field for the event -- ev_extra2 - Extra data field for the event -- ev_extra3 - Extra data field for the event -- ev_extra4 - Extra data field for the event -- -- Returns: -- Event ID -- ---------------------------------------------------------------------- begin return pgq.insert_event_raw(queue_name, null, now(), null, null, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4); end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq/expected/0000755000175000017500000000000011727601174015666 5ustar markomarkoskytools-2.1.13/sql/pgq/expected/logutriga.out0000644000175000017500000000436711670174255020430 0ustar markomarkodrop function pgq.insert_event(text, text, text, text, text, text, text); create or replace function pgq.insert_event(que text, ev_type text, ev_data text, x1 text, x2 text, x3 text, x4 text) returns bigint as $$ begin raise notice 'insert_event(%, %, %, %)', que, ev_type, ev_data, x1; return 1; end; $$ language plpgsql; create table udata ( id serial primary key, txt text, bin bytea ); NOTICE: CREATE TABLE will create implicit sequence "udata_id_seq" for serial column "udata.id" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "udata_pkey" for table "udata" create trigger utest AFTER insert or update or delete ON udata for each row execute procedure pgq.logutriga('udata_que'); insert into udata (txt) values ('text1'); NOTICE: insert_event(udata_que, I:id, id=1&txt=text1&bin, public.udata) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" insert into udata (bin) values (E'bi\tn\\000bin'); NOTICE: insert_event(udata_que, I:id, id=2&txt&bin=bi%5c011n%5c000bin, public.udata) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" -- test missing pkey create table nopkey2 (dat text); create trigger nopkey_triga2 after insert or update or delete on nopkey2 for each row execute procedure pgq.logutriga('que3'); insert into nopkey2 values ('foo'); NOTICE: insert_event(que3, I:, dat=foo, public.nopkey2) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update nopkey2 set dat = 'bat'; ERROR: Update/Delete on table without pkey delete from nopkey2; ERROR: Update/Delete on table without pkey -- test custom pkey create table ucustom_pkey (dat1 text not null, dat2 int2 not null, dat3 text); create trigger ucustom_triga after insert or update or delete on ucustom_pkey --for each row execute procedure pgq.logutriga('que3', 'pkey=dat1,dat2'); for each row execute procedure pgq.logutriga('que3'); insert into ucustom_pkey values ('foo', '2'); NOTICE: insert_event(que3, I:, dat1=foo&dat2=2&dat3, public.ucustom_pkey) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update ucustom_pkey set dat3 = 'bat'; ERROR: Update/Delete on table without pkey delete from ucustom_pkey; ERROR: Update/Delete on table without pkey skytools-2.1.13/sql/pgq/expected/pgq_init.out0000644000175000017500000000001711670174255020231 0ustar markomarko\set ECHO none skytools-2.1.13/sql/pgq/expected/pgq_core.out0000644000175000017500000001700511670174255020223 0ustar markomarkoselect * from pgq.maint_tables_to_vacuum(); maint_tables_to_vacuum ------------------------ pgq.subscription pgq.consumer pgq.queue pgq.tick pgq.retry_queue (5 rows) select * from pgq.maint_retry_events(); maint_retry_events -------------------- 0 (1 row) select pgq.create_queue('tmpqueue'); create_queue -------------- 1 (1 row) select pgq.register_consumer('tmpqueue', 'consumer'); register_consumer ------------------- 1 (1 row) select pgq.unregister_consumer('tmpqueue', 'consumer'); unregister_consumer --------------------- 1 (1 row) select pgq.drop_queue('tmpqueue'); drop_queue ------------ 1 (1 row) select pgq.create_queue('myqueue'); create_queue -------------- 1 (1 row) select pgq.register_consumer('myqueue', 'consumer'); register_consumer ------------------- 1 (1 row) select pgq.next_batch('myqueue', 'consumer'); next_batch ------------ (1 row) select pgq.next_batch('myqueue', 'consumer'); next_batch ------------ (1 row) select pgq.ticker(); ticker -------- 1 (1 row) select pgq.next_batch('myqueue', 'consumer'); next_batch ------------ 1 (1 row) select pgq.next_batch('myqueue', 'consumer'); next_batch ------------ 1 (1 row) select queue_name, consumer_name, prev_tick_id, tick_id, lag < '1 second' from pgq.get_batch_info(1); queue_name | consumer_name | prev_tick_id | tick_id | ?column? ------------+---------------+--------------+---------+---------- myqueue | consumer | 1 | 2 | t (1 row) select queue_name, queue_ntables, queue_cur_table, queue_rotation_period, queue_switch_time <= now() as switch_time_exists, queue_external_ticker, queue_ticker_max_count, queue_ticker_max_lag, queue_ticker_idle_period, ticker_lag < '2 hours' as ticker_lag_exists from pgq.get_queue_info() order by 1; queue_name | queue_ntables | queue_cur_table | queue_rotation_period | switch_time_exists | queue_external_ticker | queue_ticker_max_count | queue_ticker_max_lag | queue_ticker_idle_period | ticker_lag_exists ------------+---------------+-----------------+-----------------------+--------------------+-----------------------+------------------------+----------------------+--------------------------+------------------- myqueue | 3 | 0 | @ 2 hours | t | f | 500 | @ 3 secs | @ 1 min | t (1 row) select queue_name, consumer_name, lag < '30 seconds' as lag_exists, last_seen < '30 seconds' as last_seen_exists, last_tick, current_batch, next_tick from pgq.get_consumer_info() order by 1, 2; queue_name | consumer_name | lag_exists | last_seen_exists | last_tick | current_batch | next_tick ------------+---------------+------------+------------------+-----------+---------------+----------- myqueue | consumer | t | t | 1 | 1 | 2 (1 row) select pgq.finish_batch(1); finish_batch -------------- 1 (1 row) select pgq.finish_batch(1); WARNING: finish_batch: batch 1 not found finish_batch -------------- 0 (1 row) select pgq.ticker(); ticker -------- 1 (1 row) select pgq.next_batch('myqueue', 'consumer'); next_batch ------------ 2 (1 row) select * from pgq.batch_event_tables(2); batch_event_tables -------------------- pgq.event_2_0 (1 row) select * from pgq.get_batch_events(2); ev_id | ev_time | ev_txid | ev_retry | ev_type | ev_data | ev_extra1 | ev_extra2 | ev_extra3 | ev_extra4 -------+---------+---------+----------+---------+---------+-----------+-----------+-----------+----------- (0 rows) select pgq.finish_batch(2); finish_batch -------------- 1 (1 row) select pgq.insert_event('myqueue', 'r1', 'data'); insert_event -------------- 1 (1 row) select pgq.insert_event('myqueue', 'r2', 'data', 'extra1', 'extra2', 'extra3', 'extra4'); insert_event -------------- 2 (1 row) select pgq.insert_event('myqueue', 'r3', 'data'); insert_event -------------- 3 (1 row) select pgq.current_event_table('myqueue'); current_event_table --------------------- pgq.event_2_0 (1 row) select pgq.ticker(); ticker -------- 1 (1 row) select pgq.next_batch('myqueue', 'consumer'); next_batch ------------ 3 (1 row) select ev_id,ev_retry,ev_type,ev_data,ev_extra1,ev_extra2,ev_extra3,ev_extra4 from pgq.get_batch_events(3); ev_id | ev_retry | ev_type | ev_data | ev_extra1 | ev_extra2 | ev_extra3 | ev_extra4 -------+----------+---------+---------+-----------+-----------+-----------+----------- 1 | | r1 | data | | | | 2 | | r2 | data | extra1 | extra2 | extra3 | extra4 3 | | r3 | data | | | | (3 rows) select * from pgq.failed_event_list('myqueue', 'consumer'); ev_failed_reason | ev_failed_time | ev_id | ev_time | ev_txid | ev_owner | ev_retry | ev_type | ev_data | ev_extra1 | ev_extra2 | ev_extra3 | ev_extra4 ------------------+----------------+-------+---------+---------+----------+----------+---------+---------+-----------+-----------+-----------+----------- (0 rows) select pgq.event_failed(3, 1, 'failure test'); event_failed -------------- 1 (1 row) select pgq.event_failed(3, 1, 'failure test'); event_failed -------------- 0 (1 row) select pgq.event_retry(3, 2, 0); event_retry ------------- 1 (1 row) select pgq.event_retry(3, 2, 0); event_retry ------------- 0 (1 row) select pgq.finish_batch(3); finish_batch -------------- 1 (1 row) select ev_failed_reason, ev_id, ev_txid, ev_retry, ev_type, ev_data from pgq.failed_event_list('myqueue', 'consumer'); ev_failed_reason | ev_id | ev_txid | ev_retry | ev_type | ev_data ------------------+-------+---------+----------+---------+--------- failure test | 1 | | 0 | r1 | data (1 row) select ev_failed_reason, ev_id, ev_txid, ev_retry, ev_type, ev_data from pgq.failed_event_list('myqueue', 'consumer', 0, 1); ev_failed_reason | ev_id | ev_txid | ev_retry | ev_type | ev_data ------------------+-------+---------+----------+---------+--------- (0 rows) select * from pgq.failed_event_count('myqueue', 'consumer'); failed_event_count -------------------- 1 (1 row) select * from pgq.failed_event_delete('myqueue', 'consumer', 0); ERROR: event not found select pgq.event_retry_raw('myqueue', 'consumer', now(), 666, now(), 0, 'rawtest', 'data', null, null, null, null); event_retry_raw ----------------- 666 (1 row) select pgq.ticker(); ticker -------- 1 (1 row) -- test maint update pgq.queue set queue_rotation_period = '0 seconds'; select queue_name, pgq.maint_rotate_tables_step1(queue_name) from pgq.queue; queue_name | maint_rotate_tables_step1 ------------+--------------------------- myqueue | 1 (1 row) select pgq.maint_rotate_tables_step2(); maint_rotate_tables_step2 --------------------------- 1 (1 row) -- test extra select nextval(queue_event_seq) from pgq.queue where queue_name = 'myqueue'; nextval --------- 4 (1 row) select pgq.force_tick('myqueue'); force_tick ------------ 5 (1 row) select nextval(queue_event_seq) from pgq.queue where queue_name = 'myqueue'; nextval --------- 1006 (1 row) skytools-2.1.13/sql/pgq/expected/sqltriga.out0000644000175000017500000001300111670174255020242 0ustar markomarko-- start testing create table rtest ( id integer primary key, dat text ); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "rtest_pkey" for table "rtest" create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure pgq.sqltriga('que'); -- simple test insert into rtest values (1, 'value1'); NOTICE: insert_event(que, I, (id,dat) values ('1','value1'), public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat = 'value2'; NOTICE: insert_event(que, U, dat='value2' where id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" delete from rtest; NOTICE: insert_event(que, D, id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" -- test new fields alter table rtest add column dat2 text; insert into rtest values (1, 'value1'); NOTICE: insert_event(que, I, (id,dat,dat2) values ('1','value1',null), public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat = 'value2'; NOTICE: insert_event(que, U, dat='value2' where id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" delete from rtest; NOTICE: insert_event(que, D, id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" -- test field ignore drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure pgq.sqltriga('que2', 'ignore=dat2'); insert into rtest values (1, '666', 'newdat'); NOTICE: insert_event(que2, I, (id,dat) values ('1','666'), public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat = 5, dat2 = 'newdat2'; NOTICE: insert_event(que2, U, dat='5' where id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat = 6; NOTICE: insert_event(que2, U, dat='6' where id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" delete from rtest; NOTICE: insert_event(que2, D, id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" -- test hashed pkey -- drop trigger rtest_triga on rtest; -- create trigger rtest_triga after insert or update or delete on rtest -- for each row execute procedure pgq.sqltriga('que2', 'ignore=dat2','pkey=dat,hashtext(dat)'); -- insert into rtest values (1, '666', 'newdat'); -- update rtest set dat = 5, dat2 = 'newdat2'; -- update rtest set dat = 6; -- delete from rtest; -- test wrong key drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure pgq.sqltriga('que3'); insert into rtest values (1, 0, 'non-null'); NOTICE: insert_event(que3, I, (id,dat,dat2) values ('1','0','non-null'), public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" insert into rtest values (2, 0, NULL); NOTICE: insert_event(que3, I, (id,dat,dat2) values ('2','0',null), public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat2 = 'non-null2' where id=1; NOTICE: insert_event(que3, U, dat2='non-null2' where id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat2 = NULL where id=1; NOTICE: insert_event(que3, U, dat2=NULL where id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update rtest set dat2 = 'new-nonnull' where id=2; NOTICE: insert_event(que3, U, dat2='new-nonnull' where id='2', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" delete from rtest where id=1; NOTICE: insert_event(que3, D, id='1', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" delete from rtest where id=2; NOTICE: insert_event(que3, D, id='2', public.rtest) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" -- test missing pkey create table nopkey (dat text); create trigger nopkey_triga after insert or update or delete on nopkey for each row execute procedure pgq.sqltriga('que3'); insert into nopkey values ('foo'); NOTICE: insert_event(que3, I, (dat) values ('foo'), public.nopkey) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update nopkey set dat = 'bat'; ERROR: Update/Delete on table without pkey delete from nopkey; ERROR: Update/Delete on table without pkey -- test custom pkey create table custom_pkey (dat1 text not null, dat2 int2 not null, dat3 text); create trigger custom_triga after insert or update or delete on custom_pkey for each row execute procedure pgq.sqltriga('que3', 'pkey=dat1,dat2'); insert into custom_pkey values ('foo', '2'); NOTICE: insert_event(que3, I, (dat1,dat2,dat3) values ('foo','2',null), public.custom_pkey) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" update custom_pkey set dat3 = 'bat'; NOTICE: insert_event(que3, U, dat3='bat' where dat1='foo' and dat2='2', public.custom_pkey) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" delete from custom_pkey; NOTICE: insert_event(que3, D, dat1='foo' and dat2='2', public.custom_pkey) CONTEXT: SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, null, null)" skytools-2.1.13/sql/pgq/structure/0000755000175000017500000000000011727601174016125 5ustar markomarkoskytools-2.1.13/sql/pgq/structure/grants.sql0000644000175000017500000000063511670174255020152 0ustar markomarko grant usage on schema pgq to public; grant select on table pgq.consumer to public; grant select on table pgq.queue to public; grant select on table pgq.tick to public; grant select on table pgq.queue to public; grant select on table pgq.subscription to public; grant select on table pgq.event_template to public; grant select on table pgq.retry_queue to public; grant select on table pgq.failed_queue to public; skytools-2.1.13/sql/pgq/structure/tables.sql0000644000175000017500000002327011670174255020126 0ustar markomarko-- ---------------------------------------------------------------------- -- Section: Internal Tables -- -- Overview: -- pgq.queue - Queue configuration -- pgq.consumer - Consumer names -- pgq.subscription - Consumer registrations -- pgq.tick - Per-queue snapshots (ticks) -- pgq.event_* - Data tables -- pgq.retry_queue - Events to be retried later -- pgq.failed_queue - Events whose processing failed -- -- Its basically generalized and simplified Slony-I structure: -- sl_node - pgq.consumer -- sl_set - pgq.queue -- sl_subscriber + sl_confirm - pgq.subscription -- sl_event - pgq.tick -- sl_setsync - pgq_ext.completed_* -- sl_log_* - slony1 has per-cluster data tables, -- pgq has per-queue data tables. -- ---------------------------------------------------------------------- set client_min_messages = 'warning'; set default_with_oids = 'off'; -- drop schema if exists pgq cascade; create schema pgq; -- ---------------------------------------------------------------------- -- Table: pgq.consumer -- -- Name to id lookup for consumers -- -- Columns: -- co_id - consumer's id for internal usage -- co_name - consumer's id for external usage -- ---------------------------------------------------------------------- create table pgq.consumer ( co_id serial, co_name text not null default 'fooz', constraint consumer_pkey primary key (co_id), constraint consumer_name_uq UNIQUE (co_name) ); -- ---------------------------------------------------------------------- -- Table: pgq.queue -- -- Information about available queues -- -- Columns: -- queue_id - queue id for internal usage -- queue_name - queue name visible outside -- queue_ntables - how many data tables the queue has -- queue_cur_table - which data table is currently active -- queue_rotation_period - period for data table rotation -- queue_switch_step1 - tx when rotation happened -- queue_switch_step2 - tx after rotation was committed -- queue_switch_time - time when switch happened -- queue_external_ticker - ticks come from some external sources -- queue_ticker_max_count - batch should not contain more events -- queue_ticker_max_lag - events should not age more -- queue_ticker_idle_period - how often to tick when no events happen -- queue_data_pfx - prefix for data table names -- queue_event_seq - sequence for event id's -- queue_tick_seq - sequence for tick id's -- ---------------------------------------------------------------------- create table pgq.queue ( queue_id serial, queue_name text not null, queue_ntables integer not null default 3, queue_cur_table integer not null default 0, queue_rotation_period interval not null default '2 hours', queue_switch_step1 bigint not null default txid_current(), queue_switch_step2 bigint default txid_current(), queue_switch_time timestamptz not null default now(), queue_external_ticker boolean not null default false, queue_ticker_max_count integer not null default 500, queue_ticker_max_lag interval not null default '3 seconds', queue_ticker_idle_period interval not null default '1 minute', queue_data_pfx text not null, queue_event_seq text not null, queue_tick_seq text not null, constraint queue_pkey primary key (queue_id), constraint queue_name_uq unique (queue_name) ); -- ---------------------------------------------------------------------- -- Table: pgq.tick -- -- Snapshots for event batching -- -- Columns: -- tick_queue - queue id whose tick it is -- tick_id - ticks id (per-queue) -- tick_time - time when tick happened -- tick_snapshot - transaction state -- ---------------------------------------------------------------------- create table pgq.tick ( tick_queue int4 not null, tick_id bigint not null, tick_time timestamptz not null default now(), tick_snapshot txid_snapshot not null default txid_current_snapshot(), constraint tick_pkey primary key (tick_queue, tick_id), constraint tick_queue_fkey foreign key (tick_queue) references pgq.queue (queue_id) ); -- ---------------------------------------------------------------------- -- Sequence: pgq.batch_id_seq -- -- Sequence for batch id's. -- ---------------------------------------------------------------------- create sequence pgq.batch_id_seq; -- ---------------------------------------------------------------------- -- Table: pgq.subscription -- -- Consumer registration on a queue. -- -- Columns: -- -- sub_id - subscription id for internal usage -- sub_queue - queue id -- sub_consumer - consumer's id -- sub_last_tick - last tick the consumer processed -- sub_batch - shortcut for queue_id/consumer_id/tick_id -- sub_next_tick - batch end pos -- ---------------------------------------------------------------------- create table pgq.subscription ( sub_id serial not null, sub_queue int4 not null, sub_consumer int4 not null, sub_last_tick bigint not null, sub_active timestamptz not null default now(), sub_batch bigint, sub_next_tick bigint, constraint subscription_pkey primary key (sub_id), constraint subscription_ukey unique (sub_queue, sub_consumer), constraint sub_queue_fkey foreign key (sub_queue) references pgq.queue (queue_id), constraint sub_consumer_fkey foreign key (sub_consumer) references pgq.consumer (co_id) ); -- ---------------------------------------------------------------------- -- Table: pgq.event_template -- -- Parent table for all event tables -- -- Columns: -- ev_id - event's id, supposed to be unique per queue -- ev_time - when the event was inserted -- ev_txid - transaction id which inserted the event -- ev_owner - subscription id that wanted to retry this -- ev_retry - how many times the event has been retried, NULL for new events -- ev_type - consumer/producer can specify what the data fields contain -- ev_data - data field -- ev_extra1 - extra data field -- ev_extra2 - extra data field -- ev_extra3 - extra data field -- ev_extra4 - extra data field -- ---------------------------------------------------------------------- create table pgq.event_template ( ev_id bigint not null, ev_time timestamptz not null, ev_txid bigint not null default txid_current(), ev_owner int4, ev_retry int4, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text ); -- ---------------------------------------------------------------------- -- Table: pgq.retry_queue -- -- Events to be retried. When retry time reaches, they will -- be put back into main queue. -- -- Columns: -- ev_retry_after - time when it should be re-inserted to main queue -- * - same as pgq.event_template -- ---------------------------------------------------------------------- create table pgq.retry_queue ( ev_retry_after timestamptz not null, like pgq.event_template, constraint rq_pkey primary key (ev_owner, ev_id), constraint rq_owner_fkey foreign key (ev_owner) references pgq.subscription (sub_id) ); alter table pgq.retry_queue alter column ev_owner set not null; alter table pgq.retry_queue alter column ev_txid drop not null; create index rq_retry_idx on pgq.retry_queue (ev_retry_after); create index rq_retry_owner_idx on pgq.retry_queue (ev_owner, ev_id); -- ---------------------------------------------------------------------- -- Table: pgq.failed_queue -- -- Events whose processing failed. -- -- Columns: -- ev_failed_reason - consumer's excuse for not processing -- ev_failed_time - when it was tagged failed -- * - same as pgq.event_template -- ---------------------------------------------------------------------- create table pgq.failed_queue ( ev_failed_reason text, ev_failed_time timestamptz not null, -- all event fields like pgq.event_template, constraint fq_pkey primary key (ev_owner, ev_id), constraint fq_owner_fkey foreign key (ev_owner) references pgq.subscription (sub_id) ); alter table pgq.failed_queue alter column ev_owner set not null; alter table pgq.failed_queue alter column ev_txid drop not null; skytools-2.1.13/sql/pgq/structure/install.sql0000644000175000017500000000023711670174255020320 0ustar markomarko \i structure/tables.sql \i structure/grants.sql \i structure/types.sql \i structure/func_internal.sql \i structure/func_public.sql \i structure/triggers.sql skytools-2.1.13/sql/pgq/structure/func_public.sql0000644000175000017500000000141711670174255021144 0ustar markomarko-- Section: Public Functions -- Group: Queue creation \i functions/pgq.create_queue.sql \i functions/pgq.drop_queue.sql -- Group: Event publishing \i functions/pgq.insert_event.sql \i functions/pgq.current_event_table.sql -- Group: Subscribing to queue \i functions/pgq.register_consumer.sql \i functions/pgq.unregister_consumer.sql -- Group: Batch processing \i functions/pgq.next_batch.sql \i functions/pgq.get_batch_events.sql \i functions/pgq.event_failed.sql \i functions/pgq.event_retry.sql \i functions/pgq.finish_batch.sql -- Group: General info functions \i functions/pgq.get_queue_info.sql \i functions/pgq.get_consumer_info.sql \i functions/pgq.version.sql \i functions/pgq.get_batch_info.sql -- Group: Failed queue browsing \i functions/pgq.failed_queue.sql skytools-2.1.13/sql/pgq/structure/types.sql0000644000175000017500000000233411670174255020016 0ustar markomarko create type pgq.ret_queue_info as ( queue_name text, queue_ntables integer, queue_cur_table integer, queue_rotation_period interval, queue_switch_time timestamptz, queue_external_ticker boolean, queue_ticker_max_count integer, queue_ticker_max_lag interval, queue_ticker_idle_period interval, ticker_lag interval ); create type pgq.ret_consumer_info as ( queue_name text, consumer_name text, lag interval, last_seen interval, last_tick bigint, current_batch bigint, next_tick bigint ); create type pgq.ret_batch_info as ( queue_name text, consumer_name text, batch_start timestamptz, batch_end timestamptz, prev_tick_id bigint, tick_id bigint, lag interval ); create type pgq.ret_batch_event as ( ev_id bigint, ev_time timestamptz, ev_txid bigint, ev_retry int4, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text ); skytools-2.1.13/sql/pgq/structure/uninstall_pgq.sql0000644000175000017500000000006411670174255021530 0ustar markomarko -- brute-force uninstall drop schema pgq cascade; skytools-2.1.13/sql/pgq/structure/triggers.sql0000644000175000017500000000017211670174255020476 0ustar markomarko -- Section: Public Triggers -- Group: Trigger Functions -- \i triggers/pgq.logutriga.sql \i triggers/pgq_triggers.sql skytools-2.1.13/sql/pgq/structure/func_internal.sql0000644000175000017500000000106111670174255021475 0ustar markomarko-- Section: Internal Functions -- Group: Low-level event handling \i functions/pgq.batch_event_sql.sql \i functions/pgq.batch_event_tables.sql \i functions/pgq.event_retry_raw.sql -- \i functions/pgq.insert_event_raw.sql \i lowlevel/pgq_lowlevel.sql -- Group: Ticker \i functions/pgq.ticker.sql -- Group: Periodic maintenence \i functions/pgq.maint_retry_events.sql \i functions/pgq.maint_rotate_tables.sql \i functions/pgq.maint_tables_to_vacuum.sql -- Group: Random utility functions \i functions/pgq.grant_perms.sql \i functions/pgq.force_tick.sql skytools-2.1.13/sql/pgq/triggers/0000755000175000017500000000000011727601174015713 5ustar markomarkoskytools-2.1.13/sql/pgq/triggers/logutriga.c0000644000175000017500000000660211727577267020077 0ustar markomarko/* * logutriga.c - Smart trigger that logs urlencoded changes. * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include #include #include #include #include #include "common.h" #include "stringutil.h" PG_FUNCTION_INFO_V1(pgq_logutriga); Datum pgq_logutriga(PG_FUNCTION_ARGS); void pgq_urlenc_row(PgqTriggerEvent *ev, TriggerData *tg, HeapTuple row, StringInfo buf) { TupleDesc tupdesc = tg->tg_relation->rd_att; bool first = true; int i; const char *col_ident, *col_value; int attkind_idx = -1; for (i = 0; i < tg->tg_relation->rd_att->natts; i++) { /* Skip dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (pgqtriga_skip_col(ev, tg, i, attkind_idx)) continue; if (first) first = false; else appendStringInfoChar(buf, '&'); /* quote column name */ col_ident = SPI_fname(tupdesc, i + 1); pgq_encode_cstring(buf, col_ident, TBUF_QUOTE_URLENC); /* quote column value */ col_value = SPI_getvalue(row, tupdesc, i + 1); if (col_value != NULL) { appendStringInfoChar(buf, '='); pgq_encode_cstring(buf, col_value, TBUF_QUOTE_URLENC); } } } /* * PgQ log trigger, takes 2 arguments: * 1. queue name to be inserted to. * * Queue events will be in format: * ev_type - operation type, I/U/D * ev_data - urlencoded column values * ev_extra1 - table name * ev_extra2 - optional urlencoded backup */ Datum pgq_logutriga(PG_FUNCTION_ARGS) { TriggerData *tg; struct PgqTriggerEvent ev; HeapTuple row; /* * Get the trigger call context */ if (!CALLED_AS_TRIGGER(fcinfo)) elog(ERROR, "pgq.logutriga not called as trigger"); tg = (TriggerData *) (fcinfo->context); if (TRIGGER_FIRED_BY_UPDATE(tg->tg_event)) row = tg->tg_newtuple; else row = tg->tg_trigtuple; if (pgq_is_logging_disabled()) goto skip_it; /* * Connect to the SPI manager */ if (SPI_connect() < 0) elog(ERROR, "logtriga: SPI_connect() failed"); pgq_prepare_event(&ev, tg, true); appendStringInfoChar(ev.ev_type, ev.op_type); appendStringInfoChar(ev.ev_type, ':'); appendStringInfoString(ev.ev_type, ev.pkey_list); appendStringInfoString(ev.ev_extra1, ev.info->table_name); /* * create type, data */ pgq_urlenc_row(&ev, tg, row, ev.ev_data); /* * Construct the parameter array and insert the log row. */ pgq_insert_tg_event(&ev); if (SPI_finish() < 0) elog(ERROR, "SPI_finish failed"); /* * After trigger ignores result, * before trigger skips event if NULL. */ skip_it: if (TRIGGER_FIRED_AFTER(tg->tg_event) || ev.skip) return PointerGetDatum(NULL); else return PointerGetDatum(row); } skytools-2.1.13/sql/pgq/triggers/pgq_triggers.sql.in0000644000175000017500000000752511670174255021551 0ustar markomarko -- ---------------------------------------------------------------------- -- Function: pgq.logtriga() -- -- Deprecated - non-automatic SQL trigger. It puts row data in partial -- SQL form into queue. It does not auto-detect table structure, -- it needs to be passed as trigger arg. -- -- Purpose: -- Used by Londiste to generate replication events. The "partial SQL" -- format is more compact than the urlencoded format but cannot be -- parsed, only applied. Which is fine for Londiste. -- -- Parameters: -- arg1 - queue name -- arg2 - column type spec string where each column corresponds to one char (k/v/i). -- if spec string is shorter than column list, rest of columns default to 'i'. -- -- Column types: -- k - pkey column -- v - normal data column -- i - ignore column -- -- Queue event fields: -- ev_type - I/U/D -- ev_data - partial SQL statement -- ev_extra1 - table name -- -- ---------------------------------------------------------------------- CREATE OR REPLACE FUNCTION pgq.logtriga() RETURNS trigger AS 'MODULE_PATHNAME', 'pgq_logtriga' LANGUAGE C; -- ---------------------------------------------------------------------- -- Function: pgq.sqltriga() -- -- Automatic SQL trigger. It puts row data in partial SQL form into -- queue. It autodetects table structure. -- -- Purpose: -- Written as more flexible version of logtriga to handle exceptional cases -- where there is no primary key index on table etc. -- -- Parameters: -- arg1 - queue name -- argX - any number of optional arg, in any order -- -- Optinal arguments: -- SKIP - The actual operation should be skipped -- ignore=col1[,col2] - don't look at the specified arguments -- pkey=col1[,col2] - Set pkey fields for the table, autodetection will be skipped -- backup - Put urlencoded contents of old row to ev_extra2 -- -- Queue event fields: -- ev_type - I/U/D -- ev_data - partial SQL statement -- ev_extra1 - table name -- ev_extra2 - optional urlencoded backup -- -- ---------------------------------------------------------------------- CREATE OR REPLACE FUNCTION pgq.sqltriga() RETURNS trigger AS 'MODULE_PATHNAME', 'pgq_sqltriga' LANGUAGE C; -- ---------------------------------------------------------------------- -- Function: pgq.logutriga() -- -- Trigger function that puts row data in urlencoded form into queue. -- -- Purpose: -- Used as producer for several PgQ standard consumers (cube_dispatcher, -- queue_mover, table_dispatcher). Basically for cases where the -- consumer wants to parse the event and look at the actual column values. -- -- Trigger parameters: -- arg1 - queue name -- argX - any number of optional arg, in any order -- -- Optinal arguments: -- SKIP - The actual operation should be skipped -- ignore=col1[,col2] - don't look at the specified arguments -- pkey=col1[,col2] - Set pkey fields for the table, autodetection will be skipped -- backup - Put urlencoded contents of old row to ev_extra2 -- -- Queue event fields: -- ev_type - I/U/D ':' pkey_column_list -- ev_data - column values urlencoded -- ev_extra1 - table name -- ev_extra2 - optional urlencoded backup -- -- Regular listen trigger example: -- > CREATE TRIGGER triga_nimi AFTER INSERT OR UPDATE ON customer -- > FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('qname'); -- -- Redirect trigger example: -- > CREATE TRIGGER triga_nimi BEFORE INSERT OR UPDATE ON customer -- > FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('qname', 'SKIP'); -- ---------------------------------------------------------------------- CREATE OR REPLACE FUNCTION pgq.logutriga() RETURNS TRIGGER AS 'MODULE_PATHNAME', 'pgq_logutriga' LANGUAGE C; skytools-2.1.13/sql/pgq/triggers/logtriga.c0000644000175000017500000000424411670174255017675 0ustar markomarko/* * logtriga.c - Dumb SQL logging trigger. * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include #include #include #include #include "common.h" #include "stringutil.h" PG_FUNCTION_INFO_V1(pgq_logtriga); Datum pgq_logtriga(PG_FUNCTION_ARGS); /* * PGQ log trigger, takes 2 arguments: * 1. queue name to be inserted to. * 2. column type string * * Queue events will be in format: * ev_type - operation type, I/U/D * ev_data - partial SQL describing operation * ev_extra1 - table name */ Datum pgq_logtriga(PG_FUNCTION_ARGS) { TriggerData *tg; PgqTriggerEvent ev; /* * Get the trigger call context */ if (!CALLED_AS_TRIGGER(fcinfo)) elog(ERROR, "pgq.logutriga not called as trigger"); tg = (TriggerData *) (fcinfo->context); if (!TRIGGER_FIRED_AFTER(tg->tg_event)) elog(ERROR, "pgq.logtriga must be fired AFTER"); if (pgq_is_logging_disabled()) goto skip_it; /* * Connect to the SPI manager */ if (SPI_connect() < 0) elog(ERROR, "logtriga: SPI_connect() failed"); pgq_prepare_event(&ev, tg, false); appendStringInfoChar(ev.ev_type, ev.op_type); appendStringInfoString(ev.ev_extra1, ev.info->table_name); /* * create sql and insert if interesting */ if (pgqtriga_make_sql(&ev, tg, ev.ev_data)) pgq_insert_tg_event(&ev); if (SPI_finish() < 0) elog(ERROR, "SPI_finish failed"); skip_it: return PointerGetDatum(NULL); } skytools-2.1.13/sql/pgq/triggers/Makefile0000644000175000017500000000035111670174255017354 0ustar markomarko include ../../../config.mak MODULE_big = pgq_triggers SRCS = common.c logtriga.c logutriga.c sqltriga.c makesql.c stringutil.c OBJS = $(SRCS:.c=.o) DATA_built = pgq_triggers.sql include $(PGXS) cs: cscope -b -f .cscope.out *.c skytools-2.1.13/sql/pgq/triggers/common.c0000644000175000017500000002733011727577235017366 0ustar markomarko/* * common.c - functions used by all trigger variants. * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include #include #include #include #include #include #include #include #include #include #include #include #include "common.h" #include "stringutil.h" /* * Module tag */ #ifdef PG_MODULE_MAGIC PG_MODULE_MAGIC; #endif /* * primary key info */ static bool tbl_cache_invalid; static MemoryContext tbl_cache_ctx; static HTAB *tbl_cache_map; static const char pkey_sql [] = "SELECT k.attnum, k.attname FROM pg_index i, pg_attribute k" " WHERE i.indrelid = $1 AND k.attrelid = i.indexrelid" " AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped" " ORDER BY k.attnum"; static void *pkey_plan; static void relcache_reset_cb(Datum arg, Oid relid); /* * helper for queue insertion. * * does not support NULL arguments. */ void pgq_simple_insert(const char *queue_name, Datum ev_type, Datum ev_data, Datum ev_extra1, Datum ev_extra2) { Datum values[5]; char nulls[5]; static void *plan = NULL; int res; if (!plan) { const char *sql; Oid types[5] = { TEXTOID, TEXTOID, TEXTOID, TEXTOID, TEXTOID }; sql = "select pgq.insert_event($1, $2, $3, $4, $5, null, null)"; plan = SPI_saveplan(SPI_prepare(sql, 5, types)); if (plan == NULL) elog(ERROR, "logtriga: SPI_prepare() failed"); } values[0] = DirectFunctionCall1(textin, (Datum)queue_name); values[1] = ev_type; values[2] = ev_data; values[3] = ev_extra1; values[4] = ev_extra2; nulls[0] = ' '; nulls[1] = ' '; nulls[2] = ' '; nulls[3] = ' '; nulls[4] = ev_extra2 ? ' ' : 'n'; res = SPI_execute_plan(plan, values, nulls, false, 0); if (res != SPI_OK_SELECT) elog(ERROR, "call of pgq.insert_event failed"); } void pgq_insert_tg_event(PgqTriggerEvent *ev) { pgq_simple_insert(ev->queue_name, pgq_finish_varbuf(ev->ev_type), pgq_finish_varbuf(ev->ev_data), pgq_finish_varbuf(ev->ev_extra1), ev->ev_extra2 ? pgq_finish_varbuf(ev->ev_extra2) : (Datum)0); } char *pgq_find_table_name(Relation rel) { NameData tname = rel->rd_rel->relname; Oid nsoid = rel->rd_rel->relnamespace; char namebuf[NAMEDATALEN * 2 + 3]; HeapTuple ns_tup; Form_pg_namespace ns_struct; NameData nspname; /* find namespace info */ ns_tup = SearchSysCache(NAMESPACEOID, ObjectIdGetDatum(nsoid), 0, 0, 0); if (!HeapTupleIsValid(ns_tup)) elog(ERROR, "Cannot find namespace %u", nsoid); ns_struct = (Form_pg_namespace) GETSTRUCT(ns_tup); nspname = ns_struct->nspname; /* fill name */ sprintf(namebuf, "%s.%s", NameStr(nspname), NameStr(tname)); ReleaseSysCache(ns_tup); return pstrdup(namebuf); } static void init_pkey_plan(void) { Oid types[1] = { OIDOID }; pkey_plan = SPI_saveplan(SPI_prepare(pkey_sql, 1, types)); if (pkey_plan == NULL) elog(ERROR, "pgq_triggers: SPI_prepare() failed"); } static void init_cache(void) { HASHCTL ctl; int flags; int max_tables = 128; /* * create own context */ tbl_cache_ctx = AllocSetContextCreate(TopMemoryContext, "pgq_triggers table info", ALLOCSET_SMALL_MINSIZE, ALLOCSET_SMALL_INITSIZE, ALLOCSET_SMALL_MAXSIZE); /* * init pkey cache. */ MemSet(&ctl, 0, sizeof(ctl)); ctl.keysize = sizeof(Oid); ctl.entrysize = sizeof(struct PgqTableInfo); ctl.hash = oid_hash; flags = HASH_ELEM | HASH_FUNCTION; tbl_cache_map = hash_create("pgq_triggers pkey cache", max_tables, &ctl, flags); } /* * Prepare utility plans and plan cache. */ static void init_module(void) { static int callback_init = 0; /* do full reset if requested */ if (tbl_cache_invalid) { if (tbl_cache_map) hash_destroy(tbl_cache_map); if (tbl_cache_ctx) MemoryContextDelete(tbl_cache_ctx); tbl_cache_map = NULL; tbl_cache_ctx = NULL; tbl_cache_invalid = false; } /* htab can be occasinally dropped */ if (tbl_cache_ctx) return; init_cache(); /* * Init plans. */ if (!pkey_plan) init_pkey_plan(); /* this must be done only once */ if (!callback_init) { CacheRegisterRelcacheCallback(relcache_reset_cb, (Datum)0); callback_init = 1; } } /* * Fill table information in hash table. */ static void fill_tbl_info(Relation rel, struct PgqTableInfo *info) { StringInfo pkeys; Datum values[1]; const char *name = pgq_find_table_name(rel); TupleDesc desc; HeapTuple row; bool isnull; int res, i, attno; /* load pkeys */ values[0] = ObjectIdGetDatum(rel->rd_id); res = SPI_execute_plan(pkey_plan, values, NULL, false, 0); if (res != SPI_OK_SELECT) elog(ERROR, "pkey_plan exec failed"); /* * Fill info */ desc = SPI_tuptable->tupdesc; pkeys = makeStringInfo(); info->n_pkeys = SPI_processed; info->table_name = MemoryContextStrdup(tbl_cache_ctx, name); info->pkey_attno = MemoryContextAlloc(tbl_cache_ctx, info->n_pkeys * sizeof(int)); for (i = 0; i < SPI_processed; i++) { row = SPI_tuptable->vals[i]; attno = DatumGetInt16(SPI_getbinval(row, desc, 1, &isnull)); name = SPI_getvalue(row, desc, 2); info->pkey_attno[i] = attno; if (i > 0) appendStringInfoChar(pkeys, ','); appendStringInfoString(pkeys, name); } info->pkey_list = MemoryContextStrdup(tbl_cache_ctx, pkeys->data); } static void clean_info(struct PgqTableInfo *info, bool found) { if (!found) goto uninitialized; if (info->table_name) pfree(info->table_name); if (info->pkey_attno) pfree(info->pkey_attno); if (info->pkey_list) pfree((void *)info->pkey_list); uninitialized: info->table_name = NULL; info->pkey_attno = NULL; info->pkey_list = NULL; info->n_pkeys = 0; } /* * the callback can be launched any time from signal callback, * only minimal tagging can be done here. */ static void relcache_reset_cb(Datum arg, Oid relid) { if (relid == InvalidOid) { tbl_cache_invalid = true; } else if (tbl_cache_map && !tbl_cache_invalid) { struct PgqTableInfo *entry; entry = hash_search(tbl_cache_map, &relid, HASH_FIND, NULL); if (entry) entry->invalid = true; } } /* * fetch insert plan from cache. */ struct PgqTableInfo * pgq_find_table_info(Relation rel) { struct PgqTableInfo *entry; bool found = false; init_module(); entry = hash_search(tbl_cache_map, &rel->rd_id, HASH_ENTER, &found); if (!found || entry->invalid) { clean_info(entry, found); /* * During fill_tbl_info() 2 events can happen: * - table info reset * - exception * To survive both, always clean struct and tag * as invalid but differently from reset. */ entry->invalid = 2; /* find info */ fill_tbl_info(rel, entry); /* * If no reset happened, it's valid. Actual reset * is postponed to next call. */ if (entry->invalid == 2) entry->invalid = false; } return entry; } static void parse_newstyle_args(PgqTriggerEvent *ev, TriggerData *tg) { int i; /* * parse args */ ev->skip = false; ev->queue_name = tg->tg_trigger->tgargs[0]; for (i = 1; i < tg->tg_trigger->tgnargs; i++) { const char *arg = tg->tg_trigger->tgargs[i]; if (strcmp(arg, "SKIP") == 0) ev->skip = true; else if (strncmp(arg, "ignore=", 7) == 0) ev->ignore_list = arg + 7; else if (strncmp(arg, "pkey=", 5) == 0) ev->pkey_list = arg + 5; else if (strcmp(arg, "backup") == 0) ev->backup = true; else elog(ERROR, "bad param to pgq trigger"); } /* * Check if we have pkey */ if (ev->op_type == 'U' || ev->op_type == 'D') { if (ev->pkey_list[0] == 0) elog(ERROR, "Update/Delete on table without pkey"); } } static void parse_oldstyle_args(PgqTriggerEvent *ev, TriggerData *tg) { const char *kpos; int attcnt, i; TupleDesc tupdesc = tg->tg_relation->rd_att; ev->skip = false; if (tg->tg_trigger->tgnargs < 2 || tg->tg_trigger->tgnargs > 3) elog(ERROR, "pgq.logtriga must be used with 2 or 3 args"); ev->queue_name = tg->tg_trigger->tgargs[0]; ev->attkind = tg->tg_trigger->tgargs[1]; ev->attkind_len = strlen(ev->attkind); if (tg->tg_trigger->tgnargs > 2) ev->table_name = tg->tg_trigger->tgargs[2]; /* * Count number of active columns */ tupdesc = tg->tg_relation->rd_att; for (i = 0, attcnt = 0; i < tupdesc->natts; i++) { if (!tupdesc->attrs[i]->attisdropped) attcnt++; } /* * look if last pkey column exists */ kpos = strrchr(ev->attkind, 'k'); if (kpos == NULL) elog(ERROR, "need at least one key column"); if (kpos - ev->attkind >= attcnt) elog(ERROR, "key column does not exist"); } /* * parse trigger arguments. */ void pgq_prepare_event(struct PgqTriggerEvent *ev, TriggerData *tg, bool newstyle) { memset(ev, 0, sizeof(*ev)); /* * Check trigger calling conventions */ if (!TRIGGER_FIRED_AFTER(tg->tg_event)) /* dont care */; if (!TRIGGER_FIRED_FOR_ROW(tg->tg_event)) elog(ERROR, "pgq trigger must be fired FOR EACH ROW"); if (tg->tg_trigger->tgnargs < 1) elog(ERROR, "pgq trigger must have destination queue as argument"); /* * check operation type */ if (TRIGGER_FIRED_BY_INSERT(tg->tg_event)) ev->op_type = 'I'; else if (TRIGGER_FIRED_BY_UPDATE(tg->tg_event)) ev->op_type = 'U'; else if (TRIGGER_FIRED_BY_DELETE(tg->tg_event)) ev->op_type = 'D'; else elog(ERROR, "unknown event for pgq trigger"); /* * load table info */ ev->info = pgq_find_table_info(tg->tg_relation); ev->table_name = ev->info->table_name; ev->pkey_list = ev->info->pkey_list; /* * parse args */ if (newstyle) parse_newstyle_args(ev, tg); else parse_oldstyle_args(ev, tg); /* * init data */ ev->ev_type = pgq_init_varbuf(); ev->ev_data = pgq_init_varbuf(); ev->ev_extra1 = pgq_init_varbuf(); /* * Do the backup, if requested. */ if (ev->backup) { ev->ev_extra2 = pgq_init_varbuf(); pgq_urlenc_row(ev, tg, tg->tg_trigtuple, ev->ev_extra2); } } /* * Check if column should be skipped */ bool pgqtriga_skip_col(PgqTriggerEvent *ev, TriggerData *tg, int i, int attkind_idx) { TupleDesc tupdesc; const char *name; if (ev->attkind) { if (attkind_idx >= ev->attkind_len) return true; return ev->attkind[attkind_idx] == 'i'; } else if (ev->ignore_list) { tupdesc = tg->tg_relation->rd_att; if (tupdesc->attrs[i]->attisdropped) return true; name = NameStr(tupdesc->attrs[i]->attname); return pgq_strlist_contains(ev->ignore_list, name); } return false; } /* * Check if column is pkey. */ bool pgqtriga_is_pkey(PgqTriggerEvent *ev, TriggerData *tg, int i, int attkind_idx) { TupleDesc tupdesc; const char *name; if (ev->attkind) { if (attkind_idx >= ev->attkind_len) return false; return ev->attkind[attkind_idx] == 'k'; } else if (ev->pkey_list) { tupdesc = tg->tg_relation->rd_att; if (tupdesc->attrs[i]->attisdropped) return false; name = NameStr(tupdesc->attrs[i]->attname); return pgq_strlist_contains(ev->pkey_list, name); } return false; } /* * Check if trigger action should be skipped. */ bool pgq_is_logging_disabled(void) { #if defined(PG_VERSION_NUM) && PG_VERSION_NUM >= 80300 if (SessionReplicationRole != SESSION_REPLICATION_ROLE_ORIGIN) return true; #endif return false; } skytools-2.1.13/sql/pgq/triggers/stringutil.c0000644000175000017500000001401511670174255020266 0ustar markomarko/* * stringutil.c - some tools for string handling * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include #include #include #include #include #include "stringutil.h" #ifndef SET_VARSIZE #define SET_VARSIZE(x, len) VARATT_SIZEP(x) = len #endif StringInfo pgq_init_varbuf(void) { StringInfo buf; buf = makeStringInfo(); appendStringInfoString(buf, "XXXX"); return buf; } Datum pgq_finish_varbuf(StringInfo buf) { SET_VARSIZE(buf->data, buf->len); return PointerGetDatum(buf->data); } /* * Find a string in comma-separated list. * * It does not support space inside tokens. */ bool pgq_strlist_contains(const char *liststr, const char *str) { int c, len = strlen(str); const char *p, *listpos = liststr; loop: /* find string fragment, later check if actual token */ p = strstr(listpos, str); if (p == NULL) return false; /* move listpos further */ listpos = p + len; /* survive len=0 and avoid unneccesary compare */ if (*listpos) listpos++; /* check previous symbol */ if (p > liststr) { c = *(p - 1); if (!isspace(c) && c != ',') goto loop; } /* check following symbol */ c = p[len]; if (c != 0 && !isspace(c) && c != ',') goto loop; return true; } /* * quoting */ static int pgq_urlencode(char *dst, const uint8 *src, int srclen) { static const char hextbl[] = "0123456789abcdef"; const uint8 *end = src + srclen; char *p = dst; while (src < end) { unsigned c = *src++; if (c == ' ') { *p++ = '+'; } else if ((c >= '0' && c <= '9') || (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || c == '_' || c == '.') { *p++ = c; } else { *p++ = '%'; *p++ = hextbl[c >> 4]; *p++ = hextbl[c & 15]; } } return p - dst; } static int pgq_quote_literal(char *dst, const uint8 *src, int srclen) { const uint8 *cp1 = src, *src_end = src + srclen; char *cp2 = dst; bool is_ext = false; *cp2++ = '\''; while (cp1 < src_end) { int wl = pg_mblen((const char *)cp1); if (wl != 1) { while (wl-- > 0 && cp1 < src_end) *cp2++ = *cp1++; continue; } if (*cp1 == '\'') { *cp2++ = '\''; } else if (*cp1 == '\\') { if (!is_ext) { /* make room for 'E' */ memmove(dst + 1, dst, cp2 - dst); *dst = 'E'; is_ext = true; cp2++; } *cp2++ = '\\'; } *cp2++ = *cp1++; } *cp2++ = '\''; return cp2 - dst; } /* * slon_quote_identifier - Quote an identifier only if needed */ static int pgq_quote_ident(char *dst, const uint8 *src, int srclen) { /* * Can avoid quoting if ident starts with a lowercase letter or * underscore and contains only lowercase letters, digits, and * underscores, *and* is not any SQL keyword. Otherwise, supply * quotes. */ int nquotes = 0; bool safe; const char *ptr; char *optr; char ident[NAMEDATALEN + 1]; /* expect idents be not bigger than NAMEDATALEN */ if (srclen > NAMEDATALEN) srclen = NAMEDATALEN; memcpy(ident, src, srclen); ident[srclen] = 0; /* * would like to use macros here, but they might yield * unwanted locale-specific results... */ safe = ((ident[0] >= 'a' && ident[0] <= 'z') || ident[0] == '_'); for (ptr = ident; *ptr; ptr++) { char ch = *ptr; if ((ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9') || (ch == '_')) continue; /* okay */ safe = false; if (ch == '"') nquotes++; } if (safe) { /* * Check for keyword. This test is overly strong, since many of * the "keywords" known to the parser are usable as column names, * but the parser doesn't provide any easy way to test for whether * an identifier is safe or not... so be safe not sorry. * * Note: ScanKeywordLookup() does case-insensitive comparison, but * that's fine, since we already know we have all-lower-case. */ #if defined(PG_VERSION_NUM) && PG_VERSION_NUM >= 90000 if (ScanKeywordLookup(ident, ScanKeywords, NumScanKeywords) != NULL) #else if (ScanKeywordLookup(ident) != NULL) #endif safe = false; } optr = dst; if (!safe) *optr++ = '"'; for (ptr = ident; *ptr; ptr++) { char ch = *ptr; if (ch == '"') *optr++ = '"'; *optr++ = ch; } if (!safe) *optr++ = '"'; return optr - dst; } static char *start_append(StringInfo buf, int alloc_len) { enlargeStringInfo(buf, alloc_len); return buf->data + buf->len; } static void finish_append(StringInfo buf, int final_len) { if (buf->len + final_len > buf->maxlen) elog(FATAL, "buffer overflow"); buf->len += final_len; } static void tbuf_encode_data(StringInfo buf, const uint8 *data, int len, enum PgqEncode encoding) { int dlen = 0; char *dst; switch (encoding) { case TBUF_QUOTE_LITERAL: dst = start_append(buf, len * 2 + 3); dlen = pgq_quote_literal(dst, data, len); break; case TBUF_QUOTE_IDENT: dst = start_append(buf, len * 2 + 2); dlen = pgq_quote_ident(dst, data, len); break; case TBUF_QUOTE_URLENC: dst = start_append(buf, len * 3 + 2); dlen = pgq_urlencode(dst, data, len); break; default: elog(ERROR, "bad encoding"); } finish_append(buf, dlen); } void pgq_encode_cstring(StringInfo tbuf, const char *str, enum PgqEncode encoding) { if (str == NULL) elog(ERROR, "tbuf_encode_cstring: NULL"); tbuf_encode_data(tbuf, (const uint8 *)str, strlen(str), encoding); } skytools-2.1.13/sql/pgq/triggers/makesql.c0000644000175000017500000002066411727577306017535 0ustar markomarko/* * makesql.c - generate partial SQL statement for row change. * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Based on Slony-I log trigger: * * Copyright (c) 2003-2006, PostgreSQL Global Development Group * Author: Jan Wieck, Afilias USA INC. * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include #include #include #include #include #include #include #include "common.h" #include "stringutil.h" static void append_key_eq(StringInfo buf, const char *col_ident, const char *col_value) { if (col_value == NULL) elog(ERROR, "logtriga: Unexpected NULL key value"); pgq_encode_cstring(buf, col_ident, TBUF_QUOTE_IDENT); appendStringInfoChar(buf, '='); pgq_encode_cstring(buf, col_value, TBUF_QUOTE_LITERAL); } static void append_normal_eq(StringInfo buf, const char *col_ident, const char *col_value) { pgq_encode_cstring(buf, col_ident, TBUF_QUOTE_IDENT); appendStringInfoChar(buf, '='); if (col_value != NULL) pgq_encode_cstring(buf, col_value, TBUF_QUOTE_LITERAL); else appendStringInfoString(buf, "NULL"); } static void process_insert(PgqTriggerEvent *ev, TriggerData *tg, StringInfo sql) { HeapTuple new_row = tg->tg_trigtuple; TupleDesc tupdesc = tg->tg_relation->rd_att; int i; int need_comma = false; int attkind_idx; /* * Specify all the columns */ appendStringInfoChar(sql, '('); attkind_idx = -1; for (i = 0; i < tupdesc->natts; i++) { char *col_ident; /* Skip dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; /* Check if allowed by colstring */ attkind_idx++; if (pgqtriga_skip_col(ev, tg, i, attkind_idx)) continue; if (need_comma) appendStringInfoChar(sql, ','); else need_comma = true; /* quote column name */ col_ident = SPI_fname(tupdesc, i + 1); pgq_encode_cstring(sql, col_ident, TBUF_QUOTE_IDENT); } /* * Append the string ") values (" */ appendStringInfoString(sql, ") values ("); /* * Append the values */ need_comma = false; attkind_idx = -1; for (i = 0; i < tupdesc->natts; i++) { char *col_value; /* Skip dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; /* Check if allowed by colstring */ attkind_idx++; if (pgqtriga_skip_col(ev, tg, i, attkind_idx)) continue; if (need_comma) appendStringInfoChar(sql, ','); else need_comma = true; /* quote column value */ col_value = SPI_getvalue(new_row, tupdesc, i + 1); if (col_value == NULL) appendStringInfoString(sql, "null"); else pgq_encode_cstring(sql, col_value, TBUF_QUOTE_LITERAL); } /* * Terminate and done */ appendStringInfoChar(sql, ')'); } static int process_update(PgqTriggerEvent *ev, TriggerData *tg, StringInfo sql) { HeapTuple old_row = tg->tg_trigtuple; HeapTuple new_row = tg->tg_newtuple; TupleDesc tupdesc = tg->tg_relation->rd_att; Datum old_value; Datum new_value; bool old_isnull; bool new_isnull; char *col_ident; char *col_value; int i; int need_comma = false; int need_and = false; int attkind_idx; int ignore_count = 0; attkind_idx = -1; for (i = 0; i < tupdesc->natts; i++) { /* * Ignore dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; old_value = SPI_getbinval(old_row, tupdesc, i + 1, &old_isnull); new_value = SPI_getbinval(new_row, tupdesc, i + 1, &new_isnull); /* * If old and new value are NULL, the column is unchanged */ if (old_isnull && new_isnull) continue; /* * If both are NOT NULL, we need to compare the values and skip * setting the column if equal */ if (!old_isnull && !new_isnull) { Oid opr_oid; FmgrInfo *opr_finfo_p; /* * Lookup the equal operators function call info using the * typecache if available */ TypeCacheEntry *type_cache; type_cache = lookup_type_cache(SPI_gettypeid(tupdesc, i + 1), TYPECACHE_EQ_OPR | TYPECACHE_EQ_OPR_FINFO); opr_oid = type_cache->eq_opr; if (opr_oid == ARRAY_EQ_OP) opr_oid = InvalidOid; else opr_finfo_p = &(type_cache->eq_opr_finfo); /* * If we have an equal operator, use that to do binary * comparision. Else get the string representation of both * attributes and do string comparision. */ if (OidIsValid(opr_oid)) { if (DatumGetBool(FunctionCall2(opr_finfo_p, old_value, new_value))) continue; } else { char *old_strval = SPI_getvalue(old_row, tupdesc, i + 1); char *new_strval = SPI_getvalue(new_row, tupdesc, i + 1); if (strcmp(old_strval, new_strval) == 0) continue; } } if (pgqtriga_skip_col(ev, tg, i, attkind_idx)) { /* this change should be ignored */ ignore_count++; continue; } if (need_comma) appendStringInfoChar(sql, ','); else need_comma = true; col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(new_row, tupdesc, i + 1); append_normal_eq(sql, col_ident, col_value); } /* * It can happen that the only UPDATE an application does is to set a * column to the same value again. In that case, we'd end up here with * no columns in the SET clause yet. We add the first key column here * with it's old value to simulate the same for the replication * engine. */ if (!need_comma) { /* there was change in ignored columns, skip whole event */ if (ignore_count > 0) return 0; for (i = 0, attkind_idx = -1; i < tupdesc->natts; i++) { if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (pgqtriga_is_pkey(ev, tg, i, attkind_idx)) break; } col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(old_row, tupdesc, i + 1); append_key_eq(sql, col_ident, col_value); } appendStringInfoString(sql, " where "); for (i = 0, attkind_idx = -1; i < tupdesc->natts; i++) { /* * Ignore dropped columns */ if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (!pgqtriga_is_pkey(ev, tg, i, attkind_idx)) continue; col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(old_row, tupdesc, i + 1); if (need_and) appendStringInfoString(sql, " and "); else need_and = true; append_key_eq(sql, col_ident, col_value); } return 1; } static void process_delete(PgqTriggerEvent *ev, TriggerData *tg, StringInfo sql) { HeapTuple old_row = tg->tg_trigtuple; TupleDesc tupdesc = tg->tg_relation->rd_att; char *col_ident; char *col_value; int i; int need_and = false; int attkind_idx; for (i = 0, attkind_idx = -1; i < tupdesc->natts; i++) { if (tupdesc->attrs[i]->attisdropped) continue; attkind_idx++; if (!pgqtriga_is_pkey(ev, tg, i, attkind_idx)) continue; col_ident = SPI_fname(tupdesc, i + 1); col_value = SPI_getvalue(old_row, tupdesc, i + 1); if (need_and) appendStringInfoString(sql, " and "); else need_and = true; append_key_eq(sql, col_ident, col_value); } } int pgqtriga_make_sql(PgqTriggerEvent *ev, TriggerData *tg, StringInfo sql) { TupleDesc tupdesc; int i; int attcnt; int need_event = 1; tupdesc = tg->tg_relation->rd_att; /* * Count number of active columns */ for (i = 0, attcnt = 0; i < tupdesc->natts; i++) { if (tupdesc->attrs[i]->attisdropped) continue; attcnt++; } /* * Determine cmdtype and op_data depending on the command type */ if (TRIGGER_FIRED_BY_INSERT(tg->tg_event)) { //appendStringInfoChar(op_type, 'I'); process_insert(ev, tg, sql); } else if (TRIGGER_FIRED_BY_UPDATE(tg->tg_event)) { //appendStringInfoChar(op_type, 'U'); need_event = process_update(ev, tg, sql); } else if (TRIGGER_FIRED_BY_DELETE(tg->tg_event)) { //appendStringInfoChar(op_type, 'D'); process_delete(ev, tg, sql); } else elog(ERROR, "logtriga fired for unhandled event"); return need_event; } skytools-2.1.13/sql/pgq/triggers/sqltriga.c0000644000175000017500000000454611670174255017720 0ustar markomarko/* * sqltriga.c - Smart SQL-logging trigger. * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include #include #include #include #include "common.h" #include "stringutil.h" PG_FUNCTION_INFO_V1(pgq_sqltriga); Datum pgq_sqltriga(PG_FUNCTION_ARGS); /* * PgQ log trigger, takes 2 arguments: * 1. queue name to be inserted to. * * Queue events will be in format: * ev_type - operation type, I/U/D * ev_data - urlencoded column values * ev_extra1 - table name * ev_extra2 - optional urlencoded backup */ Datum pgq_sqltriga(PG_FUNCTION_ARGS) { TriggerData *tg; PgqTriggerEvent ev; /* * Get the trigger call context */ if (!CALLED_AS_TRIGGER(fcinfo)) elog(ERROR, "pgq.logutriga not called as trigger"); tg = (TriggerData *) (fcinfo->context); if (pgq_is_logging_disabled()) goto skip_it; /* * Connect to the SPI manager */ if (SPI_connect() < 0) elog(ERROR, "logtriga: SPI_connect() failed"); pgq_prepare_event(&ev, tg, true); appendStringInfoChar(ev.ev_type, ev.op_type); appendStringInfoString(ev.ev_extra1, ev.info->table_name); /* * create sql and insert if interesting */ if (pgqtriga_make_sql(&ev, tg, ev.ev_data)) pgq_insert_tg_event(&ev); if (SPI_finish() < 0) elog(ERROR, "SPI_finish failed"); /* * After trigger ignores result, * before trigger skips event if NULL. */ skip_it: if (TRIGGER_FIRED_AFTER(tg->tg_event) || ev.skip) return PointerGetDatum(NULL); else if (TRIGGER_FIRED_BY_UPDATE(tg->tg_event)) return PointerGetDatum(tg->tg_newtuple); else return PointerGetDatum(tg->tg_trigtuple); } skytools-2.1.13/sql/pgq/triggers/common.h0000644000175000017500000000313011670174255017353 0ustar markomarko /* * Per-event temporary data. */ struct PgqTriggerEvent { const char *table_name; const char *queue_name; const char *ignore_list; const char *pkey_list; const char *attkind; int attkind_len; char op_type; bool skip; bool backup; struct PgqTableInfo *info; StringInfo ev_type; StringInfo ev_data; StringInfo ev_extra1; StringInfo ev_extra2; }; typedef struct PgqTriggerEvent PgqTriggerEvent; /* * Per-table cached info. * * Can be shared between triggers on same table, * so nothing trigger-specific should be stored. */ struct PgqTableInfo { Oid oid; /* must be first, used by htab */ int n_pkeys; /* number of pkeys */ const char *pkey_list; /* pk column name list */ int *pkey_attno; /* pk column positions */ char *table_name; /* schema-quelified table name */ int invalid; /* set if the info was invalidated */ }; /* common.c */ struct PgqTableInfo *pgq_find_table_info(Relation rel); void pgq_prepare_event(struct PgqTriggerEvent *ev, TriggerData *tg, bool newstyle); char *pgq_find_table_name(Relation rel); void pgq_simple_insert(const char *queue_name, Datum ev_type, Datum ev_data, Datum ev_extra1, Datum ev_extra2); bool pgqtriga_skip_col(PgqTriggerEvent *ev, TriggerData *tg, int i, int attkind_idx); bool pgqtriga_is_pkey(PgqTriggerEvent *ev, TriggerData *tg, int i, int attkind_idx); void pgq_insert_tg_event(PgqTriggerEvent *ev); bool pgq_is_logging_disabled(void); /* makesql.c */ int pgqtriga_make_sql(PgqTriggerEvent *ev, TriggerData *tg, StringInfo sql); /* logutriga.c */ void pgq_urlenc_row(PgqTriggerEvent *ev, TriggerData *tg, HeapTuple row, StringInfo buf); skytools-2.1.13/sql/pgq/triggers/stringutil.h0000644000175000017500000000047711670174255020302 0ustar markomarko enum PgqEncode { TBUF_QUOTE_IDENT, TBUF_QUOTE_LITERAL, TBUF_QUOTE_URLENC, }; StringInfo pgq_init_varbuf(void); Datum pgq_finish_varbuf(StringInfo buf); bool pgq_strlist_contains(const char *liststr, const char *str); void pgq_encode_cstring(StringInfo tbuf, const char *str, enum PgqEncode encoding); skytools-2.1.13/sql/pgq/README.pgq0000644000175000017500000000045511670174255015541 0ustar markomarko Schema overview =============== pgq.consumer consumer name <> id mapping pgq.queue queue information pgq.subscription consumer registrations pgq.tick snapshots that group events into batches pgq.retry_queue events to be retried pgq.failed_queue events that have failed pgq.event_* data tables skytools-2.1.13/sql/pgq/Makefile0000644000175000017500000000331711727577156015545 0ustar markomarko DOCS = README.pgq DATA_built = pgq.sql pgq.upgrade.sql DATA = structure/uninstall_pgq.sql SRCS = $(wildcard structure/*.sql) \ $(wildcard functions/*.sql) \ $(wildcard triggers/*.sql) \ lowlevel/pgq_lowlevel.sql \ triggers/pgq_triggers.sql REGRESS = pgq_init pgq_core logutriga sqltriga REGRESS_OPTS = --load-language=plpgsql include ../../config.mak include $(PGXS) NDOC = NaturalDocs NDOCARGS = -r -o html docs/html -p docs -i docs/sql CATSQL = ../../scripts/catsql.py SUBDIRS = lowlevel triggers # PGXS does not have subdir support, thus hack to recurse into lowlevel/ all: sub-all install: sub-install clean: sub-clean distclean: sub-distclean sub-all sub-install sub-clean sub-distclean: for dir in $(SUBDIRS); do \ $(MAKE) -C $$dir $(subst sub-,,$@) DESTDIR=$(DESTDIR) \ || exit 1; \ done lowlevel/pgq_lowlevel.sql: sub-all triggers/pgq_triggers.sql: sub-all # # combined SQL files # pgq.sql: $(SRCS) $(CATSQL) structure/install.sql > $@ pgq.upgrade.sql: $(SRCS) $(CATSQL) structure/func_internal.sql structure/func_public.sql > $@ # # docs # dox: cleandox $(SRCS) mkdir -p docs/html mkdir -p docs/sql $(CATSQL) --ndoc structure/tables.sql structure/types.sql > docs/sql/schema.sql $(CATSQL) --ndoc structure/func_public.sql > docs/sql/external.sql $(CATSQL) --ndoc structure/func_internal.sql > docs/sql/internal.sql $(CATSQL) --ndoc structure/triggers.sql > docs/sql/triggers.sql $(NDOC) $(NDOCARGS) cleandox: rm -rf docs/html docs/Data docs/sql clean: cleandox upload: dox rsync -az --delete docs/html/* data1:public_html/pgq-new/ # # regtest shortcuts # test: pgq.sql $(MAKE) install installcheck || { less regression.diffs; exit 1; } ack: cp results/*.out expected/ skytools-2.1.13/sql/pgq/old/0000755000175000017500000000000011727601174014643 5ustar markomarkoskytools-2.1.13/sql/pgq/old/pgq.logutriga.sql0000644000175000017500000000600511670174255020152 0ustar markomarko create or replace function pgq.logutriga() returns trigger as $$ # -- ---------------------------------------------------------------------- # -- Function: pgq.logutriga() # -- # -- Trigger function that puts row data urlencoded into queue. # -- # -- Trigger parameters: # -- arg1 - queue name # -- arg2 - optionally 'SKIP' # -- # -- Queue event fields: # -- ev_type - I/U/D # -- ev_data - column values urlencoded # -- ev_extra1 - table name # -- ev_extra2 - primary key columns # -- # -- Regular listen trigger example: # -- > CREATE TRIGGER triga_nimi AFTER INSERT OR UPDATE ON customer # -- > FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('qname'); # -- # -- Redirect trigger example: # -- > CREATE TRIGGER triga_nimi AFTER INSERT OR UPDATE ON customer # -- > FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('qname', 'SKIP'); # -- ---------------------------------------------------------------------- # this triger takes 1 or 2 args: # queue_name - destination queue # option return code (OK, SKIP) SKIP means op won't happen # copy-paste of db_urlencode from skytools.quoting from urllib import quote_plus def db_urlencode(dict): elem_list = [] for k, v in dict.items(): if v is None: elem = quote_plus(str(k)) else: elem = quote_plus(str(k)) + '=' + quote_plus(str(v)) elem_list.append(elem) return '&'.join(elem_list) # load args queue_name = TD['args'][0] if len(TD['args']) > 1: ret_code = TD['args'][1] else: ret_code = 'OK' table_oid = TD['relid'] # on first call init plans if not 'init_done' in SD: # find table name q = "SELECT n.nspname || '.' || c.relname AS table_name"\ " FROM pg_namespace n, pg_class c"\ " WHERE n.oid = c.relnamespace AND c.oid = $1" SD['name_plan'] = plpy.prepare(q, ['oid']) # find key columns q = "SELECT k.attname FROM pg_index i, pg_attribute k"\ " WHERE i.indrelid = $1 AND k.attrelid = i.indexrelid"\ " AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped"\ " ORDER BY k.attnum" SD['key_plan'] = plpy.prepare(q, ['oid']) # insert data q = "SELECT pgq.insert_event($1, $2, $3, $4, $5, null, null)" SD['ins_plan'] = plpy.prepare(q, ['text', 'text', 'text', 'text', 'text']) # shorter tags SD['op_map'] = {'INSERT': 'I', 'UPDATE': 'U', 'DELETE': 'D'} # remember init SD['init_done'] = 1 # load & cache table data if table_oid in SD: tbl_name, tbl_keys = SD[table_oid] else: res = plpy.execute(SD['name_plan'], [table_oid]) tbl_name = res[0]['table_name'] res = plpy.execute(SD['key_plan'], [table_oid]) tbl_keys = ",".join(map(lambda x: x['attname'], res)) SD[table_oid] = (tbl_name, tbl_keys) # prepare args if TD['event'] == 'DELETE': data = db_urlencode(TD['old']) else: data = db_urlencode(TD['new']) # insert event plpy.execute(SD['ins_plan'], [ queue_name, SD['op_map'][TD['event']], data, tbl_name, tbl_keys]) # done return ret_code $$ language plpythonu; skytools-2.1.13/sql/pgq/old/pgq.insert_event_raw.sql0000644000175000017500000000625211670174255021537 0ustar markomarkocreate or replace function pgq.insert_event_raw( queue_name text, ev_id bigint, ev_time timestamptz, ev_owner integer, ev_retry integer, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text) returns bigint as $$ # -- ---------------------------------------------------------------------- # -- Function: pgq.insert_event_raw(11) # -- # -- Deprecated function, replaced by C code in pgq_lowlevel.so. # -- # -- Actual event insertion. Used also by retry queue maintenance. # -- # -- Parameters: # -- queue_name - Name of the queue # -- ev_id - Event ID. If NULL, will be taken from seq. # -- ev_time - Event creation time. # -- ev_owner - Subscription ID when retry event. If NULL, the event is for everybody. # -- ev_retry - Retry count. NULL for first-time events. # -- ev_type - user data # -- ev_data - user data # -- ev_extra1 - user data # -- ev_extra2 - user data # -- ev_extra3 - user data # -- ev_extra4 - user data # -- # -- Returns: # -- Event ID. # -- ---------------------------------------------------------------------- # load args queue_name = args[0] ev_id = args[1] ev_time = args[2] ev_owner = args[3] ev_retry = args[4] ev_type = args[5] ev_data = args[6] ev_extra1 = args[7] ev_extra2 = args[8] ev_extra3 = args[9] ev_extra4 = args[10] if not "cf_plan" in SD: # get current event table q = "select queue_data_pfx, queue_cur_table, queue_event_seq "\ " from pgq.queue where queue_name = $1" SD["cf_plan"] = plpy.prepare(q, ["text"]) # get next id q = "select nextval($1) as id" SD["seq_plan"] = plpy.prepare(q, ["text"]) # get queue config res = plpy.execute(SD["cf_plan"], [queue_name]) if len(res) != 1: plpy.error("Unknown event queue: %s" % (queue_name)) tbl_prefix = res[0]["queue_data_pfx"] cur_nr = res[0]["queue_cur_table"] id_seq = res[0]["queue_event_seq"] # get id - bump seq even if id is given res = plpy.execute(SD['seq_plan'], [id_seq]) if ev_id is None: ev_id = res[0]["id"] # create plan for insertion ins_plan = None ins_key = "ins.%s" % (queue_name) if ins_key in SD: nr, ins_plan = SD[ins_key] if nr != cur_nr: ins_plan = None if ins_plan == None: q = "insert into %s_%s (ev_id, ev_time, ev_owner, ev_retry,"\ " ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4)"\ " values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)" % ( tbl_prefix, cur_nr) types = ["int8", "timestamptz", "int4", "int4", "text", "text", "text", "text", "text", "text"] ins_plan = plpy.prepare(q, types) SD[ins_key] = (cur_nr, ins_plan) # insert the event plpy.execute(ins_plan, [ev_id, ev_time, ev_owner, ev_retry, ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4]) # done return ev_id $$ language plpythonu; -- event inserting needs no special perms skytools-2.1.13/sql/pgq/old/pgq.sqltriga.sql0000644000175000017500000001420711670174255020006 0ustar markomarko -- listen trigger: -- create trigger triga_nimi after insert or update on customer -- for each row execute procedure pgq.sqltriga('qname'); -- redirect trigger: -- create trigger triga_nimi after insert or update on customer -- for each row execute procedure pgq.sqltriga('qname', 'ret=SKIP'); create or replace function pgq.sqltriga() returns trigger as $$ # -- ---------------------------------------------------------------------- # -- Function: pgq.sqltriga() # -- # -- Trigger function that puts row data in partial SQL form into queue. # -- # -- Parameters: # -- arg1 - queue name # -- arg2 - optional urlencoded options # -- # -- Extra options: # -- # -- ret - return value for function OK/SKIP # -- pkey - override pkey fields, can be functions # -- ignore - comma separated field names to ignore # -- # -- Queue event fields: # -- ev_type - I/U/D # -- ev_data - partial SQL statement # -- ev_extra1 - table name # -- # -- ---------------------------------------------------------------------- # this triger takes 1 or 2 args: # queue_name - destination queue # args - urlencoded dict of options: # ret - return value: OK/SKIP # pkey - comma-separated col names or funcs on cols # simple: pkey=user,orderno # hashed: pkey=user,hashtext(user) # ignore - comma-separated col names to ignore # on first call init stuff if not 'init_done' in SD: # find table name plan q = "SELECT n.nspname || '.' || c.relname AS table_name"\ " FROM pg_namespace n, pg_class c"\ " WHERE n.oid = c.relnamespace AND c.oid = $1" SD['name_plan'] = plpy.prepare(q, ['oid']) # find key columns plan q = "SELECT k.attname FROM pg_index i, pg_attribute k"\ " WHERE i.indrelid = $1 AND k.attrelid = i.indexrelid"\ " AND i.indisprimary AND k.attnum > 0 AND NOT k.attisdropped"\ " ORDER BY k.attnum" SD['key_plan'] = plpy.prepare(q, ['oid']) # data insertion q = "SELECT pgq.insert_event($1, $2, $3, $4, null, null, null)" SD['ins_plan'] = plpy.prepare(q, ['text', 'text', 'text', 'text']) # shorter tags SD['op_map'] = {'INSERT': 'I', 'UPDATE': 'U', 'DELETE': 'D'} # quoting from psycopg import QuotedString def quote(s): if s is None: return "null" s = str(s) return str(QuotedString(s)) s = s.replace('\\', '\\\\') s = s.replace("'", "''") return "'%s'" % s # TableInfo class import re, urllib class TableInfo: func_rc = re.compile("([^(]+) [(] ([^)]+) [)]", re.I | re.X) def __init__(self, table_oid, options_txt): res = plpy.execute(SD['name_plan'], [table_oid]) self.name = res[0]['table_name'] self.parse_options(options_txt) self.load_pkey() def recheck(self, options_txt): if self.options_txt == options_txt: return self.parse_options(options_txt) self.load_pkey() def parse_options(self, options_txt): self.options = {'ret': 'OK'} if options_txt: for s in options_txt.split('&'): k, v = s.split('=', 1) self.options[k] = urllib.unquote_plus(v) self.options_txt = options_txt def load_pkey(self): self.pkey_list = [] if not 'pkey' in self.options: res = plpy.execute(SD['key_plan'], [table_oid]) for krow in res: col = krow['attname'] expr = col + "=%s" self.pkey_list.append( (col, expr) ) else: for a_pk in self.options['pkey'].split(','): m = self.func_rc.match(a_pk) if m: col = m.group(2) fn = m.group(1) expr = "%s(%s) = %s(%%s)" % (fn, col, fn) else: # normal case col = a_pk expr = col + "=%s" self.pkey_list.append( (col, expr) ) if len(self.pkey_list) == 0: plpy.error('sqltriga needs primary key on table') def get_insert_stmt(self, new): col_list = [] val_list = [] for k, v in new.items(): col_list.append(k) val_list.append(quote(v)) return "(%s) values (%s)" % (",".join(col_list), ",".join(val_list)) def get_update_stmt(self, old, new): chg_list = [] for k, v in new.items(): ov = old[k] if v == ov: continue chg_list.append("%s=%s" % (k, quote(v))) if len(chg_list) == 0: pk = self.pkey_list[0][0] chg_list.append("%s=%s" % (pk, quote(new[pk]))) return "%s where %s" % (",".join(chg_list), self.get_pkey_expr(new)) def get_pkey_expr(self, data): exp_list = [] for col, exp in self.pkey_list: exp_list.append(exp % quote(data[col])) return " and ".join(exp_list) SD['TableInfo'] = TableInfo # cache some functions def proc_insert(tbl): return tbl.get_insert_stmt(TD['new']) def proc_update(tbl): return tbl.get_update_stmt(TD['old'], TD['new']) def proc_delete(tbl): return tbl.get_pkey_expr(TD['old']) SD['event_func'] = { 'I': proc_insert, 'U': proc_update, 'D': proc_delete, } # remember init SD['init_done'] = 1 # load args table_oid = TD['relid'] queue_name = TD['args'][0] if len(TD['args']) > 1: options_str = TD['args'][1] else: options_str = '' # load & cache table data if table_oid in SD: tbl = SD[table_oid] tbl.recheck(options_str) else: tbl = SD['TableInfo'](table_oid, options_str) SD[table_oid] = tbl # generate payload op = SD['op_map'][TD['event']] data = SD['event_func'][op](tbl) # insert event plpy.execute(SD['ins_plan'], [queue_name, op, data, tbl.name]) # done return tbl.options['ret'] $$ language plpythonu; skytools-2.1.13/sql/pgq/lowlevel/0000755000175000017500000000000011727601174015716 5ustar markomarkoskytools-2.1.13/sql/pgq/lowlevel/pgq_lowlevel.sql.in0000644000175000017500000000221511670174255021546 0ustar markomarko -- ---------------------------------------------------------------------- -- Function: pgq.insert_event_raw(11) -- -- Actual event insertion. Used also by retry queue maintenance. -- -- Parameters: -- queue_name - Name of the queue -- ev_id - Event ID. If NULL, will be taken from seq. -- ev_time - Event creation time. -- ev_owner - Subscription ID when retry event. If NULL, the event is for everybody. -- ev_retry - Retry count. NULL for first-time events. -- ev_type - user data -- ev_data - user data -- ev_extra1 - user data -- ev_extra2 - user data -- ev_extra3 - user data -- ev_extra4 - user data -- -- Returns: -- Event ID. -- ---------------------------------------------------------------------- CREATE OR REPLACE FUNCTION pgq.insert_event_raw( queue_name text, ev_id bigint, ev_time timestamptz, ev_owner integer, ev_retry integer, ev_type text, ev_data text, ev_extra1 text, ev_extra2 text, ev_extra3 text, ev_extra4 text) RETURNS int8 AS 'MODULE_PATHNAME', 'pgq_insert_event_raw' LANGUAGE C; skytools-2.1.13/sql/pgq/lowlevel/insert_event.c0000644000175000017500000001502611727577202020577 0ustar markomarko/* * insert_event.c - C implementation of pgq.insert_event_raw(). * * Copyright (c) 2007 Marko Kreen, Skype Technologies OÜ * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include "postgres.h" #include "funcapi.h" #include "catalog/pg_type.h" #include "executor/spi.h" #include "lib/stringinfo.h" #include "utils/builtins.h" #include "utils/datetime.h" #include "utils/hsearch.h" /* * Module tag */ #ifdef PG_MODULE_MAGIC PG_MODULE_MAGIC; #endif /* * Function tag */ Datum pgq_insert_event_raw(PG_FUNCTION_ARGS); PG_FUNCTION_INFO_V1(pgq_insert_event_raw); /* * Queue info fetching. * * Always touch ev_id sequence, even if ev_id is given as arg, * to notify ticker about new event. */ #define QUEUE_SQL \ "select queue_id::int4, queue_data_pfx::text," \ " queue_cur_table::int4, nextval(queue_event_seq)::int8 " \ " from pgq.queue where queue_name = $1" #define COL_QUEUE_ID 1 #define COL_PREFIX 2 #define COL_TBLNO 3 #define COL_EVENT_ID 4 /* * Plan cache entry in HTAB. */ struct InsertCacheEntry { Oid queue_id; /* actually int32, but we want to use oid_hash */ int cur_table; void *plan; }; /* * helper structure to pass values. */ struct QueueState { int queue_id; int cur_table; char *table_prefix; Datum next_event_id; }; /* * Cached plans. */ static void *queue_plan; static HTAB *insert_cache; /* * Prepare utility plans and plan cache. */ static void init_cache(void) { static int init_done = 0; Oid types[1] = { TEXTOID }; HASHCTL ctl; int flags; int max_queues = 128; if (init_done) return; /* * Init plans. */ queue_plan = SPI_saveplan(SPI_prepare(QUEUE_SQL, 1, types)); if (queue_plan == NULL) elog(ERROR, "pgq_insert: SPI_prepare() failed"); /* * init insert plan cache. */ MemSet(&ctl, 0, sizeof(ctl)); ctl.keysize = sizeof(Oid); ctl.entrysize = sizeof(struct InsertCacheEntry); ctl.hash = oid_hash; flags = HASH_ELEM | HASH_FUNCTION; insert_cache = hash_create("pgq_insert_raw plans cache", max_queues, &ctl, flags); init_done = 1; } /* * Create new plan for insertion into current queue table. */ static void *make_plan(struct QueueState *state) { void *plan; StringInfo sql; static Oid types[10] = { INT8OID, TIMESTAMPTZOID, INT4OID, INT4OID, TEXTOID, TEXTOID, TEXTOID, TEXTOID, TEXTOID, TEXTOID }; /* * create sql */ sql = makeStringInfo(); appendStringInfo(sql, "insert into %s_%d (ev_id, ev_time, ev_owner, ev_retry," " ev_type, ev_data, ev_extra1, ev_extra2, ev_extra3, ev_extra4)" " values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)", state->table_prefix, state->cur_table); /* * create plan */ plan = SPI_prepare(sql->data, 10, types); return SPI_saveplan(plan); } /* * fetch insert plan from cache. */ static void *load_insert_plan(struct QueueState *state) { struct InsertCacheEntry *entry; Oid queue_id = state->queue_id; bool did_exist = false; entry = hash_search(insert_cache, &queue_id, HASH_ENTER, &did_exist); if (did_exist) { if (entry->plan && state->cur_table == entry->cur_table) return entry->plan; if (entry->plan) SPI_freeplan(entry->plan); } entry->cur_table = state->cur_table; entry->plan = NULL; /* this can fail, struct must be valid before */ entry->plan = make_plan(state); return entry->plan; } /* * load queue info from pgq.queue table. */ static void load_queue_info(Datum queue_name, struct QueueState *state) { Datum values[1]; int res; TupleDesc desc; HeapTuple row; bool isnull; values[0] = queue_name; res = SPI_execute_plan(queue_plan, values, NULL, false, 0); if (res != SPI_OK_SELECT) elog(ERROR, "Queue fetch failed"); if (SPI_processed == 0) elog(ERROR, "No such queue"); row = SPI_tuptable->vals[0]; desc = SPI_tuptable->tupdesc; state->queue_id = DatumGetInt32(SPI_getbinval(row, desc, COL_QUEUE_ID, &isnull)); if (isnull) elog(ERROR, "queue id NULL"); state->cur_table = DatumGetInt32(SPI_getbinval(row, desc, COL_TBLNO, &isnull)); if (isnull) elog(ERROR, "table nr NULL"); state->table_prefix = SPI_getvalue(row, desc, COL_PREFIX); if (state->table_prefix == NULL) elog(ERROR, "table prefix NULL"); state->next_event_id = SPI_getbinval(row, desc, COL_EVENT_ID, &isnull); if (isnull) elog(ERROR, "Seq name NULL"); } /* * Arguments: * 0: queue_name text NOT NULL * 1: ev_id int8 if NULL take from SEQ * 2: ev_time timestamptz if NULL use now() * 3: ev_owner int4 * 4: ev_retry int4 * 5: ev_type text * 6: ev_data text * 7: ev_extra1 text * 8: ev_extra2 text * 9: ev_extra3 text * 10:ev_extra4 text */ Datum pgq_insert_event_raw(PG_FUNCTION_ARGS) { Datum values[11]; char nulls[11]; struct QueueState state; int64 ret_id; void *ins_plan; Datum ev_id, ev_time; int i, res; if (PG_NARGS() < 6) elog(ERROR, "Need at least 6 arguments"); if (PG_ARGISNULL(0)) elog(ERROR, "Queue name must not be NULL"); if (SPI_connect() < 0) elog(ERROR, "SPI_connect() failed"); init_cache(); load_queue_info(PG_GETARG_DATUM(0), &state); if (PG_ARGISNULL(1)) ev_id = state.next_event_id; else ev_id = PG_GETARG_DATUM(1); if (PG_ARGISNULL(2)) ev_time = DirectFunctionCall1(now, 0); else ev_time = PG_GETARG_DATUM(2); /* * Prepare arguments for INSERT */ values[0] = ev_id; nulls[0] = ' '; values[1] = ev_time; nulls[1] = ' '; for (i = 3; i < 11; i++) { int dst = i - 1; if (i >= PG_NARGS() || PG_ARGISNULL(i)) { values[dst] = (Datum)NULL; nulls[dst] = 'n'; } else { values[dst] = PG_GETARG_DATUM(i); nulls[dst] = ' '; } } /* * Perform INSERT into queue table. */ ins_plan = load_insert_plan(&state); res = SPI_execute_plan(ins_plan, values, nulls, false, 0); if (res != SPI_OK_INSERT) elog(ERROR, "Queue insert failed"); /* * ev_id cannot pass SPI_finish() */ ret_id = DatumGetInt64(ev_id); if (SPI_finish() < 0) elog(ERROR, "SPI_finish failed"); PG_RETURN_INT64(ret_id); } skytools-2.1.13/sql/pgq/lowlevel/Makefile0000644000175000017500000000022411670174255017356 0ustar markomarko include ../../../config.mak MODULE_big = pgq_lowlevel DATA_built = pgq_lowlevel.sql SRCS = insert_event.c OBJS = $(SRCS:.c=.o) include $(PGXS) skytools-2.1.13/sql/pgq/sql/0000755000175000017500000000000011727601174014664 5ustar markomarkoskytools-2.1.13/sql/pgq/sql/logutriga.sql0000644000175000017500000000244711670174255017413 0ustar markomarko drop function pgq.insert_event(text, text, text, text, text, text, text); create or replace function pgq.insert_event(que text, ev_type text, ev_data text, x1 text, x2 text, x3 text, x4 text) returns bigint as $$ begin raise notice 'insert_event(%, %, %, %)', que, ev_type, ev_data, x1; return 1; end; $$ language plpgsql; create table udata ( id serial primary key, txt text, bin bytea ); create trigger utest AFTER insert or update or delete ON udata for each row execute procedure pgq.logutriga('udata_que'); insert into udata (txt) values ('text1'); insert into udata (bin) values (E'bi\tn\\000bin'); -- test missing pkey create table nopkey2 (dat text); create trigger nopkey_triga2 after insert or update or delete on nopkey2 for each row execute procedure pgq.logutriga('que3'); insert into nopkey2 values ('foo'); update nopkey2 set dat = 'bat'; delete from nopkey2; -- test custom pkey create table ucustom_pkey (dat1 text not null, dat2 int2 not null, dat3 text); create trigger ucustom_triga after insert or update or delete on ucustom_pkey --for each row execute procedure pgq.logutriga('que3', 'pkey=dat1,dat2'); for each row execute procedure pgq.logutriga('que3'); insert into ucustom_pkey values ('foo', '2'); update ucustom_pkey set dat3 = 'bat'; delete from ucustom_pkey; skytools-2.1.13/sql/pgq/sql/pgq_core.sql0000644000175000017500000000570411670174255017214 0ustar markomarko select * from pgq.maint_tables_to_vacuum(); select * from pgq.maint_retry_events(); select pgq.create_queue('tmpqueue'); select pgq.register_consumer('tmpqueue', 'consumer'); select pgq.unregister_consumer('tmpqueue', 'consumer'); select pgq.drop_queue('tmpqueue'); select pgq.create_queue('myqueue'); select pgq.register_consumer('myqueue', 'consumer'); select pgq.next_batch('myqueue', 'consumer'); select pgq.next_batch('myqueue', 'consumer'); select pgq.ticker(); select pgq.next_batch('myqueue', 'consumer'); select pgq.next_batch('myqueue', 'consumer'); select queue_name, consumer_name, prev_tick_id, tick_id, lag < '1 second' from pgq.get_batch_info(1); select queue_name, queue_ntables, queue_cur_table, queue_rotation_period, queue_switch_time <= now() as switch_time_exists, queue_external_ticker, queue_ticker_max_count, queue_ticker_max_lag, queue_ticker_idle_period, ticker_lag < '2 hours' as ticker_lag_exists from pgq.get_queue_info() order by 1; select queue_name, consumer_name, lag < '30 seconds' as lag_exists, last_seen < '30 seconds' as last_seen_exists, last_tick, current_batch, next_tick from pgq.get_consumer_info() order by 1, 2; select pgq.finish_batch(1); select pgq.finish_batch(1); select pgq.ticker(); select pgq.next_batch('myqueue', 'consumer'); select * from pgq.batch_event_tables(2); select * from pgq.get_batch_events(2); select pgq.finish_batch(2); select pgq.insert_event('myqueue', 'r1', 'data'); select pgq.insert_event('myqueue', 'r2', 'data', 'extra1', 'extra2', 'extra3', 'extra4'); select pgq.insert_event('myqueue', 'r3', 'data'); select pgq.current_event_table('myqueue'); select pgq.ticker(); select pgq.next_batch('myqueue', 'consumer'); select ev_id,ev_retry,ev_type,ev_data,ev_extra1,ev_extra2,ev_extra3,ev_extra4 from pgq.get_batch_events(3); select * from pgq.failed_event_list('myqueue', 'consumer'); select pgq.event_failed(3, 1, 'failure test'); select pgq.event_failed(3, 1, 'failure test'); select pgq.event_retry(3, 2, 0); select pgq.event_retry(3, 2, 0); select pgq.finish_batch(3); select ev_failed_reason, ev_id, ev_txid, ev_retry, ev_type, ev_data from pgq.failed_event_list('myqueue', 'consumer'); select ev_failed_reason, ev_id, ev_txid, ev_retry, ev_type, ev_data from pgq.failed_event_list('myqueue', 'consumer', 0, 1); select * from pgq.failed_event_count('myqueue', 'consumer'); select * from pgq.failed_event_delete('myqueue', 'consumer', 0); select pgq.event_retry_raw('myqueue', 'consumer', now(), 666, now(), 0, 'rawtest', 'data', null, null, null, null); select pgq.ticker(); -- test maint update pgq.queue set queue_rotation_period = '0 seconds'; select queue_name, pgq.maint_rotate_tables_step1(queue_name) from pgq.queue; select pgq.maint_rotate_tables_step2(); -- test extra select nextval(queue_event_seq) from pgq.queue where queue_name = 'myqueue'; select pgq.force_tick('myqueue'); select nextval(queue_event_seq) from pgq.queue where queue_name = 'myqueue'; skytools-2.1.13/sql/pgq/sql/pgq_init.sql0000644000175000017500000000007711670174255017225 0ustar markomarko \set ECHO none \i ../txid/txid.sql \i pgq.sql \set ECHO all skytools-2.1.13/sql/pgq/sql/sqltriga.sql0000644000175000017500000000431211670174255017235 0ustar markomarko -- start testing create table rtest ( id integer primary key, dat text ); create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure pgq.sqltriga('que'); -- simple test insert into rtest values (1, 'value1'); update rtest set dat = 'value2'; delete from rtest; -- test new fields alter table rtest add column dat2 text; insert into rtest values (1, 'value1'); update rtest set dat = 'value2'; delete from rtest; -- test field ignore drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure pgq.sqltriga('que2', 'ignore=dat2'); insert into rtest values (1, '666', 'newdat'); update rtest set dat = 5, dat2 = 'newdat2'; update rtest set dat = 6; delete from rtest; -- test hashed pkey -- drop trigger rtest_triga on rtest; -- create trigger rtest_triga after insert or update or delete on rtest -- for each row execute procedure pgq.sqltriga('que2', 'ignore=dat2','pkey=dat,hashtext(dat)'); -- insert into rtest values (1, '666', 'newdat'); -- update rtest set dat = 5, dat2 = 'newdat2'; -- update rtest set dat = 6; -- delete from rtest; -- test wrong key drop trigger rtest_triga on rtest; create trigger rtest_triga after insert or update or delete on rtest for each row execute procedure pgq.sqltriga('que3'); insert into rtest values (1, 0, 'non-null'); insert into rtest values (2, 0, NULL); update rtest set dat2 = 'non-null2' where id=1; update rtest set dat2 = NULL where id=1; update rtest set dat2 = 'new-nonnull' where id=2; delete from rtest where id=1; delete from rtest where id=2; -- test missing pkey create table nopkey (dat text); create trigger nopkey_triga after insert or update or delete on nopkey for each row execute procedure pgq.sqltriga('que3'); insert into nopkey values ('foo'); update nopkey set dat = 'bat'; delete from nopkey; -- test custom pkey create table custom_pkey (dat1 text not null, dat2 int2 not null, dat3 text); create trigger custom_triga after insert or update or delete on custom_pkey for each row execute procedure pgq.sqltriga('que3', 'pkey=dat1,dat2'); insert into custom_pkey values ('foo', '2'); update custom_pkey set dat3 = 'bat'; delete from custom_pkey; skytools-2.1.13/sql/pgq_ext/0000755000175000017500000000000011727601174014745 5ustar markomarkoskytools-2.1.13/sql/pgq_ext/functions/0000755000175000017500000000000011727601174016755 5ustar markomarkoskytools-2.1.13/sql/pgq_ext/functions/track_tick.sql0000644000175000017500000000155111670174255021620 0ustar markomarko create or replace function pgq_ext.get_last_tick(a_consumer text) returns int8 as $$ declare res int8; begin select last_tick_id into res from pgq_ext.completed_tick where consumer_id = a_consumer; return res; end; $$ language plpgsql security definer; create or replace function pgq_ext.set_last_tick(a_consumer text, a_tick_id bigint) returns integer as $$ begin if a_tick_id is null then delete from pgq_ext.completed_tick where consumer_id = a_consumer; else update pgq_ext.completed_tick set last_tick_id = a_tick_id where consumer_id = a_consumer; if not found then insert into pgq_ext.completed_tick (consumer_id, last_tick_id) values (a_consumer, a_tick_id); end if; end if; return 1; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq_ext/functions/track_batch.sql0000644000175000017500000000173011670174255021746 0ustar markomarko create or replace function pgq_ext.is_batch_done( a_consumer text, a_batch_id bigint) returns boolean as $$ declare res boolean; begin select last_batch_id = a_batch_id into res from pgq_ext.completed_batch where consumer_id = a_consumer; if not found then return false; end if; return res; end; $$ language plpgsql security definer; create or replace function pgq_ext.set_batch_done( a_consumer text, a_batch_id bigint) returns boolean as $$ begin if pgq_ext.is_batch_done(a_consumer, a_batch_id) then return false; end if; if a_batch_id > 0 then update pgq_ext.completed_batch set last_batch_id = a_batch_id where consumer_id = a_consumer; if not found then insert into pgq_ext.completed_batch (consumer_id, last_batch_id) values (a_consumer, a_batch_id); end if; end if; return true; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq_ext/functions/track_event.sql0000644000175000017500000000333011670174255022004 0ustar markomarko create or replace function pgq_ext.is_event_done( a_consumer text, a_batch_id bigint, a_event_id bigint) returns boolean as $$ declare res bigint; begin perform 1 from pgq_ext.completed_event where consumer_id = a_consumer and batch_id = a_batch_id and event_id = a_event_id; return found; end; $$ language plpgsql security definer; create or replace function pgq_ext.set_event_done( a_consumer text, a_batch_id bigint, a_event_id bigint) returns boolean as $$ declare old_batch bigint; begin -- check if done perform 1 from pgq_ext.completed_event where consumer_id = a_consumer and batch_id = a_batch_id and event_id = a_event_id; if found then return false; end if; -- if batch changed, do cleanup select cur_batch_id into old_batch from pgq_ext.partial_batch where consumer_id = a_consumer; if not found then -- first time here insert into pgq_ext.partial_batch (consumer_id, cur_batch_id) values (a_consumer, a_batch_id); elsif old_batch <> a_batch_id then -- batch changed, that means old is finished on queue db -- thus the tagged events are not needed anymore delete from pgq_ext.completed_event where consumer_id = a_consumer and batch_id = old_batch; -- remember current one update pgq_ext.partial_batch set cur_batch_id = a_batch_id where consumer_id = a_consumer; end if; -- tag as done insert into pgq_ext.completed_event (consumer_id, batch_id, event_id) values (a_consumer, a_batch_id, a_event_id); return true; end; $$ language plpgsql security definer; skytools-2.1.13/sql/pgq_ext/functions/version.sql0000644000175000017500000000016611670174255021170 0ustar markomarko create or replace function pgq_ext.version() returns text as $$ begin return '2.1.6'; end; $$ language plpgsql; skytools-2.1.13/sql/pgq_ext/expected/0000755000175000017500000000000011727601174016546 5ustar markomarkoskytools-2.1.13/sql/pgq_ext/expected/test_pgq_ext.out0000644000175000017500000000362311670174255022013 0ustar markomarko\set ECHO off -- -- test batch tracking -- select pgq_ext.is_batch_done('c', 1); is_batch_done --------------- f (1 row) select pgq_ext.set_batch_done('c', 1); set_batch_done ---------------- t (1 row) select pgq_ext.is_batch_done('c', 1); is_batch_done --------------- t (1 row) select pgq_ext.set_batch_done('c', 1); set_batch_done ---------------- f (1 row) select pgq_ext.is_batch_done('c', 2); is_batch_done --------------- f (1 row) select pgq_ext.set_batch_done('c', 2); set_batch_done ---------------- t (1 row) -- -- test event tracking -- select pgq_ext.is_batch_done('c', 3); is_batch_done --------------- f (1 row) select pgq_ext.is_event_done('c', 3, 101); is_event_done --------------- f (1 row) select pgq_ext.set_event_done('c', 3, 101); set_event_done ---------------- t (1 row) select pgq_ext.is_event_done('c', 3, 101); is_event_done --------------- t (1 row) select pgq_ext.set_event_done('c', 3, 101); set_event_done ---------------- f (1 row) select pgq_ext.set_batch_done('c', 3); set_batch_done ---------------- t (1 row) select * from pgq_ext.completed_event order by 1,2; consumer_id | batch_id | event_id -------------+----------+---------- c | 3 | 101 (1 row) -- -- test tick tracking -- select pgq_ext.get_last_tick('c'); get_last_tick --------------- (1 row) select pgq_ext.set_last_tick('c', 1); set_last_tick --------------- 1 (1 row) select pgq_ext.get_last_tick('c'); get_last_tick --------------- 1 (1 row) select pgq_ext.set_last_tick('c', 2); set_last_tick --------------- 1 (1 row) select pgq_ext.get_last_tick('c'); get_last_tick --------------- 2 (1 row) select pgq_ext.set_last_tick('c', NULL); set_last_tick --------------- 1 (1 row) select pgq_ext.get_last_tick('c'); get_last_tick --------------- (1 row) skytools-2.1.13/sql/pgq_ext/structure/0000755000175000017500000000000011727601174017005 5ustar markomarkoskytools-2.1.13/sql/pgq_ext/structure/tables.sql0000644000175000017500000000156611670174255021012 0ustar markomarko set client_min_messages = 'warning'; set default_with_oids = 'off'; create schema pgq_ext; grant usage on schema pgq_ext to public; -- -- batch tracking -- create table pgq_ext.completed_batch ( consumer_id text not null, last_batch_id bigint not null, primary key (consumer_id) ); -- -- event tracking -- create table pgq_ext.completed_event ( consumer_id text not null, batch_id bigint not null, event_id bigint not null, primary key (consumer_id, batch_id, event_id) ); create table pgq_ext.partial_batch ( consumer_id text not null, cur_batch_id bigint not null, primary key (consumer_id) ); -- -- tick tracking for SerialConsumer() -- no access functions provided here -- create table pgq_ext.completed_tick ( consumer_id text not null, last_tick_id bigint not null, primary key (consumer_id) ); skytools-2.1.13/sql/pgq_ext/Makefile0000644000175000017500000000066711670174255016420 0ustar markomarko DOCS = README.pgq_ext DATA_built = pgq_ext.sql SRCS = structure/tables.sql functions/track_batch.sql functions/track_event.sql \ functions/track_tick.sql functions/version.sql REGRESS = test_pgq_ext REGRESS_OPTS = --load-language=plpgsql include ../../config.mak include $(PGXS) pgq_ext.sql: $(SRCS) cat $(SRCS) > $@ test: pgq_ext.sql make installcheck || { less regression.diffs ; exit 1; } ack: cp results/* expected/ skytools-2.1.13/sql/pgq_ext/README.pgq_ext0000644000175000017500000000150611670174255017277 0ustar markomarko Track processed batches and events in target DB ================================================ Batch tracking is OK. Event tracking is OK if consumer does not use retry queue. Batch tracking -------------- is_batch_done(consumer, batch) returns: true - batch is done already false - batch is not done yet set_batch_done(consumer, batch) returns: true - tagging successful, batch was not done yet false - batch was done already Event tracking -------------- is_batch_done(consumer, batch, event) returns: true - event is done false - event is not done yet set_batch_done(consumer, batch, event) returns: true - tagging was successful, event was not done false - event is done already Fastvacuum ---------- pgq.ext.completed_batch pgq.ext.completed_event pgq.ext.completed_tick pgq.ext.partial_batch skytools-2.1.13/sql/pgq_ext/sql/0000755000175000017500000000000011727601174015544 5ustar markomarkoskytools-2.1.13/sql/pgq_ext/sql/test_pgq_ext.sql0000644000175000017500000000163411670174255021001 0ustar markomarko\set ECHO off \i pgq_ext.sql \set ECHO all -- -- test batch tracking -- select pgq_ext.is_batch_done('c', 1); select pgq_ext.set_batch_done('c', 1); select pgq_ext.is_batch_done('c', 1); select pgq_ext.set_batch_done('c', 1); select pgq_ext.is_batch_done('c', 2); select pgq_ext.set_batch_done('c', 2); -- -- test event tracking -- select pgq_ext.is_batch_done('c', 3); select pgq_ext.is_event_done('c', 3, 101); select pgq_ext.set_event_done('c', 3, 101); select pgq_ext.is_event_done('c', 3, 101); select pgq_ext.set_event_done('c', 3, 101); select pgq_ext.set_batch_done('c', 3); select * from pgq_ext.completed_event order by 1,2; -- -- test tick tracking -- select pgq_ext.get_last_tick('c'); select pgq_ext.set_last_tick('c', 1); select pgq_ext.get_last_tick('c'); select pgq_ext.set_last_tick('c', 2); select pgq_ext.get_last_tick('c'); select pgq_ext.set_last_tick('c', NULL); select pgq_ext.get_last_tick('c');