The bucardo script is the main interaction to a running Bucardo instance. It can be used to start and stop Bucardo, add new items, kick syncs, and even install and upgrade Bucardo itself. For more complete documentation, please view the wiki.
COMMANDS
Run bucardo help <command> for additional details
install
Installs the Bucardo configuration database.
upgrade
Upgrades the Bucardo configuration database to the latest schema.
Sends a message to all CTL and KID processes asking them to reload the Bucardo configuration.
reopen
Sends a message to all Bucardo processes asking them to reopen any log files they may have open. Call this after you have rotated the log file(s).
show all|<setting> [<setting>...]
Shows the current Bucardo settings.
<set <setting=value [<setting=value>...] >>
Sets one or more configuration setting..
ping [<timeout>]
Sends a ping notice to the MCP process to see if it will respond.
status [<status options>] <syncname> [<syncname>...]
Shows the brief status of syncs in a tabular format.
activate <syncname> [<syncname>...] [<timeout>]
Activates one or more named syncs.
deactivate <syncname> [<syncname>...] [<timeout>]
Deactivates one or more named syncs.
message '<body>'
Sends a message to the running Bucardo logs.
reload [<syncname> [<syncname>...]]
Sends a message to one or more sync processes, instructing them to reload.
inspect <type> <name> [<name>...]
Inspects one or more objects of a particular type.
validate all|<syncname> [<syncname>...]
Validates one or more syncs.
purge all|<table> [<table>...]
Purges the delta and track tables for one or more tables, for one or more databases.
delta [<database(s)>]
Show the delta counts for each source target.
help [<command> [<action>]]
Shows help.
OPTIONS
-d --db-name NAME Database name.
-U --db-user USER Database user name.
-P --db-pass PASS Database password.
-h --db-host HOST Database server host name.
-p --db-port PORT Database server port number.
--bucardorc FILE Use specified .bucardorc file.
--no-bucardorc Do not use .bucardorc file.
--quiet Incremental quiet.
--verbose Incremental verbose mode.
-? --help Output basic help and exit.
--version Print the version number and exit.
--dryrun Do not perform any actual actions.
--confirm Require direct confirmation before changes.
COMMAND DETAILS
Most of the commands take parameters. These may be passed after the command name and, where appropriate, an object name. Parameters take the form of key/value pairs separated by an equal sign (=). For example:
bucardo add db sea_widgets dbname=widgets host=db.example.com
Here dbname and <host> are parameters.
Many of the commands also use command-line options, which are specified in the normal way. For example, the bucardo add db command could also be written as:
bucardo add db sea_widgets --dbname widgets --dbhost db.example.com
However, parameters and options are not directly interchangeable in all cases. See the documentation for individual commands for their supported options.
install
bucardo install
Installs the Bucardo schema from the file bucardo.schema into an existing Postgres cluster. The user "bucardo" and database "bucardo" will be created first as needed. This is an interactive installer, but you can supply the following values from the command line:
--dbuser
defaults to postgres
--dbname
defaults to postgres
--dbport
defaults to 5432
--pid-dir
defaults to /var/run/bucardo/
upgrade
bucardo upgrade
Upgrades an existing Bucardo installation to the current version of the bucardo database script. Requires that bucardo and the bucardo.schema file be the same version. All changes should be backwards compatible, but you may need to re-validate existing scripts to make sure changes get propagated to all databases.
start
bucardo start "Reason"
Starts Bucardo. Fails if the MCP process is running (determined if its PID file is present). Otherwise, starts cleanly by first issuing the equivalent of a stop to ask any existing Bucardo processes to exit, and then starting a new Bucardo MCP process. A short reason and name should be provided - these are written to the reason_file file (./bucardo.restart.reason.txt by default) and sent in the email sent when Bucardo has been started up. It is also appended to the reason log, which has the same name as the the reason_file but ends in .log.
The options for the start command are:
--sendmail
Tells Bucardo whether or not to send mail on interesting events: startup, shutdown, and errors. Default is on.
--extra-name string
A short string that will be appended to the version string as output by the Bucardo process names. Mostly useful for debugging.
--log-destination destination
Determines the destination for logging output. The supported values are:
stderr
stdout
syslog
none
A file system directory.
May be specified more than once, which is useful for, e.g., logging both to a directory and to syslog. If --log-destination is not specified at all, the default is to log to files in /var/log/bucardo.
--log-separate
Forces creation of separate log files for each Bucardo process of the form "log.bucardo.X.Y", where X is the type of process (MCP, CTL, or KID), and Y is the process ID.
--log-extension string
Appends the given string to the end of the default log file name, log.bucardo. A dot is added before the name as well, so a log extension of "rootdb" would produce a log file named log.bucardo.rootdb.
--log-clean
Forces removal of all old log files before running.
--debug
--no-debug
Enable or disable debugging output. Disabled by default.
--exit-on-nosync
--no-exit-on-nosync
On startup, if Bucardo finds no active syncs, it normally will continue to run, requiring a restart once syncs are added. This is useful for startup scripts and whatnot.
If, however, you want it to exit when there are no active syncs, pass the --exit-on-nosync option. You can also be explicit that it should not exit when there are no syncs by passing --no-exit-on-nosync. This is the default value.
stop
bucardo stop "Reason"
Forces Bucardo to quit by creating a stop file which all MCP, CTL, and KID processes should detect and cause them to exit. Note that active syncs will not exit right away, as they will not look for the stop file until they have finished their current run. Typically, you should scan the list of processes after running this program to make sure that all Bucardo processes have stopped. One should also provide a reason for issuing the stop - usually this is a short explanation and your name. This is written to the reason_file file (./bucardo.restart.reason.txt by default) and is also used by Bucardo when it exits and sends out mail about its death. It is also appended to the reason log, which has the same name as the the reason_file but ends in .log.
restart
bucardo restart "Reason"
Stops bucardo, waits for the stop to complete, and then starts it again. Supports the same options as <start/start>. Useful for start scripts. For getting just CTL and KID processes to recognize newly added, updated, or removed objects, use the reload command, instead.
list
bucardo list <type> <regex>
Lists summary information about Bucardo objects. The supported types are:
database
dbgroup
relgroup
sync
table
sequence
customcode
customname
customcols
all
The all option will list information about all object types.
The optional regex option can be used to filter the list to only those matching a regular expression.
add
bucardo add <type> <name> <parameters>
Adds a new object to Bucardo. The type specifies the type of object to add, while the name should be the name of the object. The supported types include:
db
dbgroup
table
sequence
all tables
all sequences
relgroup
sync
customname
customcols
add db
bucardo add db <name> dbname=actual_name port=xxx host=xxx user=xxx
Adds one or more new databases. The name is the name by which the database will be known to Bucardo, and must be unique. This may vary from the actual database name, as multiple hosts might have databases with the same name. Multiple databases can be added by separating the names with commas. Options that differ between the databases should be separated by a matching commas. Example:
bucardo add db alpha,beta dbname=sales host=aa,bb user=bucardo
This command will attempt an immediate test connection to the added database(s). The supported named parameters are:
dbname
The actual name of the database. Required unless using a service file or setting it via dbdsn.
type
The type of the database. Defaults to postgres. Currently supported values are:
postgres
drizzle
mongo
mysql
maria
oracle
redis
sqlite
dbdsn
A direct DSN to connect to a database. Will override all other connection options if set.
user
The username Bucardo should use when connecting to this database.
pass
The password Bucardo should use when connecting to this database. It is recommended that you use a .pgpass file rather than entering the password here.
host
The host Bucardo should use when connecting to this database. Defaults to the value of the $PGHOSTADDR or $PGHOST environment variables, if present.
port
The port Bucardo should use when connecting to this database. Defaults to the value of the $PGPORT environment variable, if present.
conn
Additional connection parameters, e.g. sslmode=require.
service
The service name Bucardo should use when connecting to this database.
status
Initial status of this database. Defaults to "active" but can be set to "inactive".
dbgroup
Name of the database group this database should belong to.
addalltables
Automatically add all tables from this database.
addallsequences
Automatically add all sequences from this database.
server_side_prepares
ssp
Set to 1 or 0 to enable or disable server-side prepares. Defaults to 1.
makedelta
Set to 1 or 0 to enable or disable makedelta. Defaults to 0.
Additional parameters:
--force
Forces the database to be added without running a connection test.
Note: As a convenience, if the dbuser value is its default value, "bucardo", in the event that Bucardo cannot connect to the database, it will try connecting as "postgres" and create a superuser named "bucardo". This is to make things easier for folks getting started with Bucardo, but will not work if it cannot connect as "postgres", or if it the connection failed due to an authentication failure.
add dbgroup
bucardo add dbgroup name db1:source db2:source db3:target ...
Adds one or more databases to the named dbgroup. If the dbgroup doesn't exist, it will be created. The database parameters should specify their roles, either "source" or "target".
Adds a table object. The table information will be read from the specified database. Supported parameters:
db
The name of the database from which to read the table information. Should be a name known to Bucardo, thanks to a previous call to add database. Required.
autokick
Boolean indicating whether or not the table should automatically send kick messages when it's modified. Overrides the autokick parameter of any syncs of which the table is a part.
rebuild_index
Boolean indicating whether or not to rebuild indexes after every sync. Off by default. Optional.
analyze_after_copy
Boolean indicating whether or not to analyze the table after every sync. Off by default. Optional.
vacuum_after_copy
Boolean indicating whether or not to vacuum the table after every sync. Off by default. Optional.
relgroup
Adds the table to the named relgroup. If the relgroup does not exist, it will be created. Optional.
makedelta
Turns makedelta magic on or off. Value is a list of databases which need makedelta for this table. Value can also be "on" to enable makedelta for all databases. Defaults to "off".
strict_checking
Boolean indicating whether or not to be strict when comparing the table between syncs. If the columns have different names or data types, the validation will fail. But perhaps the columns are allowed to have different names or data types. If so, disable strict_checking and column differences will result in warnings rather than failing the validation. Defaults to true.
The name of the database from which to read the sequence information. Should be a name known to Bucardo, thanks to a previous call to add database. Required.
relgroup
Adds the sequence to the named relgroup. If the relgroup does not exist, it will be created. Optional.
add all tables
bucardo add all tables [relgroup=xxx] [pkonly]
Adds all the tables in all known databases or in a specified database. Excludes tables in the pg_catalog, information_schema, and bucardo schemas. (Yes, this means that you cannot replicate the Bucardo configuration database using Bucardo. Sorry about that.) Supported options and parameters:
db
--db
Name of the database from which to find all the tables to add. If not provided, tables will be added from all known databases.
schema
--schema
-n
Limit to the tables in the specified comma-delimited list of schemas. The options may be specified more than once.
exclude-schema
--exclude-schema
-N
Exclude tables in the specified comma-delimited list of schemas. The options may be specified more than once.
table
--table
-t
Limit to the specified tables. The options may be specified more than once.
exclude-table
--exclude-table
-T
Exclude the specified tables. The options may be specified more than once.
relgroup
--relgroup
Name of the relgroup to which to add new tables.
pkonly
Exclude tables without primary keys.
add all sequences
bucardo add all sequences relgroup=xxx
Adds all the sequences in all known databases or in a specified database. Excludes sequences in the pg_catalog, information_schema, and bucardo schemas. (Yes, this means that you cannot replicate the Bucardo configuration database using Bucardo. Sorry about that.) Supported options and parameters:
db
--db
Name of the database from which to find all the sequences to add. If not provided, sequences will be added from all known databases.
schema
--schema
-n
Limit to the sequences in the specified comma-delimited list of schemas. The options may be specified more than once.
exclude-schema
--exclude-schema
-N
Exclude sequences in the specified comma-delimited list of schemas. The options may be specified more than once.
relgroup
--relgroup
Name of the relgroup to which to add new tables or sequences.
add relgroup
bucardo add relgroup name
bucardo add relgroup name table, sequence, ...
Adds a relgroup. After the name, pass in an optional list of tables and/or sequences and they will be added to the group.
add sync
bucardo add sync name relgroup=xxx dbs=xxx
Adds a sync, which is a named replication event containing information about what to replicate from where to where. The supported parameters are:
dbs
The name of a dbgroup or comma-delimited list of databases. All of the specified databases will be synchronized. Required.
dbgroup
The name of a dbgroup. All of the databases within this group will be part of the sync. If the dbgroup does not exists and a separate list of databases is given, the group will be created and populated.
relgroup
The name of a relgroup to synchronize. All of the tables and/or sequences in the relgroup will be synchronized. Required unless tables is specified.
tables
List of tables to add to the sync. This implicitly creates a relgroup with the same name as the sync. Required unless relgroup is specified.
status
Indicates whether or not the sync is active. Must be either "active" or "inactive". Defaults to "active".
rebuild_index
Boolean indicating whether or not to rebuild indexes after every sync. Defaults to off.
lifetime
Number of seconds a KID can live before being reaped. No limit by default.
maxkicks
Number of times a KID may be kicked before being reaped. No limit by default.
conflict_strategy
The conflict resolution strategy to use in the sync. Supported values:
bucardo_source
The rows on the "source" database always "win". In other words, in a conflict, Bucardo copies rows from source to target.
bucardo_target
The rows on the "target" database always win.
bucardo_skip
Any conflicting rows are simply not replicated. Not recommended for most cases.
bucardo_random
Each database has an equal chance of winning each time. This is the default.
bucardo_latest
The row that was most recently changed wins.
bucardo_abort
The sync is aborted on a conflict.
onetimecopy
Determines whether or not a sync should switch to a full copy mode for a single run. Supported values are:
0: off
1: always full copy
2: only copy tables that are empty on the target
stayalive
Boolean indicating whether or not the sync processes (CTL) should be persistent. Defaults to false.
kidsalive
Boolean indicating whether or not the sync child processes (KID) should be persistent. Defaults to false.
autokick
Boolean indicating whether or not tables in the sync should automatically send kick messages when they're modified. May be overridden by the autokick parameter of individual tables.
checktime
An interval specifying the maximum time a sync should go before being kicked. Useful for busy systems where you don't want the overhead of notify triggers.
priority
An integer indicating the priority of the sync. Lower numbers are higher priority. Currently used only for display purposes.
analyze_after_copy
Boolean indicating whether or not to analyze tables after every sync. Off by default. Optional.
overdue
An interval specifying the amount of time after which the sync has not run that it should be considered overdue. check_bucardo_sync issues a warning when a sync has not been run in this amount of time.
expired
An interval specifying the amount of time after which the sync has not run that it should be considered expired. check_bucardo_sync issues a critical message when a sync has not been run in this amount of time.
track_rates
Boolean indicating whether or not to track synchronization rates.
rebuild_index
Boolean indicating whether or not to rebuild indexes after every sync. Off by default. Optional.
strict_checking
Boolean indicating whether or not to be strict when comparing tables in the sync. If the columns have different names or data types, the validation will fail. But perhaps the columns are allowed to have different names or data types. If so, disable strict_checking and column differences will result in warnings rather than failing the validation. Defaults to true.
Creates a new Bucardo custom name mapping. This allows the tables involved in replication to have different names on different databases. The oldname must contain the schema as well as the table name (if the source database supports schemas). The optional parameters limit it to one or more databases, and/or to one or more syncs. Supported parameters:
sync
A sync to which to add the customname. May be specified multiple times.
database
db
A database for which to add the customname. May be specified multiple times.
Specify the list of columns to select from when syncing. Rather than the default SELECT * behavior, you can specify any columns you want, including the use of function call return values and things not in the source column list. The optional parameters limit it to one or more databases, and/or to one or more syncs. Some examples:
bucardo add customcols public.foobar "select a, b, c"
bucardo add customcols public.foobar "select a, upper(b) AS b, c" db=foo
bucardo add customcols public.foobar "select a, b, c" db=foo sync=abc
Supported parameters:
sync
A sync to which to add the customcols. May be specified multiple times.
database
db
A database for which to add the customcols. May be specified multiple times.
Adds a customcode, which is a Perl subroutine that can be run at certain points in the sync process. It might handle exceptions, handle conflicts, or just run at certain times with no expectation of functionality (e.g., before Bucardo drops triggers). Metadata about that point will be passed to the subroutine as a hash reference.
Supported parameters:
name
The name of the custom code object.
about
A short description of the custom code.
whenrun
when_run
A string indicating when the custom code should be run. Supported values include:
before_txn
before_check_rows
before_trigger_drop
after_trigger_drop
after_table_sync
exception
conflict
before_trigger_enable
after_trigger_enable
after_txn
before_sync
after_sync
getdbh
Boolean indicating whether or not Perl DBI database handles should be provided to the custom code subroutine. If true, database handles will be provided under the dbh key of the hash reference passed to the subroutine. The value under this key will be a hash reference mapping database names to their respective handles.
sync
Name of the sync with which to associate the custom code. Cannot be used in combination with relation.
relation
Name of the table or sequence with which to associate the custom code. Cannot be used in combination with sync.
status
The current status of this customcode. Anything other than "active" means the code is not run.
priority
Number indicating the priority in which order to execute custom codes. Lower numbers are higher priority. Useful for subroutines that set lastcode in order to cancel the execution of subsequent custom codes for the same when_run.
src_code
File from which to read the custom code Perl source.
The body of the Perl subroutine should be implemented in the src_code file, and not inside a sub declaration. When called, it will be passed a single hash reference with the following keys:
syncname
The name of the currently-executing sync.
version
The version of Bucardo executing the sync.
sourcename
The name of the source database.
targetname
The name of the target database.
sendmail
A code reference that can be used to send email messages.
sourcedbh
A DBI database handle to the sync source database. Provided only to custom code executed by the controller.
rellist
An array reference of hash references, each representing a relation in the sync. Provided only to custom code executed by the controller. The keys in the hash are the same as the parameters supported by "add table" and "add sequence", as appropriate.
schemaname
The schema for the table that triggered the exception. Provided only to "exception" custom codes.
tablename
The name of the table that triggered the exception. Provided only to "exception" custom codes.
error_string
The string containing the actual error message. Provided only to "exception" custom codes.
deltabin
A hash reference with the name of each source database as a key and a list of all primary keys joined together with "\0". Provided only to "exception" custom codes.
attempts
The number of times the sync has been attempted. Provided only to "exception" custom codes.
conflicts
A hash reference of conflicting rows. The keys are the primary key values, and the values are hash references with the names of the databases containing the conflicting rows and true values. Provided only to "conflict" custom codes.
The custom code subroutine may set any of these keys in the hash reference to change the behavior of the sync:
message
Message to send to the logs.
warning
A warning to emit after the subroutine has returned.
error
An error to be thrown after the subroutine has returned.
nextcode
Set to send execution to the next custom code of the same type. Mainly useful to exception custom codes, and supported only by custom codes executed by the controller.
lastcode
Set to true to have any subsequent custom codes of the same type to be skipped.
endsync
Cancels the sync altogether.
An example:
use strict;
use warnings;
use Data::Dumper;
my $info = shift;
# Let's open a file.
my $file = '/tmp/bucardo_dump.txt';
open my $fh, '>:encoding(UTF-8)', $file or do {
$info->{warning} = "Cannot open $file: $!\n";
return;
};
# Inspect $info for fun.
print $fh Dumper $info;
close $fh or $info->{warning} = "Error closing $file: $!\n";
# Log a message and return.
$info->{message} = 'IN UR DATABASEZ NORMALIZIN UR RELAYSHUNS';
return;
update
bucardo update <type> <name> <parameters>
Updates a Bucardo object. The type specifies the type of object to update, while the name should be the name of the object. The supported parameters for each type are the same as those for "add". The supported types are:
customcode
db
sync
table
sequence
update customcode
bucardo update customcode <name> setting=value
Updates an existing customcode. Items that can be changed are:
about
A short description of the custom code.
getdbh
Boolean indicating whether or not Perl DBI database handles should be provided to the custom code subroutine. If true, database handles will be provided under the dbh key of the hash reference passed to the subroutine. The value under this key will be a hash reference mapping database names to their respective handles.
name
The name of the custom code object.
priority
Number indicating the priority in which order to execute custom codes. Lower numbers are higher priority. Useful for subroutines that set lastcode in order to cancel the execution of subsequent custom codes for the same when_run.
src_code
File from which to read the custom code Perl source.
status
The current status of this customcode. Anything other than "active" means the code is not run.
whenrun
A string indicating when the custom code should be run. Supported values include:
before_txn
before_check_rows
before_trigger_drop
after_trigger_drop
after_table_sync
exception
conflict
before_trigger_enable
after_trigger_enable
after_txn
before_sync
after_sync
update db
bucardo udpate db <name> port=xxx host=xxx user=xxx pass=xxx
Updates a database. The name is the name by which the database is known to Bucardo. This may vary from the actual database name, as multiple hosts might have databases with the same name.
The supported named parameters are:
dbname
db
The actual name of the database.
type
dbtype
The type of the database. Currently supported values are:
postgres
drizzle
mongo
mysql
maria
oracle
redis
sqlite
username
dbuser
dbdsn
A direct DSN to connect to a database. Will override all other connection options if set.
user
The username Bucardo should use to connect to the database.
password
dbpass
pass
The password Bucardo should use when connecting to the database.
Status of the database in Bucardo. Must be either "active" or "inactive".
dbgroup
server_side_prepares
ssp
Enable or disable server-side prepares. Pass 1 to enable them or 0 to disable them.
makedelta
Enable or disable makedelta for this database.
dbservice
service
The service name to use for a Postgres database.
dbgroup
A comma-separated list of dbgroups to which to add the database. The database will be removed from any other dbgroups of which it was previously a member.
update sync
bucardo update sync syncname relgroup=xxx dbs=xxx
Updates a sync, which is a named replication event containing information about what to replicate from where to where. The supported parameters are:
name
The name of the sync. Required.
dbs
The name of a dbgroup or comma-delimited list of databases.
relgroup
The name of a relgroup to synchronize.
status
Indicates whether or not the sync is active. Must be either "active" or "inactive". Note that this will not change the current run status of the sync, just mark whether it should be active or inactive on the next reload. Use the activate sync and <deactivate sync> commands to actually activate or deactivate a sync.
rebuild_index
Boolean indicating whether or not to rebuild indexes after every sync.
lifetime
Number of seconds a KID can live before being reaped.
maxkicks
Number of times a KID may be kicked before being reaped.
isolation_level
The transaction isolation level this sync should use. Only choices are "serializable" and "repeatable read"
conflict_strategy
The conflict resolution strategy to use in the sync. Supported values:
bucardo_source
The rows on the "source" database always "win". In other words, in a conflict, Bucardo copies rows from source to target.
bucardo_target
The rows on the "target" database always win.
bucardo_latest
The row that was most recently changed wins.
bucardo_abort
The sync is aborted on a conflict.
onetimecopy
Determines whether or not a sync should switch to a full copy mode for a single run. Supported values are:
0: off
1: always full copy
2: only copy tables that are empty on the target
stayalive
Boolean indicating whether or not the sync processes (CTL) should be persistent.
kidsalive
Boolean indicating whether or not the sync child processes (KID) should be persistent.
autokick
Boolean indicating whether or not tables in the sync should automatically send kick messages when they're modified. May be overridden by the autokick parameter of individual tables.
checktime
An interval specifying the maximum time a sync should go before being kicked. Useful for busy systems where you don't want the overhead of notify triggers.
priority
An integer indicating the priority of the sync. Lower numbers are higher priority. Currently used only for display purposes.
analyze_after_copy
Boolean indicating whether or not to analyze tables after every sync. Off by default.
overdue
An interval specifying the amount of time after which the sync has not run that it should be considered overdue. check_bucardo_sync issues a warning when a sync has not been run in this amount of time.
expired
An interval specifying the amount of time after which the sync has not run that it should be considered expired. check_bucardo_sync issues a critical message when a sync has not been run in this amount of time.
track_rates
Boolean indicating whether or not to track synchronization rates.
rebuild_index
Boolean indicating whether or not to rebuild indexes after every sync.
strict_checking
Boolean indicating whether or not to be strict when comparing tables in the sync. If the columns have different names or data types, the validation will fail. But perhaps the columns are allowed to have different names or data types. If so, disable strict_checking and column differences will result in warnings rather than failing the validation. Defaults to true.
Updates a table object. The table information will be read from the specified database. Supported parameters:
db
The name of the database from which to read the table information. Should be a name known to Bucardo.
schemaname
The name of the schema in which the table is found.
tablename
The actual name of the table.
autokick
Boolean indicating whether or not the table should automatically send kick messages when it's modified. Overrides the autokick parameter of any syncs of which the table is a part.
rebuild_index
Boolean indicating whether or not to rebuild indexes after every sync.
analyze_after_copy
Boolean indicating whether or not to analyze the table after every sync.
vacuum_after_copy
Boolean indicating whether or not to vacuum the table after every sync.
relgroup
Adds the table to the named relgroup. May be specified more than once. The table will be removed from any other relgroups.
makedelta
Specifies which databases need makedelta enabled for this table.
strict_checking
Boolean indicating whether or not to be strict when comparing the table between syncs. If the columns have different names or data types, the validation will fail. But perhaps the columns are allowed to have different names or data types. If so, disable strict_checking and column differences will result in warnings rather than failing the validation. Defaults to true.
The name of the database where the sequence lives.
schemaname
The name of the schema in which the sequence is found.
relgroup
Adds the sequence to the named relgroup. May be speci<fied more than once. The sequence will be removed from any other relgroups.
remove
bucardo remove <item_type> <item_name>
Removes one or more objects from Bucardo. Valid item types are;
db or database
Use the --force option to clear out related tables and groups instead of erroring out.
dbgroup
relgroup
sync
table
sequence
customcols
customname
customcode
kick
bucardo kick <syncname(s)> [timeout]
Tells one or more named syncs to fire as soon as possible. Note that this simply sends a request that the sync fire: it may not start right away if the same sync is already running, or if the source or target database has exceeded the number of allowed Bucardo connections. If the final argument is a number, it is treated as a timeout. If this number is zero, the bucardo command will not return until the sync has finished. For any other number, the sync will wait at most that number of seconds. If any sync has not finished before the timeout, an exit value of 1 will be returned. Errors will cause exit values of 2 or 3. In all other cases, an exit value of 0 will be returned.
If a timeout is given, the total completion time in seconds is also displayed. If the sync is going to multiple targets, the time that each target takes from the start of the kick is also shown as each target finishes. Options:
--retry
The number of times to retry a sync if it fails. Defaults to 0.
--retry-sleep
How long to sleep, in seconds, between each retry attempt.
--notimer
By default, kicks with a timeout argument give a running real-time summary of time elapsed by using the backspace character. This may not be wanted if running a kick, for example, via a cronjob, so turning --notimer on will simply print the entire message without backspaces.
pause
bucardo pause <syncname(s)>
bucardo pause all
bucardo resume <syncname(s)>
bucardo resume all
Tells one or more named syncs to temporarily pause, or to resume from a previous pause. This only applies to active syncs and only takes effect if Bucardo is currently running. The keyword 'all' can be used as well to pause or resume all known active syncs.
reload config
bucardo reload config
bucardo reload config 30
Sends a message to all CTL and KID processes asking them to reload the Bucardo configuration. This configuration is a series of key/value pairs that configure Bucardo's behavior, and not any of the objects managed by the add, remove, or update commands.
By default, Bucardo will send the message and then exit. Pass an optional number and Bucardo will instead wait up to that length of time for all child processes to report completion.
set
bucardo set setting1=value [setting2=value]
Sets one or more configuration setting table. Setting names are case-insensitive. The available settings are:
autosync_ddl
Which DDL changing conditions do we try to remedy automatically? Default: newcol.
bucardo_version
Current version of Bucardo. Default: 5.6.0.
bucardo_vac
Do we want the automatic VAC daemon to run? Default: 1.
bucardo_initial_version
Bucardo version this schema was created with. Default: 5.6.0.
ctl_checkonkids_time
How often does the controller check on the kids health? Default: 10.
ctl_createkid_time
How long do we sleep to allow kids-on-demand to get on their feet? Default: 0.5.
ctl_sleep
How long does the controller loop sleep? Default: 0.2.
default_conflict_strategy
Default conflict strategy for all syncs. Default: bucardo_latest.
default_email_from
Who the alert emails are sent as. Default: nobody@example.com.
default_email_host
Which host to send email through. Default: localhost.
default_email_to
Who to send alert emails to. Default: nobody@example.com.
email_debug_file
File to save a copy of all outgoing emails to. Default: None.
endsync_sleep
How long do we sleep when custom code requests an endsync? Default: 1.0.
flatfile_dir
Directory to store the flatfile output inside of. Default: ..
host_safety_check
Regex to make sure we don't accidentally run where we should not. Default: None.
isolation_level
The transaction isolation level all sync should use. Defaults to 'serializable'. The only other valid option is 'repeatable read'
kid_deadlock_sleep
How long to sleep in seconds if we hit a deadlock error. Default: 0.5. Set to -1 to prevent the kid from retrying.
kid_nodeltarows_sleep
How long do kids sleep if no delta rows are found? Default: 0.5.
kid_pingtime
How often do we ping check the KID? Default: 60.
kid_restart_sleep
How long to sleep in seconds when restarting a kid? Default: 1.
kid_serial_sleep
How long to sleep in seconds if we hit a serialization error. Default: 0.5. Set to -1 to prevent the kid from retrying.
kid_sleep
How long does a kid loop sleep? Default: 0.5.
log_conflict_file
Name of the conflict detail log file. Default: bucardo_conflict.log.
log_level
How verbose to make the logging. Higher is more verbose. Default: normal.
log_microsecond
Show microsecond output in the timestamps? Default: 0.
log_showlevel
Show log level in the log output? Default: 0.
log_showline
Show line number in the log output? Default: 0.
log_showpid
Show PID in the log output? Default: 1.
log_showtime
Show timestamp in the log output? 0=off 1=seconds since epoch 2=scalar gmtime 3=scalar localtime. Default: 3.
log_timer_format
The strftime format to use to format the log timestamp when log_showtime is set to 2 or 3. Defaults to simply the scalar output of the time.
mcp_dbproblem_sleep
How many seconds to sleep before trying to respawn. Default: 15.
mcp_loop_sleep
How long does the main MCP daemon sleep between loops? Default: 0.2.
mcp_pingtime
How often do we ping check the MCP? Default: 60.
mcp_vactime
How often in seconds do we check that a VAC is still running? Default: 60.
File to hold reasons for stopping and starting. Default: bucardo.restart.reason.txt.
reload_config_timeout
Number of seconds the reload_config command should wait for the reload to complete. Default: 30.
semaphore_table
Table to let apps know a sync is ongoing. Default: bucardo_status.
statement_chunk_size
How many primary keys to shove into a single statement. Default: 10000.
stats_script_url
Location of the stats script. Default: http://www.bucardo.org/.
stopfile
Name of the semaphore file used to stop Bucardo processes. Default: fullstopbucardo.
syslog_facility
Which syslog facility level to use. Default: log_local1.
tcp_keepalives_count
How many probes to send. 0 indicates sticking with system defaults. Default: 0.
tcp_keepalives_idle
How long to wait between each keepalive probe. Default: 0.
tcp_keepalives_interval
How long to wait for a response to a keepalive probe. Default: 0.
vac_run
How often does the VAC process run? Default: 30.
vac_sleep
How long does VAC process sleep between runs? Default: 120.
warning_file
File containing all log lines starting with "Warning". Default: bucardo.warning.log.
show
bucardo show all|changed|<setting> [<setting>...]
Shows the current Bucardo settings. Use the keyword "all" to see all the settings, "changed" to see settings different than the installed defaults, or specify one or more search terms. See "set" for complete details on the configuration settings.
config
bucardo config show all|<setting> [<setting>...]
bucardo config set <setting=value> [<setting=value>...]
Deprecated interface for showing and setting configuration settings. Use the "show" and "set" commands, instead.
ping
bucardo ping
bucardo ping 60
bucardo ping 0
Sends a ping notice to the MCP process to see if it will respond. By default, it will wait 15 seconds. A numeric argument will change this timeout. Using a 0 as the timeout indicates waiting forever. If a response was returned, the program will exit with a value of 0. If it times out, the value will be 1. Returns a Nagios like message starting with "OK" or "CRITICAL" for success or failure.
status
bucardo status [syncname(s)] [--sort=#] [--show-days] [--compress]
Shows the brief status of all known syncs in a tabular format. If given one or more sync names, shows detailed information for each one. To see detailed information for all syncs, simply use "status all"
When showing brief information, the columns are:
1. Name
The name of the sync
2. State
The state of the sync. Can be 'Good', 'Bad', 'Empty', 'No records found', 'Unknown', or the run state for a currently-running sync.
3. Last good
When the sync last successfully ran.
4. Time
How long it has been since the last sync success
5. Last I/U
The number of insert and deletes performed by the last successful sync. May also show the number of rows truncated (T) or conflicted (C), if applicable.
6. Last bad
When the sync last failed.
7. Time
How long it has been since the last sync failure
The options for status are:
--show-days
Specifies whether or not do list the time interval with days, or simply show the hours. For example, "3d 12h 6m 3s" vs. "48h 6m 3s"
--compress
Specifies whether or not to compress the time interval by removing spaces. Mostly used to limit the width of the 'status' display.
--sort=#
Requests sorting of the 'status' output by one of the nine columns. Use a negative number to reverse the sort order.
Activates one or more named syncs. If given a timeout argument, it will wait until it has received confirmation from Bucardo that each sync has been successfully activated.
Deactivates one or more named syncs. If given a timeout argument, it will wait until it has received confirmation from Bucardo that the sync has been successfully deactivated.
message
bucardo message 'I WAS HERE'
Sends a message to the running Bucardo logs. This message will appear prefixed with "MESSAGE: ". If Bucardo is not running, the message will go to the logs the next time Bucardo runs and someone adds another message.
reload
bucardo reload [syncname2 syncname3 ...]
Sends a message to one or more sync processes, instructing them to reload. Waits for each to reload before going on to the next. Reloading consists of deactivating a sync, reloading its information from the database, and activating it again.
inspect
bucardo inspect <type> <name> [<name2>...]
Inspects one or more objects of a particular type. The results are sent to STDOUT. The supported types include:
table
sync
relgroup
validate
bucardo validate all|<sync> [<sync>...]
Validates one or more syncs. Use the keyword "all" to validate all syncs, or specify one or more syncs to validate.
Note that this command executes a subset of all the validation done when a sync is started or activated.
purge
bucardo purge all|<table> [<table>...]
Purges the delta and track tables for one or more tables, for one or more databases. Use the keyword "all" to validate all tables, or specify one or more tables to validate.
delta
bucardo delta [total] [<database>...]
Show the current delta count for each source target. Provide a list of databases to limit it to just the given ones. Wildcards are allowed. Use the special name "totals" to show only the grand total.
help
bucardo help
bucardo help <command>
bucardo help <command> <action>
Get help. General help can be returned, as well as help for a single command or a command and its action. Some examples:
bucard help list
bucard help add table
OPTIONS DETAILS
It is usually easier to set most of these options at the top of the script, or make an alias for them, as they will not change very often if at all.
-d
--db-name
bucardo --db-name widgets
bucardo -d bricolage
Name of the Bucardo database to which to connect.
-U
--db-user
bucardo --db-user postgres
bucardo -U Mom
User name to use when connecting to the Bucardo database.
-P
--db-pass
bucardo --db-pass s3cr1t
bucardo -P lolz
Password to use when connecting to the Bucardo database.
Host name to use when connecting to the Bucardo database.
-p
--db-port
bucardo --db-port 7654
Port number to connect to when connecting to the Bucardo database.
--bucardorc
bucardo --bucardorc myrcfile
Use the specified file for configuration instead of the default ./.bucardorc.
--no-bucardorc
Do not use the ./.bucardorc configuration file.
--verbose
Makes bucardo run verbosely. Default is off.
--quiet
Tells bucardo to be as quiet as possible. Default is off.
--help
Shows a brief summary of usage for bucardo.
FILES
In addition to command-line configurations, you can put any options inside of a file. The file .bucardorc in the current directory will be used if found. If not found, then the file ~/.bucardorc will be used. Finally, the file /etc/bucardorc will be used if available. The format of the file is option = value, one per line. Any line starting with a '#' will be skipped. Any values loaded from a bucardorc file will be overwritten by command-line options. All bucardorc files can be ignored by supplying a --no-bucardorc argument. A specific file can be forced with the --bucardorc=file option; if this option is set, bucardo will refuse to run unless that file can be read.
ENVIRONMENT VARIABLES
The bucardo script uses $ENV{HOME} to look for a .bucardorc file.
This program is free to use, subject to the limitations in the LICENSE file.
Bucardo-5.6.0/UPGRADE 000644 000770 000024 00000004214 12061433623 014642 0 ustar 00machack staff 000000 000000 These directions are for upgrading an existing Bucardo installation.
If you encounter any problems, please email the mailing list at
bucardo-general@bucardo.org.
Ideally, upgrading Bucardo takes just five steps:
1. Install the new Bucardo programs
2. Stop Bucardo
3. Upgrade the Bucardo database
4. Re-validate all syncs
5. Start Bucardo
Each step is detailed below.
1. Install the new Bucardo programs
As with an install, just untar the file (or git pull), then run:
perl Makefile.PL
make
make test
make install
The first three steps should be done as a non-superuser. Thus, the final
step should be done on some systems as:
sudo make install
This will replace the existing Bucardo.pm file, the bucardo script, the
bucardo.schema file, and the associated man pages.
2. Stop Bucardo
Because we may be making schema changes, it is safest to stop Bucardo while
the upgrade is performed. This can be done by running:
./bucardo stop
3. Upgrade the Bucardo database
The bucardo script has an 'upgrade' command that connects to the
existing Bucardo database and upgrades the schema as needed.
./bucardo upgrade
Note:
* This must be run on every Bucardo instance you have (most sites only use one)
You should see a summary of any changes made, and then given a chance to
accept or reject the changes. You may want to run it again after, to double
check that the upgrade worked and no further changes are needed.
A summary of all changes is stored in the bucardo.upgrade_log table.
4. Re-validate all syncs
Because there may have been schema changes on the remote databases, you may
have to run the validate_sync() function against each of your syncs. This command
will connect to each remote database and do things like verify that all
bucardo-related functions and tables are installed correctly. The easiest way to
do this is by running:
./bucardo validate all
Alternatively, you can specify one or more syncs like so:
./bucardo validate sync1 sync2
5. Restart Bucardo
The final step is to restart Bucardo:
./bucardo start "Upgrade"
It's probably a good idea to check out the Bucardo log at this point and make
sure everything is working properly.
Bucardo-5.6.0/INSTALL 000644 000770 000024 00000003274 13523370351 014670 0 ustar 00machack staff 000000 000000 Installation instructions for Bucardo.
For a possibly more up to date and complete version, please visit:
https://bucardo.org/wiki/Bucardo/Installation
Before installing, please make sure you have the latest version.
Please visit:
https://bucardo.org/wiki/Bucardo
It is recommended that you create a 'bucardo' user on your system
that will be responsible for running Bucardo.
Install Bucardo like any other Perl module:
perl Makefile.PL
make
make test
make install
The last step must be done as root. You may be able to do this by running:
sudo make install
Note that 'make test' is a rather long and involved process.
However, because Bucardo is fairly complex, testing is highly
recommended before installing.
If you are using a MongoDB target, you will also need to install the 'boolean'
Perl module.
Once the above has been run, you will need to create a Bucardo database.
This can be done by running:
bucardo install
There should be no errors: if there are, please report them to
bucardo-general@bucardo.org or ask on IRC for help.
IRC: #bucardo irc.freenode.net
Bucardo by default logs to syslog, and does so very verbosely. It is
recommended that you reroute your bucardo syslog traffic to another file.
This can be done by adding this line to your /etc/syslog.conf file:
local1.* -/var/log/bucardo.log
Of course, if "local1" is already being used for something else, you will
need to change it - see the "syslog_facility" entry in the bucardo_config
table.
Once this is done, make sure you restart your syslog daemon so it rereads the
new configuration file.
Please see the documentation at https://bucardo.org for information on how to
start using Bucardo once it is installed.
Bucardo-5.6.0/LICENSE 000644 000770 000024 00000002426 13616304302 014636 0 ustar 00machack staff 000000 000000 Copyright (c) 2005-2020
Greg Sabino Mullane . All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
OF SUCH DAMAGE.
Bucardo-5.6.0/SIGNATURE 000644 000770 000024 00000014204 13626305405 015120 0 ustar 00machack staff 000000 000000 This file contains message digests of all files listed in MANIFEST,
signed via the Module::Signature module, version 0.83.
To verify the content in this distribution, first make sure you have
Module::Signature installed, then type:
% cpansign -v
It will check each file's integrity, as well as the signature's
validity. If "==> Signature verified OK! <==" is not displayed,
the distribution may already have been compromised, and you should
not run its Makefile.PL or Build.PL.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
SHA256 54289050c1d5f4c3232f266a362a446c5e0cad036076c6d9b4561469c0280ea6 .gitignore
SHA256 df4ab0f3f03db9feb9bb8a164f51c50c0b8330e05d7ab6b6571001537a88dda9 .perlcriticrc
SHA256 c5b0720e78dad153754603103ef91e0c3c2f0a58b68ac781489710259e2c0e73 Bucardo.pm
SHA256 cdc0b45c3b707fd47f8c8bb5affe81ead20468b6dbc5d03bf494433ec4da6a30 Bucardo.pm.html
SHA256 d024f66f14f6c30c2f12314b0fcb22192587b527669d0cdb5a4cc4deceb5bb62 Changes
SHA256 e13fcbe4bd8eda7bb22368879c3ae274973e1eafb521d4dfdd7bf84d6e7efbf8 INSTALL
SHA256 0e16031937359e9049c63491b153485c15cf95c013ea3dbb51ca2ff833bcd454 LICENSE
SHA256 b444bbca1dbcac10955eef4b38383cb23722ef19bf4cde0cf47e165e8cf7eb62 MANIFEST
SHA256 ba3958c32919ceffaf02a9f20542b936044318b9c13cc6b237aa2604b0fd68c9 MANIFEST.SKIP
SHA256 28731a01493f37a05fc9c0a72daefc4c4fe66326d66b9f38f89398f053e7df03 META.yml
SHA256 5ca453f73154241cabbdc2d877c0d564f03de2ef321ac3e7893575d0d47b6732 Makefile.PL
SHA256 78875d244c4b983a857ace7a88c4d0433840e3a748c1db8fb68fec9f43244b95 README
SHA256 72c941a68ce4210f187b73a726ad66278c3362557ee50d6364ecb6e9028e0d3b README.dev
SHA256 853cab84b69d3639231ec01309dc669e117d28134b58c24bddc77083c5bcdeda TODO
SHA256 0e544d8c34cfbdbc22099a346c338ad962d688727774bb0620099ad413d7507f UPGRADE
SHA256 eb0bc0d569f567a099fbcc363a3f701adadba159729ecaae767d232adbf0bac2 bucardo
SHA256 1bd087e574ba58ec6b10ba98da2dc618f5c4beaaf55cb7c9194d958c52f5a299 bucardo.html
SHA256 f82c9a766e42405238f513eb7c7ca88c9a782c411fc67fd42925febe1f05b1ac bucardo.schema
SHA256 9d060de5d4cf109d77f8d69666d5a7572ce2aaac0b3739349e267d7cf3c51613 patches/Bucardo.4.4.0.fix.search.path.patch.txt
SHA256 5b698c0904fe10dfca2ed22ffd5facc57f9de9001f305c52b3f7a54e0623a107 scripts/README
SHA256 08d7df5ed4b8c3d7f36056eca1a1c950b56d6eb98312720eb231db327c39fac0 scripts/bucardo-report
SHA256 1e556f54e1a47b0e0b8d004fa329b0f3477ae88092f33b3da44cfde24b25c75a scripts/bucardo_rrd
SHA256 58300e4312f74310eabdfcb644f5d4ee29ccca3a220cc492e722b4c373b73b62 scripts/check_bucardo_sync
SHA256 1ad92de01077a0508ae08ee3d35d977fd8ff6ddd4b89ace9528c814bbf0a1925 scripts/slony_migrator.pl
SHA256 8d1b50f17b3d93203acb826b6d2d0f9490dfb85cf63f9f1007194b64f313dcf4 t/01-basic.t
SHA256 68e1eb5ebe7b88e82ce023b43a45a8b90df759cf1775c74a79abf3b41e60da1b t/02-bctl-customname.t
SHA256 dd4575f59f57448abebd7a2c7b8ff6a31f93a9e343aedb2da35e3f3092f659c6 t/02-bctl-db.t
SHA256 8c800f5e1d3689e5eb6c4139ac632d3a42dfbd9419044e45fa4540e642719730 t/02-bctl-dbg.t
SHA256 98852aa197595f6296f659803dd863f758f79838da9d14f62900943752b0bc43 t/02-bctl-herd.t
SHA256 08f3446fa9b8037d5f0812fda380581b1e112b054e677fcef3ac1b227e443439 t/02-bctl-sync.t
SHA256 3ca67e7abfc392f626098b135db089ba01215f9f7c34105d6c2ffc9a220910c7 t/02-bctl-table.t
SHA256 de5710308891d3c71c95c259c79b2c1cdccf2e37c6d99b8a07c8e67f19c3022c t/10-fullcopy.t
SHA256 99106c47629c31d0058471d1b88403698edbe4e146b0aacec7e870a689a26564 t/10-makedelta.t
SHA256 ba0636552517bc13ca6751d7269b9b0448906649d076645862e7e612af9f00b7 t/10-object-names.t
SHA256 ab67412e70fdb3f92146481ad218f694ab211f914c7d4228ce6d4e77a37486d9 t/20-drizzle.t
SHA256 2cc4d779fa25adb57a82e0e44284e03ff11c97fcb66cdfef80df933c0404e05b t/20-firebird.t
SHA256 037148afd1734f1d7246032492feb685fa0cea61a1bf681f19490f0e95cb4b01 t/20-mariadb.t
SHA256 61976021f99a566734ea71e072b3c8f50892d986bf0f39278b3b31a401e0d338 t/20-mongo.t
SHA256 3853d7141c7cdcd1a84166d7b3b3163d57740afd23fbc6a71c03281af6d98273 t/20-mysql.t
SHA256 36a7e65ea927b697e45979eac0664b9800a6c828b20e53be3ee84e5bdf9c2834 t/20-oracle.t
SHA256 818baa9c8fd899d2e0f359bc3ddba6cccb50294417fc1dba9b785bf47f6fd6d9 t/20-postgres.t
SHA256 b86d76717275b901ea79cb7652a7eeff66b36c67d73e3db7240d2c2607150af3 t/20-redis.t
SHA256 783647719ab866487072425a8936ae2a82c8a8c621fb14ebe32ae02cd5b7a794 t/20-sqlite.t
SHA256 8447a6691bbeadb7fa7c459f4f316026eda0639d5ede6b601c46eafe5ea7b7f0 t/30-crash.t
SHA256 fe84905b96296f08bf24a253f298ceea7507cf927a6040c43d8841b3eea031c1 t/30-delta.t
SHA256 19c515dd2111a10c95d8117fb6592faf72c6d43c4930468ac1752871cae3c9b0 t/40-conflict.t
SHA256 9fb6d2f0493182c580d9146804c24ae94ad0c967fe5726d057fdd01bf219a4ee t/40-customcode-exception.t
SHA256 a6b980bcd8f127171a052dbf89d4e7e517bbf8e443ae167f8fdeb5dc97521411 t/40-serializable.t
SHA256 8c831610d6eadb561ea71020bd59768fe9c99082024dd4d0b2d6979461a3d146 t/50-star.t
SHA256 467ff3fc21cd0b89dda96ba91268af58c795b974569dc03d353a8518d78877ee t/98-cleanup.t
SHA256 1893502aab0416d7bed7a532e3144f58ca625feaaf436443093cd47c6def7e13 t/99-signature.t
SHA256 d8ad6c8e41f7b4c3e82b877bdc363cb42b8267b154f013efe9a71f3e07876cf3 t/BucardoTesting.pm
SHA256 2994e3712e39b0de1844d639421dc1311891a72fc917bcb57b011de674476fe5 t/ctest1.pl
SHA256 fee67a2df6ae7958e266a9582302ab3bb281c3723b6af83bae6266c9c8b6b330 t/ctest2.pl
SHA256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 t/ctest3.pl
SHA256 0f14ec4c21ee08a7c0417d2c828452cf2530c4e09b7c4ef10a847ee8fbe9f456 t/customcode.exception.bucardotest.pl
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEECKA1l4YEfF4eK4/eN0ojyd+bZbgFAl5ZiwUACgkQN0ojyd+b
Zbgehg/+JYOHDfbNRZq7XepMK1NYYcGOzkYdQ3nkrmh4J+DafMeELK06utmR1nr3
8aza4yH0FU2Z5VpHU2UewOAvLG7yO8RijpkvjEtmsU8TNpXM9oRec6KcuiZZf6wI
s4+Hwzc1xdditPd2tUWzXJzIMhkNwaPd1pFhtGFkjuQ6zzFjrquDV5ocqQRtZ4BH
Wn7IR+Up4lwgGeplMOK6mL+w9hyB9ViPXhrQ+v6Fg8+dZWfQyhyr8AOtW84lAsZE
1cN+FvUMSpK2w3LxY0uqpEq/U5F5fXb02Au1hllMNSkJx4ydF39SpT1aM1mjpNc9
d4ZmxS8xiSA+SIcDBUp7pN4LYKgiis9HAePoK5j+YqLmV4IkPdd3YUgDb/w3Htk5
uVDGSK/aEAIpVSuUrqJnbpqB5T4iBNmV2AwPIT/qHD9Y6Q22+aONTRcXqieHnHix
n+5Z8IIn2Tf60pkmH18NccuqV0C6xf6tV84tlo3T6D4ngSJBgbccvNBdlZ0E/kCz
O/xWCdBaL7Km3PnShaEUbGTE5aAUI1VhpwQkc88LKbs0KgD7vZVGe0azQeqrCoRb
MCAfw6XgNOlowhDOL2o6LC5MN++e5DXJOYnpSvSav1l56Q3GoE9CJjvCxnOWwB/t
9eFH90OyruKax43SzwvpAp/0l8MybbHxNiGZrLq8Y/Tebg9eomc=
=1rwd
-----END PGP SIGNATURE-----
Bucardo-5.6.0/README.dev 000644 000770 000024 00000027752 13523370351 015303 0 ustar 00machack staff 000000 000000
This file is for those interested in developing Bucardo.
It is hoped that it will be a good introduction as well as a continual
reference. Suggestions are always welcome.
Sections:
* File List
* Editing
* Test Files
* Version Numbers
* New Files
* Making a New Release
* Tips and Tricks
===============
== File List ==
===============
Here is what each file in the distribution does:
* Text files:
Changes - lists changes made to each version. Please be consistent and use
tabs, not spaces, to indent. Try to list who found the bug, and who
fixed it (if not the same person). Put the person who made the actual
changes in brackets. This file contains a version number.
README.dev - you are reading it.
README - the main file that explains the module, where to get it, and guides
people in installing it. This file contain a version number.
TODO - Rough list of ideas on improving Bucardo.
SIGNATURE - Checksum verification via PGP, generated by Module::Signature.
INSTALL - Basic install instructions
UPGRADE - Basic upgrade instructions
* Build files:
Makefile.PL - Standard Perl file that creates the makefile via "perl Makefile.PL"
Note that this file also creates the html files. Contains a version number.
Makefile - Generated automatically by Makefile.PL
META.yml - YAML description file. Updated by hand and contains version
number in two places.
* Distribution files:
MANIFEST - lists which files should be included in the release tarball.
Used by the "make dist*" set of commands.
MANIFEST.SKIP - files that are known to be safe to exclude from the release
tarball. Used by the "make dist", "make distcheck" and "make skipcheck" commands.
* Program files:
Bucardo.pm - The main program. This is the core of Bucardo - everything else supports
this file. Contains a version number in two places.
bucardo - The main command-line interface to Bucardo. Contains a version number.
scripts/* - Various scripts
* Documentation files:
Bucardo.pm.html - generated by "make html" from the POD in Bucardo.pm
bucardo.html - generated by "make html" from the POD in bucardo
* Supporting files:
bucardo.schema - The schema for Bucardo. Contains a version number in two places.
.perlcriticrc - Customized Perl::Critic rules for the t/99perlcritic.t file.
.gitignore - Things we don't want git to look at
* Test files:
t/BucardoTesting.pm - Shared routines for setup and teardown.
t/00setup.t - Shared code for the Bucardo tests.
t/01basic.t - Test of very basic things.
t/02-simple.t - Very simply replication tests
t/03-goat.t - Test goat access methods.
t/04-pushdelta.t - Test pushdelta syncs.
t/02-monkey_with_db_table.t - Test db manipulation via bucardo
t/02-monkey_with_goat_table.t - Test goat manipulation via bucardo
t/02-monkey_with_herd.t - Test herd manipulation via bucardo
t/02-monkey_with_sync_table.t - Test sync manipulation via bucardo
t/05-fullcopy.t - Test fullcopy
t/06-multicolpk.t - Test multi-column primary keys
t/06-multicolpushdelta.t - Test multi-column primary keys for pushdelta syncs
t/07-multicolswap.t - Test multi-column primary keys for swap syncs
t/08-wildkick.t - Test kicking more than one sync at a time
t/09-uniqueconstraint.t - Test of customcode with pushdelta
t/10-makedelta.t - Test of makedelta functionality
t/12-addtable.t - Test adding new tables
t/13-ddl.t - Test pushing DDL (not complete yet)
t/14-truncate.t - Test of truncation replication (requires Postgres 8.4 or greater)
t/50-star.t - Test of star replication scheme
t/98-cleanup.t - Shuts down test databases. Called by "make distclean"
t/99-perlcritic.t - Uses Perl::Critic to check Bucardo.pm, bucardo, and all the
test files (including itself). Requires that TEST_CRITIC is set. It is recommended
that you get all the Perl::Critic policies via Bundle::Perl::Critic::IncludingOptionalDependencies.
t/99-pod.t - Verifies the POD of Pg.pm. Requires Test::POD version 0.95, and
Test::Pod::Coverage 1.04.
t/99-signature.t - Uses Module::Signature to verify SIGNATURE file. All tests are skipped
if the environment variable TEST_SIGNATURE is not set.
t/99-spellcheck.t - Spell checks all files and comments
t/99-yaml.t - Uses Test::YAML::Meta to verify the META.yml file.
* Developer test files are in the dev/ directory
=============
== Editing ==
=============
All the perl files should have a cperl pragma at the top of the file, for easy use in emacs.
Please use tabs and not spaces everywhere, and keep the indenting to the cperl standard.
When in doubt, use the guidelines from the Perl Best Practices book, with the following
exceptions:
* Use tabs, not spaces. When using spaces, make them 4 characters wide.
================
== Test Files ==
================
The test files are an important part of the module. Much work has gone into making
the tests as complete, thorough, and clean as possible. Please try to follow these
guidelines when developing:
* Whenever you add a new feature, no matter how minor, add a test. Better yet, add
many tests to make sure that it not only works correctly, but that it breaks when
it is supposed to (e.g. when it is fed the wrong output). Try to conceive of every
possible way your feature will be used and mis-used.
* If someone files a bug report that is not revealed by a test, please add one in,
no matter how simple the fix maybe, or how stupid the bug is.
* Don't create a new test file unless necessary - use the existing ones whenever possible.
* If you do create a new test, keep the name short, start it with a number and a dash,
and use an existing test as a template.
* Tests should be as "standalone" as possible. Most will use BucardoTesting.pm to
automate setting up and tearing down supporting test infrastructure.
* Don't call DBI->connect inside of your tests, but use connect_database() from the
dbdpg_test_setup.pl file instead. If you don't want it to blow away and recreate the
current test table and other objects, use connect_database({nosetup => 1}).
* Use the standard format for tests, and always provide an appropriate output text.
Abbreviations are encouraged, but be consistent throughout the file.
* Make sure to test on different versions of PostgreSQL, DBI, and Perl. Use the SKIP
tag with an appropriate message if a test does not work on a particular version of
something.
* To run a single test, use: prove --blib . -v t/testname.t
=====================
== Version Numbers ==
=====================
Version numbers follow the Postgres convention: major, minor, and revision.
The major number should very, very rarely change, and is saved for the truly major changes
(e.g. those that may cause backwards compatibility problems). The minor revision is used to
indicate a change in functionality, new features, etc. The revision number is used for small
tweaks and bug fixes, and must be completely compatible with the version before it.
Version numbers are currently set in nine files:
README
Bucardo.pm (two places)
Bucardo.pm.html
bucardo
bucardo.html
bucardo.schema (three places)
Changes
Makefile.PL
META.yml (three places)
===============
== New Files ==
===============
If you are adding a new file to the distribution (and this should be a rare event),
please check that you have done the following items:
* Added it to via 'git add filename' and 'git commit'
* Added it to the MANIFEST file
* Added it to Makefile.PL if needed, to make sure all build dependencies are met
* Updated/created necessary tests for it
* Added it to the "File List" section above.
==========================
== Making a New Release ==
==========================
This is a comprehensive checklist of all the steps required to release a
new version, whether beta or regular. It is assumed you are very familiar with the
other sections referenced herein.
* Test on variety of versions of Postgres
* Make sure everything is up to date and committed in git.
* Update the versions in all relevant files. Use t/00_release.t to verify this.
* If a final version, put the release date into the Changes file.
* If a beta version, please put a large warning at the top of the README file. Here is a sample:
===================================================
WARNING!!
THIS IS A TEST VERSION (1.9.1_2) AND SHOULD BE USED
FOR TESTING PURPOSES ONLY. PLEASE USE A STABLE
VERSION (no underscore) BY VISITING:
https://bucardo.org
===================================================
* If not a beta version, remove the above warning as needed from the README.
* Completely update the Changes file
The best way to do this (other than doing it as you go along) is to check the git logs.
* Update the documentation
Make sure that anything new has been documented properly, usually as pod inside of Bucardo.pm
A good way to do this is to use the tests in 99-pod.t - they will run automatically as
part of the test suite if the right modules are installed.
* Make sure the wiki is updated as well.
* Run "perl Makefile.PL"
* Run "make html". This will create the html docs for Bucardo.pm and bucardo. Make sure you
git commit them if anything has changed.
* Run "make dist". Double check that the tarball created has the correct version name.
* Run "make distcheck". This will show you a list of all files that are in the current directory
but not inside the MANIFEST file (or the MANIFEST.SKIP file). If there are any new files here
that should be part of the distribution, add them to the MANIFEST file, commit your changes,
and then re-run. Note that files ending in ".tmp" are currently skipped, so this is a good
extension for any miscellaneous files you have that use often.
* Run "make skipcheck". This will show you a list of files that will not be packed into the
release tarball. Make sure there is nothing important here.
* Run "make disttest". This unpacks the tarball, then runs "make" and "make test" on it.
You may also want to remove the directory it creates later by using "make realclean"
* Update the SIGNATURE file with Module::Signature. This should be done as the
very last step. Running "make signature" will do the signing for you.
* Create a tag for the new version with "git tag".
Example: git tag -u 14964ac8 -m "Version 4.0.1" 4.0.1
* Create a new tarball with "make dist". Sign it: "gpg -ba "
* Put the tarball and detached sig file in the downloads directory on bucardo.org
* Update the wiki pages for the new version: Bucardo/Downloads, Bucardo/Changes
* Update the bucardo/latest_version.txt file on the website.
* Change the topic on #bucardo on irc.freenode.net
* Announce to the bucardo-announce list.
* Post to pgsql-announce if this is a major or important version.
* Post to the "PostgreSQL news"
https://www.postgresql.org/account/news/new/
The content should be roughly the same as the announcement.
* Update the Postgres wiki as needed
https://wiki.postgresql.org/wiki/Bucardo (may be other pages too)
* PostgreSQL weekly news summary
The maintainers of the weekly news are usually pretty good about catching the update
and adding it in. If not, bug them.
http://www.postgresql.org/community/weeklynews/
* Tell Greg to post on PlanetPostgresql.
* On the bucardo.org server:
Put the new tarball and tarball.asc file in place
Update the download page
Update any pages that refer to the latest version
Update the latest_release.txt file
* Add the new version to Bugzilla
* Mention on our Facebook page
=====================
== Tips and Tricks ==
=====================
Also known as, the section to put things that don't fit anywhere else. Anything
that may make life easier for other developers can go here.
To test different versions of Bucardo, set two environment variables
before running the tests:
PGBINDIR: Should be a directory holding Postgres binaries for the version you want,
especially initdb and pg_config.
LD_LIBRARY_PATH: Should point to the "lib" directory of the Postgres version you
want. May not be needed if the currently installed version is close enough
to the one you want to test.
Example:
make clean
export PGBINDIR=/home/greg/code/pg/8.0.26/bin/
export LD_LIBRARY_PATH=/home/greg/code/pg/8.0.26/lib/
prove --blib . t/20-postgres.t
Bucardo-5.6.0/Bucardo.pm.html 000644 000770 000024 00000004645 13623760761 016533 0 ustar 00machack staff 000000 000000
Bucardo - Postgres multi-master replication system
VERSION
This document describes version 5.6.0 of Bucardo
WEBSITE
The latest news and documentation can always be found at:
https://bucardo.org/
DESCRIPTION
Bucardo is a Perl module that replicates Postgres databases using a combination of Perl, a custom database schema, Pl/Perlu, and Pl/Pgsql.
Bucardo is unapologetically extremely verbose in its logging.
Full documentation can be found on the website, or in the files that came with this distribution. See also the documentation for the bucardo program.
DEPENDENCIES
DBI (1.51 or better)
DBD::Pg (2.0.0 or better)
Sys::Hostname
Sys::Syslog
DBIx::Safe ## Try 'yum install perl-DBIx-Safe' or visit bucardo.org
boolean (only if using MongoDB)
BUGS
Bugs should be reported to bucardo-general@bucardo.org. A list of bugs can be found at https://bucardo.org/bugs.html
CREDITS
Bucardo was originally developed and funded by Backcountry.com, who have been using versions of it in production since 2002. Jon Jensen <jon@endpoint.com> wrote the original version.
This software is free to use: see the LICENSE file for details.
Bucardo-5.6.0/patches/ 000755 000770 000024 00000000000 13626305670 015266 5 ustar 00machack staff 000000 000000 Bucardo-5.6.0/Changes 000644 000770 000024 00000112261 13626305126 015131 0 ustar 00machack staff 000000 000000
Bucardo version 5.6.0, released February 28, 2020
- Minor PostgreSQL 11/12 tweaks in `bucardo install`
[David Christensen]
- Add config option log_timer_format to glog() to customize timestamp output
[David Christensen, suggested by David Wheeler]
- Change handling of file-path config settings so that they are no longer
lower-cased. The new log_timer_format config will also be case-preserved.
[David Christensen and David Wheeler]
- Fixed the relation parameter to the add customcode command to support
schema-qualified relation names.
[David Wheeler]
- Optimized table lookup when validating syncs to a single query, rather than
separate queries for each table. Also added checks to avoid purging "toast"
tables and old delta tables
[Gustavo Tonini] (https://github.com/bucardo/bucardo/pull/182)
- Improve the unique conflict exception handler sample code and test
[David E. Wheeler] (https://github.com/bucardo/bucardo/pull/194)
- Map timestamp from PostgreSQL to MongoDB datetime
[Github user redsysman] (https://github.com/bucardo/bucardo/pull/114)
- Safer system for determining Postgres version on install
[Greg Sabino Mullane] ( inspired by https://github.com/bucardo/bucardo/issues/131 )
- Correction that array_agg was introduced in Postgres 8.4, not 8.3
[Jaroslav Crhonek] (https://github.com/bucardo/bucardo/pull/103)
Bucardo version 5.5.0, released November 5, 2018
- PostgreSQL 11 support
[David Christensen]
- Fix race condition in reload/reload_config
[Christoph Berg]
- Fix detection of unnamed columns
[Ross Lafferty]
- Some increased portability fixes
[David Christensen]
- Framework for reporting PostgreSQL and Bucardo logs when encountering test
errors.
[David Christensen]
- Teach bucardo about `show changed` to display any config settings different
than the default.
[David Christensen]
- Add initial values for settings in bucardo_config.
[David Christensen]
- Fix bug in fork/cleanup code that may prevent a killed CTL from restarting.
[Greg Sabino Mullane]
- Fix bug in onetimecopy=2 mode
[Greg Sabino Mullane]
- Allow one-character schema names to work.
[David Christensen]
- Allow a direct DSN via the "dbdsn" attribute for the "add db" command.
[Greg Sabino Mullane] ( https://github.com/bucardo/bucardo/issues/129 )
- Rewrite code to support the new version of the Perl MongoDB driver.
[Greg Sabino Mullane]
- Improve the push_rows() function, and use ANY(?) whenever possible.
[Greg Sabino Mullane]
- Write things out more often, making 'bucardo status' more useful.
[Greg Sabino Mullane]
- Log the version of all loaded modules.
[Greg Sabino Mullane]
- Add basic support for Firebird as a target database type.
[Greg Sabino Mullane]
- Allow commas in the list for conflict_strategy.
[Greg Sabino Mullane]
- Remove 'boolean' as a global dependency, only require it for MongoDB use
[Greg Sabino Mullane]
Bucardo version 5.4.1, released September 26, 2015
- Ensure all sequences used in the bucardo database are in the bucardo schema.
[Greg Sabino Mullane]
Bucardo version 5.4.0, released August 6, 2015 (git commit a0eff9f6558cc2c6b19e4e675604250ad00ce420)
- Allow dashes in valid schema and table names
[David Christensen]
- New command "bucardo delta [syncname...]" to show number of pending
delta rows on source databases.
[Greg Sabino Mullane]
- Fix incorrect usage of the 'dbconn' parameter in the bucardo.db table
[Greg Sabino Mullane, reported by Alexandre Busquets]
- Fix case where VAC sometimes skipped databases used in multiple syncs.
[Greg Sabino Mullane, reported by Andrey Solovjov]
- Adjustments for new version of ExtUtils::MakeMaker
[Mathieu Arnold]
- When doing target deletes, use the quoted version of the primary key column names.
[Greg Sabino Mullane]
- Make sure we do not signal other syncs during makedelta runs if the other
syncs are inactive or have autokick off.
[Greg Sabino Mullane]
- Allow code with 'array_agg' to work on Postgres <= 8.2,
along with many other minor changes to support older versions.
[Greg Sabino Mullane]
- Add new internal function bucardo_purge_delta_oid which helps the
VAC process work better on older versions of Postgres (<= 8.2)
[Greg Sabino Mullane]
- Add new function bucardo_purge_sync_track which removes all track
and delta entries for a given sync. Use with care.
[Greg Sabino Mullane]
- Allow 'bucardo validate' to drop the autokick triggers if they are no longer needed.
[Greg Sabino Mullane]
- Add 'PRAGMA defer_foreign_keys' for sqlite targets
[Greg Sabino Mullane]
- Do not try to apply 'READ WRITE' when doing the initial SET TRANSACTION for MySQL and MariaDB,
as that option is not supported across all versions.
[Greg Sabino Mullane]
- Remove default value from sync.isolation_level, so we can default to
the global one when needed.
[Greg Sabino Mullane]
- Improve workings of the 'pkonly' argument to "bucardo add tables"
[Greg Sabino Mullane]
- Make 'bucardo list relgroup foo' show the tables in priority order.
[Greg Sabino Mullane]
- Make 'bucardo list all' a little less verbose for some not-so-common items
[Greg Sabino Mullane]
- Fix up the "add relgroup" call to allow it to work for capitalized relations,
and remove a prepare_cached error that could occur.
[Greg Sabino Mullane]
- Allow validate_sync to check the contents of functions for upgrade,
not just their existence.
[Greg Sabino Mullane]
- Do not 'RESET search_path' inside some of the internal functions.
[Greg Sabino Mullane]
- Sleep longer between stop and start when doing 'bucardo restart'
[Greg Sabino Mullane]
Bucardo version 5.3.1, released January 27, 2015 (git commit a4e59ea351410acd4fe750384a19393c261f7e91)
- When clearing out the async mode, do not do so unless the state() is sane.
This will solve some of the "pg_cancel failed" errors seen
[Greg Sabino Mullane]
- Throw a cleaner and earlier exception when connect_database fails
[Greg Sabino Mullane]
- Give a warning when a bucardo rc file has an unparseable line
[Greg Sabino Mullane]
Bucardo version 5.3.0, released December 22, 2014 (git commit 07d884fea5ba113b5d314c318bcff2cced55b6bb)
- Fix race condition when makedelta used which could cause some
rows to be incorrectly entered into the track table
[Greg Sabino Mullane]
- Fix bug when makedelta used on tables with binary primary key columns
[Greg Sabino Mullane]
- Allow 'makedelta' to be specified at the database level
[Greg Sabino Mullane]
- Give customcodes access to the list of changed rows via
the 'rows' key in the passed-in hashref
[Greg Sabino Mullane]
- Allow specification of non-default port when sending email.
Attempt support for SASL auth.
[Greg Sabino Mullane, per suggestion from Michael Wagner]
- Better support for continuing once a database has crashed
[Greg Sabino Mullane]
- Allow removing all tables or sequences at once via command line,
e.g. 'bucardo remove all tables'
[Greg Sabino Mullane]
- Allow --makedelta=foo to work on command line when adding all tables
[Greg Sabino Mullane]
- New test file t/50-star.t and much improvement to testing framework
[Greg Sabino Mullane]
- Added new logging level, DEBUG2, for things that are just too verbose
even for DEBUG!
[Greg Sabino Mullane]
- When the VAC exits, disconnect from any databases cleanly we
may still be connected to.
[Greg Sabino Mullane]
- Allow --dryrun to work when adding items, esp. tables
[Greg Sabino Mullane]
- Allow msg as alias for message e.g. 'bucardo msg foobar'
[Greg Sabino Mullane]
- When showing microseconds in the log, force the third and final zero to appear
[Greg Sabino Mullane]
Bucardo version 5.2.0, released November 5, 2014 (git commit 003b827a73a6e2e51cc5e833a5b4bc21631bd216)
- quickstart: new parameter to give on startup (e.g. "bucardo start --quickstart").
When set, skips checking the tables and columns on remote databases.
[Greg Sabino Mullane]
- Fix problem of not clearing out stagetable before using it. There was a
chance that a previous KID left some rows in there, in which case duplicate
entries will appear in bucardo_track.
[Greg Sabino Mullane]
- Make sure makedelta only adds matching track table rows if the current database is
a source, not a target, for the current sync.
[Greg Sabino Mullane]
- Fix some errant global variables which were causing subtle bugs in the makedelta
system.
[Greg Sabino Mullane]
Bucardo version 5.1.2, released October 15, 2014 (git commit 6b8f02feaef5b116dfba9b57f6a0617d73ba3657)
- Fix problem with async failures
[Greg Sabino Mullane]
- Fix add_customcols params parsing
[David Christensen]
- Improve MCP loop handling on errors
[Greg Sabino Mullane]
- Make sure makedelta can be set for all databases, not just source ones
[Greg Sabino Mullane]
- Fix minor bug in send_mail subroutine
[Greg Sabino Mullane]
- Remove Encode::Locale dependency
[Joshua Tolley]
- Allow 'git list config*' to work as an alias for git config.
Make plain 'git config' act as 'git show all'
[Greg Sabino Mullane]
Bucardo version 5.1.1, released July 23, 2014 (git commit c91753ef834cdcac311d66851947d1e4cb8acc7c)
- Minor fixes to conflict handling.
[Greg Sabino Mullane]
- Fix upgrade to add updating of defaults.
[Greg Sabino Mullane]
Bucardo version 5.1.0, released July 14, 2014 (git commit 7465d87ec60d29555fc2a38e70e08cc09e33a7db)
- Many fixes and features added to the conflict resolution system.
[Greg Sabino Mullane]
- Fix the upgrade so new constraints are handled better.
[Greg Sabino Mullane]
Bucardo version 5.0.0, released June 23, 2014 (git commit 2349582733df31a4115b952bfcc2f5732d5af16a)
- Complete rework of Bucardo: we now allow as many source and target
databases as wanted in a single sync.
[Greg Sabino Mullane]
- Allow alternate targets: Drizzle, MariaDB, MongoDB, MySQL, Oracle, Redis,
SQLite, as well as "flat files"
[Greg Sabino Mullane]
- Rename "bucardo_ctl" to simply "bucardo"
[Jon Jensen]
- Rename "goat" to "relation" or, more commonly, "table" and "sequence".
[David Wheeler]
- Rename "standard_conflict" option to "conflict_strategy".
- Rename "herd" to "relgroup".
[David Wheeler]
- Change the way that swap syncs work: should be much faster
[Greg Sabino Mullane]
- Use asynchronous queries when possible
[Greg Sabino Mullane]
- Redo the makedelta system, and add makedelta option for relations, which
inherit from db. Add makedelta as on/off for db.
[Greg Sabino Mullane]
- Overhaul of sequence handling: now allows for all attributes to be
replicated, such as INCREMENT_BY and MAXVALUE
[Greg Sabino Mullane]
- Fix for failing swap syncs when column is not all lowercase.
[Greg Sabino Mullane]
- Fix for proper sourcelimit and targetlimit calculations.
[Greg Sabino Mullane]
- Add slony_migrator.pl script
[Josh Tolley]
- Add example scripts in 'bucardo_examples' directory
[Selena Deckelmann]
- Allow 'list sync' to filter by 'active' and 'inactive'
[Greg Sabino Mullane]
- Remove unused config variable 'upsert_attempts'
[Greg Sabino Mullane]
- Show detailed information on KID error exit.
[Greg Sabino Mullane]
- Fix bug in bucardo_purge_q_table in index detection logic.
[Aolmezov]
- Show the reason why a sync failed on a failed bucardo kick.
[Greg Sabino Mullane]
- Add new subprocess (VAC) to run bucardo_purge_delta
[Greg Sabino Mullane]
- Fix errors with bytea and swap syncs
[Greg Sabino Mullane]
- Do not replicate null values to MongoDB
[Ali Asad Lotia]
- Add 'default_conflict_strategy' option
[Greg Sabino Mullane, idea from Franz Dürr]
- Force time zone to UTC to resolve timestamptz issues with bucardo_delta
[Michelle Sullivan, Greg Sabino Mullane]
- Add a log_level parameter, and classify all messages inside Bucardo.pm
with a log_level of TERSE, NORMAL, VERBOSE, or DEBUG.
[Rosser Schwarz, Greg Sabino Mullane]
- Relax SELECT version() regex in bucardo, to let modified versions
of Postgres tell us their version.
[Greg Sabino Mullane]
- Fix incorrect automatic population of database when using
'bucardo add database' and only one database exists.
[Greg Sabino Mullane]
- Fix failing makedelta when using multi-column primary keys.
[Greg Sabino Mullane, reported by Michelle Sullivan]
- Add new vacuum_after_copy variable for goats and syncs.
[Greg Sabino Mullane]
- Do not perform a checktime kick if onetimecopy mode is on.
[Greg Sabino Mullane]
- New syncs of type fullcopy now default to false for ping, stayalive,
and kidsalive.
[Greg Sabino Mullane, idea from Andrew Spencer]
- Refuse to start if the bucardo database version does not match that of
bucardo (the former is set by 'bucardo upgrade')
[Greg Sabino Mullane]
- Set the search_path to prevent bogus warnings on startup about sequence
mismatches based on schema names.
[Greg Sabino Mullane] (Bug #17)
- Add a --force option so we can remove databases via 'bucardo remove db'
along with all related goats and syncs.
[Greg Sabino Mullane]
- Change bucardo.dbrun table column pgpid to INTEGER
[Wim Lewis]
- Remove race condition from bucardo_purge_q_table()
[Greg Sabino Mullane]
- Add new options for 'bucardo add db': addalltables and addallsequences
[Greg Sabino Mullane]
- Allows syncs to be paused and resumed while they are running.
[Greg Sabino Mullane]
- Allow 'bucardo status' to run even if piddir is not available.
[Greg Sabino Mullane]
- Don't bother comparing the 'log_cnt' field of sequences.
[Rosser Schwarz]
- Allow "schema.table" format for 'bucardo list tables'
[Greg Sabino Mullane]
- Add new option "log_microsecond" to show sub-second output in logs.
[Greg Sabino Mullane]
- Make sure we update bucardo_config.bucardo_current_version when
doing a 'bucardo upgrade'
[Greg Sabino Mullane] (Bug #5)
- Automatically create the bucardo (superuser) if we can connect as 'postgres'.
[Greg Sabino Mullane]
- Add 'bucardo list all' feature.
[Christian Recktenwald]
- Allow removal of tables from relgroups via bucardo.
[Greg Sabino Mullane]
- Add create_child_q_table() function.
[Greg Sabino Mullane]
- Allow 'bucardo update relgroup foo remove '
and 'bucardo update relgroup foo add '
[Greg Sabino Mullane]
- Rename the default reason_file to bucardo.restart.reason.txt. The reason
log is then named bucardo.restart.reason.log (instead of
bucardo.restart.reason.log.log as before).
[David Wheeler]
- Better formatting of bucardo.reason.txt.log, and log startup failures.
[Greg Sabino Mullane]
- Log the current configuration settings on startup.
[Greg Sabino Mullane]
- Fix incorrect assignment of ppid in the bucardo.q table.
[Greg Sabino Mullane]
- Remove unused 'kick_sleep' config variable.
[Greg Sabino Mullane]
- Show the database name when listing relgroups via bucardo.
[Greg Sabino Mullane]
- Show the database name when listing sequences via bucardo.
[Douglas Tan]
- Show each target's Postgres version, time, and timezone on first connect.
[Greg Sabino Mullane]
- Change main trigger names to 'bucardo_delta' and 'bucardo_kick'
[Greg Sabino Mullane]
- Allow databases to have the same dsn, as we are moving
towards in-database replication.
[Greg Sabino Mullane]
- Add complete documentation of all public commands, actions, parameters,
and options to the bucardo Pod (and therefore the man page, as well).
[David Wheeler]
- Use the Pod docs to show the output of 'help command' and
'help command action' instead of duplicating that information in code.
[David Wheeler]
- Fixed PostgreSQL 9.2 compatibility issues.
[David Wheeler]
- Fix syntax issues to restore support for PostgreSQL 8.1
[Greg Sabino Mullane]
- Fix "arguments of row IN must all be row expressions" error.
[Greg Sabino Mullane]
- Fix empty primary key join/split bug.
[Greg Sabino Mullane]
- Add Linux int script and RPM spec file
[Devrim GÜNDÜZ, David Wheeler]
- Add --exit-on-nosync option. Bucardo no longer exits from 'start' when
there are no syncs. Use this option to restore that behavior.
[David Wheeler]
- Switch to pg_ctl in tests for more reliable (and faster!) Postgres
startup and shutdown.
[David Wheeler]
- Fix incorrect shutdown when Postgres is not running. The pid file is now
properly cleaned up on shutdown even if Postgres is down.
[David Wheeler]
- Tests now properly report the reasons for failed bucardo commands.
[David Wheeler]
- Revamp retry on serialization failure so that it actually works. The KID
no longer sleeps and exits, but sleeps and retries the sync on its own,
only sending a NOTICE to other processes that it is doing so.
[David Wheeler, Greg Sabino Mullane]
- Change the default serialization failure sleep time from 10s to 0.5s.
- Added new 'bucardo reopen' command to reopen all open file handles
Useful for log rotation.
[Greg Sabino Mullane].
- The 'update' command no longer exits with an error code on success.
- Fixed issue with missing database handles when a KID shuts down.
[David Wheeler]
- Remove the customselect parameter. Use the customcols command, instead.
[David Wheeler]
- Remove the do_listen parameter, which was just another name for ping.
- Rename the 'ping' parameter to 'autokick', to better describe what it does
and to prevent confusion with the 'ping' command, which does something
totally different.
[David Wheeler]
- Rename standard_conflict to conflict_strategy.
[David Wheeler]
- Move conflict_strategy from add/update table to add/update sync.
[David Wheeler]
- Fix issue that prevented the VAC process from shutting down
immediately on 'stop'.
[David Wheeler]
- Fix failure to restart after database disconnections.
- Add 'remove sequence'
[David Wheeler]
- Rename most --debug* options to --log-*:
+ --debugdir is now --log-destination
+ --debugfile is deprecated; use '--log-destination none' to disable all logging
+ --debugsyslog is deprecated; use --log-destination syslog'
+ Allow logging to stderr or stdout via --log-destination
+ --log-destination may be specified multiple times for multiple logs
+ --debugfilesep is now --log-separate
+ --debugname is now --log-extension
+ --cleandebugs is now --log-clean
[David Wheeler]
- Eliminated 'Use of "goto" to jump into a construct has been deprecated'
warnings on Perl 5.12 and higher.
- Removed the 'trigrules' attribute for customcodes. Callers should
(carefully!) change session_replication_role themselves.
[Greg Sabino Mullane]
[ GSM is Greg Sabino Mullane ]
Bucardo version 4.4.8, released November 30, 2011
- Fix incorrect checking of hostname for host_safety_check [GSM]
- Fix wrong version number in the bucardo_ctl script.
Bucardo version 4.4.7, released November 8, 2011
- Fix bug when using multi-column primary keys where the first
is an int, and the second is not. Reported by Brady S Edwards [GSM]
- Do not rely on $@ to catch important evals [GSM]
Bucardo version 4.4.6, released July 10, 2011
- Fix bug resulting in wrong escapes in swap syncs for int4 primary keys
Reported by Kiriakos Tsourapas. [GSM]
Bucardo version 4.4.5, released June 18, 2011
- Set SECURITY DEFINER on triggerkick functions. [GSM]
Bucardo version 4.4.4, released May 14, 2011
- Make sure we escape backslashes when building the
DELETE queries for pushdelta syncs [GSM]. Reported
by Gabriel Weinberg
- Report correct number of inserts/deletions in
bucardo_ctl status [GSM]. Reported by Gabriel
Weinberg
Bucardo version 4.4.3, released April 15, 2011
- Fix rare race condition when target database dies. [GSM]
Thanks to Kaveh Mousavi Zamani for the report.
Bucardo version 4.4.2, released March 17, 2011
- Fix incorrect SET TIME ZONE inside the bucardo_delta functions. [GSM]
Bucardo version 4.4.1, released November 9, 2010
- Set the search_path to prevent bogus warnings on startup about sequence
mismatches based on schema names. [GSM] (Bug #17)
- Fix failing makedelta when using multi-column primary keys.
[GSM, reported by Michelle Sullivan]
- Fix for failing swap syncs when column is not all lowercase. [GSM]
- Fix errors with bytea and swap syncs. [GSM]
- Use clock_timestamp if Postgres version is 8.2 or better. (Bug #25)
Thanks to David Christensen for digging in
and solving this.
- Fix error in bind_param for target to source. [GSM]
- Make sure we truly reset onetimecopy for persistent controllers and children. [GSM]
- If in onetimecopy mode, always create a q entry, regardless of synctype. [GSM]
- Rename the ENCODE call for bytea selects. [GSM]
- Always make the source transaction serializable. [GSM]
- Force time zone to GMT to resolve timestamptz issues with bucardo_delta
[GSM, Michelle Sullivan]
- Allow '2' as a valid sqlstate after exception thrown. [GSM]
- Fix typo with 'lowest/highest' standard conflict methods. Caught by Don Drake.
- Remove race condition from bucardo_purge_q_table() [GSM]
- Make search_path declaration LOCAL in bucardo_audit() [GSM]
- Fix bug in bucardo_purge_q_table in index detection logic. [Aolmezov]
- Don't bother comparing the 'log_cnt' field of sequences. [Rosser Schwarz]
- Do not perform a checktime kick if onetimecopy mode is on. [GSM]
- Add quote() methods to safe list for DBIx::Safe call [GSM]
- Add -vv option to bucardo_ctl as a shortcut for --verbose --verbose [GSM]
- Don't allow kicking of inactive syncs. [GSM]
- Relax SELECT version() regex in bucardo_ctl, to let modified versions
of Postgres tell us their version. [GSM]
- Change DESTDIR to INSTALL_BASE in the makefile [GSM]
- No need to look at anything but ONLY master_q for populate_child_q function. [GSM]
- Automatically create the bucardo (superuser) if we can connect as 'postgres'. [GSM]
- Allow 'bucardo_ctl status' to run even if piddir is not available. [GSM]
- Allow "schema.table" format for bucardo_ctl list tables [GSM]
- Show the database name when listing herds via bucardo_ctl. [GSM]
- Show the database name when listing sequences via bucardo_ctl. [Douglas Tan]
- Make sure we update bucardo_config.bucardo_current_version when doing
a bucardo_ctl upgrade [GSM] (Bug #5)
- Add additional clause to the the q_cleanup index. [GSM]
- Remove unused config variable 'upsert_attempts' [GSM]
- Remove unused 'kick_sleep' config variable. [GSM]
- Show additional version information on startup for debugging [GSM]
Bucardo version 4.4.0, released October 14, 2009
- Allow validate_goat() to work correctly against pgbouncer databases by
respecting the server_side_prepares parameter. [GSM]
- Loosen restrictions on customcode - now the derived columns must match
the target, but the target and source tables can be completely different. [GSM]
- Add the bucardo_audit() function that runs on all master databases, ensures
that everything related to the bucardo_* control tables is correct.
[Josh Tolley and GSM]
- Fix bucardo_ctl add sync ... tables=x - now adds the sync on first pass,
even if the tables have not been added. [GSM]
- Adding customcode via bucardo_ctl can now specify sync, goat, priority,
and active columns. [GSM]
- When viewing customcode via 'bucardo_ctl list code foo', the src_code column
is shown last. [GSM]
- Add some small sleeps before checking for stale pid files, to allow their owners
a chance to clean them up first. [GSM]
- Give a warning when kicking an inactive sync. [GSM]
- Remove 'text' casting which prevented 'bucardo_ctl inspect' from working on
older versions of Postgres. [GSM]
- Allow testing of multiple versions of Postgres at the same time by setting the
environment variables PGBINDIR[A-Z]. [GSM]
- Don't look for hard-coded string to determine if cluster is up [GSM, thanks
to Andre Felipe Machado for suggesting]
Bucardo version 4.3.0, released October 8, 2009
- Allow MCP and CTL to kill based on piddir, not just audit_pid. This will prevent duplicate
kids from existing if audit_pid is switched off. [GSM]
- Fix error in lowest/highest conflict handling. [Mathieu Arnold]
- Add new column to customcode: trigrules. Defaults to false: if true,
we turn on triggers and rules for the duration of the custom code. [GSM]
- Add a new configuration value 'host_safety_check', to help prevent
Bucardo from being copied from one host to another and then run
without modifying at least bucardo.db.dbhost first. [GSM]
- For custom code exception handlers, make the 'dbi_error' hash key return
a flattened string of the actual exception thrown. [GSM]
- Ensure exiting processes always remove their pidfile. [GSM]
- Have the MCP clean up old pid files from the piddir on startup. [GSM]
- Remove the 'pidfile' configuration option: now hard-coded to 'bucardo.mcp.pid' [GSM]
- Make the pid file names more consistent, add new ones for kid processes. [GSM]
- Refactor the killing of pids, make safer by checking they are bucardo-like [GSM]
- Remove the PIDCLEANUP option from Bucardo.pm [GSM]
- Better upgrading of very old versions of Bucardo. [GSM]
- Allow add, remove, and list customcode through bucardo_ctl [GSM]
- Allow bucardo_ctl upgrade to remove configuration settings. [GSM]
Bucardo version 4.2.2, released October 2, 2009
- Fix so that warning_file works properly again. [GSM]
Bucardo version 4.2.1, released October 1, 2009
- Improve ability of 'bucardo_ctl upgrade' to work against older
versions of Bucardo [GSM]
- Changed the chunking threshold for pushdelta to 100,000 and 10,000 [GSM]
- Make bucardo_ctl list table|sequence follow same wildcard rules as
other list nouns. [GSM]
- Remove the old 'alter' verb from bucardo_ctl: use 'update' instead [GSM]
- Fix a SQL error in 'bucardo_ctl status [GSM]
- Remove the unused t/bucardo.test.data file [GSM]
Bucardo version 4.2.0, released September 30, 2009
- Add the "delta_bypass" feature, which allows a pushdelta sync to switch
to 'fullcopy' mode on the fly, depending on the number of rows that
have changed. [GSM]
- Fixed an error preventing customselect from working in certain situations. [GSM]
- Have 'bucardo_ctl install' check for successful plperlu installation. [GSM]
- Clean up and standardize all the bucardo_ctl list/add/remove actions. [GSM]
Bucardo version 4.1.2, released September 25, 2009
- Fix sequence replication by setting is_called properly on the targets
[GSM, thanks to Selena Deckelmann for discovering this]
- Respect server_side_prepares db setting in validate_sync. Previously,
this function would fail if validating more than one table at a time
and the database was set as server_side_prepares off [GSM]
- Make sure we cast rowid* to text within the bucardo_add_delta-called
functions, for picky versions of Postgres. [GSM]
Bucardo version 4.1.1, released September 24, 2009
- Fix case where Bucardo was too eager to lock tables post serialization
failures. Since we can't really do this automatically and safely,
remove the code in question for now. [GSM, hat tip to Selena Deckelmann]
Bucardo version 4.1.0, released September 24, 2009
- Fix problem with possible function name collision when the names
of the combined columns in the primary key is greater than NAMEDATALEN,
which we'll assume is 63. [GSM]
- Add "bucardo_ctl inspect table" to show all dependencies, as well
as cross referencing against herds the table is in [GSM]
- Fix an error when validate_sync tries to recreate an existing trigger;
occurred when a fullcopy sync was set to ping=true [GSM]
- When using "bucardo_ctl add all tables db=", make sure we only
look for the already added tables for that specific database.
[Selena Deckelmann and Josh Tolley]
- Add new boolean column to the "db" table named "server_side_prepares", which
is true by default. If off, Bucardo will not use server-side prepared
statements, and thus can be used in PgBouncer's transaction mode.
[GSM, idea by Andrew Spencer]
- During pushdelta, break large number of rows for a single table until multiple
DELETE and COPY calls, to help debugging, to allow for a progress indicator
both at the Bucardo and Postgres levels, and to prevent crashes seen on
older versions of Postgres when very large IN() lists are used. [GSM]
- When using "bucardo_ctl add all tables", ignore any tables inside
schemas starting with 'bucardo' [GSM]
- Allow --verbose flag for list syncs to show linked tables [GSM]
- Allow --verbose flag for list tables to show linked herds and syncs [GSM]
- Change rebuild_index to a smallint: a 1 only runs it with a fullcopy.
If you really want to use it otherwise, set it to 2 [GSM]
- Allow the dbhost to be turned off (local socket only) in the db table
when using "bucardo_ctl update db host=non. [GSM]
Bucardo version 4.0.3, released September 16, 2009
- Fix some minor problems with bucardo_ctl and "add sync" [GSM]
- Allow ping to be specified for new tables added via "add sync" [GSM]
Bucardo version 4.0.2, released September 16, 2009
- Add the name of the database to the install choices, as some systems
have mismatched names (BSD's 'pgsql' user) [Vilem Kebrt]
- Clear out stale q entries on startup, created when a kid was hit with a
kill -9. This was causing Bucardo to think the q slot was still active. [GSM]
- Fix cases where we were not quoting the UNLISTEN names [GSM]
- Remove duplicate PID numbers from log [GSM]
- Prohibit syncs from having names reserved for synctypes. [GSM]
- Add new delete method of "truncate_cascade" for syncs [GSM]
- Allow "foo*", "*foo", and "*foo*" wildcard format in bucardo_ctl
when specifying syncs, e.g. bucardo_ctl kick ABC* [GSM]
- Allow special cases for bucardo_ctl kick: "all [synctype] [active]" [GSM]
- Allow filtering of 'bucardo_ctl list syncs' by synctype [GSM]
- Allow 'bucardo_ctl add sync tables=...' to work with schema.table format [GSM]
- Add --verbose option to 'bucardo_ctl list syncs' to show tables [GSM]
- Allow name changes for syncs, herds, dbs, dbgroups [GSM]
- Automatically add missing tables when doing
"bucardo_ctl add sync source=dbname ... tables=x,y,z" [GSM]
- Allow multiple items to be changed at one time with 'bucardo_ctl update item' [GSM]
- Change validate_sync notice to LOG [GSM]
Bucardo version 4.0.1, released September 8, 2009
- Fix to allow "add all tables db=foo" to work as expected. [GSM]
- Set audit_pid to be off by default. [GSM]
- Fix broken WHERE clause inside bucardo_ctl for "add table" [GSM]
Version 4.0.0, released September 7, 2009
New features:
- Add support for multi-column primary keys. Thanks to Backcountry.com. [GSM]
- Add new 'onetimecopy' attribute to sync table, to allow recreation on
demand of pushdelta tables on targets via COPY. [GSM]
- Support for replication after TRUNCATE. This will only work on
Postgres version 8.4 and higher, and only pushdelta syncs. [GSM]
- Add log_conflict_details to allow logging of conflict information
to a separate file. [Adam Wendt]
- Add 'maxkicks' and 'lifetime' to syncs to force restart of controller
and kids after a certain number of kicks or time. [Josh Tolley]
Thanks to Backcountry.com.
- Allow replication of sequences. [GSM]
- Add the bucardo_custom_trigger table, which allows tight control of exactly
what items in a table will be replicated. [Ben Allen]
- Add the bucardo_rate table to track exactly how long replication takes
on a per-sync and per-table basis. [GSM]
- Add more bucardo_ctl actions: install, reload, validate, restart, update [GSM]
- Allow adding of all tables and sequences in a database at once via bucardo_ctl [GSM]
- Add in bucardo_compress_delta function, to remove duplicate rows from
the bucardo_delta table. [GSM]
- Allow the injection of custom messages into the Bucardo logs, and make
an interface for same in bucardo_ctl. [GSM]
- Add the "warning_file" config parameter to log all 'Warning' messages. [GSM]
- Add validate_all_syncs() function, and allow an argument to both that
and validate_sync() to force recreation of supporting objects. [GSM]
Enhancements:
- Remove 'Moose' dependency. [GSM]
- Remove disable_triggers and disable_rules from the sync table.
We simply choose pg_class or replica based on the version of the
database in question. [GSM]
- Rewrite the pushdelta sync method to be more efficient and safer. [GSM]
- Change from Mail::Sendmail to Net::SMTP [Ben Allen]
- Allow wildcards in the 'bucardo_ctl kick' [Josh Tolley]
- Improve the FreeBSD scripts/bucardo_ctl.rc file [Ben Allen]
- Add "sendmail" option to the customcode input. [Ben Allen]
- Add 'bucardo_ctl upgrade' to make intra-version upgrades easy
and automatic. [Selena Deckelmann and GSM]
- Add more tests and improve existing ones. [Josh Tolley]
- Force READ WRITE mode so Bucardo works on read only databases. [GSM]
- Make the audit_pid table more useful. Use sequences instead of
PIDs to find children. Allow for the audit_pid table to be toggled off. [GSM]
- Remove the debugstderr and debugstdout options, as they never worked
correctly anyway. [GSM]
- When truncating during fullcopy, fallback to delete if it fails. [GSM]
- On startup, all syncs that may have trigger-based notifies are
initially kicked. [GSM]
- MCP listens for pings during table validations. [GSM]
- Have master_q use a trigger, not multiple rules. Remove the need
for a separate create_child_q() function. [GSM]
- Default for log_showtime is now 'scalar localtime' rather than
epoch seconds. [GSM]
- No longer require a reason for the "stop" action. [GSM]
- Add INSTALL_BUCARDODIR for quick and dirty 'make install' [GSM]
Bug fixes:
- Fix failure to disable/enable triggers when using pg_class and table
and schema are words changed by quote_ident. [Ben Allen]
- Tolerate incompatibility in DEFAULT of negative numbers between
Postgres 8.2 and 8.3. [Jon Jensen]
- Do better checking of bucardo_ctl executable when MCP is restarting. [Ben Allen]
- Change timestamp to timestamptz where needed inside Bucardo.pm [Yan Farmawan]
- Put schema in db_getconn calls [Alex Bahlai]
- Record the proper PID for the MCP in the pid file [Goran Gugic]
- Set the tcp_* defaults in bucardo_config to zero (system default). [GSM]
- Allow for dead columns when checking attnums. [GSM]
- Do not attempt respawn of MCP if mcp_dbproblem_sleep is 0. [GSM]
Bucardo version 3.1.0, released April 6, 2008
- Fix race condition in bucardo_delta/bucardo_track updates
for pushdelta syncs. [GSM]
- Auto-detect primary keys (or unique indexes) for added tables. [GSM]
- Add update_bucardo_schema.pl to facilitate version changes. [GSM]
- Use correct quoted/non-quoted versions of primary keys to allow
primary keys with spaces, etc. [GSM]
- Add new options to log_showtime to show non-epoch time strings [GSM]
- Make bucardo_ctl use the global version number, not its own. [GSM]
- Support bytea in primary key and regular columns. [GSM]
Bucardo version 3.0.9, released March 26, 2008
- Fix problem causing killed controller to not re-do interrupted syncs
on restart. [GSM]
Bucardo version 3.0.8, released February 5, 2008
- Fix error in 'latest' conflict code. [Adam Wendt].
- Filter out inactive target dbs from bucardo-report script.
[Spencer Christensen ]
- Fix race condition in bucardo_delta/bucardo_track updates
for pushdelta syncs. [GSM]
- Fix a sorting problem in the bucardo-report script. [Jon Jensen ]
- Fix problem with after_sync code not firing at correct time for stayalive kids. [GSM]
Bucardo version 3.0.7, released October 11, 2007
- Allow bucardo_ctl options to be specified in a .bucardorc file. [GSM]
- Add new index to bucardo_delta for faster purging. [GSM]
- Add a BSD-style rc file in the scripts directory. [Ben Allen ]
- Activate the "latest" standard_conflict method. [GSM]
- Fix error for tables with primary key and no other columns in pushdelta/swap syncs,
per report from Ben Allen. [GSM]
- Change pid and ppid columns in table "q" from smallint to integer. [Ben Allen ]
- Ensure sourcedbh used by customcode has inactivedestroy set. [GSM]
- Find system grep rather than hard-coding the path [Jeff Boes ]
- Fix for tests: make third database have a default password. [GSM]
- Add ping method to CTL and KID processes. [GSM]
Bucardo version 3.0.6 - First public release, September 1, 2007
Bucardo-5.6.0/MANIFEST 000644 000770 000024 00000001565 13370127576 015002 0 ustar 00machack staff 000000 000000 .gitignore
.perlcriticrc
bucardo
bucardo.html
Bucardo.pm
Bucardo.pm.html
bucardo.schema
Changes
INSTALL
LICENSE
Makefile.PL
MANIFEST
MANIFEST.SKIP
META.yml
patches/Bucardo.4.4.0.fix.search.path.patch.txt
README
README.dev
scripts/bucardo-report
scripts/bucardo_rrd
scripts/check_bucardo_sync
scripts/README
scripts/slony_migrator.pl
SIGNATURE
t/01-basic.t
t/02-bctl-customname.t
t/02-bctl-db.t
t/02-bctl-dbg.t
t/02-bctl-herd.t
t/02-bctl-sync.t
t/02-bctl-table.t
t/10-fullcopy.t
t/10-makedelta.t
t/10-object-names.t
t/20-drizzle.t
t/20-firebird.t
t/20-mariadb.t
t/20-mongo.t
t/20-mysql.t
t/20-oracle.t
t/20-postgres.t
t/20-redis.t
t/20-sqlite.t
t/30-crash.t
t/30-delta.t
t/40-conflict.t
t/40-customcode-exception.t
t/40-serializable.t
t/50-star.t
t/98-cleanup.t
t/99-signature.t
t/BucardoTesting.pm
t/ctest1.pl
t/ctest2.pl
t/ctest3.pl
t/customcode.exception.bucardotest.pl
TODO
UPGRADE
Bucardo-5.6.0/bucardo 000755 000770 000024 00001307275 13616304302 015211 0 ustar 00machack staff 000000 000000 #!/usr/bin/env perl
# -*-mode:cperl; indent-tabs-mode: nil; cperl-indent-level: 4-*-
## Script to control Bucardo
##
## Copyright 2006-2020 Greg Sabino Mullane
##
## Please see https://bucardo.org/ for full documentation
##
## Run with a --help argument for some basic instructions
package bucardo;
use strict;
use warnings;
use utf8;
use 5.008003;
use open qw( :std :utf8 );
use DBI;
use IO::Handle qw/ autoflush /;
use File::Basename qw/ dirname /;
use Time::HiRes qw/ sleep gettimeofday tv_interval /;
use POSIX qw/ ceil setsid localeconv /;
use Config qw/ %Config /;
use Encode qw/ decode /;
use File::Spec;
use Data::Dumper qw/ Dumper /;
$Data::Dumper::Indent = 1;
use Getopt::Long;
Getopt::Long::Configure(qw/ no_ignore_case pass_through no_autoabbrev /);
require I18N::Langinfo;
our $VERSION = '5.6.0';
## For the tests, we want to check that it compiles without actually doing anything
return 1 if $ENV{BUCARDO_TEST};
## No buffering on the standard streams
*STDOUT->autoflush(1);
*STDERR->autoflush(1);
my $locale = I18N::Langinfo::langinfo(I18N::Langinfo::CODESET());
for (@ARGV) {
$_ = decode($locale, $_);
}
## All the variables we use often and want to declare here without 'my'
use vars qw/$dbh $SQL $sth %sth $count $info %global $SYNC $GOAT $TABLE $SEQUENCE $DB $DBGROUP $HERD $RELGROUP
$CUSTOMCODE $CUSTOMNAME $CUSTOMCOLS $CLONE /;
## How to show dates from the database, e.g. start time of a sync
my $DATEFORMAT = $ENV{BUCARDO_DATEFORMAT} || q{Mon DD, YYYY HH24:MI:SS};
my $SHORTDATEFORMAT = $ENV{BUCARDO_SHORTDATEFORMAT} || q{HH24:MI:SS};
## How long (in seconds) we hang out between checks after a kick - or when waiting for notices
my $WAITSLEEP = 1;
## Determine how we were called
## If we were called from a different directory, and the base directory is in our path,
## we strip out the directory part
my $progname = $0;
if (exists $ENV{PATH} and $progname =~ m{(.+)/(.+)}) {
my ($base, $name) = ($1,$2);
for my $seg (split /\:/ => $ENV{PATH}) {
if ($seg eq $base) {
$progname = $name;
last;
}
}
}
## We must have at least one argument to do anything
help(1) unless @ARGV;
## Default arguments - most are for the bc constructor
my $bcargs = {
quiet => 0,
verbose => 0,
quickstart => 0,
bcverbose => 1,
dbname => 'bucardo',
dbuser => 'bucardo',
dbpass => undef,
sendmail => 0,
extraname => '',
logseparate => 0,
logextension => '',
logclean => 0,
batch => 0,
};
## These options must come before the main GetOptions call
my @opts = @ARGV;
GetOptions(
$bcargs,
'no-bucardorc',
'bucardorc=s',
);
## Values are first read from a .bucardorc, either in the current dir, or the home dir.
## If those do not exist, check for a global rc file
## These will be overwritten by command-line args.
my $file;
if (! $bcargs->{'no-bucardorc'}) {
if ($bcargs->{bucardorc}) {
-e $bcargs->{bucardorc} or die qq{Could not find the file "$bcargs->{bucardorc}"\n};
$file = $bcargs->{bucardorc};
}
elsif (-e '.bucardorc') {
$file = '.bucardorc';
}
elsif (defined $ENV{HOME} && -e "$ENV{HOME}/.bucardorc") {
$file = "$ENV{HOME}/.bucardorc";
}
elsif (-e '/etc/bucardorc') {
$file = '/etc/bucardorc';
}
}
if (defined $file) {
open my $rc, '<', $file or die qq{Could not open "$file": $!\n};
while (<$rc>) {
## Skip any lines starting with a hash
next if /^\s*#/;
## Format is foo=bar or foo:bar, with whitespace allowed
if (/^\s*(\w[\w-]+)\s*[:=]\s*(.+?)\s*$/o) {
my ($name,$value) = ($1,$2); ## no critic (ProhibitCaptureWithoutTest)
$bcargs->{$name} = $name eq 'logdest' ? [$value] : $value;
}
else {
warn qq{Could not parse line $. of file "$file"\n};
}
}
close $rc or die;
}
Getopt::Long::Configure(qw(no_pass_through autoabbrev));
GetOptions ## no critic (ProhibitCallsToUndeclaredSubs)
($bcargs,
'verbose+',
'vv',
'vvv',
'vvvv',
'quiet+',
'quickstart',
'notimer',
'help|?',
'debug+',
'version',
'sort=i',
'showdays|show-days',
'compress',
'retry=i',
'retrysleep|retry-sleep=i',
'batch',
'dryrun|dry-run',
'confirm',
'tsep=s',
'exit-on-nosync!',
## These are sent to the constructor:
'bcverbose',
'dbport|db-port|p=i',
'dbhost|db-host|h=s',
'dbname|db-name|d=s',
'dbuser|db-user|U=s',
'dbpass|db-pass|P=s',
'sendmail=i',
'extraname|extra-name=s',
'debugsyslog=i', # legacy
'debugdir=s', # legacy
'debugfile=i', # legacy
'cleandebugs=i', # legacy
'logdest|log-dest|log-destination=s@', # stderr, syslog, none, or file path
'logseparate|log-sep|log-separate|debugfilesep!',
'logextension|log-extension|log-ext|debugname=s',
'logclean|log-clean!',
'loglevel|log-level=s',
'logshowline|log-showline|log-show-line=s',
## Used internally
'force',
'schema|n=s@',
'exclude-schema|N=s@',
'table|t=s@',
'exclude-table|T=s@',
'db|database=s',
'herd|relgroup=s',
'piddir|pid-dir=s',
) or die "\n";
## If --help is set, ignore everything else, show help, then exit
help() if $bcargs->{help};
## If --version is set, ignore everything else, show the version, and exit
if ($bcargs->{version}) {
print "$progname version $VERSION\n";
exit 0;
}
## Allow some options to be set by env
if ($ENV{BUCARDO_CONFIRM} and ! exists $bcargs->{confirm}) {
$bcargs->{confirm} = $ENV{BUCARDO_CONFIRM};
}
# Determine the logging destination.
if (exists $bcargs->{logdest}) {
if (! ref $bcargs->{logdest}) {
$bcargs->{logdest} = [$bcargs->{logdest}];
}
}
else {
if (exists $bcargs->{debugfile} && !delete $bcargs->{debugfile}) {
# Old --debugfile option can disable logging.
$bcargs->{logdest} = [];
}
elsif (my $dir = $bcargs->{debugdir}) {
# Old --debugdir option determines log directory.
$bcargs->{logdest} = [$dir];
}
else {
# Default value.
$bcargs->{logdest} = ['/var/log/bucardo'];
}
if ($bcargs->{debugsyslog}) {
# Old --debugsyslog option enables syslog logging.
push @{ $bcargs->{logdest} } => 'syslog';
}
}
# Handle legacy --cleandebugs option.
$bcargs->{logclean} = 1
if delete $bcargs->{cleandebugs} && !exists $bcargs->{logclean};
## Sometimes we want to be as quiet as possible
my $QUIET = delete $bcargs->{quiet};
## Quick shortcuts for lots of verbosity
$bcargs->{vv} and $bcargs->{verbose} = 2;
$bcargs->{vvv} and $bcargs->{verbose} = 3;
$bcargs->{vvvv} and $bcargs->{verbose} = 4;
## Set some global arguments
my $VERBOSE = delete $bcargs->{verbose};
my $DEBUG = delete $bcargs->{debug} || $ENV{BUCARDO_DEBUG} || 0;
## Do we compress time outputs by stripping out whitespace?
my $COMPRESS = delete $bcargs->{compress} || 0;
## Do we retry after a sleep period on failed kicks?
my $RETRY = delete $bcargs->{retry} || 0;
my $RETRYSLEEP = delete $bcargs->{retrysleep} || 0;
## Allow people to turn off the cool timer when kicking syncs
my $NOTIMER = delete $bcargs->{notimer} || 0;
## Anything left over is the verb and noun(s)
my $verb = shift || '';
## No verb? Show a help message and exit
help(1, "Missing required command\n") unless $verb;
## Standardize the verb as lowercase, and grab the rest of the args as the "nouns"
$verb = lc $verb;
my @nouns = @ARGV;
## Allow alternate underscore format
if ($verb =~ /^(\w+)_(\w+)$/) {
$verb = $1;
unshift @nouns => $2;
}
## Make a single string version, mostly for output in logs
my $nouns = join ' ' => @nouns;
## The verb may have a helper, usually a number
my $adverb;
## Installation must happen before we try to connect!
install() if $verb =~ /instal/i;
## Display more detailed help than --help
superhelp() if $verb eq 'help';
my ($STOPFILE,$REASONFILE,$REASONFILE_LOG);
## If we are trying a stop, and piddir is already set, do it now
if ('stop' eq $verb and $bcargs->{piddir}) {
$STOPFILE = "$bcargs->{piddir}/fullstopbucardo";
$REASONFILE = 'bucardo.restart.reason.txt';
$REASONFILE_LOG = 'bucardo.restart.reason.log';
stop();
}
## For everything else, we need to connect to a previously installed Bucardo database
## Create a quick data source name
my $DSN = "dbi:Pg:dbname=$bcargs->{dbname}";
$bcargs->{dbhost} and length $bcargs->{dbhost} and $DSN .= ";host=$bcargs->{dbhost}";
$bcargs->{dbport} and length $bcargs->{dbport} and $DSN .= ";port=$bcargs->{dbport}";
## Connect to the database
$dbh = DBI->connect($DSN, $bcargs->{dbuser}, $bcargs->{dbpass}, {AutoCommit=>0,RaiseError=>1,PrintError=>0});
## We only want to concern ourselves with things in the bucardo schema
$dbh->do('SET search_path = bucardo');
## Make sure we find a valid Postgres version
## Why do we check this after a successful install?
## In case they get pg_dumped to a different (older) database. It has happened! :)
check_version($dbh); ## dies on invalid version
## Listen for the MCP. Not needed for old-school non-payload LISTEN/NOTIFY, but does no harm
$dbh->do('LISTEN bucardo');
$dbh->commit();
## Set some global variables based on information from the bucardo_config table
## The reason file records startup and shutdown messages
$REASONFILE = get_config('reason_file');
($REASONFILE_LOG = $REASONFILE) =~ s{(?:[.][^.]+)?$}{.log};
## The directory Bucardo.pm writes PID and other information to
my $PIDDIR = $bcargs->{piddir} || get_config('piddir');
## The PID file of the master control file (MCP)
## If this exists, it is a good bet that Bucardo is currently running
my $PIDFILE = "$PIDDIR/bucardo.mcp.pid";
## The stop file whose existence tells all Bucardo processes to exit immediately
my $stopfile = get_config('stopfile');
$STOPFILE = "$PIDDIR/$stopfile";
## Aliases for terms people may shorten, misspell, etc.
## Mostly used for database columns when doing an 'update'
our %alias = (
'ssp' => 'server_side_prepares',
'server_side_prepare' => 'server_side_prepares',
'port' => 'dbport',
'host' => 'dbhost',
'name' => 'dbname',
'user' => 'dbuser',
'pass' => 'dbpass',
'password' => 'dbpass',
'service' => 'dbservice',
'dsn' => 'dbdsn',
);
## Columns that cannot be changed: used in the update_* subroutines
my %column_no_change = (
'id' => 1,
'cdate' => 1,
);
## Regular expression for a valid dbgroup name
my $re_dbgroupname = qr{\w[\w\d]*};
## Regular expression for a valid database name
my $re_dbname = qr{\w[\w\d]*};
## Send a ping to the MCP to make sure it is alive and responding
ping() if $verb eq 'ping';
## Make sure the Bucardo database has the latest schema
upgrade() if $verb =~ /^upgr/ or $verb eq 'uprgade' or $verb eq 'ugprade';
## All the rest of the verbs require use of global information
## Thus, we load everything right now
load_bucardo_info();
## View the status of one or more syncs
status_all() if $verb eq 'status' and ! @nouns;
status_detail() if $verb eq 'status';
## Stop, start, or restart the main Bucardo daemon
stop() if $verb eq 'stop';
start() if $verb eq 'start' or $verb eq 'strt';
restart() if $verb eq 'restart';
## Reload the configuration file
reload_config() if $verb eq 'reload' and defined $nouns[0] and $nouns[0] eq 'config';
## Reload the mcp (if args, we want reload_sync)
reload() if $verb eq 'reload' and ! defined $nouns[0];
# Reopen the log files
reopen() if $verb eq 'reopen';
## Show information about something: database, table, sync, etc.
list_item() if $verb eq 'list' or $verb eq 'l' or $verb eq 'lsit' or $verb eq 'liast'
or $verb eq 'lisy' or $verb eq 'lit';
## Add something
add_item() if $verb eq 'add';
## Remove something
remove_item() if $verb eq 'remove' or $verb eq 'delete' or $verb eq 'del';
## Update something
update_item() if $verb eq 'update' or $verb eq 'upd' or $verb eq 'udpate';
## Inspect something
inspect() if $verb eq 'inspect';
## Inject a message into the Bucardo logs
message() if $verb eq 'message' or $verb eq 'msg';
## Show or set an item from the bucardo.config table
config() if $verb eq 'set' or $verb eq 'show' or $verb eq 'config';
## Validate a sync
validate() if $verb =~ /^vali/;
## Purge the delta/track tables
purge() if $verb eq 'purge';
## Clone a database
clone() if $verb eq 'clone';
## View delta statistics
count_deltas() if $verb eq 'delta' or $verb eq 'deltas';
## There are only a few valid verbs left, so we check for them now
if ($verb ne 'kick' and $verb ne 'activate' and $verb ne 'deactivate'
and $verb ne 'reload'
and $verb ne 'pause' and $verb ne 'resume') {
## Show help and exit
help(1, qq{Unknown command "$verb"\n});
}
## For all remaining verbs, we expect a list of syncs with an optional decimal "timeout"
## If there are no syncs, no sense in going on!
if (! keys %$SYNC) {
die qq{No syncs have been created yet!\n};
}
## The final list of syncs we are going to do something to
my @syncs;
## The fail msg on a non-match
my $msg;
## Loop through each noun and handle it
SYNCMATCH: for my $sync (@nouns) {
## Quick skipping of noise word 'sync'
next if $sync =~ /^syncs?$/;
## If this is a number, it's a timeout, so set it and skip to the next noun
if ($sync =~ /^\d+$/) {
$adverb = $sync;
next SYNCMATCH;
}
## If they want all syncs, grab them all and stop reading any more nouns
if ($sync eq 'all') {
undef @syncs;
for my $name (sort keys %$SYNC) {
push @syncs => $name;
}
last SYNCMATCH;
}
## The rest are all ways of finding the sync they want
## Change the name to a Perl-regex friendly form
(my $term = $sync) =~ s/%/\*/g;
$term =~ s/([^\.])\*/$1.*/g;
$term =~ s/^\*/.*/;
if ($term =~ /\*/) {
for my $name (sort keys %$SYNC) {
push @syncs => $name if $name =~ /^$term$/;
}
next SYNCMATCH;
}
## Now that wildcards are out, we must have an absolute match
if (! exists $SYNC->{$sync}) {
$msg = qq{Sync "$sync" does not appear to exist\n};
## No sense in going on
last SYNCMATCH;
}
## Got a direct match, so store it away
push @syncs => $sync;
}
## If syncs is empty, a regular expression search failed
if (!@syncs) {
$msg = qq{No matching syncs were found\n};
}
## If we have a message, something is wrong
if (defined $msg) {
## Be nice and print a list of active syncs
my @goodsyncs;
for my $s (sort keys %$SYNC) {
push @goodsyncs => $s if $SYNC->{$s}{status} eq 'active';
}
if (@goodsyncs) {
$msg .= "Active syncs:\n";
$msg .= join "\n" => map { " $_" } @goodsyncs;
}
die "$msg\n";
}
## Activate or deactivate one or more syncs
vate_sync() if $verb eq 'activate' or $verb eq 'deactivate';
## Kick one or more syncs
kick() if $verb eq 'kick';
## Pause or resume one or more syncs
pause_resume($verb) if $verb eq 'pause' or $verb eq 'resume';
## Reload one or more syncs
reload_sync() if $verb eq 'reload';
## If we reach here (and we should not), display help and exit
help(1);
exit;
## Everything from here on out is subroutines
sub get_config {
## Given a name, return the matching value from the bucardo_config table
## Arguments: one
## 1. setting name
## Returns: bucardo_config.value string
my $name = shift;
$SQL = 'SELECT setting FROM bucardo.bucardo_config WHERE LOWER(name) = ?';
$sth = $dbh->prepare_cached($SQL);
$count = $sth->execute(lc $name);
if ($count < 1) {
$sth->finish();
die "Invalid bucardo_config setting: $name\n";
}
return $sth->fetchall_arrayref()->[0][0];
} ## end of get_config
sub numbered_relations {
## Sorting function
## Arguments: none (implicit $a / $b via Perl sorting)
## Returns: winning value
## Sorts relations of the form schema.table
## in which we do alphabetical first, but switch to numeric order
## for any numbers at the end of the schema or the table
## Thus, public.foobar1 will come before public.foobar10
## Pull in the names to be sorted, dereference as needed
my $uno = ref $a ? "$a->{schemaname}.$a->{tablename}" : $a;
my $dos = ref $b ? "$b->{schemaname}.$b->{tablename}" : $b;
## Break apart the first item into schema and table
die if $uno !~ /(.+)\.(.+)/;
my ($schema1,$sbase1,$table1,$tbase1) = ($1,$1,$2,$2);
## Store ending numbers if available: if not, use 0
my ($snum1, $tnum1) = (0,0);
$sbase1 =~ s/(\d+)$// and $snum1 = $1;
$tbase1 =~ s/(\d+)$// and $tnum1 = $1;
## Break apart the second item into schema and table
die if $dos !~ /(.+)\.(.+)/;
my ($schema2,$sbase2,$table2,$tbase2) = ($1,$1,$2,$2);
my ($snum2, $tnum2) = (0,0);
$sbase2 =~ s/(\d+)$// and $snum2 = $1;
$tbase2 =~ s/(\d+)$// and $tnum2 = $1;
return (
$sbase1 cmp $sbase2
or $snum1 <=> $snum2
or $tbase1 cmp $tbase2
or $tnum1 <=> $tnum2);
} ## end of numbered_relations
sub check_version {
## Quick check that we have the minumum supported version
## This is for the bucardo database itself
## Arguments: one
## 1. Database handle
## Returns: undef (may die if the version is not good)
my $dbh = shift;
my $res = $dbh->selectall_arrayref('SELECT version()')->[0][0];
if ($res !~ /\D+(\d+)(.+?)\s/) {
die "Sorry, unable to determine the database version\n";
}
my ($maj,$extra) = ($1,$2);
if ($maj < 8 or (8 == $maj and $extra =~ /\.0/)) {
die "Sorry, Bucardo requires Postgres version 8.1 or higher.\n";
}
return;
} ## end of check_version
sub _pod2usage {
require Pod::Usage;
Pod::Usage::pod2usage(
'-verbose' => 99,
'-exitval' => 2,
@_
);
return;
}
sub help {
my ($exitval, $message) = @_;
## Give detailed help about usage of this program
## Arguments: none
## Returns: never, always exits
## Nothing to do if we are being quiet
exit 0 if $QUIET;
_pod2usage(
'-message' => $message,
'-sections' => '^(?:USAGE|COMMANDS|OPTIONS)$',
'-exitval' => $exitval || 0,
);
return;
} ## end of help
sub superhelp {
## Show detailed help by examining the verb and nouns
## Arguments: none
## Returns: never, always exits
## If there are no nouns, we can only show the generic help
help() if ! @nouns;
# Make sure all commands and actions, as well as their aliases, are here.
my %names = (
( map { $_ => 'relgroup' } qw(relgroup herd) ),
( map { $_ => 'db' } qw(db database) ),
( map { $_ => 'list' } qw(l lsit liast lisy lit) ),
( map { $_ => 'upgrade' } qw(upgrade uprgade ugprade) ),
( map { $_ => 'start' } qw(start strt) ),
( map { $_ => 'remove' } qw(remove delete del) ),
( map { $_ => 'update' } qw(update upd udpate) ),
map { $_ => $_ } qw(
activate
add
all
config
customcode
customcols
customname
dbgroup
deactivate
delta
help
inspect
install
kick
list
message
ping
purge
reload
reload
restart
sequence
sequences
set
show
status
stop
sync
table
tables
validate
),
);
# Standardize names.
my @names;
for my $noun (@nouns) {
push @names => $names{ lc $noun } || $names{ standardize_name($noun) }
|| help( 1, 'Unknown command: ' . join ' ' => @nouns );
}
my @command = ($names[0]);
if (@names > 1) {
## Actions are documented in Pod as "=head3 $action $command".
push @command, join ' ', @names;
}
else {
## Don't show subsections for commands that have them.
push @command, => '!.+' if $names[0] eq 'add' || $names[0] eq 'update';
}
usage_exit(join('/' => @command), 0);
return;
} ## end of superhelp
sub ping {
## See if the MCP is alive and responds to pings
## Default is to wait 15 seconds
## Arguments: none, but looks in @nouns for a timeout
## Returns: never, exits
## Set the default timeout, but override if any remaining args start with a number
my $timeout = 15;
for (@nouns) {
if (/^(\d+)/) {
$timeout = $1;
last;
}
}
$VERBOSE and print "Pinging MCP, timeout = $timeout\n";
$dbh->do('LISTEN bucardo_mcp_pong');
$dbh->do('NOTIFY bucardo_mcp_ping');
$dbh->commit();
my $starttime = time;
sleep 0.1;
## Loop until we timeout or get a confirmation from the MCP
P:{
## Grab any notices that have come in
my $notify = $dbh->func('pg_notifies');
if (defined $notify) {
## Extract the PID that sent this notice
my ($name, $pid, $payload) = @$notify;
## We are done: ping successful
$QUIET or print "OK: Got response from PID $pid\n";
exit 0;
}
## Rollback, sleep, and check for a timeout
$dbh->rollback();
sleep 0.5;
my $totaltime = time - $starttime;
if ($timeout and $totaltime >= $timeout) {
## We are done: ping failed
$QUIET or print "CRITICAL: Timed out ($totaltime s), no ping response from MCP\n";
exit 1;
}
redo;
}
return;
} ## end of ping
sub start {
## Attempt to start the Bucardo daemon
## Arguments: none
## Returns: undef
## Write a note to the 'reason' log file
## This will automatically write any nouns in as well
append_reason_file('start');
## Refuse to go on if we get a ping response within 5 seconds
$QUIET or print "Checking for existing processes\n";
## We refuse to start if the MCP PID file exists and looks valid
if (-e $PIDFILE) {
open my $fh, '<', $PIDFILE or die qq{Could not open "$PIDFILE": $!\n};
my $pid = <$fh> =~ /(\d+)/ ? $1 : 0;
close $fh or warn qq{Could not close $PIDFILE: $!\n};
$msg = qq{Cannot start, PID file "$PIDFILE" exists\n};
if (!$pid) {
warn qq{File "$PIDFILE" does not start with a PID!\n};
}
else {
## We have a PID, see if it is still alive
my $res = kill 0 => $pid;
if (0 == $res) {
warn qq{Removing file "$PIDFILE" with stale PID $pid\n};
unlink $PIDFILE;
$msg = '';
}
}
if ($msg) {
$QUIET or print $msg;
append_reason_file('fail');
exit 1;
}
}
## Verify that the version in the database matches our version
my $dbversion = get_config('bucardo_version')
or die "Could not find Bucardo version!\n";
if ($dbversion ne $VERSION) {
my $message = "Version mismatch: bucardo is $VERSION, but bucardo database is $dbversion\n";
append_reason_file('fail');
warn $message;
warn "Perhaps you need to run 'bucardo upgrade' ?\n";
exit 1;
}
## Create a new Bucardo daemon
## If we are a symlink, put the source directory in our path
if (-l $progname and readlink $progname) {
my $dir = dirname( readlink $progname );
unshift @INC, $dir;
}
require Bucardo;
$bcargs->{exit_on_nosync} = delete $bcargs->{'exit-on-nosync'}
if exists $bcargs->{'exit-on-nosync'};
my $bc = Bucardo->new($bcargs);
## Verify that the version of Bucardo.pm matches our version
my $pm_version = $bc->{version} || 'unknown';
if ($VERSION ne $pm_version) {
my $message = "Version mismatch: bucardo is $VERSION, but Bucardo.pm is $pm_version\n";
append_reason_file('fail');
die $message;
}
my $had_stopfile = -e $STOPFILE;
## Just in case, stop it
stop_bucardo();
if ($had_stopfile) {
print qq{Removing file "$STOPFILE"\n} unless $QUIET;
}
unlink $STOPFILE;
$QUIET or print qq{Starting Bucardo\n};
## Disconnect from our local connection before we fork
$dbh->disconnect();
## Remove nouns from @opts.
## XXX Will fail if an option value is the same as a noun.
my %remove = map { $_ => undef } @nouns;
@opts = grep { ! exists $remove{$_} } @opts;
## Fork and setsid to disassociate ourselves from the daemon
if (fork) {
## We are the kid, do nothing
}
else {
setsid() or die;
## Here we go!
$bc->start_mcp( \@opts );
}
exit 0;
} ## end of start
sub stop {
## Attempt to stop the Bucardo daemon
## Arguments: none
## Returns: undef
## Write a note to the 'reason' log file
append_reason_file('stop');
print "Creating $STOPFILE ... " unless $QUIET;
stop_bucardo();
print "Done\n" unless $QUIET;
## If this was called directly, just exit now
exit 0 if $verb eq 'stop';
return;
} ## end of stop
sub stop_bucardo {
## Create the semaphore that tells all Bucardo processes to exit
## Arguments: none
## Returns: undef
## Create the file, and write some quick debug information into it
## The only thing the processe care about is if the file exists
open my $stop, '>', $STOPFILE or die qq{Could not create "$STOPFILE": $!\n};
print {$stop} "Stopped by $progname on " . (scalar localtime) . "\n";
close $stop or warn qq{Could not close "$STOPFILE": $!\n};
return;
} ## end of stop_bucardo
sub restart {
## Simple, really: stop, wait, start!
## Arguments: none
## Returns: undef
stop();
sleep 3;
start();
return;
} ## end of restart
sub reload {
## Reload the MCP daemon
## Effectively restarts everything
## Arguments: none
## Returns: never, exits
## Is Bucardo active?
my $pong = 'bucardo_mcp_pong';
$dbh->do("LISTEN $pong");
$dbh->do('NOTIFY bucardo_mcp_ping');
$dbh->commit();
## Wait a little bit, then scan for the confirmation message
sleep 0.1;
if (! wait_for_notice($dbh, $pong, 2)) {
die "Looks like Bucardo is not running, so there is no need to reload\n";
}
## We want to wait to hear from the MCP that it is done
my $done = 'bucardo_reloaded_mcp';
$dbh->do("LISTEN $done");
$dbh->do('NOTIFY bucardo_mcp_reload');
$dbh->commit();
## Wait a little bit, then scan for the confirmation message
sleep 0.1;
my $timeout = $adverb || get_config('reload_config_timeout') || 30;
if (! wait_for_notice($dbh, $done, $timeout) ) {
die "Waited ${timeout}s, but Bucardo never confirmed the reload!\n"
. "HINT: Pass a longer timeout to the reload_config command or set the\n"
. "reload_config_timeout configuration setting to wait longer\n";
}
print "DONE!\n";
exit 0;
} ## end of reload
sub reload_config {
## Reload configuration settings from the bucardo database,
## then restart all controllers and kids
## Arguments: none directly (but processes the nouns to check for numeric arg)
## Returns: never, exits
## Scan the nouns for a numeric argument.
## If found, set as the adverb.
## This will cause us to wait for confirmation or reload before exiting
for (@nouns) {
if (/^(\d+)$/) {
$adverb = $1;
last;
}
}
$QUIET or print qq{Forcing Bucardo to reload the bucardo_config table\n};
## Is Bucardo active?
my $pong = 'bucardo_mcp_pong';
$dbh->do("LISTEN $pong");
$dbh->do('NOTIFY bucardo_mcp_ping');
$dbh->commit();
## Wait a little bit, then scan for the confirmation message
sleep 0.1;
if (! wait_for_notice($dbh, $pong, 2)) {
die "Looks like Bucardo is not running, so there is no need to reload\n";
}
## We want to wait to hear from the MCP that it is done
my $done = 'bucardo_reload_config_finished';
$dbh->do("LISTEN $done");
$dbh->do('NOTIFY bucardo_reload_config');
$dbh->commit();
## Wait a little bit, then scan for the confirmation message
sleep 0.1;
my $timeout = $adverb || get_config('reload_config_timeout') || 30;
if (! wait_for_notice($dbh, $done, $timeout) ) {
die "Waited ${timeout}s, but Bucardo never confirmed the configuration reload!\n"
. "HINT: Pass a longer timeout to the reload_config command or set the\n"
. "reload_config_timeout configuration setting to wait longer\n";
}
print "DONE!\n";
exit 0;
} ## end of reload_config
sub wait_for_notice {
## Keep hanging out until we get the notice we are waiting for
## Arguments: three
## 1. Database handle
## 2. String(s) to listen for
## 3. How long to wait (default is forever)
## Returns: 1
## If the strings argument is an array ref, this will return a hash ref
## where each key is a string we found, and the value is how many times we
## found it. Note that we return as soon as we've found at least one
## matching NOTIFY; we don't wait for the full timeout to see which
## messages show up.
my ($ldbh, $string, $howlong) = @_;
my ($num_strings, %search_strings, %matches);
my $found = 0;
if (ref $string eq 'ARRAY') {
$num_strings = scalar @$string;
map { $search_strings{$_} = 1 } @$string;
}
else {
$num_strings = 1;
$search_strings{$string} = 1;
}
my $start_time = [gettimeofday];
WAITIN: {
for my $notice (@{ db_get_notices($ldbh) }) {
my ($name) = @$notice;
if (exists $search_strings{$name}) {
$found = 1;
$matches{$name}++;
}
}
last WAITIN if $found;
if (defined $howlong) {
my $elapsed = tv_interval( $start_time );
return 0 if ($elapsed >= $howlong and (scalar keys %matches == 0));
}
$dbh->commit();
sleep($WAITSLEEP);
redo;
}
if (scalar keys %matches) {
if ($num_strings == 1) {
return 1;
}
else {
return \%matches;
}
}
else {
if ($num_strings == 1) {
return 0;
}
else {
return {};
}
}
} ## end of wait_for_notice
sub reload_sync {
## Ask for one or more syncs to be reloaded
## Arguments: none directly (but processes the nouns for a list of syncs)
## Returns: never, exits
my $doc_section = 'reload';
usage_exit($doc_section) unless @nouns;
for my $syncname (@nouns) {
## Be nice and allow things like $0 reload sync foobar
next if $syncname eq 'sync';
## Make sure this sync exists, and grab its status
$SQL = 'SELECT status FROM bucardo.sync WHERE name = ?';
$sth = $dbh->prepare($SQL);
$count = $sth->execute($syncname);
if ($count != 1) {
warn "Invalid sync: $syncname\n";
$sth->finish();
next;
}
my $status = $sth->fetch()->[0];
## Skip any syncs that are not active
if ($status ne 'active') {
warn qq{Cannot reload: status of sync "$syncname" is $status\n};
next;
}
## We wait for the MCP to tell us that each sync is done reloading
my $done = "bucardo_reloaded_sync_$syncname";
my $err = "bucardo_reload_error_sync_$syncname";
print "Reloading sync $syncname...";
$dbh->do(qq{LISTEN "$done"});
$dbh->do(qq{LISTEN "$err"});
$dbh->do(qq{NOTIFY "bucardo_reload_sync_$syncname"});
$dbh->commit();
## Sleep a little, then wait until we hear a confirmation from the MCP
sleep 0.1;
my $res = wait_for_notice($dbh, [$err, $done], 10);
if ($res == 0 or scalar keys %$res == 0) {
print "Reload of sync $syncname failed; reload response message never received\n";
}
elsif (exists $res->{$done}) {
print "Reload of sync $syncname successful\n";
}
elsif (exists $res->{$err}) {
print "Reload of sync $syncname failed\n";
}
else {
print "ERROR. Reload results unavailable, because something weird happened.\n";
}
print "\n";
} ## end each sync to be reloaded
exit 0;
} ## end of reload_sync
sub reopen {
## Signal the bucardo processes that they should reopen any log files
## Used after a log rotation
## Sends a USR2 to all Bucardo processes
## Arguments: none
## Returns: never, exits
open my $fh, '<', $PIDFILE
or die qq{Could not open pid file $PIDFILE: is Bucardo running?\n};
## Grab the PID of the MCP
if (<$fh> !~ /(\d+)/) { ## no critic
die qq{Could not find a PID in file $PIDFILE!\n};
}
close $fh or warn qq{Could not close $PIDFILE: $!\n};
my $gid = getpgrp $1;
$gid =~ /^\d+$/ or die qq{Unable to obtain the process group\n};
## Quick mapping of names to numbers so we can kill effectively
my $x = 0;
my %signumber;
for (split(' ', $Config{sig_name})) {
$signumber{$_} = $x++;
}
my $signumber = $signumber{USR2};
## The minus indicates we are sending to the whole group
my $num = kill -$signumber, $gid;
if ($num < 1) {
warn "Unable to signal any processed with USR2\n";
exit 1;
}
$QUIET or print "Sent USR2 to Bucardo processes\n";
exit 0;
} ## end of reopen
sub validate {
## Attempt to validate one or more syncs
## Arguments: none directly (but processes the nouns for a list of syncs)
## Returns: never, exits
my $doc_section = 'validate';
usage_exit($doc_section) unless @nouns;
## Build the list of syncs to validate
my @synclist;
## Nothing specific is the same as 'all'
if ($nouns[0] eq 'all' and ! defined $nouns[1]) {
@synclist = sort keys %$SYNC;
if (! @synclist) {
print "Sorry, there are no syncs to validate!\n";
exit 0;
}
}
else {
for my $name (@nouns) {
## Be nice and allow things like $0 validate sync foobar
next if $name eq 'sync';
if (! exists $SYNC->{$name}) {
die qq{Sorry, there is no sync named "$name"\n};
}
push @synclist => $name;
}
}
## Get the largest sync name so we can line up the dots all pretty
my $maxsize = 1;
for my $name (@synclist) {
$maxsize = length $name if length $name > $maxsize;
}
$maxsize += 3;
## Loop through and validate each in turn,
## waiting for a positive response from the MCP
my $exitval = 0;
for my $name (@synclist) {
printf "Validating sync $name %s ",
'.' x ($maxsize - length $name);
my ($evalok, $success);
eval {
my ($message) = $dbh->selectrow_array(
'SELECT validate_sync(?)',
undef, $name
);
$dbh->commit;
if ($message eq 'MODIFY') {
$success = 1;
}
else {
warn "$message\n";
$exitval++;
}
$evalok = 1;
};
if ($evalok) {
print "OK\n" if $success;
}
else {
warn $dbh->errstr || $@;
$exitval++;
}
}
exit $exitval;
} ## end of validate
sub count_deltas {
## Count up rows in the delta tables
## Does not remove "unvacuumed" rows: assumes delta tables are getting emptied out by VAC
## Arguments: optional
## Returns: nothing, exits
## May want to see totals only
my $total_only = (defined $nouns[0] and $nouns[0] =~ /totals?/i) ? 1 : 0;
## See if we want to limit it to specific databases
my %dblimit;
for my $name (@nouns) {
## Do not limit if doing a total, even if other names are specified
next if $total_only;
## Allow wildcards
if ($name =~ s/[%*]/.*/) {
for (grep { $_ =~ /$name/ } keys %$DB) {
$dblimit{$_}++;
}
}
elsif (exists $DB->{$name}) {
$dblimit{$name}++;
}
}
## No matches means we stop right away
if (@nouns and !keys %dblimit and !$total_only) {
warn qq{No matching databases were found: try "bucardo list dbs"\n};
exit 1;
}
my $total = { grand => 0 };
for my $dbname (sort keys %$DB) {
my $db = $DB->{$dbname};
## Only sources should get checked
if (! $db->{issource}) {
if (delete $dblimit{$dbname}) {
print "Skipping database $dbname: not a source\n";
}
elsif ($VERBOSE >= 1) {
print "Skipping $dbname: not a source\n";
}
next;
}
## If we are limiting, possibly skip this one
next if keys %dblimit and ! exists $dblimit{$dbname};
## Make sure it has a bucardo schema.
## May not if validate_sync has never been run!
my $dbh = connect_database($dbname);
if (! schema_exists('bucardo')) {
warn "Cannot check database $dbname: no bucardo schema!\n";
next;
}
## Grab all potential delta tables
$SQL = 'SELECT deltaname FROM bucardo.bucardo_delta_names';
for my $row (@{ $dbh->selectall_arrayref($SQL) }) {
my $tname = $row->[0];
$SQL = "SELECT count(*) FROM bucardo.$tname";
$count = $dbh->selectall_arrayref($SQL)->[0][0];
$total->{grand} += $count;
$total->{database}{$dbname} += $count;
if ($db->{status} ne 'active') {
$total->{databaseinactive}{$dbname} = 1;
}
}
$dbh->disconnect();
}
## Stop here if we did not actually scan any databases because they are all non-source
if (! keys %{ $total->{database} }) {
print "No databases to check\n";
exit 1;
}
## Figure out our sizes for a pretty alignment
my $grandmessage = 'Total deltas across all targets';
my $dbmessage = 'Total deltas for database';
my $size = { db => 0, largest => length $grandmessage, };
for my $db (keys %{ $total->{database} }) {
$size->{db} = length $db if length $db > $size->{db};
my $len = length " $dbmessage $db";
$size->{largest} = $len if $len > $size->{largest};
}
printf "%*s: %s\n", $size->{largest}, $grandmessage, pretty_number($total->{grand});
## Break it down by database
for my $db (sort keys %{ $total->{database} }) {
next if $total_only;
printf "%*s: %s%s\n",
$size->{largest},
" $dbmessage $db",
pretty_number($total->{database}{$db}),
$total->{databaseinactive}{$db} ? ' (not active)' : '';
}
exit 0;
} ## end of count_deltas
sub purge {
## Purge the delta and track tables for one or more tables, for one or more databases
## Arguments: variable
## Returns: never, exits
## TODO: databases, tables, timeslices
my $doc_section = 'purge';
## Nothing specific is the same as 'all'
my $doall = 0;
if (!@nouns or ($nouns[0] eq 'all' and ! defined $nouns[1])) {
$doall = 1;
for my $dbname (sort keys %$DB) {
my $db = $DB->{$dbname};
## Do not purge inactive databases
next if $db->{status} ne 'active';
## Do not purge unless they are a source
next if ! $db->{issource};
print "Checking db $dbname\n";
## Make sure it has a bucardo schema.
## May not if validate_sync has never been run!
my $dbh = connect_database($dbname);
if (! schema_exists('bucardo')) {
warn "Cannot purge database $dbname: no bucardo schema!\n";
next;
}
## Run the purge_delta on this database
$SQL = 'SELECT bucardo.bucardo_purge_delta(?)';
$sth = $dbh->prepare($SQL);
$sth->execute('1 second');
my $results = $sth->fetchall_arrayref()->[0][0];
## Dump the resulting message back to the user
## Should be like this: Tables processed: 3
print "$dbname: $results\n";
$dbh->commit();
}
}
if (! $doall) {
for my $name (@nouns) {
die "Purging name $name\n";
}
}
exit 0;
} ## end of purge
sub add_item {
## Add an item to the internal bucardo database
## Arguments: none directly (but processes the nouns)
## Returns: never, exits
my $doc_section = 'add/!.+';
usage_exit($doc_section) unless @nouns;
## First word is the type of thing we are adding
my $thing = shift @nouns;
## Account for variations and abbreviations
$thing = standardize_name($thing);
## All of these will exit and do not return
add_customcode() if $thing eq 'customcode';
add_customname() if $thing eq 'customname';
add_customcols() if $thing eq 'customcols';
add_database() if $thing eq 'database';
add_dbgroup() if $thing eq 'dbgroup';
add_herd() if $thing eq 'herd';
add_sync() if $thing eq 'sync';
## The rest is tables and sequences
## We need to support 'add table all' as well as 'add all tables'
my $second_arg = $nouns[0] || '';
## Rearrange the args as needed, and determine if we want 'all'
my $do_all = 0;
if ($thing eq 'all') {
$do_all = 1;
$thing = shift @nouns;
$thing = standardize_name($thing);
}
elsif (lc $second_arg eq 'all') {
$do_all = 1;
shift @nouns;
}
## Quick check in case someone thinks they should add a goat
if ($thing =~ /^goat/i) {
warn qq{Cannot add a goat: use add table or add sequence instead\n};
exit 1;
}
## Add a table
if ($thing eq 'table') {
if ($do_all) {
## Add all the tables, and return the output
print add_all_tables();
## The above does not commit, so make sure we do it here
confirm_commit();
exit 0;
}
else {
add_table('table');
}
}
## Add a sequence
if ($thing eq 'sequence') {
if ($do_all) {
## Add all the sequences, and return the output
print add_all_sequences();
## The above does not commit, so make sure we do it here
$dbh->commit();
exit 0;
}
else {
add_table('sequence');
}
}
## Anything past this point is an error
if ($do_all) {
warn qq{The 'all' option can only be used with 'table' and 'sequence'\n};
exit 1;
}
usage_exit($doc_section);
return;
} ## end of add_item
sub update_item {
## Update some object in the database
## This merely passes control on to the more specific update_ functions
## Arguments: none (but parses nouns)
## Returns: undef
my $doc_section = 'update/!.+';
## Must have at least three nouns
usage_exit($doc_section) if @nouns < 3;
## What type of thing are we updating?
my $thing = shift @nouns;
## Account for variations and abbreviations
$thing = standardize_name($thing);
my $code = $thing eq 'customcode' ? \&update_customcode
: $thing eq 'database' ? \&update_database
: $thing eq 'dbgroup' ? \&update_dbgroup
: $thing eq 'sync' ? \&update_sync
: $thing eq 'table' ? \&update_table
: $thing eq 'sequence' ? \&update_table
: usage_exit($doc_section)
;
## The update function returns, due to recursion, so we must exit.
$code->(@nouns);
exit 0;
} ## end of update_item
sub list_item {
## Show information about one or more items in the bucardo database
## Arguments: none, but parses nouns
## Returns: 0 on success, -1 on error
my $doc_section = 'list';
usage_exit($doc_section) unless @nouns;
## First word is the type if thing we are listing
my $thing = shift @nouns;
## Account for variations and abbreviations
$thing = standardize_name($thing);
SWITCH: {
$thing eq 'clone' and do {
list_clones();
last SWITCH;
};
$thing eq 'config' and do {
$verb = 'config';
config();
exit;
};
$thing eq 'customcode' and do {
list_customcodes();
last SWITCH;
};
$thing eq 'customname' and do {
list_customnames();
last SWITCH;
};
$thing eq 'customcols' and do {
list_customcols();
last SWITCH;
};
## The dbgroup must be checked before the database (dbg vs db)
$thing eq 'dbgroup' and do {
list_dbgroups();
last SWITCH;
};
$thing eq 'database' and do {
list_databases();
last SWITCH;
};
$thing eq 'herd' and do {
list_herds();
last SWITCH;
};
$thing eq 'sync' and do {
list_syncs();
last SWITCH;
};
$thing eq 'table' and do {
list_tables();
last SWITCH;
};
$thing eq 'sequence' and do {
list_sequences();
last SWITCH;
};
$thing eq 'all' and do {
## Not shown on purpose: clones
if (keys %$CUSTOMCODE) {
print "-- customcodes:\n"; list_customcodes();
}
if (keys %$CUSTOMNAME) {
print "-- customnames:\n"; list_customnames();
}
if (keys %$CUSTOMCOLS) {
print "-- customcols:\n"; list_customcols();
}
print "-- dbgroups:\n"; list_dbgroups();
print "-- databases:\n"; list_databases();
print "-- relgroup:\n"; list_herds();
print "-- syncs:\n"; list_syncs();
print "-- tables:\n"; list_tables();
print "-- sequences:\n"; list_sequences();
print "\n";
last SWITCH;
};
## catch all
## Cannot list anything else
usage_exit($doc_section);
} # SWITCH
exit 0;
} ## end of list_item
sub remove_item {
## Delete from the bucardo database
## Arguments: none, but parses nouns
## Returns: never, exits
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
## First word is the type if thing we are removing
my $thing = shift @nouns;
## Account for variations and abbreviations
$thing = standardize_name($thing);
my $second_arg = $nouns[0] || '';
## Allow the keyword 'all' to appear before or after the noun
my $do_all = 0;
if ($thing eq 'all') {
$do_all = 1;
$thing = shift @nouns;
$thing = standardize_name($thing);
}
elsif (lc $second_arg eq 'all') {
$do_all = 1;
shift @nouns;
}
my $arg = $do_all ? 'all' : '';
## All of these will exit and do not return
remove_customcode($arg) if $thing eq 'customcode';
remove_customname($arg) if $thing eq 'customname';
remove_customcols($arg) if $thing eq 'customcols';
## The dbgroup must be checked before the database (dbg vs db)
remove_database($arg) if $thing eq 'database';
remove_dbgroup($arg) if $thing eq 'dbgroup';
remove_herd($arg) if $thing eq 'herd';
remove_sync($arg) if $thing eq 'sync';
remove_relation('table', $arg) if $thing eq 'table';
remove_relation('sequence', $arg) if $thing eq 'sequence';
## Do not know how to remove anything else
usage_exit($doc_section);
return;
} ## end of remove_item
##
## Database-related subroutines: add, remove, update, list
##
sub add_database {
## Add one or more databases. Inserts to the bucardo.db table
## By default, we do a test connection as well (turn off with the --force argument)
## Arguments: two or more
## 1. The internal name Bucardo uses to refer to this database
## 2+ name=value parameters, dash-dash arguments
## Returns: undef
## Example: bucardo add db nyc1 dbname=nyc1 dbhost=nyc1.example.com dbgroup=sales
## Example: bucardo add dbs nyc1,nyc2 dbname=nyc1,nyc2 dbgroup=sales
## Grab our generic usage message
my $doc_section = 'add/add db';
## The first word is the internal name (bucardo.db.name) - may have commas
my $item_name = shift @nouns || '';
## No name is a problem
usage_exit($doc_section) unless length $item_name;
## We may have more than one database specified at once
## Assign to an array, and set the role as well in case a dbgroup is set
my $db_names = [];
my $newsource = 0;
for my $entry (split /\s*,\s*/ => $item_name) {
## First database defaults to source, others to targets
if (! @$db_names and $entry !~ /:/) {
$entry .= ':source';
$newsource = 1;
}
push @{ $db_names } => [ extract_name_and_role($entry) ];
}
## Inputs and aliases, database column name, flags, default value
my $validcols = q{
db|dbname dbname 0 null
type|dbtype dbtype 0 postgres
pass|password|dbpass dbpass 0 null
host|dbhost|pghost dbhost 0 ENV:PGHOSTADDR|PGHOST
port|dbport|pgport dbport 0 ENV:PGPORT
conn|dbconn|pgconn dbconn 0 null
service|dbservice dbservice 0 null
dsn|dbdsn dbdsn 0 null
stat|status status =active|inactive null
group|dbgroup dbgroup 0 null
addalltables none 0 null
addallsequences none 0 null
server_side_prepares|ssp server_side_prepares TF null
makedelta makedelta TF null
};
## Include the value for the dbuser only if a service or dsn is not specified, or
## a user was explicitly included. In other words, don't default the user
## name when there's a service.
$validcols .= "user|username|dbuser dbuser 0 bucardo\n"
if ((! grep { /^(db)?service=/ or /dsn/ } @nouns) || grep { /^(db)?user(name)?=/ } @nouns);
my ($dbcols) = process_simple_args({
cols => $validcols,
list => \@nouns,
doc_section => $doc_section,
});
## Must have a database name unless using a service or dsn
if (! exists $dbcols->{dbname} && ! exists $dbcols->{dbservice} && ! exists $dbcols->{dbdsn}) {
print qq{Cannot add database: must supply a database name to connect to\n};
exit 1;
}
## Cannot add if already there
for my $db (map { $_->[0] } @$db_names) {
if (exists $DB->{ $db }) {
print qq{Cannot add database: the name "$db" already exists\n};
exit 1;
}
}
## Clean up and standardize the type name
my $dbtype = $dbcols->{dbtype} = standardize_rdbms_name($dbcols->{dbtype});
## If we have a service or DSN, strip the host and port as they may have been set via ENV
if (exists $dbcols->{dbservice} or exists $dbcols->{dbdsn}) {
delete $dbcols->{dbport};
delete $dbcols->{dbhost};
}
## We do not want some things to hang around in the dbcols hash
my $dbgroup = delete $dbcols->{dbgroup};
## Map each value into individual databases
my %dbinfo;
for my $k (sort keys %$dbcols) {
## Each db in db_names needs to have an associated value for each dbcol entry
## Hence, we only use dbcols to build list of columns: values are kept in a hash
next if $dbcols->{$k} !~ /,/;
my @list = split /\s*,\s*/ => $dbcols->{$k};
my $value;
## The dbnames can contain role information: strip it out from here
if ('dbname' eq $k) {
@list = map { [extract_name_and_role($_)]->[0] } @list;
}
for (my $x=0; defined $db_names->[$x]; $x++) {
$value = $list[$x] if defined $list[$x];
$dbinfo{$k}[$x] = $value;
}
}
## Attempt to insert into the bucardo.db table
my $columns = join ',' => keys %$dbcols;
my $qs = '?,' x keys %$dbcols;
$SQL = "INSERT INTO bucardo.db (name,$columns) VALUES (${qs}?)";
debug("SQL: $SQL");
$sth = $dbh->prepare($SQL);
for (my $x = 0; defined $db_names->[$x]; $x++) {
my @args;
for my $key (keys %$dbcols) {
push @args => exists $dbinfo{$key} ? $dbinfo{$key}->[$x] : $dbcols->{$key};
}
my $evalok = 0;
debug(Dumper $db_names->[$x]);
debug(Dumper \@args);
eval {
$sth->execute($db_names->[$x][0], @args);
$evalok = 1;
};
if (! $evalok) {
if ($@ =~ /"db_name_sane"/) {
die qq{Invalid name: you cannot refer to this database as "$db_names->[$x]"\n};
}
die "Failed to add database: $@\n";
}
}
## Store certain messages so we can output them in a desired order
my $finalmsg = '';
## Test database handle
my $testdbh;
## May want to do a test connection to each databases
TESTCONN: {
## Nothing else to do for flatfiles
last TESTCONN if 'flatfile' eq $dbtype;
## Get the module name, the way to refer to its database
## This also makes sure we have a valid type
my %dbtypeinfo = (
drizzle => ['DBD::drizzle', 'Drizzle database'],
firebird => ['DBD::Firebird', 'Firebird database'],
mongo => ['MongoDB', 'MongoDB'],
mysql => ['DBD::mysql', 'MySQL database'],
mariadb => ['DBD::mysql', 'MariaDB database'],
oracle => ['DBD::Oracle', 'Oracle database'],
postgres => ['DBD::Pg', 'PostgreSQL database'],
redis => ['Redis', 'Redis database'],
sqlite => ['DBD::SQLite', 'SQLite database'],
);
if (! exists $dbtypeinfo{$dbtype}) {
die qq{Unknown database type: $dbtype\n};
}
my ($module,$fullname) = @{ $dbtypeinfo{$dbtype} };
## Gather connection information from the database via db_getconn
$SQL = 'SELECT bucardo.db_getconn(?)';
$sth = $dbh->prepare($SQL);
for my $db (map { $_->[0] } @$db_names) {
$sth->execute($db);
my $dbconn = $sth->fetchall_arrayref()->[0][0];
## Must be able to load the Perl driver
my $evalok = 0;
eval {
eval "require $module";
$evalok = 1;
};
if (! $evalok) {
die "Cannot add unless the Perl module '$module' is available: $@\n";
}
## Reset for the evals below
$evalok = 0;
## Standard args for the DBI databases
## We put it here as we may move around with the Postgres bucardo user trick
my ($type,$dsn,$user,$pass) = split /\n/ => $dbconn;
## Handle all of the ones that do not use standard DBI first
if ('mongo' eq $dbtype) {
## Catch this nice and early - but also have a check in Bucardo.pm
my $gotboolean = 0;
eval {
require boolean;
$gotboolean = 1;
};
if (! $gotboolean) {
warn qq{Unable to load the Perl 'boolean' module: needed for MongoDB support\n};
}
my $mongoURI = 'mongodb://';
if ($dsn =~ s/^DSN://) {
## Just in case:
if ($dsn !~ /^mongodb:/) {
$mongoURI .= $dsn;
}
else {
$mongoURI = $dsn;
}
}
else {
my $mongodsn = {};
for my $line (split /\n/ => $dbconn) {
next if $line !~ /(\w+):\s+(.+)/;
$mongodsn->{$1} = $2;
}
if (exists $mongodsn->{dbuser}) {
my $pass = $mongodsn->{dbpass} || '';
$mongoURI .= "$mongodsn->{dbuser}:$pass\@";
}
$mongoURI .= $mongodsn->{host} || 'localhost';
$mongoURI .= ":$mongodsn->{port}" if exists $mongodsn->{port};
}
my $mongoversion = $MongoDB::VERSION;
my $oldversion = $mongoversion =~ /^0\./ ? 1 : 0;
eval {
$testdbh = $oldversion ? MongoDB::MongoClient->new(host => $mongoURI) : MongoDB->connect($mongoURI);
$evalok = 1;
};
}
elsif ('redis' eq $dbtype) {
my $tempdsn = {};
for my $line (split /\n/ => $dbconn) {
next if $line !~ /(\w+):\s+(.+)/;
$tempdsn->{$1} = $2;
}
my $server;
if (exists $tempdsn->{host}) {
$server = $tempdsn->{host};
}
if (exists $tempdsn->{port}) {
$server .= ":$tempdsn->{port}";
}
my @dsn;
if (defined $server) {
push @dsn => 'server', $server;
}
my ($pass, $index);
if (exists $tempdsn->{pass}) {
$pass = $tempdsn->{pass};
}
if (exists $tempdsn->{name} and $tempdsn->{name} !~ /\D/) {
$index = $tempdsn->{name};
}
push @dsn => 'on_connect', sub {
$_[0]->client_setname('bucardo');
$_[0]->auth($pass) if $pass;
$_[0]->select($index) if $index;
};
$evalok = 0;
eval {
$testdbh = Redis->new(@dsn);
$evalok = 1;
};
}
## Anything else must be something with a standard DBI driver
else {
$dsn =~ s/^DSN://;
eval {
$testdbh = DBI->connect($dsn, $user, $pass, {AutoCommit=>0,RaiseError=>1,PrintError=>0});
$evalok = 1;
};
}
## At this point, we have eval'd a connection
if ($evalok) {
## Disconnect from DBI.
$testdbh->disconnect if $module =~ /DBD/;
}
else {
my $err = $DBI::errstr || $@;
## For Postgres, we get a little fancy and try to account for instances
## where the bucardo user may not exist yet, by reconnecting and
## creating said user if needed.
if ($DBI::errstr
and 'postgres' eq $dbtype
and $user eq 'bucardo'
and $DBI::errstr =~ /bucardo/
and eval { require Digest::MD5; 1 }) {
# Try connecting as postgres instead.
print qq{Connection to "$db" ($fullname) as user bucardo failed.\nError was: $DBI::errstr\n\n};
print qq{Will try to connect as user postgres and create superuser $user...\n\n};
my $dbh = eval {
DBI->connect($dsn, 'postgres', $pass, {AutoCommit=>1,RaiseError=>1,PrintError=>0});
};
if ($dbh) {
## Create the bucardo user now. We'll need a password;
## create one if we don't have one.
my $connok = 0;
eval {
my $newpass = $pass || generate_password();
my $encpass = Digest::MD5::md5_hex($newpass);
$dbh->do(qq{CREATE USER $user SUPERUSER ENCRYPTED PASSWORD '$encpass'});
$dbh->disconnect;
my $extrauser = $pass ? '' : qq{ with password "$newpass"};
warn "Created superuser '$user'$extrauser\n\n";
$pass = $newpass;
$connok = 1;
};
goto TESTCONN if $connok;
$err = $DBI::errstr || $@;
$msg = "Unable to create superuser $user";
}
else {
$err = $DBI::errstr || $@;
$msg = 'Connection as postgres failed, too';
}
}
else {
$msg = qq{Connection to "$db" ($fullname) failed};
}
die "$msg. You may force add it with the --force argument.\nError was: $err\n\n"
unless $bcargs->{force};
warn "$msg, but will add anyway.\nError was: $err\n";
}
} ## End each database to connect to
} ## end of TESTCONN
## If we got a group, process that as well
if (defined $dbgroup) {
## If the dbnames had supplied role information, extract that now
if (exists $dbcols->{dbname} and $dbcols->{dbname} =~ /:/) {
my $x=0;
for my $namerole (split /\s*,\s*/ => $dbcols->{dbname}) {
my ($name,$role) = extract_name_and_role($namerole);
debug("$namerole gave us $name and $role");
$db_names->[$x++][1] = $role;
}
}
## If it has an attached role, strip it out and force that everywhere
my $master_role = $dbgroup =~ s/:(\w+)// ? $1 : 0;
## We need to store this away as the function below changes the global hash
my $isnew = exists $DBGROUP->{$dbgroup} ? 0 : 1;
my $firstrow = 1;
for my $row (@$db_names) {
my ($db,$role) = @$row;
## If we set this source ourself, change to target if the group already exists
if ($firstrow) {
$firstrow = 0;
if ($newsource and ! $isnew) {
$role = 'target';
}
}
## The master role trumps everything
$role = $master_role if $master_role;
my ($newgroup, $newrole) = add_db_to_group($db, "$dbgroup:$role");
if ($isnew) {
$finalmsg .= qq{Created dbgroup "$newgroup"\n};
$isnew = 0;
}
$finalmsg .= qq{ Added database "$db" to dbgroup "$newgroup" as $newrole\n};
}
}
## Adjust the db name so add_all_* can use it
$bcargs->{db} = $db_names->[0][0];
## Make sure $DB gets repopulated for the add_all_* calls below
load_bucardo_info(1);
## Add in all tables for this database
$finalmsg .= add_all_tables() if grep /addalltab/i, @nouns;
## Add in all sequences for this database
$finalmsg .= add_all_sequences() if grep /addallseq/i, @nouns;
if (!$QUIET) {
my $list = join ',' => map { qq{"$_->[0]"} } @$db_names;
printf qq{Added %s %s\n},
$list =~ /,/ ? 'databases' : 'database', $list;
$finalmsg and print $finalmsg;
}
confirm_commit();
exit 0;
} ## end of add_database
sub remove_database {
## Remove one or more databases. Updates the bucardo.db table
## Use the --force argument to clear out related tables and groups
## Arguments: one or more
## 1+ Name of a database
## Returns: undef
## Example: bucardo remove db nyc1 nyc2 --force
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
## Make sure all named databases exist
for my $name (@nouns) {
if (! exists $DB->{$name}) {
die qq{No such database "$name"\n};
}
}
## Prepare the SQL to delete each database
$SQL = 'DELETE FROM bucardo.db WHERE name = ?';
$sth = $dbh->prepare($SQL);
## Loop through and attempt to delete each given database
for my $name (@nouns) {
## Wrap in an eval so we can handle known exceptions
my $evalok = 0;
$dbh->pg_savepoint('try_remove_db');
eval {
$sth->execute($name);
$evalok = 1;
};
if (! $evalok) {
if ($bcargs->{force} and $@ =~ /"goat_db_fk"|"dbmap_db_fk"/) {
$QUIET or warn qq{Dropping all tables and dbgroups that reference database "$name"\n};
$dbh->pg_rollback_to('try_remove_db');
$dbh->do('DELETE FROM bucardo.goat WHERE db = ' . $dbh->quote($name));
$dbh->do('DELETE FROM bucardo.dbmap WHERE db = ' . $dbh->quote($name));
## Try the same query again
eval {
$sth->execute($name);
};
}
## We've failed: output a reasonable message when possible
if ($@ =~ /"goat_db_fk"/) {
die qq{Cannot delete database "$name": must remove all tables that reference it first (try --force)\n};
}
if ($@ =~ /"dbmap_db_fk"/) {
die qq{Cannot delete database "$name": must remove all dbmap references first (try --force)\n};
}
$@ and die qq{Could not delete database "$name"\n$@\n};
}
}
for my $name (@nouns) {
$QUIET or print qq{Removed database "$name"\n};
}
confirm_commit();
exit 0;
} ## end of remove_database
sub update_database {
## Update one or more databases.
## This may modify the bucardo.db, bucardo.dbgroup, and bucardo.dbmap tables
## Arguments: two plus
## 1. Name of the database to update. Can be "all" and can have wildcards
## 2+ What exactly we are updating.
## Returns: undef
## Example: bucardo update db nyc1 port=6543 group=nycservers:source,globals
my @actions = @_;
## Grab our generic usage message
my $doc_section = 'update/update db';
usage_exit($doc_section) unless @actions;
my $name = shift @actions;
## Recursively call ourselves for wildcards and 'all'
return if ! check_recurse($DB, $name, @actions);
## Make sure this database exists!
if (! exists $DB->{$name}) {
die qq{Could not find a database named "$name"\nUse 'list dbs' to see all available.\n};
}
## Everything is a name=value setting after this point
## We will ignore and allow noise word "set"
for my $arg (@actions) {
next if $arg =~ /set/i;
next if $arg =~ /\w+=\w+/o;
usage_exit($doc_section);
}
## Change the arguments into a hash
my $args = process_args(join ' ' => @actions);
## Track what changes we made
my %change;
## Walk through and handle each argument pair
for my $setting (sort keys %$args) {
next if $setting eq 'extraargs';
## Change the name to a more standard form, to better figure out what they really mean
## This also excludes all non-alpha characters
my $newname = transform_name($setting);
## Exclude ones that cannot / should not be changed (e.g. cdate)
if (exists $column_no_change{$newname}) {
print "Sorry, the value of $setting cannot be changed\n";
exit 1;
}
## Standardize the values as well
my $value = $args->{$setting};
my $newvalue = transform_value($value);
my $oldvalue = $DB->{$name}{$newname};
## We want certain booleans to appear as "off/on"
if ($setting =~ /makedelta|server_side_prepares/) {
$oldvalue = $oldvalue ? 'on' : 'off';
## Clean up, but lightly so invalid entries fall through for later
if ($newvalue =~ /^[1tT]/ or $newvalue =~ /^on/i) {
$newvalue = 'on';
}
elsif ($newvalue =~ /^[0fF]/ or $newvalue =~ /^off/i) {
$newvalue = 'off';
}
}
## Handle all the non-standard columns
if ($newname =~ /^group/) {
## Track the changes and publish at the end
my @groupchanges;
## Grab the current hash of groups
my $oldgroup = $DB->{$name}{group} || '';
## Keep track of what groups they end up in, so we can remove as needed
my %donegroup;
## Break apart into individual groups
for my $fullgroup (split /\s*,\s*/ => $newvalue) {
my ($group,$role,$extra) = extract_name_and_role($fullgroup);
## Note that we've found this group
$donegroup{$group}++;
## Does this group exist?
if (! exists $DBGROUP->{$group}) {
create_dbgroup($group);
push @groupchanges => qq{Created dbgroup "$group"};
}
## Are we a part of it already?
if ($oldgroup and exists $oldgroup->{$group}) {
## Same role?
my $oldrole = $oldgroup->{$group}{role};
if ($oldrole eq $role) {
$QUIET or print qq{No change: database "$name" already belongs to dbgroup "$group" as $role\n};
}
else {
change_db_role($role,$group,$name);
push @groupchanges => qq{Changed role for database "$name" in dbgroup "$group" from $oldrole to $role};
}
}
else {
## We are not a part of this group yet
add_db_to_group($name, "$group:$role");
push @groupchanges => qq{Added database "$name" to dbgroup "$group" as $role};
}
## Handle any extra modifiers
if (keys %$extra) {
update_dbmap($name, $group, $extra);
my $list = join ',' => map { "$_=$extra->{$_}" } sort keys %$extra;
push @groupchanges => qq{For database "$name" in dbgroup "$group", set $list};
}
} ## end each group specified
## See if we are removing any groups
if ($oldgroup) {
for my $old (sort keys %$oldgroup) {
next if exists $donegroup{$old};
## Remove this database from the group, but do not remove the group itself
remove_db_from_group($name, $old);
push @groupchanges => qq{Removed database "$name" from dbgroup "$old"};
}
}
if (@groupchanges) {
for (@groupchanges) {
chomp;
$QUIET or print "$_\n";
}
confirm_commit();
}
## Go to the next setting
next;
} ## end of 'group' adjustments
## This must exist in our hash
if (! exists $DB->{$name}{$newname}) {
print qq{Cannot change "$newname"\n};
next;
}
## Has this really changed?
if ($oldvalue eq $newvalue) {
print "No change needed for $newname\n";
next;
}
## Add to the queue. Overwrites previous ones
$change{$newname} = [$oldvalue, $newvalue];
} ## end each setting
## If we have any changes, attempt to make them all at once
if (%change) {
my $SQL = 'UPDATE bucardo.db SET ';
$SQL .= join ',' => map { "$_=?" } sort keys %change;
$SQL .= ' WHERE name = ?';
my $sth = $dbh->prepare($SQL);
eval {
$sth->execute((map { $change{$_}[1] } sort keys %change), $name);
};
if ($@) {
$dbh->rollback();
$dbh->disconnect();
print "Sorry, failed to update the bucardo.db table. Error was:\n$@\n";
exit 1;
}
for my $item (sort keys %change) {
my ($old,$new) = @{ $change{$item} };
print "Changed bucardo.db $item from $old to $new\n";
}
confirm_commit();
}
return;
} ## end of update_database
sub list_databases {
## Show information about databases. Queries the bucardo.db table
## Arguments: zero or more
## 1+ Databases to view. Can be "all" and can have wildcards
## Returns: 0 on success, -1 on error
## Example: bucardo list db sale%
## Might be no databases yet
if (! keys %$DB) {
print "No databases have been added yet\n";
return -1;
}
## If not doing all, keep track of which to show
my %matchdb;
for my $term (@nouns) {
## Special case for all: same as no nouns at all, so simply remove them!
if ($term =~ /\ball\b/i) {
undef %matchdb;
undef @nouns;
last;
}
## Check for wildcards
if ($term =~ s/[*%]/.*/) {
for my $name (keys %$DB) {
$matchdb{$name} = 1 if $name =~ /^$term$/;
}
next;
}
## Must be an exact match
for my $name (keys %$DB) {
$matchdb{$name} = 1 if $name eq $term;
}
} ## end each term
## No matches?
if (@nouns and ! keys %matchdb) {
print "No matching databases found\n";
return -1;
}
## We only show the type if they are different from each other
my %typecount;
## Figure out the length of each item for a pretty display
my ($maxdb,$maxtype,$maxstat,$maxlim1,$maxlim2,$showlim) = (1,1,1,1,1,0);
for my $name (sort keys %$DB) {
next if @nouns and ! exists $matchdb{$name};
my $info = $DB->{$name};
$typecount{$info->{dbtype}}++;
$maxdb = length $info->{name} if length $info->{name} > $maxdb;
$maxtype = length $info->{dbtype} if length $info->{dbtype} > $maxtype;
$maxstat = length $info->{status} if length $info->{status} > $maxstat;
}
## Do we show types?
my $showtypes = keys %typecount > 1 ? 1 : 0;
## Now do the actual printing
for my $name (sort keys %$DB) {
next if @nouns and ! exists $matchdb{$name};
my $info = $DB->{$name};
my $type = sprintf 'Type: %-*s ',
$maxtype, $info->{dbtype};
printf 'Database: %-*s %sStatus: %-*s ',
$maxdb, $info->{name},
$showtypes ? $type : '',
$maxstat, $info->{status};
my $showhost = length $info->{dbhost} ? " -h $info->{dbhost}" : '';
my $showport = $info->{dbport} =~ /\d/ ? " -p $info->{dbport}" : '';
my $dbname = length $info->{dbname} ? "-d $info->{dbname}" : '';
if (length $info->{dbconn}) {
$dbname = qq{-d "dbname=$info->{dbname} $info->{dbconn}"};
}
my $dbtype = $info->{dbtype};
if ($dbtype eq 'postgres') {
my $showuser = defined $info->{dbuser} ? "-U $info->{dbuser}" : '';
my $showdb = defined $info->{dbname} ? " -d $info->{dbname}" : '';
my $showservice = (defined $info->{dbservice} and length $info->{dbservice})
? qq{ "service=$info->{dbservice}"} : '';
my $showdsn = (defined $info->{dbdsn} and length $info->{dbdsn})
? qq{ (DSN=$info->{dbdsn})} : '';
print "Conn: psql$showport $showuser$showdb$showhost$showservice$showdsn";
if (! $info->{server_side_prepares}) {
print ' (SSP is off)';
}
if ($info->{makedelta}) {
print ' (makedelta on)';
}
}
if ($dbtype eq 'drizzle') {
$showport = (length $info->{dbport} and $info->{dbport} != 3306)
? " --port $info->{dbport}" : '';
printf 'Conn: drizzle -u %s -D %s%s%s',
$info->{dbuser},
$info->{dbname},
$showhost,
$showport;
}
if ($dbtype eq 'flatfile') {
print "Prefix: $info->{dbname}";
}
if ($dbtype eq 'mongo') {
if (length $info->{dbhost}) {
print "Host: $info->{dbhost}";
}
}
if ($dbtype eq 'mysql' or $dbtype eq 'mariadb') {
$showport = (length $info->{dbport} and $info->{dbport} != 3306)
? " --port $info->{dbport}" : '';
printf 'Conn: mysql -u %s -D %s%s%s',
$info->{dbuser},
$info->{dbname},
$showhost,
$showport;
}
if ($dbtype eq 'firebird') {
printf 'Conn: isql-fb -u %s %s',
$info->{dbuser},
$info->{dbname};
}
if ($dbtype eq 'oracle') {
printf 'Conn: sqlplus %s%s',
$info->{dbuser},
$showhost ? qq{\@$showhost} : '';
}
if ($dbtype eq 'redis') {
my $showindex = (length $info->{dbname} and $info->{dbname} !~ /\D/) ? " -n $info->{dbname}" : '';
printf 'Conn: redis-cli %s%s%s',
$showhost,
$showport,
$showindex;
}
if ($dbtype eq 'sqlite') {
printf 'Conn: sqlite3 %s',
$info->{dbname};
}
print "\n";
if ($VERBOSE) {
## Which dbgroups is this a member of?
if (exists $info->{group}) {
for my $group (sort keys %{ $info->{group} }) {
my $i = $info->{group}{$group};
my $role = $i->{role};
my $pri = $i->{priority};
print " Belongs to dbgroup $group ($role)";
$pri and print " Priority:$pri";
print "\n";
}
}
## Which syncs are using it, and as what role
if (exists $info->{sync}) {
for my $syncname (sort keys %{ $info->{sync} }) {
print " Used in sync $syncname in a role of $info->{sync}{$syncname}{role}\n";
}
}
$VERBOSE >= 2 and show_all_columns($info);
}
}
return 0;
} ## end of list_databases
##
## Database-group-related subroutines: add, remove, update, list
##
sub add_dbgroup {
## Add one or more dbgroups. Inserts to the bucardo.dbgroup table
## May also insert to the bucardo.dbmap table
## Arguments: one plus
## 1. The name of the group we are creating
## 2+ Databases to add to this group, with optional role information attached
## Returns: undef
## Example: bucardo add dbgroup nycservers nyc1:source nyc2:source lax1
## Grab our generic usage message
my $doc_section = 'add/add dbgroup';
my $name = shift @nouns || '';
## Must have a name
usage_exit($doc_section) unless length $name;
## Create the group if it does not exist
if (! exists $DBGROUP->{$name}) {
create_dbgroup($name);
$QUIET or print qq{Created dbgroup "$name"\n};
}
## Add all these databases to the group
for my $dblist (@nouns) {
for my $fulldb (split /\s*,\s*/ => $dblist) {
## Figure out the optional role
my ($db,$role) = extract_name_and_role($fulldb);
## This database must exist!
if (! exists $DB->{$db}) {
print qq{The database "$db" does not exist\n};
exit 1;
}
add_db_to_group($db, "$name:$role");
$QUIET or print qq{Added database "$db" to dbgroup "$name" as $role\n};
}
}
confirm_commit();
exit 0;
} ## end of add_dbgroup
sub remove_dbgroup {
## Remove one or more entries from the bucardo.dbgroup table
## Arguments: one or more
## 1+ Name of a dbgroup
## Returns: undef
## Example: bucardo remove dbgroup sales
my $doc_section = 'remove';
## Must have at least one name
usage_exit($doc_section) unless @nouns;
## Make sure all the groups exist
for my $name (@nouns) {
if (! exists $DBGROUP->{$name}) {
die qq{No such dbgroup: $name\n};
}
}
## Prepare the SQL to delete each group
$SQL = q{DELETE FROM bucardo.dbgroup WHERE name = ?};
$sth = $dbh->prepare($SQL);
for my $name (@nouns) {
## Wrap in an eval so we can handle known exceptions
eval {
$sth->execute($name);
};
if ($@) {
if ($@ =~ /"sync_dbs_fk"/) {
if ($bcargs->{force}) {
$QUIET or warn qq{Dropping all syncs that reference the dbgroup "$name"\n};
$dbh->rollback();
$dbh->do('DELETE FROM bucardo.sync WHERE dbs = ' . $dbh->quote($name));
eval {
$sth->execute($name);
};
goto NEND if ! $@;
}
else {
die qq{Cannot remove dbgroup "$name": it is being used by one or more syncs\n};
}
}
die qq{Could not delete dbgroup "$name"\n$@\n};
}
NEND:
$QUIET or print qq{Removed dbgroup "$name"\n};
}
confirm_commit();
exit 0;
} ## end of remove_dbgroup
sub update_dbgroup {
## Update one or more dbgroups
## This may modify the bucardo.dbgroup and bucardo.dbmap tables
## Arguments: two or more
## 1. Group to be updated
## 2. Databases to be adjusted, or name change request (name=newname)
## Returns: undef
## Example: bucardo update dbgroup sales A:target
my @actions = @_;
my $doc_section = 'update/update dbgroup';
usage_exit($doc_section) unless @actions;
my $name = shift @actions;
## Recursively call ourselves for wildcards and 'all'
exit 0 if ! check_recurse($DBGROUP, $name, @actions);
## Make sure this dbgroup exists!
if (! exists $DBGROUP->{$name}) {
die qq{Could not find a dbgroup named "$name"\nUse 'list dbgroups' to see all available.\n};
}
## From this point on, we have either:
## 1. A rename request
## 2. A database to add/modify
## Track dbs and roles
my %dblist;
## Track if we call confirm_commit or not
my $changes = 0;
for my $action (@actions) {
## New name for this group?
if ($action =~ /name=(.+)/) {
my $newname = $1;
if ($newname !~ /^$re_dbgroupname$/) {
die qq{Invalid dbgroup name "$newname"\n};
}
next if $name eq $newname; ## Duh
$SQL = 'UPDATE bucardo.dbgroup SET name=? WHERE name=?';
$sth = $dbh->prepare($SQL);
$sth->execute($newname, $name);
$QUIET or print qq{Changed dbgroup name from "$name" to "$newname"\n};
$changes++;
next;
}
## Assume the rest is databases to modify
## Default role is always target
my ($db,$role) = extract_name_and_role($action);
$dblist{$db} = $role;
}
## Leave now if no databases to handle
if (! %dblist) {
$changes and confirm_commit();
exit 0;
}
## The old list of databases:
my $oldlist = $DBGROUP->{$name}{db} || {};
## Walk through the old and see if any were changed or removed
for my $db (sort keys %$oldlist) {
if (! exists $dblist{$db}) {
remove_db_from_group($db, $name);
$QUIET or print qq{Removed database "$db" from dbgroup "$name"\n};
$changes++;
next;
}
my $oldrole = $oldlist->{$db}{role};
my $newrole = $dblist{$db};
if ($oldrole ne $newrole) {
change_db_role($newrole, $name, $db);
$QUIET or print qq{Changed role of database "$db" in dbgroup "$name" from $oldrole to $newrole\n};
$changes++;
}
}
## Walk through the new and see if any are truly new
for my $db (sort keys %dblist) {
next if exists $oldlist->{$db};
my $role = $dblist{$db};
add_db_to_group($db, "$name:$role");
$QUIET or print qq{Added database "$db" to dbgroup "$name" as $role\n};
$changes++;
}
confirm_commit() if $changes;
return;
} ## end of update_dbgroup
sub list_dbgroups {
## Show information about all or some subset of the bucardo.dbgroup table
## Arguments: zero or more
## 1+ Groups to view. Can be "all" and can have wildcards
## Returns: 0 on success, -1 on error
## Example: bucardo list dbgroups
## Might be no groups yet
if (! keys %$DBGROUP) {
print "No dbgroups have been added yet\n";
return -1;
}
## If not doing all, keep track of which to show
my %matchdbg;
for my $term (@nouns) {
## Special case for all: same as no nouns at all, so simply remove them!
if ($term =~ /\ball\b/i) {
undef %matchdbg;
undef @nouns;
last;
}
## Check for wildcards
if ($term =~ s/[*%]/.*/) {
for my $name (keys %$DBGROUP) {
$matchdbg{$name} = 1 if $name =~ /$term/;
}
next;
}
## Must be an exact match
for my $name (keys %$DBGROUP) {
$matchdbg{$name} = 1 if $name eq $term;
}
} ## end each term
## No matches?
if (@nouns and ! keys %matchdbg) {
print "No matching dbgroups found\n";
return -1;
}
## Figure out the length of each item for a pretty display
my ($maxlen) = (1);
for my $name (sort keys %$DBGROUP) {
next if @nouns and ! exists $matchdbg{$name};
my $info = $DBGROUP->{$name};
$maxlen = length $info->{name} if length $info->{name} > $maxlen;
}
## Print it
for my $name (sort keys %$DBGROUP) {
next if @nouns and ! exists $matchdbg{$name};
my $info = $DBGROUP->{$name};
## Does it have associated databases?
my $dbs = '';
if (exists $DBGROUP->{$name}{db}) {
$dbs = ' Members:';
for my $dbname (sort keys %{ $DBGROUP->{$name}{db} }) {
my $i = $DBGROUP->{$name}{db}{$dbname};
$dbs .= " $dbname:$i->{role}";
## Only show the priority if <> 0
if ($i->{priority} != 0) {
$dbs .= ":pri=$i->{priority}";
}
}
}
printf "dbgroup: %-*s%s\n",
$maxlen, $name, $dbs;
$VERBOSE >= 2 and show_all_columns($info);
}
return 0;
} ## end of list_dbgroups
##
## Customname-related subroutines: add, exists, remove, list
##
sub add_customname {
## Add an item to the customname table
## Arguments: none, parses nouns for tablename|goatid, syncname, database name
## Returns: never, exits
## Examples:
## bucardo add customname public.foobar foobarz
## bucardo add customname public.foobar foobarz sync=bee
## bucardo add customname public.foobar foobarz db=baz
## bucardo add customname public.foobar foobarz db=baz sync=bee
my $item_name = shift @nouns || '';
my $doc_section = 'add/add customname';
my $newname = shift @nouns || '';
usage_exit($doc_section) unless length $item_name && length $newname;
## Does this number or name exist?
my $goat;
if (exists $GOAT->{by_fullname}{$item_name}) {
$goat = $GOAT->{by_fullname}{$item_name};
}
elsif (exists $GOAT->{by_table}{$item_name}) {
$goat = $GOAT->{by_table}{$item_name};
}
elsif (exists $GOAT->{by_id}{$item_name}) {
$goat = $GOAT->{by_id}{$item_name};
}
else {
print qq{Could not find a matching table for "$item_name"\n};
exit 1;
}
## If this is a ref due to it being an unqualified name, just use the first one
$goat = $goat->[0] if ref $goat eq 'ARRAY';
my ($sname,$tname) = ($goat->{schemaname},$goat->{tablename});
## The new name can have a schema. If it does not, use the "old" one
my $Sname;
my $Tname = $newname;
if ($Tname =~ /(.+)\.(.+)/) {
($Sname,$Tname) = ($1,$2);
}
else {
$Sname = $sname;
}
## If the new name contains an equal sign, treat as an error
usage_exit($doc_section) if $Tname =~ /=/;
## Names cannot be the same
if ($sname eq $Sname and $tname eq $Tname) {
print qq{The new name cannot be the same as the old\n};
exit 1;
}
## Parse the rest of the arguments
my (@sync,@db);
for my $arg (@nouns) {
## Name of a sync
if ($arg =~ /^sync\s*=\s*(.+)/) {
my $sync = $1;
if (! exists $SYNC->{$sync}) {
print qq{No such sync: "$sync"\n};
exit 1;
}
push @sync => $sync;
}
elsif ($arg =~ /^(?:db|database)\s*=\s*(.+)/) {
my $db = $1;
if (! exists $DB->{$db}) {
print qq{No such database: "$db"\n};
exit 1;
}
push @db => $db;
}
else {
usage_exit($doc_section);
}
}
## Loop through and start adding rows to customname
my $goatid = $goat->{id};
$SQL = "INSERT INTO bucardo.customname(goat,newname,db,sync) VALUES ($goatid,?,?,?)";
$sth = $dbh->prepare($SQL);
## We may have multiple syncs or databases, so loop through
my $x = 0;
my @msg;
{
## Setup common message post scripts
my $message = '';
defined $db[$x] and $message .= " (for database $db[$x])";
defined $sync[$x] and $message .= " (for sync $sync[$x])";
## Skip if this exact entry already exists
if (customname_exists($goatid,$newname,$db[$x],$sync[$x])) {
if (!$QUIET) {
printf "Already have an entry for %s to %s%s\n",
$item_name, $newname, $message;
}
next;
}
$sth->execute($newname, $db[$x], $sync[$x]);
push @msg => "Transformed $sname.$tname to $newname$message";
## Always go at least one round
## We go a second time if there is another sync or db waiting
$x++;
redo if defined $db[$x] or defined $sync[$x];
last;
}
if (!$QUIET) {
for (@msg) {
chomp; ## Just in case we forgot above
print "$_\n";
}
}
confirm_commit();
exit 0;
} ## end of add_customname
sub remove_customname {
## Remove one or more entries from the bucardo.customname table
## Arguments: one or more
## 1+ IDs to be deleted
## Returns: undef
## Example: bucardo remove customname 7
## Grab our generic usage message
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
## Make sure each argument is a number
for my $name (@nouns) {
usage_exit($doc_section) if $name !~ /^\d+$/;
}
## We want the per-id hash here
my $cn = $CUSTOMNAME->{id};
## Give a warning if a number does not exist
for my $name (@nouns) {
if (! exists $cn->{$name}) {
$QUIET or warn qq{Customname number $name does not exist\n};
}
}
## Prepare the SQL to delete each customname
$SQL = 'DELETE FROM bucardo.customname WHERE id = ?';
$sth = $dbh->prepare($SQL);
## Go through and delete any that exist
for my $number (@nouns) {
## We've already handled these in the loop above
next if ! exists $cn->{$number};
## Unlike other items, we do not need an eval,
## because it has no cascading dependencies
$sth->execute($number);
my $cc = sprintf '%s => %s%s%s',
$cn->{$number}{tname},
$cn->{$number}{newname},
(length $cn->{$number}{sync} ? " Sync: $cn->{$number}{sync}" : ''),
(length $cn->{$number}{db} ? " Database: $cn->{$number}{db}" : '');
$QUIET or print qq{Removed customcode $number: $cc\n};
}
confirm_commit();
exit 0;
} ## end of remove_customname
sub customname_exists {
## See if an entry already exists in the bucardo.customname table
## Arguments: four
## 1. Goat id
## 2. New name
## 3. Database name (can be null)
## 4. Sync name (can be null)
## Returns: true or false (1 or 0)
my ($id,$newname,$db,$sync) = @_;
## Easy if there are no entries yet!
return 0 if ! keys %$CUSTOMNAME;
my $cn = $CUSTOMNAME->{goat};
## Quick filtering by the goatid
return 0 if ! exists $cn->{$id};
my $matchdb = defined $db ? $db : '';
my $matchsync = defined $sync ? $sync : '';
return exists $cn->{$id}{$matchdb}{$matchsync};
} ## end of customname_exists
sub list_customnames {
## Show information about all or some subset of the bucardo.customname table
## Arguments: zero or more
## 1+ Names to view. Can be "all" and can have wildcards
## Returns: 0 on success, -1 on error
## Example: bucardo list customname
## Grab our generic usage message
my $doc_section = 'list';
## Might be no entries yet
if (! keys %$CUSTOMNAME) {
print "No customnames have been added yet\n";
return -1;
}
my $cn = $CUSTOMNAME->{list};
## If not doing all, keep track of which to show
my $matches = 0;
for my $term (@nouns) {
## Special case for all: same as no nouns at all, so simply remove them!
if ($term =~ /\ball\b/i) {
undef @nouns;
last;
}
## Check for wildcards
if ($term =~ s/[*%]/.*/) {
for my $row (@$cn) {
if ($row->{tname} =~ /$term/) {
$matches++;
$row->{match} = 1;
}
}
next;
}
## Must be an exact match
for my $row (@$cn) {
if ($row->{tname} eq $term) {
$matches++;
$row->{match} = 1;
}
}
} ## end each term
## No matches?
if (@nouns and ! $matches) {
print "No matching customnames found\n";
return -1;
}
## Figure out the length of each item for a pretty display
my ($maxid,$maxname,$maxnew,$maxsync,$maxdb) = (1,1,1,1,1);
for my $row (@$cn) {
next if @nouns and ! exists $row->{match};
$maxid = length $row->{id} if length $row->{id} > $maxid;
$maxname = length $row->{tname} if length $row->{tname} > $maxname;
$maxnew = length $row->{newname} if length $row->{newname} > $maxnew;
$maxsync = length $row->{sync} if length $row->{sync} > $maxsync;
$maxdb = length $row->{db} if length $row->{db} > $maxdb;
}
## Now do the actual printing
## Sort by tablename, then newname, then sync, then db
for my $row (sort {
$a->{tname} cmp $b->{tname}
or
$a->{newname} cmp $b->{newname}
or
$a->{sync} cmp $b->{sync}
or
$a->{db} cmp $b->{db}
} @$cn) {
next if @nouns and ! exists $row->{match};
printf '%-*s Table: %-*s => %-*s',
1+$maxid, "$row->{id}.",
$maxname, $row->{tname},
$maxnew, $row->{newname};
if ($row->{sync}) {
printf ' Sync: %-*s',
$maxsync, $row->{sync};
}
if ($row->{db}) {
printf ' Database: %-*s',
$maxsync, $row->{db};
}
print "\n";
}
return 0;
} ## end of list_customnames
sub find_goat_by_item {
## Finds a goat in the %GOAT hash, using one argument as a search key
## Arguments: name. Can be a goat id or a name, possibly including schema, or wildcards
## nouns. Ref to array of other args; right now only supports "db=###"
## Results: An array of goat objects that match these keys
my $name = shift;
my $lnouns = shift;
my @lnouns = ( defined $lnouns ? @$lnouns : ());
$DEBUG and warn "Finding goats with name $name, noun: " . Dumper(@lnouns);
my @results;
## Handle ID values
if ($name =~ /^\d+$/) {
$DEBUG and warn "$name is an ID value";
push @results, $GOAT->{by_id}{$name};
}
## Handle names, with or without schemas, and with or without wildcards
else {
$DEBUG and warn "$name is a name value";
my @found_keys;
## Find GOAT keys that may include matches
map {
if (exists $GOAT->{$_}{$name}) {
push @found_keys, [ $_, $name ];
}
} qw/by_table by_fullname/;
## Handle wildcards
if (index($name, '*') >= 0 || index($name, '%') >= 0) {
my $reg_name = $name;
## Change to a regexier form
$reg_name =~ s/\./\\./g;
$reg_name =~ s/[*%]/\.\*/g;
$reg_name = "$reg_name" if $reg_name !~ /^[\^\.\%]/;
$reg_name .= '$' if $reg_name !~ /[\$\*]$/;
$DEBUG and warn "There's a wildcard here. This is the regex version: $reg_name";
map {
push @found_keys, [ 'by_fullname', $_ ];
} grep { /$reg_name/ } keys %{$GOAT->{by_fullname}};
}
## The found goat keys point to arrayrefs. Turn all that into a
## one-dimensional array of goats
$DEBUG and warn 'Found these candidate keys: '. Dumper(@found_keys);
map {
for my $b (@{$GOAT->{$_->[0]}{$_->[1]}}) {
push(@results, $b);
}
} @found_keys;
$DEBUG and warn q{Here are the goats we've found, before filtering: } . Dumper(@results);
}
if (@results && defined $results[0] && @lnouns && defined $lnouns[0]) {
my @filters = grep(/^(?:db|database)\s*=/, @lnouns);
if (@filters) {
## The @lnouns array will only contain one db= value, even if the command includes several
my $db_filter = $filters[0];
$DEBUG and warn "Database filter starting value: $db_filter";
$db_filter =~ /^(?:db|database)\s*=\s*(.+)/;
$db_filter = $1;
$DEBUG and warn "Database filter value: $db_filter";
@results = grep {
$DEBUG and warn "Comparing $_->{db} to filter value $db_filter";
$_->{db} eq $db_filter;
} @results;
}
}
$DEBUG and warn 'Here are the filtered results: ' . Dumper(@results);
@results = () if (@results and !defined $results[0]);
return @results;
} ## end of find_goat_by_item
##
## Customcols-related subroutines: add, exists, remove, list
##
sub add_customcols {
## Add an item to the customcols table
## Arguments: none, parses nouns for tablename|goatid, syncname, database name
## Returns: never, exits
## Examples:
## bucardo add customcols public.foobar "select a,b,c"
## bucardo add customcols public.foobar "select a,b,c" db=foo
## bucardo add customcols public.foobar "select a,b,c" db=foo sync=abc
my $item_name = shift @nouns || '';
my $doc_section = 'add';
## Must have a clause as well
my $clause = shift @nouns || '';
usage_exit($doc_section) unless length $item_name && length $clause;
## Does this number or name exist?
my @candidate_goats = find_goat_by_item($item_name);
if (! @candidate_goats) {
print qq{Could not find a matching table for "$item_name"\n};
exit 1;
}
# The code lower in the function is meant to handle multiple matching goats,
# but if we didn't want that, this would bleat when we ran into multiple goats.
# if ($#candidate_goats > 0) {
# print qq{Could not uniquely identify the desired table for "$item_name"\n};
# print qq{Possible choices:\n};
# print "\tdb: $_->{db}\tschema: $_->{schemaname}\ttable: $_->{tablename}\n"
# for @candidate_goats;
# exit 1;
# }
my $goat = $candidate_goats[0];
my ($sname,$tname) = ($goat->{schemaname},$goat->{tablename});
## Make sure the clause looks sane
if ($clause !~ /^\s*SELECT /i) {
warn "\nThe clause must start with SELECT\n";
usage_exit($doc_section);
}
## Parse the rest of the arguments
my (@sync,@db);
for my $arg (@nouns) {
## Name of a sync
if ($arg =~ /^sync\s*=\s*(.+)/) {
my $sync = $1;
if (! exists $SYNC->{$sync}) {
print qq{No such sync: "$sync"\n};
exit 1;
}
push @sync => $sync;
}
elsif ($arg =~ /^(?:db|database)\s*=\s*(.+)/) {
my $db = $1;
if (! exists $DB->{$db}) {
print qq{No such database: "$db"\n};
exit 1;
}
push @db => $db;
}
else {
usage_exit($doc_section);
}
}
## Loop through and start adding rows to customcols
my $goatid = $goat->{id};
$SQL = "INSERT INTO bucardo.customcols(goat,clause,db,sync) VALUES ($goatid,?,?,?)";
$sth = $dbh->prepare($SQL);
## We may have multiple syncs or databases, so loop through
my $x = 0;
my @msg;
{
## Skip if this exact entry already exists
next if customcols_exists($goatid,$clause,$db[$x],$sync[$x]);
$count = $sth->execute($clause, $db[$x], $sync[$x]);
my $message = qq{New columns for $sname.$tname: "$clause"};
defined $db[$x] and $message .= " (for database $db[$x])";
defined $sync[$x] and $message .= " (for sync $sync[$x])";
push @msg => $message;
## Always go at least one round
## We go a second time if there is another sync or db waiting
$x++;
redo if defined $db[$x] or defined $sync[$x];
last;
}
if (!$QUIET) {
for (@msg) {
chomp; ## Just in case we forgot above
print "$_\n";
}
}
confirm_commit();
exit 0;
} ## end of add_customcols
sub remove_customcols {
## Remove one or more entries from the bucardo.customcols table
## Arguments: one or more
## 1+ IDs to be deleted
## Returns: undef
## Example: bucardo remove customcols 7
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
## Make sure each argument is a number
for my $name (@nouns) {
usage_exit($doc_section) if $name !~ /^\d+$/;
}
## We want the per-id hash here
my $cc = $CUSTOMCOLS->{id};
## Give a warning if a number does not exist
for my $name (@nouns) {
if (! exists $cc->{$name}) {
$QUIET or warn qq{Customcols number $name does not exist\n};
}
}
## Prepare the SQL to delete each customcols
$SQL = 'DELETE FROM bucardo.customcols WHERE id = ?';
$sth = $dbh->prepare($SQL);
## Go through and delete any that exist
for my $name (@nouns) {
## We've already handled these in the loop above
next if ! exists $cc->{$name};
## Unlike other items, we do not need an eval,
## because it has no cascading dependencies
$sth->execute($name);
my $cc2 = sprintf '%s => %s%s%s',
$cc->{$name}{tname},
$cc->{$name}{clause},
(length $cc->{$name}{sync} ? " Sync: $cc->{$name}{sync}" : ''),
(length $cc->{$name}{db} ? " Database: $cc->{$name}{db}" : '');
$QUIET or print qq{Removed customcols $name: $cc2\n};
}
confirm_commit();
exit 0;
} ## end of remove_customcols
sub customcols_exists {
## See if an entry already exists in the bucardo.customcols table
## Arguments: four
## 1. Goat id
## 2. Clause
## 3. Database name (can be null)
## 4. Sync name (can be null)
## Returns: true or false (1 or 0)
my ($id,$clause,$db,$sync) = @_;
## Easy if there are no entries yet!
return 0 if ! keys %$CUSTOMCOLS;
my $cc = $CUSTOMCOLS->{goat};
## Quick filtering by the goatid
return 0 if ! exists $cc->{$id};
## And by the clause therein
return 0 if ! exists $cc->{$id}{$clause};
## Is there a match for this db and sync combo?
for my $row (@{ $cc->{$id}{$clause} }) {
if (defined $db) {
next if (! length $row->{db} or $row->{db} ne $db);
}
else {
next if length $row->{db};
}
if (defined $sync) {
next if (! length $row->{sync} or $row->{sync} ne $sync);
}
else {
next if length $row->{sync};
}
## Complete match!
return 1;
}
return 0;
} ## end of customcols_exists
sub list_customcols {
## Show information about all or some subset of the bucardo.customcols table
## Arguments: zero or more
## 1+ Names to view. Can be "all" and can have wildcards
## Returns: 0 on success, -1 on error
## Example: bucardo list customcols
my $doc_section = 'list';
## Might be no entries yet
if (! keys %$CUSTOMCOLS) {
print "No customcols have been added yet\n";
return -1;
}
my $cc = $CUSTOMCOLS->{list};
## If not doing all, keep track of which to show
my $matches = 0;
for my $term (@nouns) {
## Special case for all: same as no nouns at all, so simply remove them!
if ($term =~ /\ball\b/i) {
undef @nouns;
last;
}
## Check for wildcards
if ($term =~ s/[*%]/.*/) {
for my $row (@$cc) {
if ($row->{tname} =~ /$term/) {
$matches++;
$row->{match} = 1;
}
}
next;
}
## Must be an exact match
for my $row (@$cc) {
if ($row->{tname} eq $term) {
$matches++;
$row->{match} = 1;
}
}
} ## end each term
## No matches?
if (@nouns and ! $matches) {
print "No matching customcols found\n";
return -1;
}
## Figure out the length of each item for a pretty display
my ($maxid,$maxname,$maxnew,$maxsync,$maxdb) = (1,1,1,1,1);
for my $row (@$cc) {
next if @nouns and ! exists $row->{match};
$maxid = length $row->{id} if length $row->{id} > $maxid;
$maxname = length $row->{tname} if length $row->{tname} > $maxname;
$maxnew = length $row->{clause} if length $row->{clause} > $maxnew;
$maxsync = length $row->{sync} if length $row->{sync} > $maxsync;
$maxdb = length $row->{db} if length $row->{db} > $maxdb;
}
## Now do the actual printing
## Sort by tablename, then newname, then sync, then db
for my $row (sort {
$a->{tname} cmp $b->{tname}
or
$a->{clause} cmp $b->{clause}
or
$a->{sync} cmp $b->{sync}
or
$a->{db} cmp $b->{db}
} @$cc) {
next if @nouns and ! exists $row->{match};
printf '%-*s Table: %-*s => %-*s',
1+$maxid, "$row->{id}.",
$maxname, $row->{tname},
$maxnew, $row->{clause};
if ($row->{sync}) {
printf ' Sync: %-*s',
$maxsync, $row->{sync};
}
if ($row->{db}) {
printf ' Database: %-*s',
$maxsync, $row->{db};
}
print "\n";
}
return 0;
} ## end of list_customcols
##
## Table-related subroutines: add, remove, update, list
##
sub add_table {
my $reltype = shift;
## Add one or more tables or sequences. Inserts to the bucardo.goat table
## May also update the bucardo.herd and bucardo.herdmap tables
## Arguments: one. Also parses @nouns for table / sequence names
## 1. Type of object to be added: table, or sequence
## Returns: undef
## Example: bucardo add table pgbench_accounts foo% myschema.abc
## Grab our generic usage message
my $doc_section = 'add/add table';
usage_exit($doc_section) unless @nouns;
## Inputs and aliases, database column name, flags, default
my $validcols = q{
db db 0 null
autokick|ping autokick TF null
rebuild_index rebuild_index numeric null
analyze_after_copy analyze_after_copy TF null
makedelta makedelta 0 null
herd|relgroup herd 0 skip
strict_checking strict_checking TF 1
};
my ( $dbcols, $cols, $phs, $vals, $extra ) = process_simple_args({
cols => $validcols,
list => \@nouns,
doc_section => $doc_section,
});
## Loop through all the args and attempt to add the tables
## This returns a hash with the following keys: relations, match, nomatch
my $goatlist = get_goat_ids(args => \@nouns, type => $reltype, dbcols => $dbcols);
## The final output. Store it up all at once for a single QUIET check
my $message = '';
## We will be nice and indicate anything that did not match
if (keys %{ $goatlist->{nomatch} }) {
$message .= "Did not find matches for the following terms:\n";
for (sort keys %{ $goatlist->{nomatch} }) {
$message .= " $_\n";
}
}
## Now we need to output which ones were recently added
if (keys %{ $goatlist->{new} }) {
$message .= "Added the following tables or sequences:\n";
for (sort keys %{ $goatlist->{new} }) {
$message .= " $_\n";
}
}
## If they requested a herd and it does not exist, create it
if (exists $extra->{relgroup}) {
my $herdname = $extra->{relgroup};
if (! exists $HERD->{$herdname}) {
$SQL = 'INSERT INTO bucardo.herd(name) VALUES(?)';
$sth = $dbh->prepare($SQL);
$sth->execute($herdname);
$message .= qq{Created the relgroup named "$herdname"\n};
}
## Now load all of these tables into this herd
$SQL = 'INSERT INTO bucardo.herdmap (herd,priority,goat) VALUES (?,?,'
. qq{ (SELECT id FROM goat WHERE schemaname||'.'||tablename=? AND db=? AND reltype='$reltype'))};
$sth = $dbh->prepare($SQL);
## Which tables were already in the herd, and which were just added
my (@oldnames,@newnames);
for my $name (sort keys %{ $goatlist->{relations} }) {
## Is it already part of this herd?
if (exists $HERD->{$herdname}{goat}{$name} and
$HERD->{$herdname}{goat}{$name}{reltype} eq $reltype) {
push @oldnames => $name;
next;
}
my $db = $goatlist->{relations}{$name}{goat}[0]{db};
my $pri = 0;
$count = $sth->execute($herdname,$pri,$name, $db);
push @newnames => $name;
}
if (@oldnames) {
$message .= qq{The following tables or sequences were already in the relgroup "$herdname":\n};
for (@oldnames) {
$message .= " $_\n";
}
}
if (@newnames) {
$message .= qq{The following tables or sequences are now part of the relgroup "$herdname":\n};
for (sort numbered_relations @newnames) {
$message .= " $_\n";
}
}
} ## end if herd
if (!$QUIET) {
print $message;
}
confirm_commit();
exit 0;
} ## end of add_table
sub remove_relation {
my $reltype = shift;
my $arg = shift || '';
my $doc_section = 'remove';
if (!@nouns and $arg ne 'all') {
usage_exit($doc_section);
}
my $db_filter;
for my $name ( @nouns ) {
next unless $name =~ /^db=(.*)/;
$db_filter = $1;
}
my @removed;
if ($arg eq 'all') {
if (! $bcargs->{batch}) {
print "Are you sure you want to remove all ${reltype}s? ";
exit if !~ /Y/i;
}
$SQL = q{DELETE FROM bucardo.goat WHERE id = ?};
$sth = $dbh->prepare($SQL);
for my $tid ( sort { $a <=> $b } keys %{$GOAT->{by_id}}) {
my $t = $GOAT->{by_id}{$tid};
next if $t->{reltype} ne $reltype;
$count = $sth->execute($tid);
if (1 == $count) {
push @removed => "$t->{schemaname}.$t->{tablename}";
}
}
}
else {
## Prepare our SQL
$SQL = q{DELETE FROM bucardo.goat WHERE reltype = ? AND schemaname||'.'||tablename = ?};
$SQL .= ' AND db = ?' if $db_filter;
$sth = $dbh->prepare($SQL);
## Bucardo won't fully support a table name that starts with "db=". Darn.
for my $name (grep { ! /^db=/ } @nouns) {
if ($name =~ /^\w[\w\d]*\.\w[\w\d]*$/) {
if (! exists $GOAT->{by_fullname}{$name}) {
print qq{No such $reltype: $name\n};
next;
}
eval {
if ($db_filter) {
$sth->execute($reltype, $name, $db_filter);
}
else {
$sth->execute($reltype, $name);
}
};
if ($@) {
die qq{Could not delete $reltype "$name"\n$@\n};
}
push @removed, $name;
}
else {
die qq{Please use the full schema.$reltype name\n};
}
}
}
if (@removed) {
print "Removed the following ${reltype}s:\n";
for my $name (sort numbered_relations @removed) {
print qq{ $name} . ($db_filter ? " (DB: $db_filter)" : '') . "\n";
}
confirm_commit();
}
else {
print "Nothing found to remove\n";
}
exit 0;
} ## end of remove_relation
sub update_table {
## Update one or more tables
## This may modify the bucardo.goat and bucardo.herdmap tables
## Arguments: two or more
## 1. Table to be updated
## 2+. Items to be adjusted (name=value)
## Returns: undef
## Example: bucardo update table quad ping=false
my @actions = @_;
my $doc_section = 'update/update table';
usage_exit($doc_section) unless @actions;
my $name = shift @actions;
## Recursively call ourselves for wildcards and 'all'
exit 0 if ! check_recurse($GOAT, $name, @actions);
## Make sure this table exists!
my @tables = find_goat_by_item($name, \@nouns);
if (!@tables) {
die qq{Didn't find any matching tables\n};
}
## If this is an array, then see how many matches we have
if ($#tables > 0) {
die qq{More than one matching table: please use a schema\n};
}
my $table = $tables[0];
## Store the id so we work with that alone whenever possible
my $id = $table->{id};
## Everything is a name=value setting after this point, except stuff that
## matches /^db=/
## We will ignore and allow noise word "set"
for my $arg (grep { ! /^db=/ } @actions) {
next if $arg =~ /set/i;
next if $arg =~ /\w+=\w+/o;
usage_exit($doc_section);
}
## Change the arguments into a hash
my $args = process_args(join ' ' => ( grep { ! /^db=/ } @actions));
## Track what changes we made
my %change;
## Walk through and handle each argument pair
for my $setting (sort keys %$args) {
next if $setting eq 'extraargs';
## Change the name to a more standard form, to better figure out what they really mean
## This also excludes all non-alpha characters
my $newname = transform_name($setting);
## Exclude ones that cannot / should not be changed (e.g. cdate)
if (exists $column_no_change{$newname}) {
print "Sorry, the value of $setting cannot be changed\n";
exit 1;
}
## Standardize the values as well
my $value = $args->{$setting};
my $newvalue = transform_value($value);
## Handle all the non-standard columns
if (lc $newname eq 'herd' || lc $newname eq 'relgroup') {
## Track the changes and publish at the end
my @herdchanges;
## Grab the current hash of herds
my $oldherd = $table->{herd} || '';
## Keep track of what groups they end up in, so we can remove as needed
my %doneherd;
## Break apart into individual herds
for my $herd (split /\s*,\s*/ => $newvalue) {
## Note that we've found this herd
$doneherd{$herd}++;
## Does this herd exist?
if (! exists $HERD->{$herd}) {
create_herd($herd);
push @herdchanges => qq{Created relgroup "$herd"};
}
## Are we a part of it already?
if ($oldherd and exists $oldherd->{$herd}) {
$QUIET or print qq{No change: table "$name" already belongs to relgroup "$herd"\n};
}
else {
## We are not a part of this herd yet
add_goat_to_herd($herd, $id);
push @herdchanges => qq{Added table "$name" to relgroup "$herd"};
}
} ## end each herd specified
## See if we are removing any herds
if ($oldherd) {
for my $old (sort keys %$oldherd) {
next if exists $doneherd{$old};
## We do not want to remove herds here, but maybe in the future
## we can allow a syntax that does
next;
remove_table_from_herd($name, $old);
push @herdchanges => qq{Removed table "$name" from relgroup "$old"};
}
}
if (@herdchanges) {
for (@herdchanges) {
chomp;
$QUIET or print "$_\n";
}
confirm_commit();
}
## Go to the next setting
next;
} ## end of 'herd' adjustments
## This must exist in our hash
## We assume it is the first entry for now
## Someday be more intelligent about walking and adjusting all matches
if (! exists $table->{$newname}) {
print qq{Cannot change "$newname"\n};
next;
}
my $oldvalue = $table->{$newname};
## May be undef!
$oldvalue = 'NULL' if ! defined $oldvalue;
## Has this really changed?
if ($oldvalue eq $newvalue) {
print "No change needed for $newname\n";
next;
}
## Add to the queue. Overwrites previous ones
$change{$newname} = [$oldvalue, $newvalue];
} ## end each setting
## If we have any changes, attempt to make them all at once
if (%change) {
my $SQL = 'UPDATE bucardo.goat SET ';
$SQL .= join ',' => map { "$_=?" } sort keys %change;
$SQL .= ' WHERE id = ?';
my $sth = $dbh->prepare($SQL);
eval {
$sth->execute((map { $change{$_}[1] } sort keys %change), $id);
};
if ($@) {
$dbh->rollback();
$dbh->disconnect();
print "Sorry, failed to update the relation. Error was:\n$@\n";
exit 1;
}
for my $item (sort keys %change) {
my ($old,$new) = @{ $change{$item} };
print "Changed relation $item from $old to $new\n";
}
confirm_commit();
}
return;
} ## end of update_table
sub list_tables {
## Show information about all or some tables in the 'goat' table
## Arguments: none (reads nouns for a list of tables)
## Returns: 0 on success, -1 on error
## Example: bucardo list tables
my $doc_section = 'list';
## Might be no tables yet
if (! keys %$TABLE) {
print "No tables have been added yet\n";
return -1;
}
## If not doing all, keep track of which to show
my %matchtable;
my @filters = grep { /^db=/ } @nouns;
for my $term (grep { ! /^db=/ } @nouns) {
## Special case for all: same as no nouns at all, so simply remove them!
if ($term =~ /\ball\b/i) {
undef %matchtable;
undef @nouns;
last;
}
map { $matchtable{$_->{id}} = 1; } find_goat_by_item($term, \@filters);
} ## end each term
## No matches?
if (@nouns and ! keys %matchtable) {
print "No matching tables found\n";
return -1;
}
## Figure out the length of each item for a pretty display
my ($maxid,$maxname,$maxdb,$maxpk) = (1,1,1,1);
for my $row (values %$TABLE) {
my $id = $row->{id};
next if @nouns and ! exists $matchtable{$id};
$maxid = length $id if length $id > $maxid;
my $name = "$row->{schemaname}.$row->{tablename}";
$maxname = length $name if length $name > $maxname;
$maxdb = length $row->{db} if length $row->{db} > $maxdb;
$row->{ppk} = $row->{pkey} ? "$row->{pkey} ($row->{pkeytype})" : 'none';
$maxpk = length $row->{ppk} if length $row->{ppk} > $maxpk;
}
## Now do the actual printing
## Sort by schemaname then tablename
for my $row (sort numbered_relations values %$TABLE) {
next if @nouns and ! exists $matchtable{$row->{id}};
printf '%-*s Table: %-*s DB: %-*s PK: %-*s',
1+$maxid, "$row->{id}.",
$maxname, "$row->{schemaname}.$row->{tablename}",
$maxdb, $row->{db},
$maxpk, $row->{ppk};
if ($row->{sync}) {
printf ' Syncs: ';
print join ',' => sort keys %{ $row->{sync} };
}
if (defined $row->{autokick}) {
printf ' autokick:%s', $row->{autokick} ? 'true' : 'false';
}
if ($row->{rebuild_index}) {
print ' rebuild_index:true';
}
if ($row->{makedelta}) {
print " (makedelta:$row->{makedelta})";
}
print "\n";
$VERBOSE >= 2 and show_all_columns($row);
}
return 0;
} ## end of list_tables
##
## Herd-related subroutines: add, remove, update, list
##
sub add_herd {
## Add a herd aka relgroup. Inserts to the bucardo.herd table
## May also insert to the bucardo.herdmap and bucardo.goat tables
## Arguments: one or more
## 1. Name of the herd
## 2+ Names of tables or sequences to add. Can have wildcards
## Returns: undef
## Example: bucardo add herd foobar tab1 tab2
my $doc_section = 'add/add relgroup';
my $herdname = shift @nouns || '';
## Must have a name
usage_exit($doc_section) unless length $herdname;
## Create the herd if it does not exist
if (exists $HERD->{$herdname}) {
print qq{Relgroup "$herdname" already exists\n};
}
else {
create_herd($herdname);
$QUIET or print qq{Created relgroup "$herdname"\n};
}
## Everything else is tables or sequences to add to this herd
## How many arguments were we given?
my $nouncount = @nouns;
## No sense going on if no nouns!
if (! $nouncount) {
confirm_commit();
exit 0;
}
## Get the list of all requested tables, adding as needed
my $goatlist = get_goat_ids(args => \@nouns, noherd => $herdname);
## The final output. Store it up all at once for a single QUIET check
my $message = '';
## We will be nice and indicate anything that did not match
if (keys %{ $goatlist->{nomatch} }) {
$message .= "Did not find matches for the following terms:\n";
for (sort keys %{ $goatlist->{nomatch} }) {
$message .= " $_\n";
}
}
## Now we need to output which ones were recently added
if (keys %{ $goatlist->{new} }) {
$message .= "Added the following tables or sequences:\n";
for (sort keys %{ $goatlist->{new} }) {
$message .= " $_ (DB: $goatlist->{relations}{$_}{goat}[0]{db})\n";
}
}
## Now load all of these tables into this herd
$SQL = 'INSERT INTO bucardo.herdmap (herd,priority,goat) VALUES (?,?,'
. q{ (SELECT id FROM goat WHERE schemaname||'.'||tablename=? AND db=?))};
$sth = $dbh->prepare($SQL);
my (@oldnames, @newnames);
for my $name (sort keys %{ $goatlist->{relations} }) {
## Is it already part of this herd?
if (exists $HERD->{goat}{$name}) {
push @oldnames => $name;
next;
}
my @a;
eval {
@a = @{$goatlist->{relations}{$name}{goat}};
};
my $doneit;
for my $tmpgoat (@a) {
next if exists $doneit->{$tmpgoat->{id}};
my $db = $tmpgoat->{db};
my $pri = 0;
$count = $sth->execute($herdname,$pri,$name,$db);
push @newnames => $name;
$doneit->{$tmpgoat->{id}}++;
}
}
if (@oldnames) {
$message .= qq{The following tables or sequences were already in the relgroup "$herdname":\n};
for (@oldnames) {
$message .= " $_\n";
}
}
if (@newnames) {
$message .= qq{The following tables or sequences are now part of the relgroup "$herdname":\n};
for (@newnames) {
$message .= " $_\n";
}
}
if (!$QUIET) {
print $message;
}
confirm_commit();
exit 0;
} ## end of add_herd
sub remove_herd {
## Usage: remove herd herdname [herd2 herd3 ...]
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
my $herd = $global{herd};
for my $name (@nouns) {
if (! exists $herd->{$name}) {
die qq{No such relgroup: $name\n};
}
}
$SQL = 'DELETE FROM bucardo.herd WHERE name = ?';
$sth = $dbh->prepare($SQL);
for my $name (@nouns) {
eval {
$sth->execute($name);
};
if ($@) {
if ($@ =~ /"sync_source_herd_fk"/) {
die qq{Cannot delete relgroup "$name": must remove all syncs that reference it first\n};
}
die qq{Could not delete relgroup "$name"\n$@\n};
}
}
for my $name (@nouns) {
print qq{Removed relgroup "$name"\n};
}
$dbh->commit();
exit 0;
} ## end of remove_herd
sub add_goat_to_herd {
die "Adding to a relgroup not implemented yet\n";
}
sub list_herds {
## Show information about all or some subset of the 'herd' table
## Arguments: none, parses nouns for herd names
## Returns: 0 on success, -1 on error
my $doc_section = 'list';
## Any nouns are filters against the whole list
my $clause = generate_clause({col => 'name', items => \@nouns});
my $WHERE = $clause ? "WHERE $clause" : '';
$SQL = "SELECT * FROM bucardo.herd $WHERE ORDER BY name";
$sth = $dbh->prepare($SQL);
$count = $sth->execute();
if ($count < 1) {
$sth->finish();
printf "There are no%s relgroups.\n",
$WHERE ? ' matching' : '';
return -1;
}
$info = $sth->fetchall_arrayref({});
## Get sizing information
my $maxlen = 1;
for my $row (@$info) {
$maxlen = length $row->{name} if length $row->{name} > $maxlen;
}
for my $row (@$info) {
my $name = $row->{name};
my $h = $HERD->{$name};
printf 'Relgroup: %-*s ',
$maxlen, $name;
printf ' DB: %s ', $h->{db} if $h->{db};
## Got goats?
if (exists $h->{goat}) {
print ' Members: ';
print join ', ' => sort {
$h->{goat}{$b}{priority} <=> $h->{goat}{$a}{priority}
or $a cmp $b
} keys %{ $h->{goat} };
}
## Got syncs?
if (exists $h->{sync}) {
print "\n Used in syncs: ";
print join ', ' => sort keys %{$h->{sync}};
}
print "\n";
$VERBOSE >= 2 and show_all_columns($row);
}
return 0;
} ## end of list_herds
##
## Sync-related subroutines: add, remove, update, list
##
sub add_sync {
## Create a new sync by adding an entry to the bucardo.sync table
## Will add tables as needed to the bucardo.goat table
## Will create implicit relgroups as needed
## May modify the goat, herd, and herdmap tables
## Arguments: none (uses nouns)
## Returns: never, exits
my $sync_name = shift @nouns || '';
## If the sync name does not exist or is empty, show a help screen
my $doc_section = 'add/add sync';
usage_exit($doc_section) if ! length $sync_name;
## If this named sync already exists, throw an exception
if (exists $SYNC->{$sync_name}) {
die qq{A sync with the name "$sync_name" already exists\n};
}
## Store a list of messages we can output once we have no errors
my @message;
## Inputs and aliases, database column name, flags, default
my $validcols = qq{
name name 0 $sync_name
relgroup|herd relgroup 0 null
stayalive stayalive TF null
kidsalive kidsalive TF null
autokick|ping autokick TF null
checktime checktime interval null
strict_checking strict_checking TF null
status status =active|inactive null
priority priority numeric null
analyze_after_copy analyze_after_copy TF null
overdue overdue interval null
expired expired interval null
track_rates track_rates TF null
onetimecopy onetimecopy =0|1|2 null
lifetime lifetime interval null
maxkicks maxkicks numeric null
isolation_level|txnmode isolation_level 0 null
rebuild_index|rebuildindex rebuild_index numeric null
dbgroup dbgroup 0 null
conflict_strategy|standard_conflict|conflict conflict_strategy 0 null
relation|relations|table|tables tables 0 null
db|databases|database|databases|dbs dbs 0 null
};
my $morph = [
## Fullcopy syncs get some of their defaults overriden
## The controllers and kids never start automatically,
## and autokick is never on
{
field => 'synctype',
value => 'fullcopy',
new_defaults => 'autokick|F stayalive|F kidsalive|F',
},
## We need isolation level to be dash free for SQL
{
field => 'isolation_level',
dash_to_white => 1,
}
];
## Parse all of our arguments, and convert them per rules above
## We don't use cols and phs and vals in this particular sub!
## Others should be modified someday to also avoid them
my ($dbcols) = process_simple_args({
cols => $validcols,
list => \@nouns,
doc_section => $doc_section,
morph => $morph,
});
## We must know what to replicate: need a relgroup or a list of tables
if (! exists $dbcols->{relgroup} and ! exists $dbcols->{tables}) {
die "Must specify a relgroup (or a list of tables) for this sync\n";
}
## We must know where to replicate: need a dbgroup or a list of databases
if (! exists $dbcols->{dbgroup} and ! exists $dbcols->{dbs}) {
die "Need to specify which dbgroup (or list of databases) for this sync\n";
}
#### RELGROUP
## Determine what relgroup to use
## If one is given, use that; else create a new one
my $relgroup_name;
if (exists $dbcols->{relgroup}) {
## Simple case where they give us the exact relgroup
if (exists $HERD->{ $dbcols->{relgroup} }) {
## We cannot have both an existing relgroup and a list of tables
if (exists $dbcols->{tables}) {
die "Cannot specify an existing relgroup and a list of tables\n";
}
$relgroup_name = $dbcols->{relgroup};
}
## If the relgroup has commas, we treat it as a list of tables
## Otherwise, we create a new relgroup
elsif ($dbcols->{relgroup} !~ /,/) {
$relgroup_name = create_herd( $dbcols->{relgroup}, 'noreload' );
}
}
## DBGROUP
## Determine which dbgroup to use
## We create a unique name as needed later on
my $dbgroup_name;
if (exists $dbcols->{dbgroup}) {
## If this dbgroup already exists, we are done
if (exists $DBGROUP->{ $dbcols->{dbgroup} }) {
## We cannot have both an existing dbgroup and a list of databases
if (exists $dbcols->{dbs}) {
die "Cannot specify an existing dbgroup and a list of databases\n";
}
$dbgroup_name = $dbcols->{dbgroup};
}
## If the dbgroup has commas, we treat it as a list of databases
## Otherwise, we create a new dbgroup
elsif ($dbcols->{dbgroup} !~ /,/) {
$dbgroup_name = create_dbgroup( $dbcols->{dbgroup}, 'noreload' );
}
}
## Before we potentially create a unique dbgroup name
## we need to process all of our databases, to see
## if this combo matches an existing dbgroup
#### DB
## Parse the list of databases to use
## Databases can come from both dbs and dbgroup - the latter only if it has commas
my @dblist;
if (exists $dbcols->{dbs}) {
@dblist = split /\s*,\s*/ => $dbcols->{dbs};
}
if (exists $dbcols->{dbgroup} and $dbcols->{dbgroup} =~ /,/) {
push @dblist => split /\s*,\s*/ => $dbcols->{dbgroup};
}
## If this is a new dbgroup, databases must be given
if (!@dblist and defined $dbgroup_name and ! exists $DBGROUP->{ $dbgroup_name }) {
die qq{Must provide a list of databases to go into the new dbgroup\n};
}
my $dbtype = ''; ## the current database type (e.g. source, target)
my %db; ## used to build matching list below
my %rolecount; ## Keep track of types for later logic
my $db_for_lookups; ## Which database to search for new tables
for my $db (@dblist) {
## Set the default type of database: first is always source
$dbtype = $dbtype eq '' ? 'source' : 'target';
## Extract the type if it has one
if ($db =~ s/[=:](.+)//) {
$dbtype = $1;
}
## If this database is not known to us, throw an exception
if (! exists $DB->{$db}) {
## This may be a dbgroup: we allow this if it is the only entry!
if (exists $DBGROUP->{ $db } and ! defined $dblist[1]) {
$dbgroup_name = $db;
undef @dblist;
last;
}
die qq{Unknown database "$db"\n};
}
## Standardize and check the types
$dbtype = 'source'
if $dbtype =~ /^s/i or $dbtype =~ /^mas/i or $dbtype =~ /^pri/;
$dbtype = 'target'
if $dbtype =~ /^t/i or $dbtype =~ /^rep/i;
$dbtype = 'fullcopy'
if $dbtype =~ /^f/i;
if ($dbtype !~ /^(?:source|target|fullcopy)$/) {
die "Invalid database type: must be source, target, or fullcopy (not $dbtype)\n";
}
$db{$db} = $dbtype;
$rolecount{$dbtype}++;
$db_for_lookups = $db if $dbtype eq 'source';
}
## If we were given dbgroup only, we still need to populate rolecount
if (! @dblist) {
for my $d (values %{ $DBGROUP->{ $dbgroup_name }{db} }) {
$rolecount{$d->{role}}++;
}
}
## Do any existing groups match this list exactly?
## We only care about this if they don't have an explicit dbgroup set
if (! defined $dbgroup_name) {
my $newlist = join ',' => map { "$_=".$db{$_} } sort keys %db;
for my $gname (sort keys %$DBGROUP) {
my $innerjoin = join ',' =>
map { "$_=".$DBGROUP->{$gname}{db}{$_}{role} }
sort keys %{$DBGROUP->{$gname}{db}};
if ($innerjoin eq $newlist) {
push @message => qq{Using existing dbgroup "$gname"};
$dbgroup_name = $gname;
last;
}
}
}
## If we still don't have a dbgroup, create one based on the sync name
if (! defined $dbgroup_name) {
## We will use the name of the sync if free
## Otherwise, keep adding numbers to it until we get a free name
my $newname = $sync_name;
for my $x (2..10_000) {
last if ! exists $DBGROUP->{$newname};
$newname = "${sync_name}_$x";
}
$dbgroup_name = create_dbgroup( $newname, 'noreload' );
}
## Give a courtesy message if we created a new dbgroup
## Also associate our dbs with this new group
if (! exists $DBGROUP->{ $dbgroup_name }) {
push @message => qq{Created a new dbgroup named "$dbgroup_name"\n};
$SQL = 'INSERT INTO bucardo.dbmap(dbgroup,db,role) VALUES (?,?,?)';
$sth = $dbh->prepare($SQL);
for my $db (sort keys %db) {
$count = $sth->execute($dbgroup_name, $db, $db{$db});
if (1 != $count) {
die qq{Unable to add database "$db" to dbgroup "$dbgroup_name"\n};
}
}
}
## Make sure we only use what the bucardo.sync table needs: dbs
delete $dbcols->{dbgroup};
$dbcols->{dbs} = $dbgroup_name; ## Someday, rename this column!
## TABLES
## Determine the tables to use
## Always check the first found source database
## We can get a list of tables via the relgroup argument,
## or from a tables argument. Handle both ways.
my @tables;
if (exists $dbcols->{relgroup} and $dbcols->{relgroup} =~ /,/) {
@tables = split /\s*,\s*/ => $dbcols->{relgroup};
}
if (exists $dbcols->{tables}) {
for my $table (split /\s*,\s*/ => $dbcols->{tables}) {
push @tables => $table;
}
}
## Keep track of what we have already done
my %table;
for my $table (@tables) {
## Skip if we have seen this already
next if exists $table{$table};
## If this table already exists, we are done
if (exists $GOAT->{by_fullname}{$table}) {
$table{$table} = $GOAT->{by_fullname}{$table}->[0];
next;
}
my $result = get_goat_ids(args => [$table], dbcols => { db => $db_for_lookups} );
my $found = keys %{ $result->{match} };
for my $name (sort keys %{ $result->{new} }) {
push @message => qq{ Added table "$name"};
}
## If a specific table is not found, throw an exception
if (!$found and $table !~ /^all/) {
die qq{Could not find a relation named "$table"\n};
}
## Save each table's information for use below
for my $tname (sort keys %{ $result->{relations} }) {
$table{$tname} ||= $result->{relations}{$tname}{goat}[0];
}
}
## If we don't have a relgroup already, see if our list matches an existing one
if (! defined $relgroup_name and keys %table) {
my $newlist = join ',' =>
map { "$table{$_}{schemaname}.$table{$_}{tablename}"}
sort { "$table{$a}->{schemaname}.$table{$a}->{tablename}"
cmp "$table{$b}->{schemaname}.$table{$b}->{tablename}"}
keys %table;
for my $rname (sort keys %$RELGROUP) {
my $innerjoin = join ',' => sort keys %{$RELGROUP->{$rname}{goat}};
if ($innerjoin eq $newlist) {
push @message => qq{Using existing relgroup "$rname"};
$relgroup_name = $rname;
last;
}
}
}
## Now we can set a default relgroup based on the sync name if needed
## If we still don't have a relgroup, create one based on the sync name
if (! defined $relgroup_name) {
## We will use the name of the sync if free
## Otherwise, keep adding numbers to it until we get a free name
my $newname = $sync_name;
for my $x (2..10_000) {
last if ! exists $HERD->{$newname};
$newname = "${sync_name}_$x";
}
$relgroup_name = create_herd( $newname, 'noreload' );
}
## Give a courtesy message if we created a new relgroup
## Also associate our tables with this new group
if (! exists $HERD->{ $relgroup_name }) {
unshift @message => qq{Created a new relgroup named "$relgroup_name"\n};
$SQL = 'INSERT INTO bucardo.herdmap(herd,goat) VALUES (?,?)';
$sth = $dbh->prepare($SQL);
for my $t (sort keys %table) {
$count = $sth->execute($relgroup_name, $table{$t}{id});
if (1 != $count) {
die qq{Unable to add table "$t" to relgroup "$relgroup_name"\n};
}
}
}
## Make sure we use relgroup but not tables for the SQL below
delete $dbcols->{tables};
delete $dbcols->{relgroup};
$dbcols->{herd} = $relgroup_name;
## If this is a pure fullcopy sync, we want to turn stayalive and kidsalive off
if ($rolecount{'source'} == 1
and $rolecount{'fullcopy'}
and ! $rolecount{'target'}) {
$dbcols->{stayalive} = 0;
$dbcols->{kidsalive} = 0;
}
## Allow some alternate way to say things
my $cs = 'conflict_strategy';
if (exists $dbcols->{$cs}) {
$dbcols->{$cs} = 'bucardo_latest'
if $dbcols->{$cs} eq 'default' or $dbcols->{$cs} eq 'latest';
}
## Attempt to insert this into the database
my $columns = join ',' => keys %$dbcols;
my $qs = '?,' x keys %$dbcols;
chop $qs;
$SQL = "INSERT INTO bucardo.sync ($columns) VALUES ($qs)";
$DEBUG and warn "SQL: $SQL\n";
$sth = $dbh->prepare($SQL);
$DEBUG and warn Dumper values %$dbcols;
eval {
$count = $sth->execute(values %$dbcols);
};
if ($@) {
die "Failed to add sync: $@\n";
}
$QUIET or print qq{Added sync "$sync_name"\n};
## Now we can output our success messages if any
for my $msg (@message) {
chomp $msg;
$QUIET or print "$msg\n";
}
confirm_commit();
exit 0;
} ## end of add_sync
sub remove_sync {
## Usage: remove sync name [name2 name3 ...]
## Arguments: none (uses nouns)
## Returns: never, exits
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
## Make sure all named syncs exist
my $s = $global{sync};
for my $name (@nouns) {
if (! exists $s->{$name}) {
die qq{No such sync: $name\n};
}
}
## Make sure none of the syncs are currently running
## XXX Is there anything we can do to check that the sync is active?
$SQL = 'DELETE FROM bucardo.sync WHERE name = ?';
$sth = $dbh->prepare($SQL);
for my $name (@nouns) {
eval {
$sth->execute($name);
};
if ($@) {
if ($@ =~ /"goat_db_fk"/) {
die qq{Cannot delete sync "$name": must remove all tables that reference it first\n};
}
die qq{Could not delete sync "$name"\n$@\n};
}
}
for my $name (@nouns) {
print qq{Removed sync "$name"\n};
print "Note: table triggers (if any) are not automatically removed!\n";
}
$dbh->commit();
exit 0;
} ## end of remove_sync
sub update_sync {
## Update one or more syncs
## Arguments: none (reads nouns for a list of syncs)
## Returns: never, exits
my @actions = @_;
my $doc_section = 'update/update sync';
usage_exit($doc_section) unless @actions;
my $name = shift @actions;
## Recursively call ourselves for wildcards and 'all'
exit 0 if ! check_recurse($SYNC, $name, @actions);
## Make sure this sync exists!
if (! exists $SYNC->{$name}) {
die qq{Could not find a sync named "$name"\nUse 'list syncs' to see all available.\n};
}
my $changes = 0;
## Current information about this sync, including column names
my $syncinfo;
my %aliases = (
standard_conflict => 'conflict_strategy',
conflict => 'conflict_strategy',
ping => 'autokick',
relgroup => 'herd',
);
for my $action (@actions) {
## Skip noise words
next if $action =~ 'set';
## Allow for some simple shortcuts
if ($action =~/^inactiv/i) {
$action = 'status=inactive';
}
elsif ($action =~ /^activ/i) {
$action = 'status=active';
}
## Look for a standard foo=bar or foo:bar format
if ($action =~ /(.+?)\s*[=:]\s*(.+)/) {
my ($setting,$value) = (lc $1,$2);
## No funny characters please, just boring column names
$setting =~ /^[a-z_]+$/ or die "Invalid setting: $setting\n";
$setting = $aliases{$setting} || $setting;
## If we have not already, grab the current information for this sync
## We also use this to get the list of valid column names to modify
if (! defined $syncinfo) {
$SQL = 'SELECT * FROM sync WHERE name = ?';
$sth = $dbh->prepare($SQL);
$count = $sth->execute($name);
## Check count
$syncinfo = $sth->fetchall_arrayref({})->[0];
for my $col (qw/ cdate /) {
delete $syncinfo->{$col};
}
}
## Is this a valid column?
if (! exists $syncinfo->{$setting}) {
die "Invalid setting: $setting\n";
}
## Do any magic we need for specific items
if ($setting eq 'isolation_level') {
$value =~ s/_/ /g;
}
elsif ($setting eq 'conflict_strategy') {
## Allow some alternative names
$value = 'bucardo_latest' if $value eq 'default' or $value eq 'latest';
$value = 'bucardo_latest_all_tables' if $value eq 'latest_all';
## If the name does not start with bucardo, it must be a list of databases
if ($value !~ /^bucardo_/) {
my $dbs = $SYNC->{$name}{dblist};
$value =~ s/[,\s]+/ /g;
for my $dbname (split / / => $value) {
if (! exists $dbs->{$dbname}) {
die qq{conflict_strategy should contain a list of databases, but "$dbname" is not a db for this sync!\n};
}
}
}
$QUIET or print qq{Set conflict strategy to '$value'\n};
}
## Try setting it
$SQL = "UPDATE sync SET $setting=? WHERE name = ?";
$sth = $dbh->prepare($SQL);
$sth->execute($value,$name);
$changes++;
next;
}
warn "\nUnknown action: $action\n";
usage_exit($doc_section);
}
confirm_commit() if $changes;
return;
} ## end of update_sync
sub list_syncs {
## Show information about all or some subset of the 'sync' table
## Arguments: none (reads nouns for a list of syncs)
## Returns: 0 on success, -1 on error
my $doc_section = 'list';
my $syncs = $global{sync};
## Do we have at least one name specified (if not, show all)
my $namefilter = 0;
for my $term (@nouns) {
## Filter out by status: only show active or inactive syncs
if ($term =~ /^(active|inactive)$/i) {
my $stat = lc $1;
for my $name (keys %$syncs) {
delete $syncs->{$name} if $syncs->{$name}{status} ne $stat;
}
next;
}
## Filter out by arbitrary attribute matches
if ($term =~ /(\w+)\s*=\s*(\w+)/) {
my ($attrib, $value) = (lc $1,$2);
for my $name (keys %$syncs) {
if (! exists $syncs->{$name}{$attrib}) {
my $message = "No such sync attribute: $attrib\n";
$message .= "Must be one of the following:\n";
my $names = join ',' =>
sort
grep { $_ !~ /\b(?:cdate|name)\b/ }
keys %{ $syncs->{$name} };
$message .= " $names\n";
die $message;
}
delete $syncs->{$name} if $syncs->{$name}{$attrib} ne $value;
}
next;
}
## Everything else should be considered a sync name
$namefilter = 1;
## Check for wildcards
if ($term =~ s/[*%]/.*/) {
for my $name (keys %$syncs) {
$syncs->{$name}{ok2show} = 1 if $name =~ /$term/;
}
next;
}
## Must be an exact match
for my $name (keys %$syncs) {
$syncs->{$name}{ok2show} = 1 if $name eq $term;
}
}
## If we filtered by name, remove all the non-matched ones
if ($namefilter) {
for my $name (keys %$syncs) {
delete $syncs->{$name} if ! exists $syncs->{$name}{ok2show};
}
}
## Nothing found? We're out of here
if (! keys %$syncs) {
print "No syncs found\n";
return -1;
}
## Determine the size of the output strings for pretty aligning later
my ($maxname, $maxherd, $maxdbs) = (2,2,2);
for my $name (keys %$syncs) {
my $s = $syncs->{$name}; $maxname = length $name if length $name > $maxname;
$maxherd = length $s->{herd}{name} if length $s->{herd}{name} > $maxherd;
$s->{d} = qq{DB group "$s->{dbs}"};
for (sort keys %{ $s->{dblist} }) {
$s->{d} .= " $_:$s->{dblist}{$_}{role}";
}
$maxdbs = length $s->{d} if length $s->{d} > $maxdbs;
}
## Now print them out in alphabetic order
for my $name (sort keys %$syncs) {
my $s = $syncs->{$name};
## Switch to multi-line if database info strings are over this
my $maxdbline = 50;
## Show basic information
printf qq{Sync %-*s Relgroup %-*s %s[%s]\n},
2+$maxname, qq{"$name"},
2+$maxherd, qq{"$s->{herd}{name}"},
$maxdbs > $maxdbline ? '' : " $s->{d} ",
ucfirst $s->{status};
## Print the second line if needed
if ($maxdbs > $maxdbline) {
print " $s->{d}\n";
}
## Show associated tables if in verbose mode
if ($VERBOSE >= 1) {
if (exists $s->{herd}{goat}) {
my $goathash = $s->{herd}{goat};
for my $relname (sort {
$goathash->{$b}{priority} <=> $goathash->{$a}{priority}
or $a cmp $b
}
keys %{ $goathash }) {
printf " %s %s\n",
ucfirst($goathash->{$relname}{reltype}),$relname;
}
}
}
## Show all the sync attributes
$VERBOSE >= 2 and show_all_columns($s);
} ## end of each sync
return 0;
} ## end of list_syncs
sub get_goat_ids {
## Returns the ids from the goat table for matching relations
## Also checks the live database and adds tables to the goat table as needed.
## Arguments: key-value pairs:
## - args: arrayref of names to match against. Can have wildcards. Can be 'all'
## - type: 'table' or 'sequence', depending on what we expect to find.
## - dbcols: optional hashref of fields to populate goat table with (e.g. autokick=1)
## - noherd: do not consider items if already in this herd for "all"
## Returns: a hash with:
## - relations: hash of goat objects, key is the fully qualified name
## - original: hash of search term(s) used to find this
## - goat: the goat object
## - nomatch: hash of non-matching terms
## - match: hash of matching terms
## - new: hash of newly added tables
my %arg = @_;
my $reltype = $arg{type};
my $names = $arg{args} or die 'Must have list of things to match';
my $dbcols = $arg{dbcols} || {};
my $noherd = $arg{noherd} || '';
## The final hash we return
my %relation;
## Args that produced a match
my %match;
## Args that produced no matches at all
my %nomatch;
## Keep track of which args we've already done, just in case there are dupes
my %seenit;
## Which tables we added to the goat table
my %new;
## Figure out which database to search in, unless already given
my $bestdb = (exists $dbcols->{db} and defined $dbcols->{db})
? $dbcols->{db} : find_best_db_for_searching();
## This check still makes sense: if no databases, there should be nothing in $GOAT!
if (! defined $bestdb) {
die "No databases have been added yet, so we cannot add tables!\n";
}
## Allow "tables=all" to become "all"
for my $item (@$names) {
$item = 'all' if $item =~ /^tables?=all/i;
}
my $rdbh = connect_database({name => $bestdb}) or die;
## SQL to find a table or a sequence
## We do not want pg_table_is_visible(c.oid) here
my $BASESQL = sub {
my $arg = shift || 'table';
## Assume we're talking about tables unless we say "sequence" explicitly
my $type = ( $arg eq 'sequence' ? 'S' : 'r' );
return qq{
SELECT nspname||'.'||relname AS name, relkind, c.oid, coalesce(i.indisprimary, false) as relhaspkey, nspname, relname
FROM pg_class c
JOIN pg_namespace n ON (n.oid = c.relnamespace)
LEFT JOIN pg_index i ON (indrelid = c.oid AND indisprimary)
WHERE relkind IN ('$type')
AND nspname <> 'information_schema'
AND nspname !~ '^pg_'
};
};
## Loop through each argument, and try and find matching goats
ITEM: for my $item (@$names) {
## In case someone entered duplicate arguments
next if $seenit{$item}++;
## Skip if this is not a tablename, but an argument of the form x=y
next if index($item, '=') >= 0;
## Determine if this item has a dot in it, and/or it is using wildcards
my $hasadot = index($item,'.') >= 0 ? 1 : 0;
my $hasstar = (index($item,'*') >= 0 or index($item,'%') >= 0) ? 1 : 0;
## Temporary list of matching items
my @matches;
## A list of tables to be bulk added to the goat table
my @addtable;
## We may mutate the arg, so stow away the original
my $original_item = $item;
## We look for matches in the existing $GOAT hash
## We may also check the live database afterwards
map {
push @matches, $_ if (! defined $reltype || $_->{reltype} eq $reltype);
} find_goat_by_item($item, \@nouns);
## Wildcards?
my $regex_item = $item;
## Setup the SQL to search the live database
if ($hasstar) {
## Change to a regexier form
$regex_item =~ s/\./\\./g;
$regex_item =~ s/[*%]/\.\*/g;
$regex_item = "^$regex_item" if $regex_item !~ /^[\^\.\%]/;
$regex_item .= '$' if $regex_item !~ /[\$\*]$/;
## Setup the SQL to search the live database
$SQL = $BASESQL->($reltype) . ($hasadot
? q{AND nspname||'.'||relname ~ ?}
: 'AND relname ~ ?');
} ## end wildcards
elsif ($hasadot) {
## A dot with no wildcards: exact match
## TODO: Allow foobar. to mean foobar.% ??
$SQL = $BASESQL->($reltype) . q{AND nspname||'.'||relname = ?};
}
else {
## No wildcards and no dot, so we match all tables regardless of the schema
$SQL = $BASESQL->($reltype);
$item eq 'all' or $SQL .= 'AND relname = ?';
}
## We do not check the live database if the match was exact
## *and* something was found. In all other cases, we go live.
if ($hasstar or !$hasadot or !@matches) {
debug(qq{NB! Found some existing matches; searching for other possibilities, because "$item" }
. ( $hasstar ? 'includes wildcard characters ' : '' )
. ( !$hasadot ? 'does not include a dot' : '' )) if @matches;
## Search the live database for matches
$sth = $rdbh->prepare($SQL);
$regex_item ||= $item;
if ('all' eq $item) {
($count = $sth->execute()) =~ s/0E0/0/;
}
else {
($count = $sth->execute($regex_item)) =~ s/0E0/0/;
}
debug(qq{Searched live database "$bestdb" for arg "$regex_item", count was $count});
debug(qq{SQL: $SQL}, 2);
debug(qq{Arg: $item ($regex_item)}, 2);
for my $row (@{ $sth->fetchall_arrayref({}) }) {
## The 'name' is combined "schema.relname"
my $name = $row->{name};
## Don't bother if we have already added this!
next if find_goat_by_item($name, [ "db=$bestdb" ]);
## If we are doing 'all', exclude the bucardo schema, and insist on a primary key
if ('all' eq $item) {
next if $name =~ /^bucardo\./;
if (!$row->{relhaspkey}) {
## Allow if we have a unique index on this table
$SQL = q{SELECT 1 FROM pg_index WHERE indisunique AND indrelid = }
. q{(SELECT c.oid FROM pg_class c JOIN pg_namespace n ON (n.oid = c.relnamespace) WHERE n.nspname=? AND c.relname=?)};
my $sthunique = $rdbh->prepare_cached($SQL);
$count = $sthunique->execute($row->{nspname},$row->{relname});
$sthunique->finish();
next if $count < 1;
}
}
## Document the string that led us to this one
$relation{$name}{original}{$item}++;
## Document the fact that we found this on a database
$new{$name}++;
## Mark this item as having produced a match
$match{$item}++;
## Set this table to be added to the goat table below
push @addtable, {name => $name, db => $bestdb, reltype => $row->{relkind}, dbcols => $dbcols};
}
}
## Add all the tables we just found from searching the live database
my $added_tables;
if (@addtable) {
$added_tables = add_items_to_goat_table(\@addtable);
}
for my $tmp (@$added_tables) {
push @matches, $GOAT->{by_id}{$tmp};
}
## If we asked for "all", add in all of our known tables (not already in this herd)
if ($names->[0] eq 'all') {
for (values %{ $GOAT->{by_db}{$bestdb} }) {
next if exists $_->{herd}{$noherd};
push @matches, $_;
}
}
## Populate the final hashes based on the match list
for my $match (@matches) {
next unless defined $match;
my $name;
if (ref $match eq 'HASH') {
$name = "$match->{schemaname}.$match->{tablename}";
}
else {
$name = $match;
}
$relation{$name}{original}{$original_item}++;
## This goat entry should be an array, if there are multiple goats
## with that name (e.g. from different databases)
if (exists $relation{$name}{goat}) {
push @{$relation{$name}{goat}}, $match;
}
else {
$relation{$name}{goat} = [ $match ];
}
$match{$item}++;
}
## If this item did not match anything, note that as well
if (! @matches and $names->[0] ne 'all') {
$nomatch{$original_item}++;
}
} ## end each given needle
return {
relations => \%relation,
nomatch => \%nomatch,
match => \%match,
new => \%new,
};
} ## end of get_goat_ids
sub add_items_to_goat_table {
## Given a list of tables, add them to the goat table as needed
## Arguments: one
## 1. Arrayref where keys are:
## - name: name of relation to add (mandatory)
## - db: the database name (mandatory)
## - reltype: table or sequence (optional, defaults to table)
## - dbcols: optional hashref of goat columns to set
## Returns: arrayref with all the new goat.ids
my $info = shift or die;
## Quick check if the entry is already there.
$SQL = 'SELECT id FROM bucardo.goat WHERE schemaname=? AND tablename=? AND db=?';
my $isthere = $dbh->prepare($SQL);
## SQL to add this new entry in
my $NEWGOATSQL = 'INSERT INTO bucardo.goat (schemaname,tablename,reltype,db) VALUES (?,?,?,?) RETURNING id';
my @newid;
for my $rel (sort { $a->{name} cmp $b->{name} } @$info) {
# XXX Is it safe to assume UTF8 encoding here? Probably not
my $name = $rel->{name};
if ($name !~ /^([-\w ]+)\.([-\w ]+)$/o) {
die qq{Invalid name, got "$name", but expected format "schema.relname"};
}
my ($schema,$table) = ($1,$2);
my $db = $rel->{db} or die q{Must provide a database};
my $reltype = $rel->{reltype} || 't';
$reltype = $reltype =~ /s/i ? 'sequence' : 'table';
## Adjust the SQL as necessary for this goat
$SQL = $NEWGOATSQL;
my @args = ($schema, $table, $reltype, $db);
if (exists $rel->{dbcols}) {
for my $newcol (sort keys %{ $rel->{dbcols} }) {
next if $newcol eq 'db';
$SQL =~ s/\)/,$newcol)/;
$SQL =~ s/\?,/?,?,/;
push @args => $rel->{dbcols}{$newcol};
}
}
$sth = $dbh->prepare($SQL);
($count = $sth->execute(@args)) =~ s/0E0/0/;
debug(qq{Added "$schema.$table" with db "$db", count was $count});
push @newid => $sth->fetchall_arrayref()->[0][0];
}
## Update the global
load_bucardo_info('force_reload');
## Return a list of goat IDs we've just added
# my %newlist;
# for my $id (@newid) {
# my $goat = $global{goat}{by_id}{$id};
# my $name = "$goat->{schemaname}.$goat->{tablename}";
# $newlist{$name} = $goat;
# }
return \@newid;
} ## end of add_items_to_goat_table
sub create_dbgroup {
## Creates a new entry in the bucardo.dbgroup table
## Caller should have alredy checked for existence
## Does not commit
## Arguments: two
## 1. Name of the new group
## 2. Boolean: if true, prevents the reload
## Returns: name of the new group
my ($name,$noreload) = @_;
$SQL = 'INSERT INTO bucardo.dbgroup(name) VALUES (?)';
$sth = $dbh->prepare($SQL);
eval {
$sth->execute($name);
};
if ($@) {
if ($@ =~ /dbgroup_name_sane/) {
print "Trying name $name\n";
print qq{Invalid characters in dbgroup name "$name"\n};
}
else {
print qq{Failed to create dbgroup "$name"\n$@\n};
}
exit 1;
}
## Reload our hashes
$noreload or load_bucardo_info(1);
return $name;
} ## end of create_dbgroup
sub get_arg_items {
## From an argument list, return all matching items
## Arguments: two
## 1. Arrayref of source items to match on
## 2. Arrayref of arguments
## Returns: an arrayref of matches, or an single scalar indicating what arg failed
my ($haystack, $needles) = @_;
my %match;
for my $needle (@$needles) {
my $hasadot = index($needle,'.') >= 0 ? 1 : 0;
my $hasstar = (index($needle,'*') >= 0 or index($needle,'%') >= 0) ? 1 : 0;
## Wildcards?
if ($hasstar) {
## Change to a regexier form
$needle =~ s/\*/\.\*/g;
## Count matches: if none found, we bail
my $found = 0;
for my $fullname (@$haystack) {
## If it has a dot, match the whole thing
if ($hasadot) {
if ($fullname =~ /^$needle$/) {
$match{$fullname} = $found++;
}
next;
}
## No dot, so match table part only
my ($schema,$table) = split /\./ => $fullname;
if ($table =~ /^$needle$/) {
$match{$fullname} = $found++;
}
}
return $needle if ! $found;
next;
} ## end wildcards
## If it has a dot, it must match exactly
if ($hasadot) {
if (grep { $_ eq $needle } @$haystack) {
$match{$needle} = 1;
next;
}
return $needle;
}
## No dot, so we match all tables regardless of the schema
my $found = 0;
for my $fullname (@$haystack) {
my ($schema,$table) = split /\./ => $fullname;
if ($table eq $needle) {
$match{$fullname} = $found++;
}
}
return $needle if ! $found;
} ## end each given needle
return \%match; ## May be undefined
} ## end of get_arg_items
sub clone {
## Put an entry in the clone table so the MCP can do some copyin'
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'clone';
usage_exit($doc_section) unless @nouns;
## Examples:
## ./bucardo clone dbs=A:source,B,C relgroup=foo
## ./bucardo clone sync=foobar
## ./bucardo clone sync=foobar prime=A
## ./bucardo clone dbs=A,B,C,D relgroup=foo notruncate=B,C
## Optional sync to associate with:
my $sync;
## Optional database group to use:
my $dbgroup;
## The prime (winning) source database.
my $prime;
## Optonal relgroup. Can be a list of tables
my $relgroup;
## Optional options :)
my $options;
for my $word (@nouns) {
## Check for an optional sync name.
if ($word =~ /(?i)sync(?-i)\s*[:=]\s*(\w.*?)\s*$/) {
my $syncname = $1;
if (! exists $SYNC->{$syncname}) {
die qq{Invalid sync "$syncname"\n};
}
## Have we already specified a sync?
if (defined $sync) {
die qq{Cannot specify more than one sync\n};
}
$sync = $syncname;
next;
}
## Check for an optional dbgroup
if ($word =~ /(?i)dbg(?-i)\w*\s*[:=]\s*(\w.*?)\s*$/) {
my $dbgroupname = $1;
if (! exists $DBGROUP->{$dbgroupname}) {
die qq{Invalid database group "$dbgroupname"\n};
}
## Have we already specified a database group?
if (defined $dbgroup) {
die qq{Cannot specify more than one database group\n};
}
$dbgroup = $dbgroupname;
next;
}
## Check for an optional relgroup
if ($word =~ /(?i)(?:relgroup|table)s?(?-i)\w*\s*[:=]\s*(\w.*?)\s*$/) {
my $relgroupname = $1;
## May be a relgroup, or a list of tables
if (exists $RELGROUP->{$relgroupname}) {
$relgroup = $relgroupname;
next;
}
## Must be one or more tables. See if we can find them, and shove into a relgroup
## Go through all the items and try to find matches
## Assumes tables are all in CSV format
my @tablelist = split /\s*,\s*/ => $relgroupname;
my $goatlist = get_goat_ids(args => \@tablelist, type => 'table');
## Cannot proceed unless we have a match for every table
if (keys %{ $goatlist->{nomatch} }) {
print "Cannot clone because the following tables were not found:\n";
for my $badname (sort keys %{ $goatlist->{nomatch} }) {
print " $badname\n";
}
exit 1;
}
## We need to generate a relgroup name
## TODO: See if any existing relgroups match exactly
my $basename = 'clone_relgroup';
my $number = 1;
{
$relgroupname = "$basename$number";
last if ! exists $RELGROUP->{$relgroupname};
$number++;
redo;
}
$SQL = 'INSERT INTO bucardo.herd(name) VALUES (?)';
$sth = $dbh->prepare($SQL);
$sth->execute($relgroupname);
$SQL = 'INSERT INTO bucardo.herdmap (herd,goat) VALUES (?,?)';
$sth = $dbh->prepare($SQL);
for my $goat (values %{ $goatlist->{relations} }) {
$sth->execute($relgroupname, $goat->{goat}[0]{id});
}
next;
}
## Check for a prime
if ($word =~ /(?i)prime(?-i)\w*\s*[:=]\s*(\w.*?)\s*$/) {
$prime = $1;
for my $candidate (split /\s*,\s*/ => $prime) {
## This must be a valid database
if (! exists $DB->{$candidate}) {
die qq{The prime option must specify a known database (not "$candidate")\n};
}
}
$options .= "prime=$prime;";
next;
}
die qq{Unknown option: $word\n};
} ## end checking each noun
## Must have at least one of sync or dbgroup
if (! defined $sync and ! defined $dbgroup) {
die qq{Must provide a sync or a database group\n};
}
## Generate a list of databases to make sure we know which is prime
my $dbrole;
if (defined $dbgroup) {
for my $row (values %{ $DBGROUP->{$dbgroup}{db} }) {
$dbrole->{ $row->{role} }{ $row->{db} } = 1;
}
}
else {
for my $db (values %{ $SYNC->{$sync}{dblist} }) {
$dbrole->{ $db->{role} }{ $db->{db} } = 1;
}
}
## If we have more than one source, make sure we know how to proceed
if (keys %{ $dbrole->{source}} > 1) {
## TODO: Allow more than one somehow
if (! defined $prime) {
warn qq{Must provide a prime so we know which database to copy from\n};
my $dblist = join ', ' => sort keys %{ $dbrole->{source} };
warn qq{Should be one of: $dblist\n};
exit 1;
}
}
## Clean up the options by removing any trailing semicolons
if (defined $options) {
$options =~ s/;$//;
}
$SQL = 'INSERT INTO bucardo.clone(status,sync,dbgroup,relgroup,options) VALUES (?,?,?,?,?) RETURNING id';
$sth = $dbh->prepare($SQL);
$sth->execute('new', $sync, $dbgroup, $relgroup, $options);
my $id = $sth->fetchall_arrayref()->[0][0];
## Tell the MCP there is a new clone
$dbh->do('NOTIFY bucardo_clone_ready');
confirm_commit();
$QUIET or print qq{Clone $id has been started. Track progress with "bucardo status clone $id"\n};
exit 0;
} ## end of clone
sub kick {
## Kick one or more syncs
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'kick';
usage_exit($doc_section) unless @nouns;
my ($exitstatus, $retries, $do_retry) = (0,0,0);
RETRY: {
$dbh->rollback();
$exitstatus = 0;
SYNC: for my $sync (@syncs) {
my $relname = "bucardo_kick_sync_$sync";
## If this sync is not active, cowardly refuse to kick it
if ($SYNC->{$sync}{status} ne 'active') {
print qq{Cannot kick inactive sync "$sync"\n};
next SYNC;
}
$dbh->do(qq{NOTIFY "bucardo_kick_sync_$sync"});
my $done = "bucardo_syncdone_$sync";
my $killed = "bucardo_synckill_$sync";
if (! defined $adverb) {
$dbh->commit();
$QUIET or print qq{Kicked sync $sync\n};
next;
}
$QUIET or print qq{Kick $sync: };
$dbh->do(qq{LISTEN "$done"});
$dbh->do(qq{LISTEN "$killed"});
$dbh->commit();
my $time = time;
sleep 0.1;
my $timeout = (defined $adverb and $adverb > 0) ? $adverb : 0;
my $printstring = $NOTIMER ? '' : '[0 s] ';
print $printstring unless $QUIET or $NOTIMER;
my $oldtime = 0;
local $SIG{ALRM} = sub { die 'Timed out' };
$do_retry = 0;
eval {
if ($timeout) {
alarm $timeout;
}
WAITIN: {
my $lastwait = '';
if (time - $time != $oldtime) {
$oldtime = time - $time;
if (!$QUIET and !$NOTIMER) {
print "\b" x length($printstring);
$printstring =~ s/\d+/$oldtime/;
print $printstring;
}
}
for my $notice (@{ db_get_notices($dbh) }) {
my ($name) = @$notice;
if ($name eq $done) {
$lastwait = 'DONE!';
}
elsif ($name eq $killed) {
$lastwait = 'KILLED!';
$exitstatus = 2;
}
elsif ($name =~ /^bucardo_syncdone_${sync}_(.+)$/) {
my $new = sprintf '%s(%ds) ', $1, ceil(time-$time);
print $new unless $QUIET;
$printstring .= $new;
}
elsif ($name =~ /^bucardo_synckill_${sync}_(.+)$/) {
my $new = sprintf '%s KILLED (%ds) ', $1, ceil(time-$time);
print $new unless $QUIET;
$printstring .= $new;
$exitstatus = 2;
$lastwait = ' ';
}
}
$dbh->rollback();
if ($lastwait) {
print $lastwait unless $QUIET;
if ($lastwait ne 'DONE!' and $RETRY and ++$retries <= $RETRY) {
print "Retry #$retries\n";
$do_retry = 1;
die "Forcing eval to exit for retry attempt\n";
}
last WAITIN;
}
sleep($WAITSLEEP);
redo WAITIN;
} ## end of WAITIN
alarm 0 if $timeout;
};
alarm 0 if $timeout;
if ($do_retry) {
$do_retry = 0;
redo RETRY;
}
if (2 == $exitstatus) {
my $reason = show_why_sync_killed($sync);
$reason and print "\n$reason\n";
}
if ($@) {
if ($@ =~ /Timed out/o) {
$exitstatus = 1;
warn "Timed out!\n";
}
else {
$exitstatus = 3;
warn "Error: $@\n";
}
next SYNC;
}
next SYNC if $QUIET;
print "\n";
} ## end each sync
} ## end RETRY
exit $exitstatus;
} ## end of kick
sub pause_resume {
## Pause or resume one or more syncs
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'pause';
usage_exit($doc_section) unless @nouns;
my $action = shift;
my @syncs_signalled;
for my $sync (@syncs) {
## Syncs can only be paused/resumed if they are active
my $status = $SYNC->{$sync}{status};
if ($status ne 'active') {
print qq{Cannot pause or resume sync "$sync" unless it is active (currently "$status")\n};
}
else {
$dbh->do(qq{NOTIFY "bucardo_${action}_sync_$sync"});
push @syncs_signalled => $sync;
}
}
$dbh->commit();
my $list = join ',' => @syncs_signalled;
$QUIET or print qq{Syncs ${action}d: $list\n};
exit 0;
} ## end of pause_resume
sub show_why_sync_killed {
## If a kick results in a "KILLED!" try and show why
## Arguments: one
## 1. Sync object
## Returns: message string
my $sync = shift;
$SQL = q{
SELECT * FROM bucardo.syncrun
WHERE sync = ?
AND lastbad
ORDER BY started DESC LIMIT 1
};
$sth = $dbh->prepare($SQL);
$count = $sth->execute($sync);
if ($count != 1) {
$sth->finish();
return '';
}
my $result = $sth->fetchall_arrayref({})->[0];
my $whydie = $result->{status} || '';
$whydie =~ s/\\n */\n /g;
$whydie =~ s/: ERROR:/:\n ERROR:/;
$whydie =~ s/ (at .+ line \d+\.)/\n $1/g;
$whydie =~ s/\t//g;
my $message = sprintf " Started: %s\n Ended: %s\n %s",
$result->{started} || '?',
$result->{ended} || '?',
$whydie;
return $message;
} ## end of show_why_sync_killed
sub status_all {
## Show status of all syncs in the database
## Arguments: none
## Returns: never, exits
## See if the MCP is running and what its PID is
if (! -e $PIDFILE) {
#print " (Bucardo does not appear to be currently running)\n";
}
else {
my $fh;
if (!open $fh, '<', $PIDFILE) {
print "\nERROR: Could not open $PIDFILE: $!";
}
else {
my $pid = <$fh>;
chomp $pid;
close $fh or warn qq{Could not close $PIDFILE: $!\n};
if ($pid =~ /^\d+$/) {
print "PID of Bucardo MCP: $pid";
}
else {
print "\nERROR: $PIDFILE contained: $pid";
}
}
}
print "\n";
if (! keys %$SYNC) {
print "No syncs have been created yet.\n";
exit 0;
}
my $orderby = $bcargs->{sort} || '1';
if ($orderby !~ /^\+?\-?\d$/) {
die "Invalid sort option, must be +- 1 through 9\n";
}
## Set the status for each sync if possible
my $max = set_sync_status();
## The titles
my %title = (
name => ' Name',
state => ' State',
lastgood => ' Last good',
timegood => ' Time',
dit => ($max->{truncate} ?
$max->{conflicts} ? ' Last I/D/T/C' : ' Last I/D/T' :
$max->{conflicts} ? ' Last I/D/C' :' Last I/D'),
lastbad => ' Last bad',
timebad => ' Time',
);
## Set the maximum as needed based on the titles
for my $name (keys %title) {
if (! exists $max->{$name}
or length($title{$name}) > $max->{$name}) {
$max->{$name} = length $title{$name};
}
}
## Account for our extra spacing by bumping everything up
for my $var (values %$max) {
$var += 2;
}
## Print the column headers
printf qq{%-*s %-*s %-*s %-*s %-*s %-*s %-*s\n},
$max->{name}, $title{name},
$max->{state}, $title{state},
$max->{lastgood}, $title{lastgood},
$max->{timegood}, $title{timegood},
$max->{dit}, $title{dit},
$max->{lastbad}, $title{lastbad},
$max->{timebad}, $title{timebad};
## Print a fancy dividing line
printf qq{%s+%s+%s+%s+%s+%s+%s\n},
'=' x $max->{name},
'=' x $max->{state},
'=' x $max->{lastgood},
'=' x $max->{timegood},
'=' x $max->{dit},
'=' x $max->{lastbad},
'=' x $max->{timebad};
## If fancy sorting desired, call the list ourself to sort
sub sortme {
my $sortcol = $bcargs->{sort} || 1;
+1 == $sortcol and return $a cmp $b;
-1 == $sortcol and return $b cmp $a;
my ($uno,$dos) = ($SYNC->{$a}, $SYNC->{$b});
## State
+3 == $sortcol and return ($uno->{state} cmp $dos->{state} or $a cmp $b);
-3 == $sortcol and return ($dos->{state} cmp $uno->{state} or $a cmp $b);
## Last good
+5 == $sortcol and return ($uno->{lastgoodsecs} <=> $dos->{lastgoodsecs} or $a cmp $b);
-5 == $sortcol and return ($dos->{lastgoodsecs} <=> $uno->{lastgoodsecs} or $a cmp $b);
## Good time
+6 == $sortcol and return ($uno->{lastgoodtime} <=> $dos->{lastgoodtime} or $a cmp $b);
-6 == $sortcol and return ($dos->{lastgoodtime} <=> $uno->{lastgoodtime} or $a cmp $b);
if ($sortcol == 7 or $sortcol == -7) {
my ($total1,$total2) = (0,0);
while ($uno->{dit} =~ /(\d+)/go) {
$total1 += $1;
}
while ($dos->{dit} =~ /(\d+)/go) {
$total2 += $1;
}
7 == $sortcol and return ($total1 <=> $total2 or $a cmp $b);
return ($total2 <=> $total1 or $a cmp $b);
}
## Last bad
+8 == $sortcol and return ($uno->{lastbadsecs} <=> $dos->{lastbadsecs} or $a cmp $b);
-8 == $sortcol and return ($dos->{lastbadsecs} <=> $uno->{lastbadsecs} or $a cmp $b);
## Bad time
+9 == $sortcol and return ($uno->{lastbadtime} <=> $dos->{lastbadtime} or $a cmp $b);
-9 == $sortcol and return ($dos->{lastbadtime} <=> $uno->{lastbadtime} or $a cmp $b);
return $a cmp $b;
}
for my $sync (sort sortme keys %$SYNC) {
my $s = $SYNC->{$sync};
## If this has been filtered out, skip it entirely
next if $s->{filtered};
## Populate any missing fields with an empty string
for my $name (keys %title) {
if (! exists $s->{$name}) {
$s->{$name} = '';
}
}
my $X = '|';
printf qq{%-*s$X%-*s$X%-*s$X%-*s$X%-*s$X%-*s$X%-*s\n},
$max->{name}," $sync ",
$max->{state}, " $s->{state} ",
$max->{lastgood}, " $s->{lastgood} ",
$max->{timegood}, " $s->{timegood} ",
$max->{dit}, " $s->{dit} ",
$max->{lastbad}, " $s->{lastbad} ",
$max->{timebad}, " $s->{timebad} ";
}
exit 0;
} ## end of status_all
sub status_detail {
## Show detailed information about one or more syncs
## Arguments: none, parses nouns
## Returns: never, exits
## Walk through and check each given sync
## It must either exist, or be a special key word
my @synclist;
for my $sync (@nouns) {
## Allow a special noise word: 'sync'
next if $sync eq 'sync';
## Add everything if we find the special word 'all'
if ($sync eq 'all') {
undef @synclist;
for my $sync (keys %$SYNC) {
## Turn off the filtering that set_sync_status may have added
$SYNC->{$sync}{filtered} = 0;
push @synclist => $sync;
}
last;
}
## If we don't know about this particular sync, give a warning
## We allow another special word: 'all'
if (!exists $SYNC->{$sync}) {
## If a number, skip for ease of "kick name #" toggling
$sync !~ /^\d+$/ and die "No such sync: $sync\n";
}
else {
push @synclist => $sync;
}
}
## Verify that all named syncs exist
my $max = set_sync_status({syncs => \@synclist});
## Present each in the order they gave
my $loops = 0;
for my $sync (@synclist) {
my $s = $SYNC->{$sync};
## Skip if it has been filtered out
next if $s->{filtered};
## Put a space between multiple entries
if ($loops++) {
print "\n";
}
print '=' x 70; print "\n";
my @items;
my $numtables = keys %{$s->{herd}{goat}};
my $sourcedb = $s->{herd}{db};
## Last good time, and number of rows affected
if (exists $s->{rowgood}) {
my $tt = pretty_time($s->{rowgood}{total_time});
push @items => ['Last good', "$s->{rowgood}{started_time} (time to run: $tt)"];
## Space out the numbers
$s->{dit} =~ s{/}{ / }g;
## Pretty comma formatting (based on ENV)
$s->{dit} =~ s/(\d+)/pretty_number($1)/ge;
## Change the title if we have any truncates
my $extra = $max->{truncates} ? '/truncates' : '';
## Change the title if we have any conflicts
$extra .= $max->{conflicts} ? '/conflicts' : '';
push @items => ["Rows deleted/inserted$extra", $s->{dit}];
}
## Last bad time, and the exact error
## The error should always be last, so we defer adding it to the queue
my $lasterror = '';
if (exists $s->{rowbad}) {
my $tt = pretty_time($s->{rowbad}{total_time});
push @items => ['Last bad', "$s->{rowbad}{started_time} (time until fail: $tt)"];
## Grab the error message, and strip out trailing whitespace
($lasterror = $s->{rowbad}{status}) =~ s/\s+$//;
## Add any newlines back in
$lasterror =~ s/\\n/\n/g;
## Remove starting whitespace
$lasterror =~ s/^\s+//;
}
## Undefined should be written as 'none'
for (qw/checktime/) {
$s->{$_} ||= 'None';
}
## Should be 'yes' or 'no'
for (qw/analyze_after_copy vacuum_after_copy stayalive kidsalive autokick/) {
$s->{$_} = (defined $s->{$_} and $s->{$_}) ? 'Yes' : 'No';
}
## If currently running, there should be a PID file
if (exists $s->{PIDFILE}) {
push @items => ['PID file' => $s->{PIDFILE}];
push @items => ['PID file created' => $s->{CREATED}];
}
## Build the display list
push @items => ['Sync name' => $sync];
push @items => ['Current state' => $s->{state}];
push @items => ['Source relgroup/database' => "$s->{herd}{name} / $sourcedb"];
push @items => ['Tables in sync' => $numtables];
push @items => ['Status' => ucfirst $s->{status}];
push @items => ['Check time' => $s->{checktime}];
push @items => ['Overdue time' => $s->{overdue}];
push @items => ['Expired time' => $s->{expired}];
push @items => ['Stayalive/Kidsalive' => "$s->{stayalive} / $s->{kidsalive}"];
push @items => ['Rebuild index' => $s->{rebuild_index} ? 'Yes' : 'No'];
push @items => ['Autokick' => $s->{autokick}];
push @items => ['Onetimecopy' => $s->{onetimecopy} ? 'Yes' : 'No'];
## Only show these if enabled
if ($s->{analyze_after_copy} eq 'Yes') {
push @items => ['Post-copy analyze', 'Yes'];
}
if ($s->{vacuum_after_copy} eq 'Yes') {
push @items => ['Post-copy vacuum', 'Yes'];
}
## Final items:
push @items => ['Last error:' => $lasterror];
## Figure out the maximum size of the left-hand items
my $leftmax = 0;
for (@items) {
$leftmax = length $_->[0] if length $_->[0] > $leftmax;
}
## Print it all out
for (@items) {
printf "%-*s : %s\n",
$leftmax, $_->[0], $_->[1];
}
print '=' x 70; print "\n";
}
exit 0;
} ## end of status_detail
sub append_reason_file {
## Add an entry to the 'reason' log file
## Arguments: one
## 1. Message to store
## Returns: undef
my $event = shift or die;
my $string = sprintf "%s | %-5s | %s\n", (scalar localtime), $event, $nouns;
open my $fh, '>', $REASONFILE or die qq{Could not open "$REASONFILE": $!\n};
print {$fh} $string;
close $fh or warn qq{Could not close $REASONFILE: $!\n};
open $fh, '>>', $REASONFILE_LOG or die qq{Could not open "$REASONFILE_LOG": $!\n};
print {$fh} $string;
close $fh or warn qq{Could not close $REASONFILE_LOG: $!\n};
return;
} ## end of append_reason_file
sub set_sync_status {
## Set detailed information about syncs from the syncrun table
## Arguments: zero or one (hashref)
## 1. Hashref containing a. syncs=arrarref of syncnames
## Returns: hashref indicating maximum lengths of inner information
## If a sync is filtered out via the 'syncs' argument, it is set to $s->{filtered} = 1
my $arg = shift || {};
## View the details of the syncs via the syncrun table
$SQL = qq{
SELECT *,
TO_CHAR(started,'$DATEFORMAT') AS started_time,
CASE WHEN current_date = ended::date
THEN TO_CHAR(ended,'$SHORTDATEFORMAT')
ELSE TO_CHAR(ended,'$DATEFORMAT') END AS ended_time,
ROUND(EXTRACT(epoch FROM ended)) AS ended_epoch,
EXTRACT(epoch FROM ended-started) AS total_time,
ROUND(EXTRACT(epoch FROM now()-started)) AS total_time_started,
ROUND(EXTRACT(epoch FROM now()-ended)) AS total_time_ended
FROM syncrun
WHERE sync = ?
AND ( lastgood IS TRUE
OR lastbad IS TRUE
OR lastempty IS TRUE
OR ended IS NULL)
};
$sth = $dbh->prepare($SQL);
## Find the maximum lengths of items so we can line things up pretty
my %max = (
name => 1,
state => 1,
dit => 1,
lastgood => 1,
timegood => 1,
lastbad => 1,
timebad => 1,
);
for my $sync (keys %$SYNC) {
my $s = $SYNC->{$sync};
## Sometimes we only want some of them
if ($arg->{syncs}) {
if (! grep { $_ eq $sync } @{$arg->{syncs}}) { ## no critic (ProhibitBooleanGrep)
$s->{filtered} = 1;
next;
}
}
$max{name} = length($sync) if length($sync) > $max{name};
$count = $sth->execute($sync);
if ($count < 1) {
$sth->finish;
$s->{state} = 'No records found';
$max{state} = length($s->{state}) if length($s->{state}) > $max{state};
next;
}
for my $row (@{ $sth->fetchall_arrayref({}) }) {
if ($row->{lastgood}) {
$s->{rowgood} = $row;
}
elsif ($row->{lastempty}) {
$s->{rowempty} = $row;
}
elsif ($row->{lastbad}) {
$s->{rowbad} = $row;
}
else {
$s->{runningrow} = $row;
}
}
## What is the state of this sync? First, is it still actively running?
if (exists $s->{runningrow}) {
$s->{state} = "$s->{runningrow}{status}";
}
else {
## What was the most recent run?
my $highepoch = 0;
undef $s->{latestrow};
my $wintype;
for my $type (qw/ bad good empty /) {
my $r = $s->{"row$type"};
next if ! defined $r;
my $etime = $r->{ended_epoch};
if ($etime >= $highepoch) {
$s->{latestrow} = $r;
$highepoch = $etime;
$wintype = $type;
}
}
if (! defined $s->{latestrow}) {
$s->{state} = 'Unknown';
$max{state} = length($s->{state}) if length($s->{state}) > $max{state};
next;
}
if ($wintype eq 'empty') {
# Empty is good, as far as status is concerned.
$s->{rowgood} = $s->{latestrow};
$wintype = 'good';
}
$s->{state} = ucfirst $wintype;
}
## deletes/inserts/truncates/conflicts
$s->{dit} = '';
if (exists $s->{rowgood}) {
$s->{dit} = "$s->{rowgood}{deletes}/$s->{rowgood}{inserts}";
if ($s->{rowgood}{truncates}) {
$max{truncates} = 1;
$s->{dit} .= "/$s->{rowgood}{truncates}";
}
if ($s->{rowgood}{conflicts}) {
$max{conflicts} = 1;
$s->{dit} .= "/$s->{rowgood}{conflicts}";
}
}
$s->{lastgood} = exists $s->{rowgood} ? $s->{rowgood}{ended_time} : 'none';
$s->{timegood} = exists $s->{rowgood} ? pretty_time($s->{rowgood}{total_time_ended}) : '';
$s->{lastbad} = exists $s->{rowbad} ? $s->{rowbad}{ended_time} : 'none';
$s->{timebad} = exists $s->{rowbad} ? pretty_time($s->{rowbad}{total_time_ended}) : '';
for my $var (qw/ state dit lastgood timegood lastbad timebad /) {
$max{$var} = length($s->{$var}) if length($s->{$var}) > $max{$var};
}
}
return \%max;
} ## end of set_sync_status
sub inspect {
## Inspect an item in the database
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'inspect';
usage_exit($doc_section) unless @nouns;
my $thing = shift @nouns;
inspect_table() if $thing =~ /tab/i or $thing eq 't';
inspect_sync() if $thing =~ /sync/i or $thing eq 's';
inspect_herd() if $thing =~ /(?:relgr|herd)/i or $thing eq 'h';
usage_exit($doc_section);
return;
} ## end of inspect
sub inspect_table {
## Inspect an item from the goat table
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'inspect';
usage_exit($doc_section) unless @nouns;
$SQL = q{SELECT * FROM bucardo.goat WHERE tablename=?};
my $sth_goat = $dbh->prepare($SQL);
$SQL = q{SELECT * FROM bucardo.goat WHERE schemaname = ? AND tablename=?};
my $sth_goat_schema = $dbh->prepare($SQL);
my @tables;
for my $name (@nouns) {
my $sthg;
if ($name =~ /(.+)\.(.+)/) {
$sthg = $sth_goat_schema;
$count = $sthg->execute($1,$2);
}
else {
$sthg = $sth_goat;
$count = $sthg->execute($name);
}
if ($count < 1) {
die "Unknown table: $name\n";
}
for my $row (@{$sthg->fetchall_arrayref({})}) {
push @tables, $row;
}
}
for my $t (@tables) {
my ($s,$t,$db,$id) = @$t{qw/schemaname tablename db id/};
print "Inspecting $s.$t on $db\n";
## Grab all other tables referenced by this one
my $tablist = get_reffed_tables($s,$t,$db);
## Check that each referenced table is in a herd with this table
my %seenit;
for my $tab (@$tablist) {
my ($type,$tab1,$tab2,$name,$def) = @$tab;
my $table = $type==1 ? $tab1 : $tab2;
if ($table !~ /(.+)\.(.+)/) {
die "Invalid table information\n";
}
my $schema = $1;
$table = $2;
next if $seenit{"$schema.$table.$type"}++;
## Make sure that each herd with this table also has this new table
my $ggoat = $global{goat};
my $hherd = $global{herd};
for my $herd (sort keys %{$ggoat->{by_id}{$id}{herd}}) {
$seenit{fktable} = 1;
next if exists $hherd->{$herd}{hasgoat}{$schema}{$table};
printf "Table %s.%s is in relgroup %s, but %s.%s (used as FK%s) is not\n",
$s, $t, $herd, $schema, $table,
$type == 1 ? '' : ' target';
}
if (! exists $seenit{fktable}) {
printf "Table %s.%s is used as FK%s by %s.%s\n",
$s,$t,$type==1 ? '' : ' target', $schema, $table;
delete $seenit{fktable};
}
}
}
exit 0;
} ## end of inspect_table
sub inspect_herd {
## Inspect an item from the herd table
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'inspect';
usage_exit($doc_section) unless @nouns;
die "Not implemented yet\n";
} ## end of inspect_herd
sub inspect_sync {
## Inspect an item from the sync table
## Arguments: none, parses nouns
## Returns: never, exits
my $doc_section = 'inspect';
usage_exit($doc_section) unless @nouns;
die "Not implemented yet\n";
} ## end of inspect_sync
sub get_reffed_tables {
## Find all tables that are references by the given one
## Arguments: three
## 1. Schema name
## 2. Table name
## 3. Database name
## Returns: arrayref of tables from the database
my ($s,$t,$db) = @_;
my $rdbh = connect_database({name => $db});
## So we get the schemas
$rdbh->do('SET search_path = pg_catalog');
$SQL= q{
SELECT CASE WHEN conrelid=x.toid THEN 1 ELSE 2 END,
confrelid::regclass,
conrelid::regclass,
conname,
pg_get_constraintdef(oid, true)
FROM pg_constraint,
(SELECT c.oid AS toid FROM pg_class c JOIN pg_namespace n
ON (n.oid=c.relnamespace) WHERE nspname=? AND relname=?
) x
WHERE contype = 'f' AND
(confrelid = x.toid OR conrelid = x.toid)
};
$sth = $rdbh->prepare($SQL);
$count = $sth->execute($s,$t);
return $sth->fetchall_arrayref();
} ## end of get_reffed_tables
sub show_all_columns {
## Give a detailed listing of a particular row in the bucardo database
## Arguments: one
## 1. Hashref of information to display
## Returns: formatted, sorted, and indented list as a single string
my $row = shift or die;
my $maxkey = 1;
for my $key (keys %$row) {
next if ref $row->{$key};
$maxkey = length $key if length $key > $maxkey;
}
for my $key (sort {
($a eq 'src_code' and $b ne 'src_code' ? 1 : 0)
or
($a ne 'src_code' and $b eq 'src_code' ? -1 : 0)
or
$a cmp $b } keys %$row
) {
next if ref $row->{$key};
printf " %-*s = %s\n", $maxkey, $key,
defined $row->{$key} ? $row->{$key} : 'NULL';
}
return;
} ## end of show_all_columns
sub process_args {
## Break apart a string of args, return a clean hashref
## Arguments: one
## 1. List of arguments
## Returns: hashref
my $string = shift or return {};
$string .= ' ';
my %arg;
while ($string =~ m/(\w+)\s*[=:]\s*"(.+?)" /g) {
$arg{lc $1} = $2;
}
$string =~ s/\w+\s*=\s*".+?" / /g;
while ($string =~ m/(\w+)\s*[=:]\s*'(.+?)' /g) {
$arg{lc $1} = $2;
}
$string =~ s/\w+\s*=\s*'.+?' / /g;
while ($string =~ m/(\w+)\s*[=:]\s*(\S+)/g) {
$arg{lc $1} = $2;
}
$string =~ s/\w+\s*=\s*\S+/ /g;
if ($string =~ /\S/) {
$string =~ s/^\s+//;
$arg{extraargs} = [split /\s+/ => $string];
}
## Clean up and standardize the names
if (exists $arg{type}) {
$arg{type} = standardize_rdbms_name($arg{type});
}
return \%arg;
} ## end of process_args
sub list_clones {
## Show information about clones. Queries the bucardo.clone table
## Arguments: zero or more
## 1+ Clone id to view.
## Returns: 0 on success, -1 on error
## Example: bucardo list clones
## Might be no clones yet
if (! keys %$CLONE) {
print "No clones have been created yet\n";
return -1;
}
## Keep track of specific requests
my $cloneids;
for my $term (@nouns) {
if ($term =~ /^(\d+)$/) {
my $id = $1;
if (! exists $CLONE->{$id}) {
die qq{No such clone id "$id": try bucardo list clones\n};
}
$cloneids->{$id}++;
}
} ## end each term
## Print them out in numeric order
for my $clone (sort { $a->{id} <=> $b->{id} } values %$CLONE) {
## TODO: right justify numbers
next if keys %$cloneids and ! exists $cloneids->{$clone->{id}};
print "Clone #$clone->{id}";
print " Status: $clone->{status}";
defined $clone->{sync} and print " Sync: $clone->{sync}";
defined $clone->{dbgroup} and print " Dbgroup: $clone->{dbgroup}";
defined $clone->{relgroup} and print " Relgroup: $clone->{relgroup}";
defined $clone->{started} and print " Started: $clone->{pstarted}";
defined $clone->{ended} and print " Ended: $clone->{pstarted}";
if (defined $clone->{options}) {
print " $clone->{options}";
}
## Print last, on a new line:
defined $clone->{summary} and print "\n Summary: $clone->{summary}";
print "\n";
}
return 0;
} ## end of list_clones
sub list_customcodes {
## Show information about all or some subset of the 'customcode' table
## Arguments: none, parses nouns for customcodes
## Returns: 0 on success, -1 on error
my $doc_section = 'list';
## Any nouns are filters against the whole list
my $clause = generate_clause({col => 'name', items => \@nouns});
my $WHERE = $clause ? "WHERE $clause" : '';
$SQL = "SELECT * FROM bucardo.customcode $WHERE ORDER BY name";
$sth = $dbh->prepare($SQL);
$count = $sth->execute();
if ($count < 1) {
$sth->finish();
printf "There are no%s entries in the 'customcode' table.\n",
$WHERE ? ' matching' : '';
return -1;
}
$info = $sth->fetchall_arrayref({});
my ($maxname,$maxwhen) = (1,1);
for my $row (@$info) {
$maxname = length $row->{name} if length $row->{name} > $maxname;
$maxwhen = length $row->{whenrun} if length $row->{whenrun} > $maxwhen;
}
for my $row (@$info) {
my $name = $row->{name};
## We never show the actual source code unless verbosity is at least three!
if ($VERBOSE < 3) {
$row->{src_code} = 'Use -vvv to see the actual source code';
}
## We want to show all associates syncs and relations (when mapping is active)
my $info2 = $CUSTOMCODE->{$name} || {};
my ($synclist, $relationlist) = ('','');
if (exists $info2->{map}) {
$synclist = join ',' => sort map { $_->{sync} }
grep { defined $_->{sync} and $_->{active} }
@{ $info2->{map} };
$relationlist = join ',' => sort
map { "$GOAT->{by_id}{$_->{goat}}{schemaname}.$GOAT->{by_id}{$_->{goat}}{tablename}" }
grep { defined $_->{goat} and $_->{active} }
@{ $info2->{map} };
}
printf "Code: %-*s When run: %-*s Get dbh: %s Status: %s\n",
$maxname, $name,
$maxwhen, $row->{whenrun},
$row->{getdbh},
$row->{status};
if (length $synclist) {
print " Syncs: $synclist\n";
}
if (length $relationlist) {
print " Relations: $relationlist\n";
}
if (defined $row->{about} and $VERBOSE) {
(my $about = $row->{about}) =~ s/(.)^/$1 /gsm;
print " About: $about\n";
}
$VERBOSE >= 2 and show_all_columns($row);
}
return 0;
} ## end of list_customcodes
sub update_customcode {
## Update one or more customcodes
## Arguments: none (reads nouns for a list of customcodes)
## Returns: never, exits
my @actions = @_;
my $doc_section = 'update/update customcode';
usage_exit($doc_section) unless @actions;
my $name = shift @actions;
## Recursively call ourselves for wildcards and 'all'
exit 0 if ! check_recurse($SYNC, $name, @actions);
## Make sure this customcode exists!
if (! exists $CUSTOMCODE->{$name}) {
die qq{Could not find a customcode named "$name"\nUse 'list customcodes' to see all available.\n};
}
my $cc = $CUSTOMCODE->{$name};
my $changes = 0;
for my $action (@actions) {
## Look for a standard foo=bar or foo:bar format
if ($action =~ /(.+?)\s*[=:]\s*(.+)/) {
my ($setting,$value) = (lc $1,$2);
## No funny characters please, just boring column names
$setting =~ /^[a-z_]+$/ or die "Invalid setting: $setting\n";
my $srcfile;
## We only allow changing a strict subset of all the columns
if ($setting =~ /^(?:about|getdbh|name|priority|status|whenrun|src_code)$/) {
my $oldvalue = defined $cc->{$setting} ? $cc->{$setting} : '';
## Allow some variation for booleans, but transform to 0/1
if ($setting =~ /^(?:getdbh)$/) {
$value = $value =~ /^[1tTyY]/ ? 1 : 0;
}
## Some things can only be numbers
elsif ($setting =~ /^(?:priority)$/) {
if ($value !~ /^\d+$/) {
die qq{Customcode setting "$setting" must be a number!\n};
}
}
## And some are very specific indeed
elsif ('whenrun' eq $setting) {
my %whenrun = map { $_ => 1 } _whenrun_values();
die qq{Invalid value for setting "whenrun"\n}
unless $whenrun{$value};
}
elsif ('src_code' eq $setting) {
$srcfile = $value;
if (! -e $srcfile) {
warn qq{Could not find a file named "$srcfile"\n};
exit 2;
}
open my $fh, '<', $srcfile or die qq{Could not open "$srcfile": $!\n};
$value = '';
{ local $/; $value = <$fh>; } ## no critic (RequireInitializationForLocalVars)
close $fh or warn qq{Could not close "$srcfile": $!\n};
}
## Make the change, if it has changed
if ($value ne $oldvalue) {
$SQL = "UPDATE customcode SET $setting=? WHERE name = ?";
$sth = $dbh->prepare($SQL);
$sth->execute($value, $name);
$changes++;
if ('src_code' eq $setting) {
print qq{Changed customcode "$name" $setting with content of file "$srcfile"\n};
}
else {
print qq{Changed customcode "$name" $setting from '$oldvalue' to '$value'\n};
}
}
}
else {
warn "Cannot change attribute '$setting'\n";
usage_exit($doc_section);
}
next;
}
warn "\nUnknown action: $action\n";
usage_exit($doc_section);
}
confirm_commit() if $changes;
return;
} ## end of update_customcode
sub _whenrun_values {
return qw(
before_txn
before_check_rows
before_trigger_drop
after_trigger_drop
after_table_sync
exception
conflict
before_trigger_enable
after_trigger_enable
after_txn
before_sync
after_sync
);
}
sub list_sequences {
## Show information about all or some sequences in the 'goat' table
## Arguments: none (reads nouns for a list of sequences)
## Returns: 0 on success, -1 on error
my $doc_section = 'list';
my $clause = generate_clause({col => 'tablename', items => \@nouns});
my $WHERE = $clause ? "AND $clause" : '';
$SQL = "SELECT * FROM bucardo.goat WHERE reltype = 'sequence' $WHERE ORDER BY schemaname, tablename";
$sth = $dbh->prepare($SQL);
$count = $sth->execute();
if ($count < 1) {
$sth->finish();
printf "There are no%s sequences.\n",
$WHERE ? ' matching' : '';
return -1;
}
$info = $sth->fetchall_arrayref({});
my $maxq = 1;
for my $row (@$info) {
my $len = length "$row->{schemaname}.$row->{tablename}";
$maxq = $len if $len > $maxq;
}
for my $row (@$info) {
printf "Sequence: %-*s DB: %s\n",
$maxq, "$row->{schemaname}.$row->{tablename}",
$row->{db};
$VERBOSE >= 2 and show_all_columns($row);
}
return 0;
} ## end of list_sequences
sub pretty_time {
## Change seconds to a prettier display with hours, minutes, etc.
## Arguments: one
## 1. Number of seconds
## Returns: formatted string
my $secs = shift;
## Round up to the nearest second if decimal places are given
$secs = ceil($secs);
## If we cannot figure out the seconds, give up and return a question mark
return '?' if ! defined $secs or $secs !~ /^\-?\d+$/o or $secs < 0;
## Initialize days, hours, minutes, and seconds
my ($D,$H,$M,$S) = (0,0,0,0);
## Some people do not want days shown, so leave it as an option
if ($bcargs->{showdays}) {
if ($secs > 60*60*24) {
$D = int $secs/(60*60*24);
$secs -= $D*60*60*24;
}
}
## Show hours if there is > 1 hour
if ($secs > 60*60) {
$H = int $secs/(60*60);
$secs -= $H*60*60;
}
## Show minutes if there is > 1 minute
if ($secs > 60) {
$M = int $secs/60;
$secs -= $M*60;
}
$secs = int $secs;
my $answer = sprintf "%s%s%s${secs}s",$D ? "${D}d " : '',$H ? "${H}h " : '',$M ? "${M}m " : '';
## Detailed listings get compressed
if ((defined $COMPRESS and $COMPRESS) or (!defined $COMPRESS and !@nouns)) {
$answer =~ s/ //g;
}
return $answer;
} ## end of pretty_time
sub pretty_number {
## Format a raw number in a more readable style
## Arguments: one
## 1. Number
## Returns: formatted number
my $number = shift;
return $number if $number !~ /^\d+$/ or $number < 1000;
## If this is our first time here, find the correct separator
if (! defined $bcargs->{tsep}) {
my $lconv = localeconv();
$bcargs->{tsep} = $lconv->{thousands_sep} || ',';
}
## No formatting at all
return $number if '' eq $bcargs->{tsep} or ! $bcargs->{tsep};
(my $reverse = reverse $number) =~ s/(...)(?=\d)/$1$bcargs->{tsep}/g;
$number = reverse $reverse;
return $number;
} ## end of pretty_number
sub vate_sync {
## Activate or deactivate a sync
## Arguments: none (reads verbs and nouns)
## Returns: never, exits
my $name = lc $verb;
my $ucname = ucfirst $name;
@nouns or die qq{${name}_sync requires at least one sync name\n};
my $wait = (defined $adverb and $adverb eq '0') ? 1 : 0;
for my $sync (@syncs) {
(my $vname = $ucname) =~ s/e$/ing/;
$QUIET or print qq{$vname sync $sync};
my $done = "bucardo_${name}d_sync_$sync";
$dbh->do(qq{NOTIFY "bucardo_${name}_sync_$sync"});
if ($wait) {
print '...';
$dbh->do(qq{LISTEN "$done"});
}
$dbh->commit();
if (!$wait) {
print "\n";
next;
}
sleep 0.1;
wait_for_notice($dbh, $done);
print "OK\n";
} ## end each sync
exit 0;
} ## end of vate_sync
sub add_customcode {
## Add an entry to the bucardo.customcode table
## Arguments: none (uses nouns)
## Returns: never, exits
my $item_name = shift @nouns || '';
my $doc_section = 'add/add customcode';
usage_exit($doc_section) unless length $item_name;
## Inputs and aliases, database column name, flags, default
my $whenrun = join '|' => _whenrun_values();
my $validcols = qq{
name name 0 $item_name
about about 0 null
whenrun|when_run whenrun =$whenrun null
getdbh getdbh TF null
sync sync 0 skip
goat|table|relation goat 0 skip
status status =active|inactive skip
priority priority number skip
src_code src_code 0 skip
};
my ( $dbcols, $cols, $phs, $vals, $extras ) = process_simple_args({
cols => $validcols,
list => \@nouns,
doc_section => $doc_section,
});
my $newname = $dbcols->{name};
## Does this already exist?
if (exists $CUSTOMCODE->{$newname}) {
warn qq{Cannot create: customcode "$newname" already exists\n};
exit 2;
}
## We must have a "whenrun"
usage_exit($doc_section) unless $dbcols->{whenrun};
## We must have a src_code as a file
usage_exit($doc_section) unless $extras->{src_code};
my $tfile = $extras->{src_code};
if (! -e $tfile) {
warn qq{Could not find a file named "$tfile"\n};
exit 2;
}
open my $fh, '<', $tfile or die qq{Could not open "$tfile": $!\n};
my $src = '';
{ local $/; $src = <$fh>; } ## no critic (RequireInitializationForLocalVars)
close $fh or warn qq{Could not close "$tfile": $!\n};
## Attempt to insert this into the database
$SQL = "INSERT INTO bucardo.customcode ($cols,src_code) VALUES ($phs,?)";
$DEBUG and warn "SQL: $SQL\n";
$DEBUG and warn Dumper $vals;
$sth = $dbh->prepare($SQL);
eval {
$count = $sth->execute((map { $vals->{$_} } sort keys %$vals), $src);
};
if ($@) {
die "Failed to add customcode: $@\n";
}
my $finalmsg = '';
## See if any updates to customcode_map need to be made
## Only one of sync or goat can be specified
if ($extras->{sync} and $extras->{relation}) {
die qq{Sorry, you must specify a sync OR a relation, not both\n};
}
## Makes no sense to specify priority or active if no goat or sync
if (($extras->{priority} or $extras->{active}) and !$extras->{sync} and ! $extras->{relation}) {
die qq{You must specify a sync or a relation when using priority or active\n};
}
## Is this a valid sync?
if ($extras->{sync} and ! exists $SYNC->{$extras->{sync}}) {
die qq{Unknown sync: $extras->{sync}\n};
}
## Is this a valid gaot?
if ($extras->{relation} and ! exists $GOAT->{by_name}{$extras->{relation}}
and ! exists $GOAT->{by_table}{$extras->{relation}}
and ! exists $GOAT->{by_fullname}{$extras->{relation}} ) {
die qq{Unknown relation: $extras->{relation}\n};
}
## Add to the customcode_map table
if ($extras->{sync} or $extras->{relation}) {
$SQL = 'INSERT INTO customcode_map(code,';
my @vals;
for my $col (qw/sync priority active/) {
if ($extras->{$col}) {
$SQL .= "$col,";
push @vals => $extras->{$col};
}
}
if ($extras->{relation}) {
$SQL .= 'goat,';
push @vals => exists $GOAT->{by_name}{$extras->{relation}}
? $GOAT->{by_name}{$extras->{relation}}->[0]{id}
: exists $GOAT->{by_table}{$extras->{relation}}->[0]{id}
? $GOAT->{by_table}{$extras->{relation}}->[0]{id}
: $GOAT->{by_fullname}{$extras->{relation}}->[0]{id}
}
my $phs2 = '?,' x @vals;
$SQL .= ") VALUES ((SELECT currval('customcode_id_seq')),$phs2)";
$SQL =~ s/,\)/)/g;
$sth = $dbh->prepare($SQL);
eval {
$count = $sth->execute(@vals);
};
if ($@) {
die "Failed to add customcode_map: $@\n";
}
}
if (!$QUIET) {
print qq{Added customcode "$newname"\n};
$finalmsg and print $finalmsg;
}
confirm_commit();
exit 0;
} ## end of add_customcode
sub _list_databases {
## Quick list of available databases
## Arguments: none
## Returns: list of databases as a single string
return if ! keys %{ $DB };
warn "The following databases are available:\n";
for (sort keys %{ $DB }) {
next if $DB->{$_}{dbtype} ne 'postgres';
print "$_\n";
}
return;
} ## end of _list_databases
sub add_all_tables {
## Add all tables, returns output from add_all_goats
## Arguments: none
## Returns: output of inner sub
return add_all_goats('table');
} ## end of add_all_tables
sub add_all_sequences {
## Add all tables, returns output from add_all_goats
## Arguments: none
## Returns: output of inner sub
return add_all_goats('sequence');
} ## end of add_all_sequences
sub add_all_goats {
## Add all tables, or sequences
## Arguments: one
## 1. The type, table or sequence
## Returns: Srting indicating what was done
my $type = shift;
## Usage: add all table(s) | add all sequence(s)
## Options:
## --db: use this database (internal name from the db table)
## --schema or -n: limit to one or more comma-separated schemas
## --exclude-schema or -N: exclude these schemas
## --table or -t: limit to the given tables
## --exclude-table or -T: exclude these tables
## --relgroup: name of the herd to add new tables to
## pkonly: exclude tables with no pkey
## Returns: text string of results, with a newline
## Transform command-line args to traditional format
## e.g. db=foo becomes the equivalent of --db=foo
## foo becomes foo=1
for my $noun (@nouns) {
if ($noun =~ /(\w+)=(\w+)/) {
$bcargs->{$1} = $2;
}
else {
$bcargs->{$noun} = 1;
}
}
$bcargs->{herd} = delete $bcargs->{relgroup} || $bcargs->{herd};
## If no databases, cowardly refuse to continue
if (! keys %$DB) {
die "Sorry, cannot add any ${type}s until at least one database has been added\n";
}
## If there is more than one database, it should be selected via db=
my $db;
my $showdbs = 0;
if (exists $bcargs->{db}) {
if (! exists $DB->{$bcargs->{db}}) {
warn qq{Sorry, could not find a database named "$bcargs->{db}"\n};
$showdbs = 1;
}
else {
$db = $DB->{$bcargs->{db}};
}
}
elsif (keys %$DB == 1) {
($db) = values %$DB;
}
else {
## Grab the most likely candidate
my $bestdb = find_best_db_for_searching();
if (! $bestdb) {
warn "Please specify which database to use with the db= option\n";
$showdbs = 1;
}
else {
$db = $DB->{$bestdb};
}
}
if ($showdbs) {
_list_databases();
exit 1;
}
## Connect to the remote database
my $dbh2 = connect_database({name => $db->{name}});
## Query to pull all tables we may possibly need
my $kind = $type eq 'table' ? 'r' : 'S';
$SQL = q{SELECT nspname, relname FROM pg_catalog.pg_class c }
. q{JOIN pg_catalog.pg_namespace n ON (n.oid = c.relnamespace) }
. qq{WHERE relkind = '$kind' };
## We always exclude information_schema, system, and bucardo schemas
$SQL .= q{AND n.nspname <> 'information_schema' AND nspname !~ '^pg' AND nspname !~ '^bucardo'};
my @clause;
## If they gave a schema option, restrict the query by namespace
push @clause => generate_clause({col => 'nspname', items => $bcargs->{schema}});
## If they have asked to exclude schemas, append that to the namespace clause
push @clause => generate_clause({col => 'nspname', items => $bcargs->{'exclude-schema'}, not => 1});
## If they gave a table option, restrict the query by relname
push @clause => generate_clause({col => 'relname', items => $bcargs->{table}});
## If they have asked to exclude tables, append that to the relname clause
push @clause => generate_clause({col => 'relname', items => $bcargs->{'exclude-table'}, not => 1});
for my $c (@clause) {
next if ! $c;
$SQL .= "\nAND ($c)";
}
## Fetch all the items, warn if no matches are found
$VERBOSE >= 2 and warn "Query: $SQL\n";
$sth = $dbh2->prepare($SQL);
$count = $sth->execute();
if ($count < 1) {
warn "Sorry, no ${type}s were found\n";
}
## Grab all current tables or sequences from the goat table.
$SQL = qq{SELECT schemaname, tablename FROM bucardo.goat WHERE reltype= '$type' AND db = '$db->{name}'};
my %hastable;
for my $row (@{$dbh->selectall_arrayref($SQL)}) {
$hastable{$row->[0]}{$row->[1]}++;
}
## Do we have a herd request? Process it if so
my $herd = '';
my $addtoherd;
if ($bcargs->{herd}) {
$herd = $bcargs->{herd};
$SQL = 'SELECT 1 FROM bucardo.herd WHERE name = ?';
my $herdcheck = $dbh->prepare($SQL);
$count = $herdcheck->execute($herd);
$herdcheck->finish();
if ($count < 1) {
print "Creating relgroup: $herd\n";
$SQL = 'INSERT INTO bucardo.herd(name) VALUES (?)';
$herdcheck = $dbh->prepare($SQL);
$herdcheck->execute($herd);
}
else {
$VERBOSE >= 1 and warn "Relgroup already exists: $herd\n";
}
$SQL = 'INSERT INTO bucardo.herdmap(herd,goat) VALUES (?,?)';
$addtoherd = $dbh->prepare($SQL);
}
## Get ready to add tables or sequences to the goat table
$SQL = q{INSERT INTO bucardo.goat (db,schemaname,tablename,reltype};
$SQL .= exists $bcargs->{makedelta} ? ',makedelta) VALUES (?,?,?,?,?)' : ') VALUES (?,?,?,?)';
my $addtable = $dbh->prepare($SQL);
## Walk through all returned tables from the remote database
my %count = (seenit => 0, added => 0);
my (%old, %new, %fail, $id);
for my $row (@{$sth->fetchall_arrayref()}) {
my ($S,$T) = @$row;
my $tinfo;
## Do we already have this one?
if (exists $hastable{$S}{$T}) {
$VERBOSE >= 2 and warn "Skipping $type already in relation: $S.$T\n";
$count{seenit}++;
$old{$S}{$T} = 1;
if ($herd) {
## In case this is not already in the herd, grab the id and jump down
$SQL = 'SELECT * FROM goat WHERE db=? AND schemaname=? AND tablename=? AND reltype=?';
$sth = $dbh->prepare($SQL);
$count = $sth->execute($db->{name},$S,$T,$type);
if ($count < 1) {
die qq{Could not find $type $S.$T in database "$db->{name}"!\n};
}
$tinfo = $sth->fetchall_arrayref({})->[0];
$id = $tinfo->{id};
goto HERD;
}
next;
}
$VERBOSE >= 2 and warn "Attempting to add relation $S.$T\n";
## We want a savepoint as we may retract the addition (e.g. no pkey and pkonly specified)
$dbh->do('SAVEPOINT newtable');
eval {
my @arg = ($db->{name},$S,$T,$type);
push @arg => $bcargs->{makedelta} if exists $bcargs->{makedelta};
$count = $addtable->execute(@arg);
};
if ($@) {
warn "$@\n";
if ($@ =~ /prepared statement.+already exists/) {
warn "This message usually indicates you are using pgbouncer\n";
warn "You can probably fix this problem by running:\n";
warn "$progname update db $db->{name} server_side_prepares=false\n";
warn "Then retry your command again\n\n";
}
exit 1;
}
if ($count != 1) {
$addtable->finish();
warn "Failed to add $type relation $S.$T!\n";
$fail{$S}{$T} = 1;
next;
}
$SQL = q{SELECT currval('bucardo.goat_id_seq')};
$id = $dbh->selectall_arrayref($SQL)->[0][0];
$VERBOSE >= 2 and warn "ID of new table $S.$T is $id\n";
## Grab it back from the database
$SQL = 'SELECT * FROM goat WHERE id = ?';
$sth = $dbh->prepare($SQL);
$sth->execute($id);
$tinfo = $sth->fetchall_arrayref({})->[0];
## If it has no primary key and pkonly is set, abandon this change
if ($bcargs->{pkonly} and 'table' eq $type and ! length $tinfo->{pkey}) {
$VERBOSE >= 1 and warn "Not adding table $S.$T: no pkey\n";
$dbh->do('ROLLBACK TO newtable');
next;
}
$count{added}++;
$new{$S}{$T} = 1;
HERD:
if ($herd) {
## Need to check again as the previous check above was only for brand new tables
if ($bcargs->{pkonly} and 'table' eq $type and ! length $tinfo->{pkey}) {
$VERBOSE >= 1 and warn "Not adding table $S.$T to relgroup: no pkey\n";
}
else {
$SQL = 'SELECT 1 FROM herdmap WHERE herd=? AND goat = ?';
$sth = $dbh->prepare($SQL);
$count = $sth->execute($herd, $id);
if ($count < 1) {
$addtoherd->execute($herd, $id);
print "Added $type $S.$T to relgroup $herd\n";
}
}
}
}
## Disconnect from the remote database
$dbh2->disconnect();
if ($VERBOSE >= 1) {
if (%new) {
print "New ${type}s:\n";
for my $s (sort keys %new) {
for my $t (sort keys %{$new{$s}}) {
print " $s.$t\n";
}
}
}
if (%fail) {
print "Failed to add ${type}s:\n";
for my $s (sort keys %fail) {
for my $t (sort keys %{$fail{$s}}) {
print " $s.$t\n";
}
}
}
}
my $message = "New ${type}s added: $count{added}\n";
if ($count{seenit}) {
$message .= "Already added: $count{seenit}\n";
}
return $message;
} ## end of add_all_goats
sub remove_customcode {
## Usage: remove customcode name [name2 name3 ...]
## Arguments: none (uses nouns)
## Returns: never, exits
my $doc_section = 'remove';
usage_exit($doc_section) unless @nouns;
## Make sure all named codes exist
my $code = $global{cc};
for my $name (@nouns) {
if (! exists $code->{$name}) {
die qq{No such code: $name\n};
}
}
$SQL = 'DELETE FROM bucardo.customcode WHERE name = ?';
$sth = $dbh->prepare($SQL);
for my $name (@nouns) {
eval {
$sth->execute($name);
};
if ($@) {
die qq{Could not delete customcode "$name"\n$@\n};
}
}
for my $name (@nouns) {
print qq{Removed customcode "$name"\n};
}
$dbh->commit();
exit 0;
} ## end of remove_customcode
sub clog {
## Output a message to stderr
## Arguments: one
## 1. Message
## Returns: undef
my $message = shift;
chomp $message;
warn "$message\n";
return;
} ## end of clog
sub schema_exists {
## Determine if a named schema exists
## Arguments: one
## 1. Schema name
## Returns: 0 or 1
my $schema = shift;
my $SQL = 'SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = ?';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($schema);
$sth->finish();
return $count < 1 ? 0 : 1;
} ## end of schema_exists
sub relation_exists {
## Determine if a named relation exists
## Arguments: two
## 1. Schema name
## 2. Relation name
## Returns: OID of the relation, or 0 if it does not exist
my ($schema,$name) = @_;
my $SQL = 'SELECT c.oid FROM pg_catalog.pg_class c, pg_catalog.pg_namespace n '.
'WHERE n.oid=c.relnamespace AND n.nspname = ? AND c.relname = ?';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($schema,$name);
if ($count == 1) {
return $sth->fetchall_arrayref()->[0][0];
}
$sth->finish();
return 0;
} ## end of relation_exists
sub domain_exists {
## Determine if a named domain exists
## Arguments: two
## 1. Schema name
## 2. Domain name
## Returns: 0 or 1
my ($schema,$name) = @_;
my $SQL =
q{SELECT 1 FROM pg_catalog.pg_type t }
. q{JOIN pg_namespace n ON (n.oid = t.typnamespace) }
. q{WHERE t.typtype = 'd' AND n.nspname = ? AND t.typname = ?};
my $sth = $dbh->prepare_cached($SQL);
$count = $sth->execute($schema,$name);
$sth->finish();
return $count < 1 ? 0 : 1;
} ## end of domain_exists
sub config_exists {
## Checks if a configuration settings exists
## Arguments: one
## 1. Name of the setting
## Returns: 0 or 1
my $name = shift;
my $SQL = 'SELECT 1 FROM bucardo.bucardo_config WHERE name = ?';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($name);
$sth->finish();
return $count < 1 ? 0 : 1;
} ## end of config_exists
sub constraint_exists {
## Determine if a named constraint exists
## Arguments: three
## 1. Schema name
## 2. Table name
## 3. Constraint name
## Returns: 0 or 1
my ($schema,$table,$constraint) = @_;
my $SQL = 'SELECT 1 FROM pg_catalog.pg_class c, pg_catalog.pg_namespace n, pg_catalog.pg_constraint o '.
'WHERE n.oid=c.relnamespace AND c.oid=o.conrelid AND n.nspname = ? AND c.relname = ? AND o.conname = ?';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($schema,$table,$constraint);
$sth->finish();
return $count < 1 ? 0 : 1;
} ## end of constraint_exists
sub column_exists {
## Determine if a named column exists
## Arguments: three
## 1. Schema name
## 2. Table name
## 3. Column name
## Returns: 0 or 1
my ($schema,$table,$column) = @_;
my $SQL = 'SELECT 1 FROM pg_catalog.pg_class c, pg_catalog.pg_namespace n, '.
'pg_catalog.pg_attribute a WHERE n.oid=c.relnamespace AND n.nspname = ? AND c.relname = ? '.
'AND a.attname = ? AND a.attrelid = c.oid';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($schema,$table,$column);
$sth->finish();
return $count < 1 ? 0 : 1;
} ## end of column_exists
sub trigger_exists {
## Determine if a named trigger exists
## Arguments: one
## 1. Trigger name
## Returns: 0 or 1
my $name = shift;
my $SQL = 'SELECT 1 FROM pg_catalog.pg_trigger WHERE tgname = ?';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($name);
$sth->finish();
return $count < 1 ? 0 : 1;
} ## end of trigger_exists
sub function_exists {
## Determine if a named function exists
## Arguments: three
## 1. Schema name
## 2. Function name
## 3. Function arguments (as one CSV string)
## Returns: MD5 of the function source if found, otherwise an empty string
my ($schema,$name,$args) = @_;
$name = lc $name;
$SQL = 'SELECT md5(prosrc) FROM pg_proc p, pg_language l '.
'WHERE p.prolang = l.oid AND proname = ? AND oidvectortypes(proargtypes) = ?';
$sth = $dbh->prepare($SQL);
$count = $sth->execute($name,$args);
if ($count < 1) {
$sth->finish();
return '';
}
return $sth->fetchall_arrayref()->[0][0];
} ## end of function_exists
sub column_default {
## Return the default value for a column in a table
## Arguments: three
## 1. Schema name
## 2. Table name
## 3. Column name
## Returns: default value if available, otherwise an empty string
my ($schema,$table,$column) = @_;
my $SQL = 'SELECT pg_get_expr(adbin,adrelid) FROM pg_catalog.pg_class c, pg_catalog.pg_namespace n, '.
'pg_catalog.pg_attribute a, pg_attrdef d '.
'WHERE n.oid=c.relnamespace AND n.nspname = ? AND c.relname = ? '.
'AND a.attname = ? AND a.attrelid = c.oid AND d.adnum = a.attnum AND d.adrelid = a.attrelid';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($schema,$table,$column);
if ($count < 1) {
$sth->finish();
return '';
}
return $sth->fetchall_arrayref()->[0][0];
} ## end of column_default
sub column_value {
## Return the value of a table's column
## Arguments: four
## 1. Schema name
## 2. Table name
## 3. Column name
## 4. Where clause
## Returns: value if available, otherwise an empty string
my ($schema,$table,$column,$where) = @_;
my $SQL = "SELECT $column FROM $schema.$table WHERE $where";
return $dbh->selectall_arrayref($SQL)->[0][0];
} ## end of column_value
sub column_type {
## Return the data type of a table column
## Arguments: three
## 1. Schema name
## 2. Table name
## 3. Column name
## Returns: data type if available, otherwise an empty string
my ($schema,$table,$column) = @_;
my $SQL = 'SELECT pg_catalog.format_type(a.atttypid, a.atttypmod) '.
'FROM pg_catalog.pg_class c, pg_catalog.pg_namespace n, '.
'pg_catalog.pg_attribute a '.
'WHERE n.oid=c.relnamespace AND n.nspname = ? AND c.relname = ? '.
'AND a.attname = ? AND a.attrelid = c.oid';
my $sth = $dbh->prepare_cached($SQL);
my $count = $sth->execute($schema,$table,$column);
if ($count eq '0E0') {
$sth->finish();
return '';
}
return $sth->fetchall_arrayref()->[0][0];
} ## end of column_type
sub constraint_definition {
## Return the definition for a constraint
## Arguments: one
## 1. Constraint name
## Returns: definition if found, otherwise an empty string
my $name = shift;
my $SQL = qq{SELECT pg_get_constraintdef(oid,true) FROM pg_constraint WHERE conname = '$name'};
my $def = $dbh->selectall_arrayref($SQL)->[0][0];
## Nothing found? Just return an empty string
return '' if ! defined $def;
## Do some cleanups to standardize across versions and match bucardo.schema cleanly
$def =~ s/\((\(.+\))\)/$1/;
$def =~ s/= ANY \(ARRAY\[(.+)\]\)/IN ($1)/;
$def =~ s/<> ALL \(ARRAY\[(.+)\]\)/NOT IN ($1)/;
$def =~ s/::text//g;
$def =~ s/(\w+) ~ '/$1 ~ E'/g;
$def =~ s/CHECK \(\((\w+)\) <>/CHECK ($1 <>/;
return $def;
} ## end of constraint_definition
sub table_comment {
## Return the comment of a table
## Arguments: two
## 1. Schema name
## 2. Table name
## Returns: comment if available, otherwise an empty string
my ($schema,$relation) = @_;
my $SQL = q{SELECT description FROM pg_description WHERE objoid = (}
. q{ SELECT c.oid FROM pg_class c JOIN pg_namespace n ON (n.oid = c.relnamespace)}
. q{ WHERE n.nspname = ? AND c.relname = ?)};
my $sth = $dbh->prepare($SQL);
$count = $sth->execute($schema,$relation);
if ($count < 1) {
$sth->finish();
return '';
}
return $sth->fetchall_arrayref()->[0][0];
} ## end of table_comment
sub domain_comment {
## Return the comment of a domain
## Arguments: two
## 1. Schema name
## 2. Domain name
## Returns: comment if available, otherwise an empty string
my ($schema,$relation) = @_;
my $SQL = q{SELECT description FROM pg_description WHERE objoid = (}
. q{ SELECT t.oid FROM pg_type t JOIN pg_namespace n ON (n.oid = t.typnamespace)}
. q{ WHERE t.typtype = 'd' AND n.nspname = ? AND t.typname = ?)};
my $sth = $dbh->prepare($SQL);
$count = $sth->execute($schema,$relation);
if ($count < 1) {
$sth->finish();
return '';
}
return $sth->fetchall_arrayref()->[0][0];
} ## end of domain_comment
sub find_bucardo_schema {
## Locate the best bucardo.schema file and return a file handle and name for it
## Arguments: none
## Returns: file handle and location of the file
my $fh;
## Start by checking the current directory
my $schema_file = 'bucardo.schema';
return ($fh, $schema_file) if open $fh, '<', $schema_file;
## Check for a symlink path back to the right directory
if (-l $progname) {
my $dir = dirname( readlink $progname );
$schema_file = File::Spec->catfile( $dir, 'bucardo.schema' );
return ($fh, $schema_file) if open $fh, '<', $schema_file;
}
## Try /usr/local/share/bucardo
$schema_file = '/usr/local/share/bucardo/bucardo.schema';
return ($fh, $schema_file) if open $fh, '<', $schema_file;
## Try /usr/share/bucardo
$schema_file = '/usr/share/bucardo/bucardo.schema';
return ($fh, $schema_file) if open $fh, '<', $schema_file;
die "Could not find the bucardo.schema file!\n";
} ## end of find_bucardo_schema
sub table_definition {
## Pull the complete table definition from the bucardo.schema file
## Returns an arrayref of sequences, and the textual table def
## Arguments: one
## 1. Name of the table
## Returns: arrayref of sequences used, table definition
my $name = shift;
my $def = '';
my ($fh, $schema_file) = find_bucardo_schema();
my @seq;
while (<$fh>) {
if (!$def) {
if (/^CREATE TABLE $name/) {
$def .= $_;
}
}
else {
$def .= $_;
last if /^\);/;
}
}
close $fh or die qq{Could not close "$schema_file": $!\n};
while ($def =~ /nextval\('(.+?)'/g) {
push @seq => $1;
}
if (! length($def)) {
die "Could not find the table definition for $name\n";
}
return \@seq, $def;
} ## end of table_definition
sub generate_clause {
## Generate a snippet of SQL for a WHERE clause
## Arguments: one
## 1. Hashref of information
## Returns: new clause
my $arg = shift or die;
return '' if ! $arg->{items} or ! defined $arg->{items}[0];
my $col = $arg->{col} or die;
my $items = $arg->{items};
my ($NOT,$NOTR) = ('','');
if (exists $arg->{not}) {
$NOT = 'NOT ';
$NOTR = '!';
}
my $andor = exists $arg->{andor} ? uc($arg->{andor}) : $NOT ? 'AND' : 'OR';
my (@oneitem,@itemlist);
for my $name (@{$items}) {
$name =~ s/^\s*(.+?)\s*$/$1/;
## Break into schema and relation
my $schema = '';
if ($col eq 'tablename' and $name =~ s/(.+\w)\.(\w.+)/$2/) {
$schema = $1;
}
my $one = 1;
## Contains:
if ($name =~ s/^\*(.+)\*$/$1/) {
push @oneitem => "$col ${NOTR}~ " . qquote($1);
}
## Starts with:
elsif ($name =~ s/^\*(.+)/$1/) {
push @oneitem => "$col ${NOTR}~ " . qquote("$1\$");
}
## Ends with:
elsif ($name =~ s/(.+)\*$/$1/) {
push @oneitem => "$col ${NOTR}~ " . qquote("^$1");
}
else {
push @itemlist => qquote($name);
$one = 0;
}
if ($schema) {
my $col2 = 'schemaname';
my $old = $one ? pop @oneitem : pop @itemlist;
if ($schema =~ s/^\*(.+)\*$/$1/) {
push @oneitem => "($old AND $col2 ${NOTR}~ " . qquote($1) . ')';
}
elsif ($schema =~ s/^\*(.+)/$1/) {
push @oneitem => "($old AND $col2 ${NOTR}~ " . qquote("$1\$") . ')';
}
elsif ($schema =~ s/(.+)\*$/$1/) {
push @oneitem => "($old AND $col2 ${NOTR}~ " . qquote("^$1") . ')';
}
else {
push @oneitem => "($col = $old AND $col2 = " . qquote($schema) . ')';
}
}
}
if (@itemlist) {
my $list = sprintf '%s %s%s (%s)', $col, $NOT, 'IN', (join ',' => @itemlist);
push @oneitem => $list;
}
my $SQL = join " $andor " => @oneitem;
return $SQL;
} ## end of generate_clause
sub qquote {
## Quick SQL quoting
## Arguments: one
## 1. String to be quoted
## Returns: modified string
my $thing = shift;
$thing =~ s/'/''/g;
return qq{'$thing'};
} ## end of qquote
sub upgrade {
## Make upgrades to an existing Bucardo schema to match the current version
## Arguments: none
## Returns: never, exits
## Ensure the bucardo.schema file is available and the correct version
my ($fh, $schema_file) = find_bucardo_schema();
my $schema_version = 0;
while (<$fh>) {
if (/\-\- Version (\d+\.\d+\.\d+)/) {
$schema_version = $1;
last;
}
}
if (! $schema_version) {
die qq{Could not find version number in the file "$schema_file"!\n};
}
if ($schema_version ne $VERSION) {
die qq{Cannot continue: bucardo is version $VERSION, but $schema_file is version $schema_version\n};
}
$dbh->do(q{SET escape_string_warning = 'OFF'});
if ($dbh->{pg_server_version} >= 80200) {
$dbh->do(q{SET standard_conforming_strings = 'ON'});
}
my $changes = 0;
## Quick sanity to make sure we don't try to cross the 4/5 boundary
if (!relation_exists('bucardo', 'syncrun')) {
print "Sorry, but Bucardo version 4 cannot be upgraded to version 5\n";
print "You will have to recreate your information (dbs, syncs, etc.)\n";
exit 1;
}
## Make sure the upgrade_log table is in place
if (!relation_exists('bucardo', 'upgrade_log')) {
my ($seqlist, $tabledef) = table_definition('bucardo.upgrade_log');
upgrade_and_log($tabledef,'CREATE TABLE bucardo.upgrade_log');
$dbh->commit();
}
my @old_sequences = (
'dbgroup_id_seq',
);
my @old_configs = (
'pidfile',
'upsert_attempts',
);
my @renamed_configs = (
['default_standard_conflict' => 'default_conflict_strategy'],
);
my @old_constraints = (
['bucardo', 'goat', 'goat_pkeytype_check'],
['bucardo', 'sync', 'sync_replica_allornone'],
['bucardo', 'sync', 'sync_disable_triggers_method'],
['bucardo', 'sync', 'sync_disable_rules_method'],
);
my @old_columns = (
['bucardo', 'dbmap', 'makedelta'],
['bucardo', 'sync', 'disable_rules'],
['bucardo', 'sync', 'disable_triggers'],
['bucardo', 'sync', 'makedelta'],
);
my @old_functions = (
['create_child_q', 'text'],
);
my @old_indexes = (
['bucardo', 'sync', 'sync_source_targetdb'],
['bucardo', 'sync', 'sync_source_targetgroup'],
);
my @old_views = (
'goats_in_herd',
);
my @new_columns = (
);
my @dropped_columns = (
['bucardo', 'sync', 'limitdbs'],
['bucardo', 'goat', 'customselect'],
['bucardo', 'sync', 'usecustomselect'],
['bucardo', 'sync', 'do_listen'],
['bucardo', 'customcode', 'getrows'],
);
my @altered_columns = (
['bucardo', 'goat', 'rebuild_index', 'BOOL2SMALLINT1'],
['bucardo', 'goat', 'schemaname', 'NO DEFAULT'],
['bucardo', 'sync', 'isolation_level', 'NO DEFAULT'],
['bucardo', 'sync', 'rebuild_index', 'BOOL2SMALLINT1'],
['bucardo', 'sync', 'standard_conflict', 'RENAME conflict_strategy'],
['bucardo', 'sync', 'ping', 'RENAME autokick'],
['bucardo', 'goat', 'ping', 'RENAME autokick'],
['bucardo', 'goat', 'standard_conflict', 'RENAME conflict_strategy'],
);
my @row_values = (
['bucardo_config','about',q{name = 'log_showtime'}, 1,
'Show timestamp in the log output? 0=off 1=seconds since epoch 2=scalar gmtime 3=scalar localtime'],
['bucardo_config', 'about', q{name = 'default_conflict_strategy'}, 1, 'Default conflict strategy for all syncs'],
);
my @drop_all_rules = (
);
## Drop all existing rules from a table:
for my $row (@drop_all_rules) {
my ($schema,$table) = @$row;
my $oid = relation_exists($schema,$table);
if (!$oid) {
warn "Could not find table $schema.$table to check!\n";
next;
}
$SQL = 'SELECT rulename FROM pg_catalog.pg_rewrite WHERE ev_class = ? ORDER BY rulename';
$sth = $dbh->prepare($SQL);
$count = $sth->execute($oid);
if ($count < 1) {
$sth->finish();
next;
}
for my $rule (map { $_->[0] } @{$sth->fetchall_arrayref()}) {
upgrade_and_log(qq{DROP RULE "$rule" ON $schema.$table});
clog "Dropped rule $rule on table $schema.$table";
$changes++;
}
}
## Drop any old views
for my $name (@old_views) {
next if !relation_exists('bucardo', $name);
upgrade_and_log("DROP VIEW $name");
clog "Dropped view $name";
$changes++;
}
## Drop any old sequences
for my $sequence (@old_sequences) {
next if !relation_exists('bucardo', $sequence);
upgrade_and_log("DROP SEQUENCE bucardo.$sequence");
clog "Dropped sequence: $sequence";
$changes++;
}
## Drop any old constraints
for my $con (@old_constraints) {
my ($schema, $table, $constraint) = @$con;
next if !constraint_exists($schema, $table, $constraint);
upgrade_and_log(qq{ALTER TABLE $schema.$table DROP CONSTRAINT "$constraint"});
clog "Dropped constraint $constraint ON $schema.$table";
$changes++;
}
## Parse the bucardo.schema file and verify the following types of objects exist:
## Functions, triggers, constraints, sequences, indexes, comments, and domains
my (@flist, @tlist, @ilist, @clist, @clist2, @slist, @tablelist, @comlist, @domlist, @collist);
my ($fname,$args,$fbody) = ('','','');
my ($tname,$tbody) = ('','');
my ($tablename,$tablebody) = ('','');
my ($altername,$alterbody,$alterstat) = ('','','');
seek $fh, 0, 0;
while (<$fh>) {
if ($fbody) {
if (/^(\$bc\$;)/) {
$fbody .= $1;
push @flist, [$fname, $args, $fbody];
$fbody = $fname = $args = '';
}
else {
$fbody .= $_;
}
next;
}
if ($tbody) {
$tbody .= $_;
if (/;/) {
push @tlist, [$tname, $tbody];
$tbody = $tname = '';
}
next;
}
if ($tablebody) {
$tablebody .= $_;
if (/^\s*CONSTRAINT\s+(\w+)\s+(.+?)\s*$/) {
my ($cname,$def) = ($1,$2);
$def =~ s/,$//;
$def =~ s/\bbucardo\.//;
push @clist2, [$tablename, $cname, $def];
}
elsif (/^\s+([a-z_]+)\s+([A-Z]+)\s*(NOT)? NULL(.*)/) {
my ($colname,$coltype,$isnull,$extra,$default) = ($1, $2, $3 ? 1 : 0, $4, undef);
if ($extra =~ /DEFAULT\s+([^,]+)/) {
$default = $1;
}
push @collist, ['bucardo', $tablename, $colname, $_, $default];
}
elsif (/;/) {
push @tablelist, [$tablename, $tablebody];
$tablebody = $tablename = '';
}
else {
die qq{Could not parse table definition: invalid column at line $. ($_)\n};
}
next;
}
if ($altername) {
$alterbody =~ s/\s+$//;
$alterbody ? s/^\s+/ / : s/^\s+//;
s/\s+$/ /;
$alterbody .= $_;
$alterstat .= $_;
if ($alterbody =~ s/;\s*$//) {
push @clist, [$altername->[0], $altername->[1], $alterbody, $alterstat];
$alterbody = $altername = $alterstat = '';
}
next;
}
if (/^CREATE (?:OR REPLACE )?FUNCTION\s+bucardo\.(.+?\))/) {
$fname = $1;
$fbody .= $_;
$fname =~ s/\((.*)\)// or die "No args found for function: $_\n";
$args = $1;
$args =~ s/,(\S)/, $1/g;
next;
}
if (/^CREATE TRIGGER (\S+)/) {
$tname = $1;
$tbody .= $_;
next;
}
if (/^CREATE TABLE bucardo\.(\w+)/) {
$tablename = $1;
$tablebody .= $_;
next;
}
if (/^CREATE (UNIQUE )?INDEX (\S+)/) {
push @ilist, [$1, $2, $_];
next;
}
if (/^ALTER TABLE bucardo\.(\S+)\s+ADD CONSTRAINT\s*(\S+)\s*(\S*.*)/) {
$altername = [$1,$2];
$alterbody = $3 || '';
$alterstat = $_;
next;
}
if (/^CREATE SEQUENCE bucardo\.(\w+)/) {
push @slist, [$1, $_];
next;
}
if (/^COMMENT ON (\w+) (\w+)\.(\w+) IS \$\$(.+)\$\$/) {
push @comlist, [lc $1, $2, $3, $4, $_];
next;
}
if (/^CREATE DOMAIN bucardo\.(\w+) (.+)/) {
push @domlist, [$1, $2];
next;
}
}
## Add any new domains, verify existing ones
for my $row (@domlist) {
my ($name,$def) = @$row;
next if domain_exists('bucardo', $name);
upgrade_and_log("CREATE DOMAIN bucardo.$name $def");
clog("Created domain: $name");
$changes++;
}
## Check for any added sequences
for my $row (@slist) {
my ($sname,$body) = @$row;
next if relation_exists('bucardo', $sname);
upgrade_and_log($body);
clog "Created sequence $sname";
$changes++;
}
## Check for any added tables
for my $row (@tablelist) {
my ($name,$body) = @$row;
next if relation_exists('bucardo', $name);
upgrade_and_log($body);
clog "Created table $name";
$changes++;
}
## Add new columns as needed from the schema
for my $row (@collist) {
my ($schema,$table,$column,$definition) = @$row;
next if column_exists($schema, $table, $column);
$definition =~ s/\-\-.+$//;
$definition =~ s/,\s*$//;
$definition =~ s/\s+/ /g;
upgrade_and_log("ALTER TABLE $schema.$table ADD COLUMN $definition");
clog "Created column: $schema.$table.$column";
$changes++;
}
## Add specific new columns as needed
for my $row (@new_columns) {
my ($schema,$table,$column,$def) = @$row;
next if column_exists($schema, $table, $column);
$def =~ s/\s+/ /g;
upgrade_and_log("ALTER TABLE $schema.$table ADD COLUMN $def");
clog "Created column: $schema.$table.$column";
$changes++;
}
## Drop columns as needed.
for my $row (@dropped_columns) {
my ($schema,$table,$column) = @$row;
next unless column_exists($schema, $table, $column);
upgrade_and_log("ALTER TABLE $schema.$table DROP COLUMN $column");
clog "Dropped column: $schema.$table.$column";
$changes++;
}
## Change any altered columns
for my $row (@altered_columns) {
my ($schema,$table,$column,$change) = @$row;
next if ! column_exists($schema, $table, $column);
if ($change eq 'NO DEFAULT') {
my $def = column_default($schema, $table, $column);
next if !$def;
upgrade_and_log("ALTER TABLE $schema.$table ALTER COLUMN $column DROP DEFAULT");
clog "Removed DEFAULT ($def) from $schema.$table.$column";
$changes++;
}
elsif ($change =~ /^RENAME\s+(\w+)/) {
my $newname = $1;
next if column_exists($schema, $table, $newname);
upgrade_and_log("ALTER TABLE $schema.$table RENAME COLUMN $column TO $newname");
clog("Renamed $schema.$table.$column to $newname");
$changes++;
}
elsif ($change =~ /^DEFAULT\s+(.+)/) {
my $newname = $1;
my $oldname = column_default($schema, $table, $column);
next if $newname eq $oldname;
upgrade_and_log("ALTER TABLE $schema.$table ALTER COLUMN $column SET DEFAULT $newname");
clog("Changed DEFAULT on $schema.$table.$column to $newname");
$changes++;
}
elsif ($change =~ /BOOL2SMALLINT(\d)/) {
my $defval = $1;
my $oldtype = column_type($schema, $table, $column);
next if $oldtype eq 'smallint';
upgrade_and_log("ALTER TABLE $schema.$table ALTER COLUMN $column DROP DEFAULT");
upgrade_and_log("ALTER TABLE $schema.$table ALTER COLUMN $column TYPE smallint "
. "USING CASE WHEN $column IS NULL OR $column IS FALSE THEN 0 ELSE $defval END");
upgrade_and_log("ALTER TABLE $schema.$table ALTER COLUMN $column SET DEFAULT 0");
clog("Changed type of $schema.$table.$column to smallint");
$changes++;
}
else {
die qq{Do not know how to handle altered column spec of "$change"};
}
}
## Change any column defaults
## Add new columns as needed from the schema
for my $row (@collist) {
my ($schema,$table,$column,$definition,$default) = @$row;
next if ! column_exists($schema, $table, $column) or ! defined $default;
my $olddefault = column_default($schema, $table, $column);
$olddefault =~ s/::text//;
$olddefault =~ s/::regclass//;
$olddefault =~ s/'00:00:00'::interval/'0 seconds'::interval/;
next if $olddefault eq $default;
upgrade_and_log("ALTER TABLE $schema.$table ALTER COLUMN $column SET DEFAULT $default");
clog "Set new default for $schema.$table.$column: $default";
$changes++;
}
## Drop any old columns
for my $row (@old_columns) {
my ($schema,$table,$column) = @$row;
next if !column_exists($schema, $table, $column);
upgrade_and_log("ALTER TABLE $schema.$table DROP COLUMN $column");
clog "Dropped column: $schema.$table.$column";
$changes++;
}
## Drop any old indexes
for my $row (@old_indexes) {
my ($schema,$table,$name) = @$row;
next if !relation_exists($schema, $name);
upgrade_and_log("DROP INDEX $name");
clog "Dropped index $name";
$changes++;
}
## Drop any old functions
for my $row (@old_functions) {
my ($name, $largs) = @$row;
next if ! function_exists('bucardo', $name, $largs);
clog "Dropped function $name($largs)";
upgrade_and_log(qq{DROP FUNCTION bucardo."$name"($largs)});
$changes++;
}
## Drop any old config items
for my $name (@old_configs) {
next if ! config_exists($name);
clog "Removed old bucardo_config name: $name";
upgrade_and_log(qq{DELETE FROM bucardo.bucardo_config WHERE name = '$name'});
$changes++;
}
## Rename configs.
for my $names (@renamed_configs) {
next if config_exists($names->[1]);
clog "Renamed bucardo_config $names->[0] to $names->[1]";
upgrade_and_log(qq{
UPDATE bucardo.bucardo_config
SET name = '$names->[1]'
WHERE name = '$names->[0]'
});
$changes++;
}
## Special case config renaming
if (config_exists('bucardo_current_version')) {
## was version and current_version; became initial_version and version
clog('Renaming bucardo_current_version to bucardo_version, and bucardo_version to bucardo_initial_version');
upgrade_and_log(q{UPDATE bucardo.bucardo_config SET name = 'bucardo_initial_version' WHERE name = 'bucardo_version'});
upgrade_and_log(q{UPDATE bucardo.bucardo_config SET name = 'bucardo_version' WHERE name = 'bucardo_current_version'});
}
## Check for any new config items
$SQL = 'SELECT setting FROM bucardo.bucardo_config WHERE lower(name) = ?';
my $cfgsth = $dbh->prepare($SQL);
$SQL = 'INSERT INTO bucardo.bucardo_config(name,setting,about) VALUES (?,?,?)';
my $newcfg = $dbh->prepare($SQL);
my %config;
my $inside = 0;
seek $fh, 0, 0;
while (<$fh>) {
chomp;
if (!$inside) {
if (/^WITH DELIMITER/) {
$inside = 1;
}
next;
}
if (/^\\/) {
$inside = 0;
next;
}
## Scoop
my ($name,$setting,$about) = split /\|/ => $_;
$config{$name} = [$setting,$about];
$count = $cfgsth->execute($name);
$cfgsth->finish();
if ($count eq '0E0') {
clog "Added new bucardo_config name: $name";
$changes++;
$newcfg->execute($name,$setting,$about);
}
}
close $fh or die qq{Could not close file "$file": $!\n};
## Apply any specific row changes
for my $row (@row_values) {
my ($table,$column,$where,$force,$value) = @$row;
my $val = column_value('bucardo',$table,$column,$where);
if (!defined $val) {
die "Failed to find $table.$column where $where!\n";
}
next if $val eq $value;
$SQL = sprintf "UPDATE bucardo.$table SET $column=%s WHERE $where",
$dbh->quote($value);
upgrade_and_log($SQL);
clog "New value set for bucardo.$table.$column WHERE $where";
$changes++;
}
$SQL = 'SELECT pg_catalog.md5(?)';
my $md5sth = $dbh->prepare($SQL);
for my $row (@flist) {
my ($name,$arg,$body) = @$row;
next if $name =~ /plperlu_test/;
my $oldbody = function_exists('bucardo',$name,$arg);
if (!$oldbody) {
upgrade_and_log($body,"CREATE FUNCTION $name($arg)");
clog "Added function $name($arg)";
$changes++;
next;
}
my $realbody = $body;
$realbody =~ s/.*?\$bc\$(.+)\$bc\$;/$1/sm;
$md5sth->execute($realbody);
my $newbody = $md5sth->fetchall_arrayref()->[0][0];
next if $oldbody eq $newbody;
$body =~ s/^CREATE FUNCTION/CREATE OR REPLACE FUNCTION/;
(my $short = $body) =~ s/^(.+?)\n.*/$1/s;
$dbh->do('SAVEPOINT bucardo_upgrade');
eval { upgrade_and_log($body,$short); };
if ($@) {
$dbh->do('ROLLBACK TO bucardo_upgrade');
(my $dropbody = $short) =~ s/CREATE OR REPLACE/DROP/;
$dropbody .= ' CASCADE';
upgrade_and_log($dropbody);
upgrade_and_log($body,$short);
}
else {
$dbh->do('RELEASE bucardo_upgrade');
}
clog "Updated function: $name($arg)";
$changes++;
}
## Check for any added triggers
for my $row (@tlist) {
my ($name,$body) = @$row;
next if trigger_exists($name);
upgrade_and_log($body);
clog "Created trigger $name";
$changes++;
}
## Check for any added indexes
for my $row (@ilist) {
my ($uniq,$name,$body) = @$row;
next if relation_exists('bucardo',$name);
upgrade_and_log($body);
clog "Created index $name";
$changes++;
}
## Check for any added constraints
for my $row (@clist) {
my ($tcname,$cname,$cdef,$body) = @$row;
if (! constraint_exists('bucardo', $tcname, $cname)) {
upgrade_and_log($body);
clog "Created constraint $cname on $tcname";
$changes++;
next;
}
## Clean up the constraint to make it match what comes back from the database:
$cdef =~ s/','/', '/g;
my $condef = constraint_definition($cname);
$condef =~ s{\\}{\\\\}g;
if ($condef ne $cdef) {
upgrade_and_log("ALTER TABLE $tcname DROP CONSTRAINT $cname");
upgrade_and_log("ALTER TABLE $tcname ADD CONSTRAINT $cname $cdef");
clog "Altered constraint $cname on $tcname";
clog "OLD: $condef\nNEW: $cdef\n";
$changes++;
}
}
## Check that any bare constraints (e.g. foreign keys) are unchanged
for my $row (@clist2) {
my ($tcname,$cname,$cdef) = @$row;
my $condef = constraint_definition($cname);
next if ! $condef or $condef eq $cdef;
if ($condef and $condef ne $cdef) {
upgrade_and_log("ALTER TABLE $tcname DROP CONSTRAINT $cname");
}
upgrade_and_log("ALTER TABLE $tcname ADD CONSTRAINT $cname $cdef");
my $action = $condef ? 'Altered' : 'Added';
clog "$action constraint $cname on $tcname";
$changes++;
}
## Check that object comments exist and match
for my $row (@comlist) {
my ($type,$schema,$relation,$comment,$full) = @$row;
my $current_comment =
$type eq 'table' ? table_comment($schema,$relation)
: $type eq 'domain' ? domain_comment($schema,$relation)
: 'Unknown type';
if ($current_comment ne $comment) {
upgrade_and_log($full);
clog (length $current_comment
? "Changed comment on $type $schema.$relation"
: "Added comment for $type $schema.$relation");
$changes++;
}
}
## The freezer.q_staging table is no longer needed, but we must empty it before dropping
if (relation_exists('freezer','q_staging')) {
upgrade_and_log('INSERT INTO freezer.master_q SELECT * FROM freezer.q_staging');
upgrade_and_log('DROP TABLE freezer.q_staging');
clog 'Dropped deprecated table freezer.q_staging';
$changes++;
}
## Make sure bucardo_config has the new schema version
$count = $cfgsth->execute('bucardo_version');
if ($count eq '0E0') {
$cfgsth->finish();
warn "Weird: could not find bucardo_version in the bucardo_config table!\n";
}
else {
my $curval = $cfgsth->fetchall_arrayref()->[0][0];
if ($curval ne $schema_version) {
$SQL = 'UPDATE bucardo.bucardo_config SET setting = ? WHERE name = ?';
my $updatecfg = $dbh->prepare($SQL);
$updatecfg->execute($schema_version, 'bucardo_version');
clog "Set bucardo_config.bucardo_version to $schema_version";
$changes++;
}
}
## Update default config settings per the parsed config
$dbh->do('CREATE TEMPORARY TABLE stage_bucardo_config (name text primary key, setting text)');
$dbh->do('COPY stage_bucardo_config (name,setting) FROM STDIN');
while (my ($name,$rec) = each %config) {
my $set = $rec->[0];
$dbh->pg_putcopydata("$name\t$set\n");
}
$dbh->pg_putcopyend;
$dbh->do('UPDATE bucardo_config c SET defval = s.setting FROM stage_bucardo_config s WHERE c.name = s.name');
## Run the magic updater
$SQL = 'SELECT bucardo.magic_update()';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $message = $sth->fetchall_arrayref()->[0][0];
if (length $message) {
clog $message;
$changes++;
}
if ($changes) {
printf "Okay to commit $changes %s? ", $changes==1 ? 'change' : 'changes';
exit if !~ /Y/i;
$dbh->commit();
print "Changes have been commited\n";
}
else {
print "No schema changes were needed\n";
exit 1;
}
print "Don't forget to run '$progname validate all' as well: see the UPGRADE file for details\n";
exit 0;
} ## end of upgrade
sub upgrade_and_log {
## Put an entry in the bucardo.upgrade_log table
## Arguments: two
## 1. Type of action
## 2. Optional message
## Returns: undef
my $action = shift;
my $short = shift || $action;
eval {
$dbh->do($action);
};
if ($@) {
my $line = (caller)[2];
die "From line $line, action $action\n$@\n";
}
$SQL = 'INSERT INTO bucardo.upgrade_log(action,version,summary) VALUES (?,?,?)';
eval {
$sth = $dbh->prepare($SQL);
$sth->execute($action,$VERSION,$short);
};
if ($@) {
my $line = (caller)[2];
die "From line $line, insert to upgrade_log failed\n$@\n";
}
return;
} ## end of upgrade_and_log
sub usage_exit {
## Grab the help string for a specific item
## Arguments: one
## 1. The thing we want help on
## Returns: nothing
my $name = shift or die;
my $exitval = defined $_[0] ? shift : 1;
if ($name =~ m{/!}) {
# Bug in Pod::Usage prevents these from working properly. Force it
# to use Pod::PlainText.
# https://rt.perl.org/rt3//Public/Bug/Display.html?id=115534
require Pod::Usage;
require Pod::PlainText;
unshift @Pod::Usage::ISA => 'Pod::PlainText';
}
_pod2usage(
'-sections' => "COMMAND DETAILS/$name",
'-exitval' => $exitval,
);
return;
} ## end of usage_exit
sub connect_database {
## Connect to a datbase and return a dbh
## Arguments: one
## 1. Hashref of connection arguments (optional)
## Returns: database handle
my $dbh2;
my $opt = shift || {};
## If given just a name, transform to a hash
if (! ref $opt) {
$opt = { name => $opt };
}
if (! exists $DB->{$opt->{name}}) {
die qq{Unknown database "$opt->{name}": try bucardo list dbs\n};
}
if (exists $opt->{name}) {
$SQL = qq{SELECT bucardo.db_getconn('$opt->{name}')};
my $conn = $dbh->selectall_arrayref($SQL)->[0][0];
my ($type,$dsn,$user,$pass) = split /\n/ => $conn;
if ($type ne 'postgres') {
die "Cannot return a handle for database type $type\n";
}
$dsn =~ s/DSN://;
eval {
$dbh2 = DBI->connect_cached($dsn, $user, $pass, {AutoCommit=>0,RaiseError=>1,PrintError=>0});
};
if ($@) {
## The bucardo user may not exist yet.
if ($user eq 'bucardo' and $@ =~ /FATAL/ and $@ =~ /bucardo/) {
$user = 'postgres';
$dbh2 = DBI->connect_cached($dsn, $user, $pass, {AutoCommit=>0,RaiseError=>1,PrintError=>0});
$dbh2->do('CREATE USER bucardo SUPERUSER');
$dbh2->commit();
$user = 'bucardo';
$dbh2 = DBI->connect_cached($dsn, $user, $pass, {AutoCommit=>0,RaiseError=>1,PrintError=>0});
}
else {
die $@;
}
}
}
return $dbh2;
} ## end of connect_database
sub config {
## View or change a value inside the bucardo_config table
## Arguments: none, reads nouns
## Returns: never, exits
my $setusage = "Usage: $progname set setting=value [setting=value ...]\n";
## Allow for old syntax
if ($verb eq 'config') {
## Plain old "config" means the same as "show all"
if (!@nouns) {
@nouns = ('show','all');
}
$verb = shift @nouns;
}
if (!@nouns) {
$verb eq 'set' and die $setusage;
die "Usage: $progname show [settting2 ...]\n";
}
$SQL = 'SELECT * FROM bucardo.bucardo_config';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $config = $sth->fetchall_hashref('name');
if ($verb eq 'show') {
my $all = $nouns[0] =~ /\ball\b/i ? 1 : 0;
my $changed = $nouns[0] =~ /\bchanged\b/i ? 1 : 0;
my $maxsize = 3;
for my $s (keys %$config) {
next if
($changed && $config->{$s}{setting} eq $config->{$s}{defval})
|| (! $all and ! $changed and ! grep { $s =~ /$_/i } @nouns);
$maxsize = length $s if length $s > $maxsize;
}
for my $s (sort keys %$config) {
next if
($changed && $config->{$s}{setting} eq $config->{$s}{defval})
|| (! $all and ! $changed and ! grep { $s =~ /$_/i } @nouns);
printf "%-*s = %s\n", $maxsize, $s, $config->{$s}{setting};
}
exit 1;
}
$SQL = 'UPDATE bucardo.bucardo_config SET setting = ? WHERE name = ?';
$sth = $dbh->prepare($SQL);
my %allow_mixed_case_config = map { $_ => 1 } qw(
log_conflict_file
warning_file
email_debug_file
flatfile_dir
reason_file
stats_script_url
stopfile
log_timer_format
);
for my $noun (@nouns) {
$noun =~ /(\w+)=(.+)/ or die $setusage;
my $setting = lc $1;
my $val = $allow_mixed_case_config{$setting} ? $2 : lc $2;
if (! exists $config->{$setting}) {
die qq{Unknown setting "$setting"\n};
}
## Sanity checks
if ($setting eq 'log_level') {
if ($val !~ /^(?:terse|normal|verbose|debug)$/oi) {
die "Invalid log_level, must be terse, normal, verbose, or debug\n";
}
}
if ($setting eq 'default_standard_conflict' || $setting eq 'default_conflict_strategy') {
if ($val !~ /^(?:source|target|skip|random|latest|none)$/oi) {
## FIXME
#die "Invalid default_standard_conflict, must be none, source, target, skip, random, or latest\n";
}
if ($val =~ /none/i) {
$val = '';
}
$setting = 'default_conflict_strategy';
}
$sth->execute($val,$setting);
$QUIET or print qq{Set "$setting" to "$val"\n};
}
$dbh->commit();
exit 0;
} ## end of config
sub message {
## Add a message to the Bucardo logs, via the bucardo_log_message table
## Note: If no MCP processes are listening, the message will hang out until an MCP processes it
## Arguments: none (reads in nouns)
## Returns: never, exits
my $doc_section = 'message';
usage_exit($doc_section) unless length $nouns;
$SQL = 'INSERT INTO bucardo.bucardo_log_message(msg) VALUES (?)';
$sth = $dbh->prepare($SQL);
$sth->execute($nouns);
$dbh->commit();
$VERBOSE and print "Added message\n";
exit 0;
} ## end of message
sub db_get_notices {
## Gather up and return a list of asynchronous notices received since the last check
## Arguments: one
## 1. Database handle
## Returns: arrayref of notices, each an arrayref of name and pid
## If using 9.0 or greater, the payload becomes the name
my ($ldbh) = @_;
my ($n, @notices);
while ($n = $ldbh->func('pg_notifies')) {
my ($name, $pid, $payload) = @$n;
if ($ldbh->{pg_server_version} >= 9999990000) {
next if $name ne 'bucardo';
$name = $payload; ## presto!
}
push @notices => [$name, $pid];
}
return \@notices;
} ## end of db_get_notices
sub install {
## Install Bucardo into a database
## Arguments: none
## Returns: never, exits
if (! $bcargs->{batch}) {
print "This will install the bucardo database into an existing Postgres cluster.\n";
print "Postgres must have been compiled with Perl support,\n";
print "and you must connect as a superuser\n\n";
}
## Setup our default arguments for the installation choices
my $host = $bcargs->{dbhost} || $ENV{PGHOST} || '';
my $port = $bcargs->{dbport} || $ENV{PGPORT} || 5432;
my $user = $bcargs->{dbuser} || $ENV{DBUSER} || 'postgres';
my $dbname = $bcargs->{dbname} || $ENV{DBNAME} || 'postgres';
## Make sure the bucardo.schema file is available, and extract some config items
my ($fh, $schema_file) = find_bucardo_schema();
my %confvar = (piddir => '');
while (<$fh>) {
for my $string (keys %confvar) {
if (/^$string\|(.+?)\|/) {
$confvar{$string} = $1;
}
}
}
close $fh or warn qq{Could not close "$schema_file": $!\n};
## Make sure each item has a default value
for my $key (keys %confvar) {
if (!$confvar{$key}) {
warn "Could not find default configuration for $key!\n";
}
}
## If the PID directory was not provided on the command line,
## use the value from the bucardo.schema file
my $piddir = $bcargs->{piddir} || $confvar{piddir};
## Keep looping until they are happy with the settings
GOOEY:
{
## We only don't print this in quiet and batch mode
if (! $QUIET or ! $bcargs->{batch}) {
print "Current connection settings:\n";
print "1. Host: $host\n";
print "2. Port: $port\n";
print "3. User: $user\n";
print "4. Database: $dbname\n";
print "5. PID directory: $piddir\n";
}
## If in batch mode, we accept everything right away and move on
last GOOEY if $bcargs->{batch};
print 'Enter a number to change it, P to proceed, or Q to quit: ';
my $ans = <>;
print "\n";
## If the answer starts with a number, try and apply it
## Can also provide the value right away
if ($ans =~ /^\s*(\d+)(.*)/) {
my ($num,$text) = (int $1,$2);
$text =~ s/^\s*(\S+)\s*$/$1/;
my $new = length $text ? $text : '';
## Host: allow anything
## Change empty string to '';
if (1 == $num) {
if (!length $new) {
print 'Change the host to: ';
$new = <>;
print "\n";
chomp $new;
}
$host = length $new ? $new : '';
print "Changed host to: $host\n";
}
## Port: only allow numbers
elsif (2 == $num) {
if (!length $new) {
print 'Change the port to: ';
$new = <>;
print "\n";
chomp $new;
}
if ($new !~ /^\d+$/) {
print "-->Sorry, but the port must be a number\n\n";
redo GOOEY;
}
$port = $new;
print "Changed port to: $port\n";
}
## User: allow anything except an empty string
elsif (3 == $num) {
if (!length $new) {
print 'Change the user to: ';
$new = <>;
print "\n";
chomp $new;
}
if (! length $new) {
print "-->Sorry, you must specify a user\n\n";
redo GOOEY;
}
$user = $new;
print "Changed user to: $user\n";
}
## Database: allow anything except an empty string
elsif (4 == $num) {
if (!length $new) {
print 'Change the database name to: ';
$new = <>;
print "\n";
chomp $new;
}
if (! length $new) {
print "-->Sorry, you must specify a database name\n\n";
redo GOOEY;
}
$dbname = $new;
print "Changed database name to: $dbname\n";
}
## PID directory: allow anything, as long as it starts with a slash
elsif (5 == $num) {
if (!length $new) {
print 'Change the PID directory to: ';
$new = <>;
print "\n";
chomp $new;
}
if (! length $new) {
print "-->Sorry, you must specify a directory\n\n";
redo GOOEY;
}
if ($new !~ m{^/}) {
print "-->Sorry, the PID directory must be absolute (start with a slash)\n";
redo GOOEY;
}
if (! -d $new) {
print "-->Sorry, that is not a valid directory\n";
redo GOOEY;
}
$piddir = $new;
print "Changed PID dir to: $piddir\n";
}
}
elsif ($ans =~ /^\s*Q/i) {
die "Goodbye!\n";
}
elsif ($ans =~ /^\s*P/i) {
## Check on the PID directory before going any further
## This is the only item that can be easily checked here
if (! -d $piddir) {
print "-->Sorry, that is not a valid PID directory\n";
redo GOOEY;
}
last GOOEY;
}
else {
print "-->Please enter Q to quit, P to proceed, or enter a number to change a setting\n";
}
redo GOOEY;
}
## Try to connect
my $PSQL = sprintf '%s -p %d -U %s -d %s',
$ENV{PGBINDIR} ? "$ENV{PGBINDIR}/psql" : 'psql',
$port, $user, $dbname;
$host !~ /&1};
## Dump any problems verbatim to stderr
my $delayed_warning;
if ($res =~ /FATAL|ERROR/ or $res =~ /psql:/) {
$delayed_warning = $res;
}
## Check for some common errors
if ($res =~ /role "(.+)" does not exist/) {
my $baduser = $1;
if ($baduser eq 'postgres' and exists $ENV{USER} and $ENV{USER} =~ /^[\w-]+$/) {
$user = $ENV{USER};
if (!$QUIET and !$bcargs->{batch}) {
print "Failed to connect as user 'postgres', will try '$user'\n";
}
}
else {
print "-->Sorry, please try using a different user\n\n";
exit 1 if $bcargs->{batch};
}
goto GOOEY;
}
## Check for some common errors
if ($res =~ /database "(.+)" does not exist/) {
my $baddb = $1;
if ($baddb ne 'postgres') {
if (!$QUIET and !$bcargs->{batch}) {
print "Failed to connect to database '$dbname', will try 'postgres'\n";
}
$dbname = 'postgres';
goto GOOEY;
}
}
if ($res !~ /pg version: \D+(\d+)(.+?)\s/) {
print "-->Sorry, unable to connect to the database\n\n";
warn $delayed_warning;
exit 1 if $bcargs->{batch};
goto GOOEY;
}
## At this point, we assume a good connection
## The version check is really just to see if we are 8.1 or higher
my ($maj,$extra) = ($1,$2);
if ($maj < 8 or (8 == $maj and $extra =~ /\.0/)) {
die "Sorry, Bucardo requires Postgres version 8.1 or higher.\n";
}
## Determine if we need to create the bucardo user
$COM = qq{$PSQL -c "SELECT 1 FROM pg_user WHERE usename = 'bucardo'"};
$res = qx{$COM 2>&1};
## If no number 1 seen, no bucardo user, so create it
if ($res !~ /1/) {
$QUIET or print "Creating superuser 'bucardo'\n";
## Generate a new random password
my $pass = generate_password();
$SQL = qq{CREATE USER bucardo SUPERUSER PASSWORD '$pass'};
$COM = qq{$PSQL -c "$SQL"};
$res = qx{$COM 2>&1};
## Put the new password into the .pgpass file
my $passfile = "$ENV{HOME}/.pgpass";
my $pfh;
if (open my $pfh, '>>', $passfile) {
printf {$pfh} "%s:%s:%s:%s:%s\n",
$host =~ /^\w/ ? $host : '*',
$port =~ /^\d/ ? $port : '*',
'*',
'bucardo',
$pass;
close $pfh or warn qq{Could not close file "$passfile": $!\n};
chmod 0600, $passfile;
}
else {
print qq{Could not append password information to file "$passfile"\n};
print qq{Password for user bucardo is: $pass\n};
print qq{You probably want to change it or put into a .pgpass file\n};
}
}
## Now we apply the bucardo.schema to the new database
$COM = "$PSQL -AX -qt -f $schema_file 2>&1";
print "Attempting to create and populate the bucardo database and schema\n"
if ! $bcargs->{batch};
$res= qx{$COM};
chomp $res;
## Detect case where bucardo is already there
## This probably needs to be i18n safe
if ($res =~ /relation .* already exists/) {
warn "\nINSTALLATION FAILED! Looks like you already have Bucardo installed there.\n";
warn "Try running 'bucardo upgrade' instead.\n";
warn "If you are trying to completely reinstall Bucardo,\n";
warn "drop the bucardo database, and the bucardo schema from all databases.\n\n";
exit 1;
}
if ($res =~ /"plperlu".*CREATE LANGUAGE/s) {
warn "\nINSTALLATION FAILED! ($res)\n\n";
warn "The Pl/PerlU language needs to be available\n";
warn "This is usually available as a separate package\n";
warn "For example, you might try: yum install postgresql-plperl\n";
warn "If compiling from source, add the --with-perl option to your ./configure command\n\n";
exit 1;
}
## This can actually happen for many reasons: lack of this message
## simply means something went wrong somewhere
if ($res !~ m{Pl/PerlU was successfully installed}) {
warn "\nINSTALLATION FAILED! ($res)\n\n";
exit 1;
}
## We made it! All downhill from here
print "Database creation is complete\n\n" if ! $bcargs->{batch};
## Whether or not we really need to, change some bucardo_config items:
my $BDSN = 'dbi:Pg:dbname=bucardo';
$host and $host ne '' and $BDSN .= ";host=$host";
$port and $BDSN .= ";port=$port";
$dbh = DBI->connect($BDSN, 'bucardo', '', {AutoCommit=>0,RaiseError=>1,PrintError=>0});
$dbh->do('SET search_path = bucardo');
$SQL = 'UPDATE bucardo.bucardo_config SET setting = ? WHERE name = ?';
$sth = $dbh->prepare($SQL);
$confvar{piddir} = $piddir;
for my $key (sort keys %confvar) {
$count = $sth->execute($confvar{$key}, $key);
if ($count != 1) {
warn "!! Failed to set $key to $confvar{$key}\n";
}
else {
print qq{Updated configuration setting "$key"\n} if ! $bcargs->{batch};
}
}
$dbh->commit();
$QUIET or print "Installation is now complete.\n";
## A little less verbose if in batch mode
if (! $bcargs->{batch}) {
print "If you see errors or need help, please email bucardo-general\@bucardo.org\n\n";
print "You may want to check over the configuration variables next, by running:\n";
print "$progname show all\n";
print "Change any setting by using: $progname set foo=bar\n\n";
}
exit 0;
} ## end of install
##
## Internal helper subs
##
sub debug {
## Print a debug line if needed
## Arguments: one or two
## 1. String to be printed
## 2. Required debug level: defaults to 1
## Returns: undef
return if ! $DEBUG;
my $string = shift;
my $level = shift || 1;
return if $DEBUG < $level;
chomp $string;
print " |DEBUG| $string\n";
return;
} ## end of debug
sub standardize_name {
## Return canonical version of certain names
## Normalizes abbreviations, misspelling, plurals, case, etc.
## Arguments: one
## 1. Name
## Returns: canonical name
my $name = shift;
return 'customcode' if $name =~ /^c?code/i or $name =~ /^custom_?code/i;
return 'customname' if $name =~ /^cname/i or $name =~ /^custom_?name/i;
return 'customcols' if $name =~ /^ccol/i or $name =~ /^custom_?col/i;
return 'dbgroup' if $name =~ /^dbg/i or $name =~ /^d.+group/i;
return 'database' if $name =~ /^db/i or $name =~ /^database/i;
return 'herd' if $name =~ /^(?:relgr|herd)/i;
return 'sync' if $name =~ /^s[yi]n[ck]/i;
return 'table' if $name =~ /^tab/i or $name =~ /^tbale/i;
return 'sequence' if $name =~ /^seq/i;
return 'all' if $name =~ /^all$/i;
return 'config' if $name =~ /^config/i;
return 'clone' if $name =~ /^clon/i;
return $name;
} ## end of standardize_name
sub generate_password {
## Generate a random 42 character password
## Arguments: none
## Returns: new password
my @chars = split // => q!ABCDEFGHJKMNPQRSTWXYZabcdefghjkmnpqrstwxyz23456789@#%^&(){}[];./!;
my $pass = join '' => @chars[map{ rand @chars }(1..42)];
return $pass;
} ## end of generate_password
sub process_simple_args {
## Process args to an inner function in the style of a=b
## Arguments: one
## 1. Custom hashref
## Returns: db column hashref, columns string, placeholders string,
## values string, and 'extra' hashref
my $arg = shift;
my $validcols = $arg->{cols} or die 'Need a list of valid cols!';
my $list = $arg->{list} or die 'Need a list of arguments!';
my $doc_section = $arg->{doc_section} or die 'Need a doc_section!';
my %item;
my %dbcol;
my %extra;
my %othername;
## Transform array of x=y into a hashref
my $xyargs = process_args(join ' ' => map { s/[=:]\s*(\w+ .*)/="$1"/; $_; } @$list);
## Parse the validcols string, and setup any non-null defaults
for my $row (split /\n/ => $validcols) {
next if $row !~ /\w/ or $row =~ /^#/;
$row =~ /^\s*(\S+)\s+(\S+)\s+(\S+)\s+(.+)/ or die "Invalid valid cols ($row)";
my ($args,$dbcol,$flag,$default) = ([split /\|/ => $1],$2,$3,$4);
my $alias = @{$args}[-1];
for my $name (@$args) {
$item{$name} = [$dbcol,$flag,$default];
$othername{$name} = $alias;
}
## Process environment variable default
if ($default =~ s/^ENV://) {
for my $env (split /\|/ => $default) {
if ($ENV{$env}) {
## Skip if it starts with PG and this is not postgres
next if $env =~ /^PG/ and exists $xyargs->{type} and $xyargs->{type} ne 'postgres';
$dbcol{$dbcol} = $ENV{$env};
last;
}
}
}
elsif ($default ne 'null' and $default ne 'skip') {
$dbcol{$dbcol} = $default;
}
}
for my $arg (sort keys %$xyargs) {
next if $arg eq 'extraargs';
if (! exists $item{$arg}) {
warn "Unknown option '$arg'\n";
usage_exit($doc_section);
}
(my $val = $xyargs->{$arg}) =~ s/^\s*(\S+)\s*$/$1/;
if ($item{$arg}[2] eq 'skip') {
$extra{$othername{$arg}} = $val;
next;
}
my ($dbcol,$flag,$default) = @{$item{$arg}};
if ($flag eq '0') {
## noop
}
elsif ($flag eq 'TF') {
$val =~ s/^\s*t(?:rue)*\s*$/1/i;
$val =~ s/^\s*f(?:alse)*\s*$/0/i;
$val =~ s/^\s*on*\s*$/1/i;
$val =~ s/^\s*off*\s*$/0/i;
$val =~ s/^\s*yes*\s*$/1/i;
$val =~ s/^\s*no*\s*$/0/i;
if ($val !~ /^[01]$/) {
die "Invalid value for '$arg': must be true or false\n";
}
}
elsif ($flag eq 'numeric') {
if ($val !~ /^\d+$/) {
die "Invalid value for '$arg': must be numeric\n";
}
}
elsif ($flag =~ /^=(.+)/) {
my $ok = 0;
for my $okval (split /\|/ => $1) {
if ($okval =~ /~/) { ## aliases - force to the first one
my @alias = split /~/ => $okval;
for my $al (@alias) {
if ($val eq $al) {
$ok = 1;
last;
}
}
if ($ok) {
$val = $alias[0];
last;
}
}
elsif (lc $val eq lc $okval) {
$ok = 1;
last;
}
}
if (!$ok) {
(my $arglist = $flag) =~ s/\|/ or /g;
$arglist =~ s/^=//;
$arglist =~ s/~\w+//g;
die "Invalid value for '$arg': must be one of $arglist\n";
}
}
elsif ($flag eq 'interval') {
## Nothing for now
}
else {
die "Unknown flag '$flag' for $arg";
}
## Value has survived our minimal checking. Store it and clobber any default
$dbcol{$dbcol} = $val;
}
## Apply any magic
if (exists $arg->{morph}) {
for my $mline (@{$arg->{morph}}) {
if (exists $mline->{field}) {
next unless exists $dbcol{$mline->{field}};
if (exists $mline->{new_defaults}) {
for my $change (split /\s+/ => $mline->{new_defaults}) {
my ($f,$v) = split /\|/ => $change;
next if exists $dbcol{$f};
$dbcol{$f} = $v;
}
}
if (exists $mline->{dash_to_white}) {
$dbcol{$mline->{field}} =~ s/_/ /g;
}
}
else {
die "Do not know how to handle that morph!\n";
}
}
}
## Automatic morphing magic
if (exists $item{status} and ! exists $dbcol{status}) {
for my $stat (qw/ active inactive /) {
if (grep { $_ eq $stat } @{ $xyargs->{extraargs} }) {
$dbcol{status} = $stat;
}
}
}
## Build the lists of columns and placeholders for the SQL statement
my ($cols,$phs,$vals) = ('','',{});
for my $col (sort keys %dbcol) {
$cols .= "$col,";
$phs .= '?,';
$vals->{$col} = $dbcol{$col};
}
$cols =~ s/,$//;
$phs =~ s/,$//;
return \%dbcol, $cols, $phs, $vals, \%extra;
} ## end of process_simple_args
sub check_recurse {
## Call a sub recursively depending on first argument
## Arguments: three or more
## 1. Type of thing (e.g. database)
## 2. Name of the thing
## 3. Any additional actions
## Returns: 0 or 1
my ($thing, $name, @actions) = @_;
my $caller = (caller(1))[3];
## If the name is 'all', recursively call on all objects of this type
if ($name =~ /^all$/i) {
for my $item (sort keys %$thing) {
&$caller($item, @actions);
}
return 0;
}
## If we have a wildcard, recursively call all matching databases
if ($name =~ s/[*%]/\.*/g) {
my @list = grep { $_ =~ /^$name$/ } keys %$thing;
if (! @list) {
die qq{No matching items found\n};
}
for my $item (sort @list) {
&$caller($item, @actions);
}
return 0;
}
return 1;
} ## end of check_recurse
sub extract_name_and_role {
## Given a group or db name with optional role information, return both
## Also returns optional list of other items, e.g. ABC:target:pri=2
## Arguments: one
## 1. Group or database name: 'foo' or 'foo:master'
## Returns: name, role name, and hashref of 'extra' info
my $name = shift or die;
## Role always defaults to 'target'
my $role = 'target';
## Check for a role attached to the group name
if ($name =~ s/:([^:]+)//) {
$role = lc $1;
}
## Look for any additional items
my %extra;
while ($name =~ s/:([^:]+)//) {
my $extra = $1;
if ($extra !~ /(\w+)=([\w\d]+)/) {
die qq{Invalid value "$extra"\n};
}
my ($lname,$val) = ($1,$2);
if ($lname =~ /make?delta/i) {
$extra{'makedelta'} = make_boolean($val);
}
elsif ($lname =~ /pri/i) {
$extra{'priority'} = $val;
}
else {
die qq{Unknown value "$lname": must be priority or makedelta\n};
}
}
## Valid group name?
if ($name !~ /^[\w\d]+$/) {
die "Invalid name: $name\n";
}
## Valid role name?
if ($role !~ /^(?:master|target|t|slave|rep|replica|source|s|fullcopy)$/) {
die "Invalid database role: $role\n";
}
## Standardize the names
$role = 'source' if $role =~ /^(?:master|s)$/;
$role = 'target' if $role =~ /^(?:slave|rep|replica|tar|t)$/;
return $name, $role, \%extra;
} ## end of extract_name_and_role
sub load_bucardo_info {
## Load of all information from the database into global hashes
## Arguments: one
## 1. Boolean: if true, force run even if we've run once already
## Returns: undef
my $force = shift || 0;
return if exists $global{db} and ! $force;
## Grab all database information
$SQL = 'SELECT *, EXTRACT(epoch FROM cdate) AS epoch FROM bucardo.db';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $db = $sth->fetchall_hashref('name');
## Grab all database information
$SQL = 'SELECT * FROM bucardo.dbgroup';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $dbgroup = $sth->fetchall_hashref('name');
## Map databases to their groups
$SQL = 'SELECT * FROM bucardo.dbmap';
$sth = $dbh->prepare($SQL);
$sth->execute();
for my $row (@{$sth->fetchall_arrayref({})}) {
$db->{$row->{db}}{group}{$row->{dbgroup}} = $row;
## Tally up the roles each database fills
$db->{$row->{db}}{roles}{$row->{role}}++;
## Mark if this db is ever used as a source, for help in adding table
$db->{$row->{db}}{issource}++ if $row->{role} eq 'source';
$dbgroup->{$row->{dbgroup}}{db}{$row->{db}} = $row;
}
## Grab all goat information
$SQL = 'SELECT * FROM bucardo.goat';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $goat;
$goat->{by_id} = $sth->fetchall_hashref('id');
$goat->{by_table} = {};
for my $key (%{$goat->{by_id}}) {
next if $key !~ /^\d/;
my $tname = $goat->{by_id}{$key}{tablename};
my $name = "$goat->{by_id}{$key}{schemaname}.$tname";
my $dbname = $goat->{by_id}{$key}{db};
## Index by database, so different databases containing matching object
## names can be handled
$goat->{by_db}{$dbname}{$name} = $goat->{by_id}{$key};
## Index by full object name
if (! exists $goat->{by_fullname}{$name}) {
$goat->{by_fullname}{$name} = [ $goat->{by_id}{$key} ];
}
else {
push @{$goat->{by_fullname}{$name}}, $goat->{by_id}{$key};
}
## Also want a table-only version:
$goat->{by_table}{$tname} = [] unless exists $goat->{by_table}{$tname};
push @{$goat->{by_table}{$tname}} => $goat->{by_id}{$key};
}
## Grab all herd information
$SQL = 'SELECT * FROM bucardo.herd';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $herd = $sth->fetchall_hashref('name');
## Grab all herdmap information, stick into previous hashes
$SQL = 'SELECT * FROM bucardo.herdmap ORDER BY priority DESC, goat ASC';
$sth = $dbh->prepare($SQL);
$sth->execute();
for my $row (@{$sth->fetchall_arrayref({})}) {
my ($g,$h,$p) = @$row{qw/goat herd priority/};
$goat->{by_id}{$g}{herd}{$h} = $p;
$herd->{$h}{goat}{"$goat->{by_id}{$g}{schemaname}.$goat->{by_id}{$g}{tablename}"} = {
id => $g,
priority => $p,
reltype => $goat->{by_id}{$g}{reltype},
schema => $goat->{by_id}{$g}{schemaname},
table => $goat->{by_id}{$g}{tablename},
};
my ($s,$t) = @{$goat->{by_id}{$g}}{qw/schemaname tablename/};
$herd->{$h}{hasgoat}{$s}{$t} = $p;
## Assign each herd to a datbase via its included goats
$herd->{$h}{db} = $goat->{by_id}{$g}{db};
}
## Grab all sync information
$SQL = 'SELECT * FROM bucardo.sync';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $sync;
for my $row (@{$sth->fetchall_arrayref({})}) {
my ($name,$p,$sherd,$dbs) = @$row{qw/name priority herd dbs/};
$sync->{$name} = $row;
## Add in herd information
$sync->{$name}{herd} = $herd->{$sherd};
## Add this sync back to the herd
$herd->{$sherd}{sync}{$name}++;
## Grab the databases used by this sync
$sync->{$name}{dblist} = $dbgroup->{$dbs}{db};
## Map each database back to this sync, along with its type (source/target)
for my $dbname (keys %{ $sync->{$name}{dblist} }) {
$db->{$dbname}{sync}{$name} = $sync->{$name}{dblist}{$dbname};
}
## Note which syncs are used by each goat
for my $row2 (sort keys %{$row->{herd}{goat}}) {
$goat->{by_id}{$row2}{sync}{$name} = 1;
}
}
## Grab all customcode information
$SQL = 'SELECT * FROM bucardo.customcode';
$sth = $dbh->prepare($SQL);
$sth->execute();
my $cc = $sth->fetchall_hashref('name');
$SQL = 'SELECT * FROM bucardo.customcode_map';
$sth = $dbh->prepare($SQL);
$sth->execute();
my %codename;
for my $row (values %$cc) {
$codename{$row->{id}} = $row->{name};
}
for my $row (@{$sth->fetchall_arrayref({})}) {
my $codename = $codename{$row->{code}};
push @{$cc->{$codename}{map}} => $row;
}
## Grab all customname information
$SQL = q{SELECT c.id, c.goat, c.newname,
COALESCE(c.sync, '') AS sync,
COALESCE(c.db, '') AS db,
g.schemaname || '.' || g.tablename AS tname
FROM bucardo.customname c
JOIN goat g ON (g.id = c.goat)
};
$sth = $dbh->prepare($SQL);
$sth->execute();
$CUSTOMNAME = {};
for my $row (@{ $sth->fetchall_arrayref({}) }) {
## We store three versions
## Look things up by the internal customname id: used for 'delete customname'
## Only one entry per id
$CUSTOMNAME->{id}{$row->{id}} = $row;
## Look things up by the goat id: used to check for existing entries
## Can have multiple entries per goat
$CUSTOMNAME->{goat}{$row->{goat}}{$row->{db}}{$row->{sync}} = $row;
## A simple list of all rows: used for 'list customnames'
push @{ $CUSTOMNAME->{list} } => $row;
}
## Grab all customcols information
$SQL = q{SELECT c.id, c.goat, c.clause,
COALESCE(c.sync, '') AS sync,
COALESCE(c.db, '') AS db,
g.schemaname || '.' || g.tablename AS tname
FROM bucardo.customcols c
JOIN goat g ON (g.id = c.goat)
};
$sth = $dbh->prepare($SQL);
$sth->execute();
$CUSTOMCOLS = {};
for my $row (@{ $sth->fetchall_arrayref({}) }) {
## We store three versions: one for quick per-goat lookup,
## one by the assigned id, and one just a big list
push @{ $CUSTOMCOLS->{goat}{$row->{goat}}{$row->{clause}} } => $row;
$CUSTOMCOLS->{id}{$row->{id}} = $row;
push @{ $CUSTOMCOLS->{list} } => $row;
}
$global{cc} = $CUSTOMCODE = $cc;
$global{dbgroup} = $DBGROUP = $dbgroup;
$global{db} = $DB = $db;
$global{goat} = $GOAT = $goat;
$global{herd} = $HERD = $RELGROUP = $herd;
$global{sync} = $SYNC = $sync;
## Separate goat into tables and sequences
for my $id (keys %{$GOAT->{by_id}}) {
## Ids only please
next if $id !~ /^\d+$/;
my $type = $GOAT->{by_id}{$id}{reltype};
if ($type eq 'table') {
$TABLE->{$id} = $GOAT->{by_id}{$id};
}
elsif ($type eq 'sequence') {
$SEQUENCE->{$id} = $GOAT->{by_id}{$id};
}
else {
die "Unknown relation type $type!";
}
}
## Grab all clone information
$SQL = qq{SELECT *,
TO_CHAR(started,'$DATEFORMAT') AS pstarted,
TO_CHAR(ended,'$DATEFORMAT') AS pended
FROM bucardo.clone};
$sth = $dbh->prepare($SQL);
$sth->execute();
$CLONE = {};
for my $row (@{ $sth->fetchall_arrayref({}) }) {
$CLONE->{$row->{id}} = $row;
}
return;
} ## end of load_bucardo_info
sub transform_name {
## Change a given word to a more standard form
## Generally used for database column names, which follow some simple rules
## Arguments: one
## 1. Name to transform
## Returns: transformed name
my $name = shift;
## Complain right away if these are not standard characters
if ($name !~ /^[\w ]+$/) {
die "Invalid name: $name\n";
}
## Change to lowercase
$name = lc $name;
## Change dashes and spaces to underscores
$name =~ s{[- ]}{_}go;
## Compress all underscores
$name =~ s{__+}{_}go;
## Fix common spelling errors
$name =~ s{perpare}{prepare}go;
## Look up standard abbreviations
if (exists $alias{$name}) {
$name = $alias{$name};
}
return $name;
} ## end of transform_name
sub transform_value {
## Change a value to a more standard form
## Used for database column SET actions
## Arguments: one
## 1. Value
## Returns: transformed value
my $value = shift;
## Remove all whitespace on borders
$value =~ s/^\s*(\S+)\s*$/$1/;
## Change booleans to 0/1
$value =~ s/^(?:t|true)$/1/io;
$value =~ s/^(?:f|false)$/0/io;
return $value;
} ## end of transform_value
sub make_boolean {
## Transform some string into a strict boolean value
## Arguments: one
## 1. String to be analyzed
## Returns: the string literals 'true' or 'false' (unquoted)
my $value = shift;
$value = lc $value;
return 'true' if $value =~ /^(?:t|true|1|yes)$/o;
return 'false' if $value =~ /^f|false|0|no$/o;
die "Invalid value: must be 'true' of 'false'\n";
} ## end of make_boolean
sub standardize_rdbms_name {
## Make the database types standard: account for misspellings, case, etc.
## Arguments: one
## 1. Name of a database type
## Returns: modified name
my $name = shift;
$name =~ s/postgres.*/postgres/io;
$name =~ s/pg.*/postgres/io;
$name =~ s/driz?zle.*/drizzle/io;
$name =~ s/firebird/firebird/io;
$name =~ s/mongo.*/mongo/io;
$name =~ s/mysql.*/mysql/io;
$name =~ s/maria.*/mariadb/io;
$name =~ s/oracle.*/oracle/io;
$name =~ s/redis.*/redis/io;
$name =~ s/sqll?ite.*/sqlite/io;
return $name;
} ## end of standardize_rdbms_name
sub find_best_db_for_searching {
## Returns the db from $DB most likely to contain tables to add
## Basically, we use source ones first, then the date added
## Arguments: none
## Returns: database name or undef if no databases defined yet
for my $db (
map { $_->[0] }
sort {
## Source databases are always first
$a->[1] <=> $b->[1]
## First created are first
or $a->[2] <=> $b->[2]
## All else fails, sort by name
or $a->[0] cmp $b->[0] }
map { [
$_,
exists $DB->{$_}{issource} ? 0 : 1,
$DB->{$_}{epoch},
lc $_,
]
}
keys %{ $DB } ) {
return $db;
}
## Probably an error, but let the caller handle it:
return undef;
} ## end of find_best_db_for_searching
##
## Subs to perform common SQL actions
##
sub confirm_commit {
## Perform a database commit unless the user does not want it
## Arguments: none
## Returns: true for commit, false for rollback
## The dryrun option overrides everything else: we never commit
if ($bcargs->{dryrun}) {
$VERBOSE and print "In dryrun mode, so not going to commit database changes\n";
return 0;
}
if ($bcargs->{confirm}) {
print 'Commit the changes? Y/N ';
if ( !~ /Y/i) {
$dbh->rollback();
print "Changes have been rolled back\n";
return 0;
}
else {
$dbh->commit();
print "Changes have been committed\n";
}
}
else {
$dbh->commit();
}
return 1;
} ## end of confirm_commit
sub add_db_to_group {
## Add a database to a group
## Will create the group as needed
## Does not commit
## Arguments: two
## 1. Database name
## 2. Group name (may have :role specifier)
## Returns: group name and role name
my ($db,$fullgroup) = @_;
## Figure out the role. Defaults to target
my ($group,$role) = extract_name_and_role($fullgroup);
if (! exists $DBGROUP->{$group}) {
## Extra argument prevents load_bucardo_info from being called by the sub
create_dbgroup($group, 1);
}
$SQL = 'INSERT INTO bucardo.dbmap(db,dbgroup,role) VALUES (?,?,?)';
$sth = $dbh->prepare($SQL);
eval {
$sth->execute($db,$group,$role);
};
if ($@) {
my $message = qq{Cannot add database "$db" to dbgroup "$group"};
if ($@ =~ /"dbmap_unique"/) {
die qq{$message: already part of the group\n};
}
die qq{$message: $@\n};
}
## Reload our hashes
load_bucardo_info(1);
return $group, $role;
} ## end of add_db_to_group
sub remove_db_from_group {
## Removes a database from a group: deletes from bucardo.dbmap
## Does not commit
## Arguments: two
## 1. Database name
## 2. Group name
## 3. Boolean: if true, prevents the reload
## Returns: undef
my ($db,$group,$noreload) = @_;
$SQL = 'DELETE FROM bucardo.dbmap WHERE db=? AND dbgroup=?';
$sth = $dbh->prepare_cached($SQL);
$sth->execute($db, $group);
## Reload our hashes
$noreload or load_bucardo_info(1);
return;
} ## end of remove_db_from_group
sub change_db_role {
## Changes the role of a database: updates bucardo.dbmap
## Does not commit
## Arguments: four
## 1. New role
## 2. Name of the dbgroup
## 3. Name of the database
## 4. Boolean: if true, prevents the reload
## Returns: undef
my ($role,$group,$db,$noreload) = @_;
$SQL = 'UPDATE bucardo.dbmap SET role=? WHERE dbgroup=? AND db=?';
$sth = $dbh->prepare_cached($SQL);
$sth->execute($role,$group,$db);
## Reload our hashes
$noreload or load_bucardo_info(1);
return;
} ## end of change_db_role
sub update_dbmap {
## Update the values in the bucardo.dbmap table
## Arguments: three
## 1. Name of the database
## 2. Name of the dbgroup
## 3. Hashref of things to change
## Returns: undef
my ($db,$group,$changes) = @_;
## This should not need quoting as they are all [\w\d]
my $list = join ',' => map { "$_=$changes->{$_}" } sort keys %$changes;
$SQL = "UPDATE bucardo.dbmap SET $list WHERE db=? AND dbgroup=?";
$sth = $dbh->prepare($SQL);
$sth->execute($db, $group);
return;
} ## end of update_dbmap
sub create_herd {
## Creates a new entry in the bucardo.herd table
## Caller should have already checked for existence
## Does not commit
## Arguments: two
## 1. Name of the new herd
## 2. Boolean: if true, prevents the reload
## Returns: name of the herd just created
my ($name,$noreload) = @_;
$SQL = 'INSERT INTO bucardo.herd(name) VALUES (?)';
$sth = $dbh->prepare($SQL);
eval {
$sth->execute($name);
};
if ($@) {
print qq{Failed to create relgroup "$name"\n$@\n};
exit 1;
}
## Reload our hashes
$noreload or load_bucardo_info(1);
return $name;
} ## end of create_herd
__END__
=head1 NAME
bucardo - utility script for controlling the Bucardo program
=head1 VERSION
This document describes version 5.6.0 of bucardo
=head1 USAGE
bucardo [] [] [] []
=head1 DESCRIPTION
The bucardo script is the main interaction to a running Bucardo instance. It
can be used to start and stop Bucardo, add new items, kick syncs, and even
install and upgrade Bucardo itself. For more complete documentation, please
view L.
=head1 COMMANDS
Run C<< bucardo help >> for additional details
=over
=item C
Installs the Bucardo configuration database.
=item C
Upgrades the Bucardo configuration database to the latest schema.
=item C<< start [] [] >>
Starts Bucardo.
=item C<< stop [] >>
Stops Bucardo.
=item C<< restart [] [] >>
Stops and starts Bucardo.
=item C<< list [] >>
Lists objects managed by Bucardo.
=item C<< add >>
Adds a new object.
=item C<< update >>
Updates an object.
=item C<< remove [...] >>
Removes one or more objects.
=item C<< kick [] [...] [] >>
Kicks off one or more syncs.
=item C
Sends a message to all CTL and KID processes asking them to reload the Bucardo
configuration.
=item C
Sends a message to all Bucardo processes asking them to reopen any log files
they may have open. Call this after you have rotated the log file(s).
=item C<< show all| [...] >>
Shows the current Bucardo settings.
=item C< [...] >>
Sets one or more configuration setting..
=item C<< ping [] >>
Sends a ping notice to the MCP process to see if it will respond.
=item C<< status [] [...] >>
Shows the brief status of syncs in a tabular format.
=item C<< activate [...] [] >>
Activates one or more named syncs.
=item C<< deactivate [...] [] >>
Deactivates one or more named syncs.
=item C<< message '' >>
Sends a message to the running Bucardo logs.
=item C<< reload [ [...]] >>
Sends a message to one or more sync processes, instructing them to reload.
=item C<< inspect [...] >>
Inspects one or more objects of a particular type.
=item C<< validate all| [...] >>
Validates one or more syncs.
=item C<< purge all|
[
...] >>
Purges the delta and track tables for one or more tables, for one or more
databases.
=item C<< delta [] >>
Show the delta counts for each source target.
=item C<< help [ []] >>
Shows help.
=back
=head1 OPTIONS
-d --db-name NAME Database name.
-U --db-user USER Database user name.
-P --db-pass PASS Database password.
-h --db-host HOST Database server host name.
-p --db-port PORT Database server port number.
--bucardorc FILE Use specified .bucardorc file.
--no-bucardorc Do not use .bucardorc file.
--quiet Incremental quiet.
--verbose Incremental verbose mode.
-? --help Output basic help and exit.
--version Print the version number and exit.
--dryrun Do not perform any actual actions.
--confirm Require direct confirmation before changes.
=head1 COMMAND DETAILS
Most of the commands take parameters. These may be passed after the command
name and, where appropriate, an object name. Parameters take the form of
key/value pairs separated by an equal sign (C<=>). For example:
bucardo add db sea_widgets dbname=widgets host=db.example.com
Here C and are parameters.
Many of the commands also use command-line options, which are specified in the
normal way. For example, the C command could also be written
as:
bucardo add db sea_widgets --dbname widgets --dbhost db.example.com
However, parameters and options are not directly interchangeable in all cases.
See the documentation for individual commands for their supported options.
=head2 install
bucardo install
Installs the Bucardo schema from the file F into an existing Postgres cluster.
The user "bucardo" and database "bucardo" will be created first as needed. This is an
interactive installer, but you can supply the following values from the command line:
=over
=item C<--dbuser>
defaults to postgres
=item C<--dbname>
defaults to postgres
=item C<--dbport>
defaults to 5432
=item C<--pid-dir>
defaults to /var/run/bucardo/
=back
=head2 upgrade
bucardo upgrade
Upgrades an existing Bucardo installation to the current version of the bucardo database
script. Requires that bucardo and the F file be the same version. All
changes should be backwards compatible, but you may need to re-validate existing scripts
to make sure changes get propagated to all databases.
=head2 start
bucardo start "Reason"
Starts Bucardo. Fails if the MCP process is running (determined if its PID file is present).
Otherwise, starts cleanly by first issuing the equivalent of a stop to ask any existing Bucardo
processes to exit, and then starting a new Bucardo MCP process. A short reason and name should
be provided - these are written to the C file (F<./bucardo.restart.reason.txt> by
default) and sent in the email sent when Bucardo has been started up. It is also appended to
the reason log, which has the same name as the the C but ends in F<.log>.
The options for the C command are:
=over
=item C<--sendmail>
Tells Bucardo whether or not to send mail on interesting events: startup,
shutdown, and errors. Default is on.
=item C<--extra-name string>
A short string that will be appended to the version string as output by the
Bucardo process names. Mostly useful for debugging.
=item C<--log-destination destination>
Determines the destination for logging output. The supported values are:
=over
=item C
=item C
=item C
=item C
=item A file system directory.
=back
May be specified more than once, which is useful for, e.g., logging both to a
directory and to syslog. If C<--log-destination> is not specified at all, the
default is to log to files in F.
=item C<--log-separate>
Forces creation of separate log files for each Bucardo process of the form
"log.bucardo.X.Y", where X is the type of process (MCP, CTL, or KID), and Y is
the process ID.
=item C<--log-extension string>
Appends the given string to the end of the default log file name,
F. A dot is added before the name as well, so a log extension of
"rootdb" would produce a log file named F.
=item C<--log-clean>
Forces removal of all old log files before running.
=item C<--debug>
=item C<--no-debug>
Enable or disable debugging output. Disabled by default.
=item C<--exit-on-nosync>
=item C<--no-exit-on-nosync>
On startup, if Bucardo finds no active syncs, it normally will continue to
run, requiring a restart once syncs are added. This is useful for startup
scripts and whatnot.
If, however, you want it to exit when there are no active syncs, pass the
C<--exit-on-nosync> option. You can also be explicit that it should I
exit when there are no syncs by passing C<--no-exit-on-nosync>. This is the
default value.
=back
=head2 stop
bucardo stop "Reason"
Forces Bucardo to quit by creating a stop file which all MCP, CTL, and KID processes should
detect and cause them to exit. Note that active syncs will not exit right away, as they
will not look for the stop file until they have finished their current run. Typically,
you should scan the list of processes after running this program to make sure that all Bucardo
processes have stopped. One should also provide a reason for issuing the stop - usually
this is a short explanation and your name. This is written to the C file
(F<./bucardo.restart.reason.txt> by default) and is also used by Bucardo when it exits and
sends out mail about its death. It is also appended to the reason log, which has the same name
as the the C but ends in F<.log>.
=head2 restart
bucardo restart "Reason"
Stops bucardo, waits for the stop to complete, and then starts it again.
Supports the same options as /start>. Useful for start scripts. For
getting just CTL and KID processes to recognize newly added, updated, or
removed objects, use the C command, instead.
=head2 list
bucardo list
Lists summary information about Bucardo objects. The supported types are:
=over
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=back
The C option will list information about all object types.
The optional C option can be used to filter the list to only those
matching a regular expression.
=head2 add
bucardo add
Adds a new object to Bucardo. The C specifies the type of object to add,
while the C should be the name of the object. The supported types
include:
=over
=item C
=item C
=item C
=item C
=item C
=item C
=item C
=item C
=item C
=item C
=back
=head3 add db
bucardo add db dbname=actual_name port=xxx host=xxx user=xxx
Adds one or more new databases. The C is the name by which the database will be
known to Bucardo, and must be unique. This may vary from the actual database
name, as multiple hosts might have databases with the same name. Multiple databases
can be added by separating the names with commas. Options that differ between the
databases should be separated by a matching commas. Example:
bucardo add db alpha,beta dbname=sales host=aa,bb user=bucardo
This command will attempt an immediate test connection to the added database(s).
The supported named parameters are:
=over
=item C
The actual name of the database. Required unless using a service file or setting it via dbdsn.
=item C
The type of the database. Defaults to C. Currently supported values are:
=over
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=item * C
=back
=item C
A direct DSN to connect to a database. Will override all other connection options if set.
=item C
The username Bucardo should use when connecting to this database.
=item C
The password Bucardo should use when connecting to this database. It is recommended
that you use a .pgpass file rather than entering the password here.
=item C
The host Bucardo should use when connecting to this database. Defaults to the value of the C<$PGHOSTADDR>
or C<$PGHOST> environment variables, if present.
=item C
The port Bucardo should use when connecting to this database. Defaults to the value of the C<$PGPORT>
environment variable, if present.
=item C
Additional connection parameters, e.g. C.
=item C
The service name Bucardo should use when connecting to this database.
=item C
Initial status of this database. Defaults to "active" but can be set to "inactive".
=item C
Name of the database group this database should belong to.
=item C
Automatically add all tables from this database.
=item C
Automatically add all sequences from this database.
=item C
=item C
Set to 1 or 0 to enable or disable server-side prepares. Defaults to 1.
=item C
Set to 1 or 0 to enable or disable makedelta. Defaults to 0.
=back
Additional parameters:
=over
=item C<--force>
Forces the database to be added without running a connection test.
=back
B As a convenience, if the C value is its default value,
"bucardo", in the event that Bucardo cannot connect to the database, it will
try connecting as "postgres" and create a superuser named "bucardo". This is
to make things easier for folks getting started with Bucardo, but will not
work if it cannot connect as "postgres", or if it the connection failed due to
an authentication failure.
=head3 add dbgroup
bucardo add dbgroup name db1:source db2:source db3:target ...
Adds one or more databases to the named dbgroup. If the dbgroup
doesn't exist, it will be created. The database parameters should specify
their roles, either "source" or "target".
=head3 add table
bucardo add table [schema].table db=actual_db_name
Adds a table object. The table information will be read from the specified
database. Supported parameters:
=over
=item C
The name of the database from which to read the table information. Should be a
name known to Bucardo, thanks to a previous call to C. Required.
=item C
Boolean indicating whether or not the table should automatically send kick
messages when it's modified. Overrides the C parameter of any syncs
of which the table is a part.
=item C
Boolean indicating whether or not to rebuild indexes after every sync. Off by
default. Optional.
=item C
Boolean indicating whether or not to analyze the table after every sync. Off
by default. Optional.
=item C
Boolean indicating whether or not to vacuum the table after every sync. Off by
default. Optional.
=item C
Adds the table to the named relgroup. If the relgroup does not
exist, it will be created. Optional.
=item C
Turns makedelta magic on or off. Value is a list of databases which need makedelta
for this table. Value can also be "on" to enable makedelta for all databases.
Defaults to "off".
=item C
Boolean indicating whether or not to be strict when comparing the table
between syncs. If the columns have different names or data types, the
validation will fail. But perhaps the columns are allowed to have different
names or data types. If so, disable C and column differences will
result in warnings rather than failing the validation. Defaults to true.
=back
=head3 add sequence
bucardo add sequence [schema].sequence relgroup=xxx
=over
=item C
The name of the database from which to read the sequence information. Should
be a name known to Bucardo, thanks to a previous call to C.
Required.
=item C
Adds the sequence to the named relgroup. If the relgroup does not
exist, it will be created. Optional.
=back
=head3 add all tables
bucardo add all tables [relgroup=xxx] [pkonly]
Adds all the tables in all known databases or in a specified database.
Excludes tables in the C, C, and C
schemas. (Yes, this means that you cannot replicate the Bucardo configuration
database using Bucardo. Sorry about that.) Supported options and parameters:
=over
=item C
=item C<--db>
Name of the database from which to find all the tables to add. If not
provided, tables will be added from all known databases.
=item C
=item C<--schema>
=item C<-n>
Limit to the tables in the specified comma-delimited list of schemas. The
options may be specified more than once.
=item C
=item C<--exclude-schema>
=item C<-N>
Exclude tables in the specified comma-delimited list of schemas. The options
may be specified more than once.
=item C
=item C<--table>
=item C<-t>
Limit to the specified tables. The options may be specified more than once.
=item C
=item C<--exclude-table>
=item C<-T>
Exclude the specified tables. The options may be specified more than once.
=item C
=item C<--relgroup>
Name of the relgroup to which to add new tables.
=item C
Exclude tables without primary keys.
=back
=head3 add all sequences
bucardo add all sequences relgroup=xxx
Adds all the sequences in all known databases or in a specified database.
Excludes sequences in the C, C, and C
schemas. (Yes, this means that you cannot replicate the Bucardo configuration
database using Bucardo. Sorry about that.) Supported options and parameters:
=over
=item C
=item C<--db>
Name of the database from which to find all the sequences to add. If not
provided, sequences will be added from all known databases.
=item C
=item C<--schema>
=item C<-n>
Limit to the sequences in the specified comma-delimited list of schemas. The
options may be specified more than once.
=item C
=item C<--exclude-schema>
=item C<-N>
Exclude sequences in the specified comma-delimited list of schemas. The
options may be specified more than once.
=item C
=item C<--relgroup>
Name of the relgroup to which to add new tables or sequences.
=back
=head3 add relgroup
bucardo add relgroup name
bucardo add relgroup name table, sequence, ...
Adds a relgroup. After the name, pass in an optional list of tables
and/or sequences and they will be added to the group.
=head3 add sync
bucardo add sync name relgroup=xxx dbs=xxx
Adds a sync, which is a named replication event containing information about
what to replicate from where to where. The supported parameters are:
=over
=item C
The name of a dbgroup or comma-delimited list of databases. All of the
specified databases will be synchronized. Required.
=item C
The name of a dbgroup. All of the databases within this group will be
part of the sync. If the dbgroup does not exists and a separate list
of databases is given, the group will be created and populated.
=item C
The name of a relgroup to synchronize. All of the tables and/or
sequences in the relgroup will be synchronized. Required unless C is
specified.
=item C
List of tables to add to the sync. This implicitly creates a relgroup
with the same name as the sync. Required unless C is specified.
=item C
Indicates whether or not the sync is active. Must be either "active" or
"inactive". Defaults to "active".
=item C
Boolean indicating whether or not to rebuild indexes after every sync.
Defaults to off.
=item C
Number of seconds a KID can live before being reaped. No limit by default.
=item C
Number of times a KID may be kicked before being reaped. No limit by default.
=item C
The conflict resolution strategy to use in the sync. Supported values:
=over
=item C
The rows on the "source" database always "win". In other words, in a conflict,
Bucardo copies rows from source to target.
=item C
The rows on the "target" database always win.
=item C
Any conflicting rows are simply not replicated. Not recommended for most
cases.
=item C
Each database has an equal chance of winning each time. This is the default.
=item C
The row that was most recently changed wins.
=item C
The sync is aborted on a conflict.
=back
=item C
Determines whether or not a sync should switch to a full copy mode for a
single run. Supported values are:
=over
=item 0: off
=item 1: always full copy
=item 2: only copy tables that are empty on the target
=back
=item C
Boolean indicating whether or not the sync processes (CTL) should be
persistent. Defaults to false.
=item C
Boolean indicating whether or not the sync child processes (KID) should be
persistent. Defaults to false.
=item C
Boolean indicating whether or not tables in the sync should automatically send
kick messages when they're modified. May be overridden by the C
parameter of individual tables.
=item C
An interval specifying the maximum time a sync should go before being
kicked. Useful for busy systems where you don't want the overhead of notify
triggers.
=item C
An integer indicating the priority of the sync. Lower numbers are higher
priority. Currently used only for display purposes.
=item C
Boolean indicating whether or not to analyze tables after every sync. Off by
default. Optional.
=item C
An interval specifying the amount of time after which the sync has not run
that it should be considered overdue. C issues a warning
when a sync has not been run in this amount of time.
=item C
An interval specifying the amount of time after which the sync has not run
that it should be considered expired. C issues a critical
message when a sync has not been run in this amount of time.
=item C
Boolean indicating whether or not to track synchronization rates.
=item C
Boolean indicating whether or not to rebuild indexes after every sync. Off by
default. Optional.
=item C
Boolean indicating whether or not to be strict when comparing tables in the
sync. If the columns have different names or data types, the validation will
fail. But perhaps the columns are allowed to have different names or data
types. If so, disable C and column differences will result in
warnings rather than failing the validation. Defaults to true.
=back
=head3 add customname
bucardo add customname oldname newname [db=name] [sync=name]
Creates a new Bucardo custom name mapping. This allows the tables involved in
replication to have different names on different databases. The C
must contain the schema as well as the table name (if the source database
supports schemas). The optional parameters limit it to one or more databases,
and/or to one or more syncs. Supported parameters:
=over
=item C
A sync to which to add the customname. May be specified multiple times.
=item C
=item C
A database for which to add the customname. May be specified multiple times.
=back
=head3 add customcols
bucardo add customcols tablename select_clause [sync=x db=x]
Specify the list of columns to select from when syncing. Rather than the
default C