flexbackup-1.2.1/ 0040775 0000764 0000764 00000000000 07741574735 013216 5 ustar edwinh edwinh flexbackup-1.2.1/Makefile 0100664 0000764 0000764 00000002077 07711524027 014643 0 ustar edwinh edwinh # Hey emacs, use -*-Makefile-*- mode ###################################################################### # $Id: Makefile.dist,v 1.8 2003/07/29 17:00:07 edwinh Exp $ # $Name: v1_2_1 $ ###################################################################### PREFIX=/usr # Where the script binary should go BINPATH = $(PREFIX)/bin # Where the manpage should go MANPATH = $(PREFIX)/share/man # Where flexbackup.conf should be stored CONFFILE = /etc/flexbackup.conf # Where perl lives PERLPATH = /usr/bin/perl ###################################################################### all: fb.install install: all install -m 0644 flexbackup.conf $(CONFFILE) install -m 0755 fb.install $(BINPATH)/flexbackup install -m 0644 flexbackup.1 $(MANPATH)/man1/flexbackup.1 install -m 0644 flexbackup.conf.5 $(MANPATH)/man5/flexbackup.conf.5 fb.install: cp flexbackup fb.install cp fb.install fb.tmp; sed -e 's%/etc/flexbackup.conf%$(CONFFILE)%g;' fb.tmp > fb.install cp fb.install fb.tmp; sed -e 's%/usr/bin/perl%$(PERLPATH)%g' fb.tmp > fb.install rm -f fb.tmp clean: rm -f fb.install flexbackup-1.2.1/CHANGES 0100664 0000764 0000764 00000073552 07741551561 014212 0 ustar edwinh edwinh $Id: CHANGES,v 1.159 2003/10/10 15:42:41 edwinh Exp $ $Name: v1_2_1 $ ---------------------------------------------------------------------- Version 1.2.1 (20031010) Changes: - Added "-ignore-errors" flag to allow backups to continue even if commands return non-zero exit status. Use at your own risk. - Added contributed manpages - Add 'rsync' type that acts like 'copy' but using rsync instead of piped cpio's. You need rsync >= 2.5.6 for this to work! - Added lzop compression support - Spinning bar indicator for long-running pkg delta operations if output is a terminal - Recognize .deb files as ar archives - Skip proc/devpts/devfs/tmpfs for traverse_fs=local or all - Add hooks for mbuffer's multivolume support (experimental) Bug fixes/cleanup: - FreeBSD pkgdelta support fixed up - Cleanup better if error detected - Fix quoting in check_remote_progs if your remote shell is csh - Make sure we don't generate conflicting index keys (as long as -newtape is not run in parallel) - Cleanup a bit regarding block devices ---------------------------------------------------------------------- Version 1.2.0 (20030707) Changes: - Merge devel -> stable - Added -pipe option to archive single directory to stdout, or list/extract/compare archive from stdin - No null padding necessary for reads, even with $pad_blocks true - only needed for writes. Should take harmless warnings way from gzip,tar, and others when reading - Added -nodefaults option for debug - Allow compression for filelist type Bugfixes/cleanup: - Fix FreeBSD find w/ prune regex - Truncate tar label to 99 chars - Error if -num flag with backup write mode or using file - Error if -erase with archive read mode or using file - Fix afio volume header print filehandle - Fix if on-disk backups, and "-rmfile all", the index for the keyfile was removed but not the keyfile itself - use $:: rather than $main:: for globals, its shorter... ---------------------------------------------------------------------- Version 1.1.8 (20030621) Changes: - Detect old config file - Config file parsing not so rigid - if values not found, it now uses defaults (w/ message printed so you know it) - Can configure buffer percent full and write sleep time options - Compare works for copy type (just diff -r -q) Bugfixes/cleanup: - Doh! index key creation broken for to-disk backups as a result of 1.1.7 changes. Fixed. - Relative links for $device get chased correctly - $pkgdelta_archive_changed = false didn't work. - README updated; example for filesystem setup & cron. - find -depth w/ -prune isn't supposed to work, no bug. (see single unix spec and various manpages). ---------------------------------------------------------------------- Version 1.1.7 (20030618) Changes: - Added "copy" type that just mirrors the files into a directory tree if archiving to disk - Initial support of "-pkgdelta" with FreeBSD packages - On FreeBSD, use mt to set 1-filemark behavior globally - Globally ignore sockets - Use "find -prune" for subtree pruning. (When this happens, we have to turn off "-depth" switch, as with GNU find 4.1.7 they don't work together. Need to check into findutils more.) Bugfixes/cleanup: - Don't use --same-order for tar. -extract -flist would fail if you had more than one file and they weren't in the expected order - Use --ignore-failed-read for tar - Remote tape drive device didn't create index key correctly - Typo/update star flags setup - Tweak quiet flags ---------------------------------------------------------------------- Version 1.1.6 (20030313) Changes beyond stable 1.0.3 release: - Different way to spec the filesystems. This lets us have arbitrarily named backup "sets". Config file: $set{'tag1'} = "/dir1 /dir2 ..."; $set{'tag2'} = "/dir3 host:/dir4 ..."; Flags: "-dir
}; foreach ( sort( keys( %temp_archive ) ) ) { print qq{Friendly name: } , $temp_archive{ $_ } , qq{}; untie( %temp_archive ); } } print qq{
\n}; } print qq{
Files:}; #$passed_file =~ s/(.*)\/$/$1/; foreach ( sort( keys( %hash ) ) ) { if ( $hash{ $_ } =~ m/^$passed_file([^\/]+\/)$/ ) { $dir_hash{ $hash{ $_ } } = $1; } #elsif ( $hash{ $_ } =~ m/^$passed_file$/ ) { elsif ( $hash{ $_ } =~ m/^$passed_file$/ ) { unless ( $_ eq "\n}; } print qq{ Click here to download directory as a .tar file | Subdirectories of } , ( $CGI -> param( 'directory' ) || $passed_file ) , qq{:}; foreach ( sort( keys( %dir_hash ) ) ) { print "[$dir_hash{ $_ }]\n"; } print qq{ |
Current Directory:}; for ( $i = 0 ; $i < ( scalar( @rootlink ) - 1 ) ; $i++ ) { $full_rootlink .= "/$rootlink[ $i ]/"; $full_rootlink =~ s/\/\/*/\//; print qq{} , $rootlink[ $i ] , qq{/}; } print pop( @rootlink ), qq{\nReturn to archive selector |
\n"; } sub edit_config_page { # mess with the flexbackup config file in etc &html_header; my( $type , $compress , $compr_level , $blksize , $fs_ref ) = &read_flexbackup_conf; my( @filesystems ) = @{ $fs_ref }; my( $hour , $m_tens , $m_ones , $day , $half_day ) = &read_cron; print $CGI -> startform( -action => $script ), qq{
Scheduled backup time }, $CGI -> popup_menu( -name => 'schedule_hour', -values => [1..12], -default => $hour ), qq{:}, $CGI -> popup_menu( -name => 'schedule_minutes_tens', -values => [0..5], -default => $m_tens ), $CGI -> popup_menu( -name => 'schedule_minutes_ones', -values => [0..9], -default => $m_ones ), $CGI -> popup_menu( -name => 'schedule_halfday', -values => [ "am" , "pm" ], -default => $half_day ), qq{ every}, $CGI -> popup_menu( -name => 'schedule_day_of_week', -values => [ "Sunday" , "Monday" , "Tuesday", "Wednesday" , "Thursday" , "Friday", "Saturday" , "Weekday" , "Day" ], -default => $day ), qq{ }, $CGI -> submit( -name => 'Change Schedule' ), qq{ |
}, qq{
Filesystems to backup \n}, $CGI -> scrolling_list( -name => 'filesystems', -values => @filesystems, -size=> 5 ), $CGI -> textfield( -name => 'add_filesystem' ), qq{ }, $CGI -> submit( -name => 'Remove Filesystem' ), $CGI -> submit( -name => 'Add Filesystem' ), qq{ | },
qq{
}, qq{WARNING: changing these options is not recommended}, qq{
Backup type \n}, $CGI -> popup_menu( -name =>'type', -values => [ "afio" , "dump" , "tar", "cpio" , "zip" ], -default => $type ), qq{ | },
qq{Compress type \n}, $CGI -> popup_menu( -name => 'compress', -values => [ "false" , "gzip" , "bzip2", "compress" , "hardware" ], -default => $compress ), qq{ |
Block size \n}, $CGI -> textfield( -name => 'blksize', -default => $blksize ), qq{ | },
qq{Compress level \n}, $CGI -> popup_menu( -name => 'compress', -values => [ 1..9 ], -default => $compr_level ), qq{ |
}, $CGI -> submit( -name => 'Change Options' ), qq{ |
Suggestions/additions welcome
FAQ Revised: Friday 10 October 2003 11:26:37
$set{'tag1'} = "/dir1 /dir2 ..."; $set{'tag2'} = "/dir3 host:/dir4 ...";
You can test exclusion, levels, pruning, and package deltas by using "-type filelist". This writes a text file listing all files that would have been backed up into your current working directory.
You can also test to see if your expression matches what you want to
exclude by:
cd <topdir>; find . -regex "<expression>"
$path{command} = 'sudo command';
in the config file.
Do this for all commands that will need privilege (find, afio/tar/etc)
Cmnd_Alias BACKUP = /usr/bin/afio, /usr/bin/find
backupuser ALL = NOPASSWD: BACKUP
$path{'dump'} = 'ufsdump';
. You can
use this mechanism to override paths or just change command names.In the future we might add a way to set up "smbtar" or something similar...
$device
dir.
Note that archive-to-disk with remote directories ($device =
host:/path/to/dir
) is not currently possible. You need to either run the
backup from the machine with the disk, or use NFS.
$set{'backup'} = "'/path/file system 1' '/path/file system 2'";
See also these sun docs for an example with some pictures.
Traditional Unix 'dump' behavior only allows levels 0-9; flexbackup removes this limit if you are using any of the non-dump archive formats.
For those people coming from other platforms that are not used to
dump-style numeric levels, the latest versions of flexbackup have added
-full
, -differential
, -incremental
that get translated into appropriate dump-style level numbers.
Definitions:
These are simply aliased to numeric levels: full = 0, differential = 1, and
incremental = (latest backup + 1)
As of now, having one directory archive span multiple tapes is not supported. True, some of the archiver programs can deal with end-of media if and only if they are writing directly to the device, but it can't be done in a shell pipeline, or in any kind of generic fashion. For now, split the directory up into workable portions, use larger media, or backup to disk...
We need a program like "buffer", but that also deals with the volume end detection & media change prompting, has configuraable "new tape" scripts, etc.
Version 1.2.1 has basic support for using mbuffer with multivolumes.
Please try it out and let me know.
$staticfiles
to true in the config file. Archives will
then have names such as "etc.3.dump.gz" rather than something like
"home.0.200301212305.afio-bz2". There is a similar parameter for the log
files, by the way.Some older programs/systems barf at 2GB files -- anything recent should deal with large files no problem, using standard APIs.
If you are using "buffer" be aware that you need to compile with extra flags, or patch the 1.19 version to handle >2GB files. (The RPMS linked to from this site have been)
See also Large File
Support in Linux or Large
file size changes to the single Unix spec for more info and hints on
how to fix things if you need to recompile something.
To quickly extract just a single file, use "-extract -onefile path/to/my/file", giving the path from the archive.
To extract a list of multiple files, put them into a text file, for
instance "restorelist", then use "-extract -flist restorelist". The format
is one line per pathname, using the path of the file in the archive. Note
if you are using afio with compression you need to append ".z" to
filenames for any compressed files (depends on threshold and exclusion patterns).
dd if=/dev/tapedevice bs=10k | gzip -dc | tar xvf -
afio -i -k -x -P bzip2 -Q -d -Z -v
flexbackup -test-tape-drive
. It writes a couple
small files to the tape, then reads & diffs them. This will flush out any
problems with parameters like blocking, filemarks, and padding; or issues
with the tape drive or driver itself.
If it fails, try these one at a time:
$buffer='false'
) until you can get things working without it.
$blksize
) in the configuration file. The default is 10k, but some tape drives like 32k or 64k much better.
$pad_blocks
to false
$mt_blocksize
.
$indexes
set to false
If you are using an IDE tape drive under Linux, and are still having trouble, you might also try the 'ide-scsi' layer with the 'st' module, and treating the thing like a SCSI tape - I've had more success that way.
You can also just try writing/reading stuff straight from the drive with
an archiver and dd (to factor flexbackup out of the loop), although that's what the
-test-tape-drive
switch tries to do. If you can get that working,
but flexbackup still seems to malfunction, let us know. Try things like:
(The example below uses afio to test, tar/cpio/others will be different)
# Backup two dirs mt -f /dev/tape setblk 32768 mt -f /dev/tape rewind mt -f /dev/tape erase find /dir1 -print | afio -o -z -v -b 32k - | dd ibs=32k obs=32k of=/dev/tape find /dir2 -print | afio -o -z -v -b 32k - | dd ibs=32k obs=32k of=/dev/tape # List mt -f /dev/tape rewind dd ibs=32k obs=32k if=/dev/tape | afio -t -z -v -b 32k - dd ibs=32k obs=32k if=/dev/tape | afio -t -z -v -b 32k -
If you are using ssh and are having a problem with it asking for a
password, then you don't have a passphrase-less authorized key set up, your
RSAAuthentication parameters are incorrect, or you have some other
public/private key problem. Section 7.2.2 of the ssh FAQ might
help:
http://www.onsight.com/faq/ssh/ssh-faq.html
In a nutshell the value of your ~/.ssh/identity.pub
needs to
be present in your ~/.ssh/authorized_keys
file for RSA
authentication to work. For root accounts you might also need to
put
PermitRootLogin yes
in your sshd_config
file, as root is treated with more care than normal user accounts. Note:
you might choose without-password or forced-commands-only instead of yes
for more security. See your ssh(1) and sshd(1) man pages.
If you do not want to use passphrase-less ssh keys with root logins, See
the section on use with "sudo". (Or else sit at the terminal and type
passwords during the backups if you *really* want to...)
$buffer_megs
to a lower value and try again. I usually
run with it set somewhere between 5-20MB.
On FreeBSD, SysV shared memory can run low using the GENERIC kernel at
its default value (especially if you are running something like GNOME). You
can modify your kernel file and change the line
options SHMMAXPGS=2048
To something really big. You can also change this dynamically at startup by
putting
kern.ipc.shmmax={large integer}
in /etc/sysctl.conf. Other BSD's will have very similar mechanisms.
$erase_rewind_only
to true - then an
"erase" operation becomes just a rewind. If an archive is then written,
filemarks and tape end-of-data should work fine.
Also, this hang can be caused by incorrect tape drive parameters on some
systems. Try "flexbackup -test-tape-drive". If it hangs, and there is a dd
or buffer process continuously running, kill it and adjust blocksizes and/or
padding.
$mt{'rdhpos'} = 'rdspos';
If flexbackup detects a non-zero exit status for any command in the pipeline, it will stop. You can override this with the "--ignore-errors" switch, but then it will just blindly continue on such cases. You have been warned.
Some archivers are more picky than others. You can try switching the archive type.
If it is something like a SQL database data file causing the problem, you
should dump the database to a secondary location and back THAT data up.
(See the mysqldump command, for instance).
Flexbackup can use a "package delta" mode that cuts down the number of files to be archived by basically doing the following:
It was orginally intended for use with RPM-based systems ("-pkgdelta rpm"). Initial support for FreeBSD packages is included ("-pkgdelta freebsd"), although there the "base" files aren't part of package so they can't be excluded.
If someone knows how to do the equivalent things for
.deb packages or others, let me know and we can add it.
Now, tell me you how to set up a cron job to run once a month on the first Sunday only.... you can't do it.
The "-wday" flag tells flexbackup to just exit if the current day of the week does not match the flag. Now, to fulfill the above requirement, you add a cron entry like:
0 3 1-7 * * flexbackup -wday 7 -set all ....Which will run at 3:00AM on the first Sunday of every month.
$indexes =
'false';
in the config file, and just use written labels on the
tapes...FAQ compilation thanks to makefaq