buffer-1.19/0040755000175000001440000000000007263146527011252 5ustar mbusersbuffer-1.19/README0100644000175000001440000000424007263146524012124 0ustar mbusersThis is a program designed to speed up writing tapes on remote tape drives. Requirements are shared memory and locks which normally means that these are supported in your kernel. Buffer has been tested under SunOS 4.0.*, SunOS 4.1.*, Solarix, HP-UX 7.0, and Gould UTX 2.1A (sv universe). The program splits itself into two processes. The first process reads (and reblocks) from stdin into a shared memory buffer. The second writes from the shared memory buffer to stdout. Doing it this way means that the writing side effectly sits in a tight write loop and doesn't have to wait for input. Similarly for the input side. It is this waiting that slows down other reblocking processes, like dd. I run an archive and need to write large chunks out to tape regularly with an ethernet in the way. Using 'buffer' in a command like: tar cvf - stuff | rsh somebox buffer -o /dev/rst8 is a factor of 5 faster than the best alternative, gnu tar with its remote tape option: tar cvf somebox:/dev/rst8 stuff We have been using buffer here at Imperial for a couple of years now for writing tar tapes and the main system dumps. Thanks to Kevin Twidle for the -p and -B code. Thanks to Bard Isley for fixes to the read loop/SIGCHLD handling. Thanks to PerSteinar.Iversen@fi.uib.no for the DEC Alpha patches. Thanks to kargard@ampex.com (Erik L. Kargard) for the AMPEX enhancements. INSTALLATION: Check that your kernel supports shared memory and semaphores. A quick way to check is to build buffer and run it. If it says "couldn't create shared memory segment" you probably need to reconfigure and rebuild your kernel. To install edit the Makefile and tailor the variables to your local systems. Then type make. DISCLAIMER: This package is under the GNU GENERAL PUBLIC LICENSE! In addtion under NO circumstances can I, or Imperial College, be held liable for any event caused by the running or storing of this program or its documentation. Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, William Penney Lab, Fax: +44 171 594 8449 Imperial College, London, SW7 2BZ, UK Email: L.McLoughlin@doc.ic.ac.uk buffer-1.19/buffer.man0100644000175000001440000001173707263146525013224 0ustar mbusers.\" Buffer. Very fast reblocking filter speedy writing of tapes. .\" Copyright (C) 1990,1991 Lee McLoughlin .\" .\" This program is free software; you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation; either version 1, or (at your option) .\" any later version. .\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. .\" .\" You should have received a copy of the GNU General Public License .\" along with this program; if not, write to the Free Software .\" Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. .\" .\" Lee McLoughlin. .\" Dept of Computing, Imperial College, .\" 180 Queens Gate, London, SW7 2BZ, UK. .\" .\" Email: L.McLoughlin@doc.ic.ac.uk .TH BUFFER 1 "14 May 1990" .SH NAME buffer \- very fast reblocking program .SH SYNTAX .B buffer [\fB\-S size\fP] [\fB\-b blocks\fP] [\fB\-s size\fP] [\fB\-m size\fP] [\fB\-p percentage\fP] [\fB\-u microseconds\fP] [\fB-B\fR] [\fB-t\fR] [\fB-Z\fR] [\fB-i filename\fR] [\fB-o filename\fR] .SH OPTIONS .TP 5 .B \-i filename Use the given file as the input file. The default is stdin. .TP .B \-o filename Use the given file as the output file. The default is stdout. .TP .B \-S size After every chunk this size has been writen print out how much been writen so far. By default this is not set. .TP .B \-s size Size in bytes of each block. The default blocksize is 10k to match the normal output of the .I tar(1) program. .TP .B \-z size Combines the .B \-S and .B \-s flags. .TP .B \-b blocks Number of blocks to allocate to shared memory circular buffer. Defaults to the number required to fill up the shared memory requested. .TP .B \-m size Maximum size of the shared memory chunk to allocate for the circular queue. Defaults to one megabyte. .TP .B \-p percentage Only start a write when the given percentage of the internal queue is full. A percentage around 75 often proves best. Defaults to zero. .TP .B \-u microseconds After every write pause for this many microseconds. Defaults to zero. (Suprisingly a small sleep, 100 usecs, after each write can greatly enhance throughput on some drives.) .TP .B \-B Force each block writen to be padded out to the blocksize. This is needed by some tape and cartridge drives. Defaults to unpadded. This only affects the last block writen. .TP .B \-t On exiting print to stderr a brief message showing the total number of bytes written. .TP .B \-Z If reading/writing directly to a character device (like a tape drive) then after each gigabyte perform an lseek to the start of the file. Use this flag with extreme care. If can only be used on devices where an lseek does not rewind the tape but does reset the kernels position flags. It is used to allow more than 2 gigabytes to be written. .PP Sizes are a number with an optional trailing character. A 'b' multiplies the size by 512, a 'k' by 1024 and an 'm' by a meg. .SH DESCRIPTION .I Buffer reads from standard input reblocking to the given blocksize and writes each block to standard output. .PP Internally .I buffer is a pair of processes communicating via a large circular queue held in shared memory. The reader process only has to block when the queue is full and the writer process when the queue is empty. .I Buffer is designed to try and keep the writer side continuously busy so that it can stream when writing to tape drives. When used to write tapes with an intervening network .I buffer can result in a considerable increase in throughput. .PP The default settings for .I buffer are normally good enough. If you are a heavy tape user then it is worth your while trying out various different combinations of options. In particular running a .I buffer at both ends of the pipe can provide a substantial increase (see last example below). .SH EXAMPLES .br $ \fBbuffer < /etc/termcap > /dev/rst8\fP .br .sp $ \fBtar cf - . | rsh somehost 'buffer > /dev/rst8'\fP .br .sp $ \fBdump fu - | rsh somehost 'buffer -s 16k > /dev/nrst8'\fP .br $ \fBtar cf - . | buffer | .br \ \ \ rsh somehost 'buffer -S 500K -p 75 > /dev/rst0'\fP .SH BUGS Internally, for printing purposes, buffer counts in terms of the number of kilobytes output. If the blocksize you use is not a whole number of kilobytes then the numbers printed will be inaccurate. .SH THANKS Thanks to Kevin Twidle for a lot of early suggestions and patches to make it work with non-tar/dump tapes to exabyte drives. Thanks to Andi Karrer , Rumi Zahir and Christoph Wicki for patches to make buffer work when trying to write single tape files of greater than 2 gigabytes. .SH COPYRIGHT .if n Copyright (C) 1990, 1991 by Lee McLoughlin. .if t Copyright \(co 1990, 1991 by Lee McLoughlin. .SH SEE ALSO dd(1), tar(1), rsh(1) buffer-1.19/Makefile0100644000175000001440000000214107263146525012703 0ustar mbusers# Make the buffer program # You might need to add the following to CGFLAGS: # # Add -DSYS5 for A System 5 (USG) version of Unix # You should also add -DSYS5 for Ultrix, AIX, and Solaris. # Add -DDEF_SHMEM=n if you can only have n bytes of shared memory # (eg: -DDEF_SHMEM=524288 if you can only have half a meg.) # Add -DAMPEX to change the default settings suitable for the high capacity # Ampex drives, such as the DST 310. CC=gcc CFLAGS=-Wall # Where to install buffer and its manual pages INSTBIN=/usr/local/bin INSTMAN=/usr/man/manl # The manual page section (normally l or 1) S=l RM=/bin/rm ALL=README buffer.man Makefile buffer.c sem.c sem.h COPYING all: buffer buffer: buffer.o sem.o $(CC) -o buffer $(CFLAGS) buffer.o sem.o clean: $(RM) -f *.o core buffer .merrs install: buffer rm -f $(INSTBIN)/buffer cp buffer $(INSTBIN)/buffer chmod 111 $(INSTBIN)/buffer rm -f $(INSTMAN)/buffer.$S cp buffer.man $(INSTMAN)/buffer.$S chmod 444 $(INSTMAN)/buffer.$S buffer.tar: $(ALL) $(RM) -f buffer.tar tar cvf buffer.tar $(ALL) buffer.shar: $(ALL) $(RM) -f buffer.shar shar $(ALL) > buffer.shar buffer-1.19/buffer.c0100644000175000001440000005265007263146526012673 0ustar mbusers/* Buffer. Very fast reblocking filter speedy writing of tapes. Copyright (C) 1990,1991 Lee McLoughlin This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. Lee McLoughlin. Dept of Computing, Imperial College, 180 Queens Gate, London, SW7 2BZ, UK. Email: L.McLoughlin@doc.ic.ac.uk */ /* This is a reblocking process, designed to try and read from stdin * and write to stdout - but to always try and keep the writing side * busy. It is meant to try and stream tape writes. * * This program runs in two parts. The reader and the writer. They * communicate using shared memory with semaphores locking the access. * The shared memory implements a circular list of blocks of data. * * L.McLoughlin, Imperial College, 1990 * * $Log: buffer.c,v $ * Revision 1.19 1995/08/24 17:46:28 lmjm * Be more careful abour EINTR errors * Ingnore child processes dying. * * Revision 1.18 1993/08/25 19:07:31 lmjm * Added Brad Isleys patchs to read/sigchld handling. * * Revision 1.17 1993/06/04 10:26:39 lmjm * Cleaned up error reporting. * Spot when the child terminating is not mine but inherited from via exec. * Use only one semaphore group. * Print out why writer died on error. * * Revision 1.16 1993/05/28 10:47:32 lmjm * Debug shutdown sequence. * * Revision 1.15 1992/11/23 23:32:58 lmjm * Oops! This should be outside the ifdef * * Revision 1.14 1992/11/23 23:29:58 lmjm * allow MAX_BLOCKSIZE and DEF_SHMEM to be configured * * Revision 1.13 1992/11/23 23:22:29 lmjm * Printf's use %lu where appropriate. * * Revision 1.12 1992/11/23 23:17:55 lmjm * Got rid of floats and use Kbyte counters instead. * * Revision 1.11 1992/11/03 23:11:51 lmjm * Forgot Andi Karrer on the patch list. * * Revision 1.10 1992/11/03 22:58:41 lmjm * Cleaned up the debugging prints. * * Revision 1.9 1992/11/03 22:53:00 lmjm * Corrected stdin, stout and showevery use. * * Revision 1.8 1992/11/03 22:41:34 lmjm * Added 2Gig patches from: * Andi Karrer * Rumi Zahir * Christoph Wicki * * Revision 1.7 1992/07/23 20:42:03 lmjm * Added 't' option to print total writen at end. * * Revision 1.6 1992/04/07 19:57:30 lmjm * Added Kevins -B and -p options. * Turn off buffering to make -S output appear ok. * Added GPL. * * Revision 1.5 90/07/22 18:46:38 lmjm * Added system 5 support. * * Revision 1.4 90/07/22 18:29:48 lmjm * Updated arg handling to be more consistent. * Make sofar printing size an option. * * Revision 1.3 90/05/15 23:27:46 lmjm * Added -S option (show how much has been writen). * Added -m option to specify how much shared memory to grab. * Now tries to fill this with blocks. * reader waits for writer to terminate and then frees the shared mem and sems. * * Revision 1.2 90/01/20 21:37:59 lmjm * Reset default number of blocks and blocksize for best thruput of * standard tar 10K Allow. * blocks number of blocks to be changed. * Don't need a hole in the circular queue since the semaphores prevent block * clash. * * Revision 1.1 90/01/17 11:30:23 lmjm * Initial revision * */ #include #include #include #include #include #include #include #include #include #include #include #include "sem.h" #ifndef lint static char *rcsid = "$Header: /a/swan/home/swan/staff/csg/lmjm/src/buffer/RCS/buffer.c,v 1.19 1995/08/24 17:46:28 lmjm Exp lmjm $"; #endif #ifndef __alpha extern char *shmat(); #endif /* __alpha */ /* General macros */ #define TRUE 1 #define FALSE 0 #define K *1024 #define M *1024*1024 /* Some forward declarations */ void byee(); void start_reader_and_writer(); void parse_args(); void set_handlers(); void buffer_allocate(); void report_proc(); int do_size(); void get_buffer(); void reader(); void writer(); void writer_end(); void wait_for_writer_end(); void get_next_free_block(); void test_writer(); int fill_block(); void get_next_filled_block(); int data_to_write(); void write_blocks_to_stdout(); void write_block_to_stdout(); void pr_out(); void end_writer(); /* When showing print a note every this many bytes writen */ int showevery = 0; #define PRINT_EVERY 10 K /* Pause after every write */ unsigned write_pause; /* This is the inter-process buffer - it implements a circular list * of blocks. */ #ifdef AMPEX #define MAX_BLOCKSIZE (4 M) #define DEF_BLOCKSIZE MAX_BLOCKSIZE #define DEF_SHMEM (32 M) #endif #ifndef MAX_BLOCKSIZE #define MAX_BLOCKSIZE (512 K) #endif #ifndef DEF_BLOCKSIZE #define DEF_BLOCKSIZE (10 K) #endif int blocksize = DEF_BLOCKSIZE; /* Which process... in error reports*/ char *proc_string = "buffer"; /* Numbers of blocks in the queue. */ #define MAX_BLOCKS 2048 int blocks = 1; /* Circular increment of a buffer index */ #define INC(i) (((i)+1) == blocks ? 0 : ((i)+1)) /* Max amount of shared memory you can allocate - can't see a way to look * this up. */ #ifndef DEF_SHMEM #define DEF_SHMEM (1 K K) #endif int max_shmem = DEF_SHMEM; /* Just a flag to show unfilled */ #define NONE (-1) /* the shared memory id of the buffer */ int buffer_id = NONE; struct block { int bytes; char *data; } *curr_block; #define NO_BUFFER ((struct buffer *)-1) struct buffer { /* Id of the semaphore group */ int semid; /* writer will hang trying to lock this till reader fills in a block */ int blocks_used_lock; /* reader will hang trying to lock this till writer empties a block */ int blocks_free_lock; int next_block_in; int next_block_out; struct block block[ MAX_BLOCKS ]; /* These actual space for the blocks is here - the array extends * pass 1 */ char data_space[ 1 ]; } *pbuffer = NO_BUFFER; int buffer_size; int fdin = 0; int fdout = 1; int in_ISCHR = 0; int out_ISCHR = 0; int padblock = FALSE; int writer_pid = 0; int reader_pid = 0; int free_shm = 1; int percent = 0; int debug = 0; int Zflag = 0; int writer_status = 0; char *progname = "buffer"; char print_total = 0; /* Number of K output */ unsigned long outk = 0; int main( argc, argv ) int argc; char **argv; { parse_args( argc, argv ); set_handlers(); buffer_allocate(); start_reader_and_writer(); byee( 0 ); /* NOTREACHED */ exit( 0 ); } void parse_args( argc, argv ) int argc; char **argv; { int c; int iflag = 0; int oflag = 0; int zflag = 0; extern char *optarg; char blocks_given = FALSE; struct stat buf; while( (c = getopt( argc, argv, "BS:Zdm:s:b:p:u:ti:o:z:" )) != -1 ){ switch( c ){ case 't': /* Print to stderr the total no of bytes writen */ print_total++; break; case 'u': /* pause after write for given microseconds */ write_pause = atoi( optarg ); break; case 'B': /* Pad last block */ padblock = TRUE; break; case 'Z': /* Zero by lseek on the tape device */ Zflag = TRUE; break; case 'i': /* Input file */ iflag++; if( iflag > 1 ){ report_proc(); fprintf( stderr, "-i given twice\n" ); byee( -1 ); } if( (fdin = open( optarg, O_RDONLY )) < 0 ){ report_proc(); perror( "cannot open input file" ); fprintf( stderr, "filename: %s\n", optarg ); byee ( -1 ); } break; case 'o': /* Output file */ oflag++; if( oflag > 1 ){ report_proc(); fprintf( stderr, "-o given twice\n" ); byee( -1 ); } if( (fdout = open( optarg, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) < 0 ){ report_proc(); perror( "cannot open output file" ); fprintf( stderr, "filename: %s\n", optarg ); byee ( -1 ); } break; case 'S': /* Show every once in a while how much is printed */ showevery = do_size( optarg ); if( showevery <= 0 ) showevery = PRINT_EVERY; break; case 'd': /* debug */ debug++; if( debug == 1 ){ setbuf( stdout, NULL ); setbuf( stderr, NULL ); fprintf( stderr, "debugging turned on\n" ); } break; case 'm': /* Max size of shared memory lump */ max_shmem = do_size( optarg ); if( max_shmem < (sizeof( struct buffer ) + (blocksize * blocks)) ){ fprintf( stderr, "max_shmem %d too low\n", max_shmem ); byee( -1 ); } break; case 'b': /* Number of blocks */ blocks_given = TRUE; blocks = atoi( optarg ); if( (blocks <= 0) || (MAX_BLOCKS < blocks) ){ fprintf( stderr, "blocks %d out of range\n", blocks ); byee( -1 ); } break; case 'p': /* percent to wait before dumping */ percent = atoi( optarg ); if( (percent < 0) || (100 < percent) ){ fprintf( stderr, "percent %d out of range\n", percent ); byee( -1 ); } if( debug ) fprintf( stderr, "percent set to %d\n", percent ); break; case 'z': zflag++; /* FALL THRU */ case 's': /* Size of a block */ blocksize = do_size( optarg ); if( (blocksize <= 0) || (MAX_BLOCKSIZE < blocksize) ){ fprintf( stderr, "blocksize %d out of range\n", blocksize ); byee( -1 ); } break; default: fprintf( stderr, "Usage: %s [-B] [-t] [-S size] [-m memsize] [-b blocks] [-p percent] [-s blocksize] [-u pause] [-i infile] [-o outfile] [-z size]\n", progname ); fprintf( stderr, "-B = blocked device - pad out last block\n" ); fprintf( stderr, "-t = show total amount writen at end\n" ); fprintf( stderr, "-S size = show amount writen every size bytes\n" ); fprintf( stderr, "-m size = size of shared mem chunk to grab\n" ); fprintf( stderr, "-b num = number of blocks in queue\n" ); fprintf( stderr, "-p percent = don't start writing until percent blocks filled\n" ); fprintf( stderr, "-s size = size of a block\n" ); fprintf( stderr, "-u usecs = microseconds to sleep after each write\n" ); fprintf( stderr, "-i infile = file to read from\n" ); fprintf( stderr, "-o outfile = file to write to\n" ); fprintf( stderr, "-z size = combined -S/-s flag\n" ); byee( -1 ); } } if (zflag) showevery = blocksize; /* If -b was not given try and work out the max buffer size */ if( !blocks_given ){ blocks = (max_shmem - sizeof( struct buffer )) / blocksize; if( blocks <= 0 ){ fprintf( stderr, "Cannot handle blocks that big, aborting!\n" ); byee( -1 ); } if( MAX_BLOCKS < blocks ){ fprintf( stderr, "Cannot handle that many blocks, aborting!\n" ); byee( -1 ); } } /* check if fdin or fdout are character special files */ if( fstat( fdin, &buf ) != 0 ){ report_proc(); perror( "can't stat input file" ); byee( -1 ); } in_ISCHR = S_ISCHR( buf.st_mode ); if( fstat( fdout, &buf ) != 0 ){ report_proc(); perror( "can't stat output file" ); byee( -1 ); } out_ISCHR = S_ISCHR( buf.st_mode ); } /* The interrupt handler */ void shutdown() { static int shutting; if( shutting ){ if( debug ) fprintf( stderr, "%s: ALREADY SHUTTING!\n", proc_string ); return; } shutting = 1; if( debug ) fprintf( stderr, "%s: shutdown on signal\n", proc_string ); byee( -1 ); } /* Shutdown because the child has ended */ void child_shutdown() { /* Find out which child has died. (They may not be my * children if buffer was exec'd on top of something that had * childred.) */ int deadpid; while( (deadpid = waitpid( -1, &writer_status, WNOHANG )) && deadpid != -1 && deadpid != 0 ){ if( debug > 2 ) fprintf( stderr, "child_shutdown %d: 0x%04x\n", deadpid, writer_status ); if( deadpid == writer_pid ){ if( debug > 2 ) fprintf( stderr, "writer has ended\n" ); writer_pid = 0; byee( 0 ); } } } void set_handlers() { if( debug ) fprintf( stderr, "%s: setting handlers\n", proc_string ); signal( SIGHUP, shutdown ); signal( SIGINT, shutdown ); signal( SIGQUIT, shutdown ); signal( SIGTERM, shutdown ); #ifdef SIGCHLD signal( SIGCHLD, child_shutdown ); #else #ifdef SIGCLD signal( SIGCLD, child_shutdown ); #endif #endif } void buffer_allocate() { /* Allow for the data space */ buffer_size = sizeof( struct buffer ) + ((blocks * blocksize) - sizeof( char )); /* Create the space for the buffer */ buffer_id = shmget( IPC_PRIVATE, buffer_size, IPC_CREAT|S_IREAD|S_IWRITE ); if( buffer_id < 0 ){ report_proc(); perror( "couldn't create shared memory segment" ); byee( -1 ); } get_buffer(); if( debug ) fprintf( stderr, "%s pbuffer is 0x%08x, buffer_size is %d [%d x %d]\n", proc_string, (char *)pbuffer, buffer_size, blocks, blocksize ); #ifdef SYS5 memset( (char *)pbuffer, '\0', buffer_size ); #else bzero( (char *)pbuffer, buffer_size ); #endif pbuffer->semid = -1; pbuffer->blocks_used_lock = -1; pbuffer->blocks_free_lock = -1; pbuffer->semid = new_sems( 2 ); /* Get a read and a write sem */ pbuffer->blocks_used_lock = 0; /* Start it off locked - it is unlocked when a buffer gets filled in */ lock( pbuffer->semid, pbuffer->blocks_used_lock ); pbuffer->blocks_free_lock = 1; /* start this off so lock() can be called on it for each block * till all the blocks are used up */ sem_set( pbuffer->semid, pbuffer->blocks_free_lock, blocks - 1 ); /* Detattach the shared memory so the fork doesnt do anything odd */ shmdt( (char *)pbuffer ); pbuffer = NO_BUFFER; } void buffer_remove() { static char removing = FALSE; /* Avoid accidental recursion */ if( removing ) return; removing = TRUE; /* Buffer not yet created */ if( buffer_id == NONE ) return; /* There should be a buffer so this must be after its detached it * but before the fork picks it up */ if( pbuffer == NO_BUFFER ) get_buffer(); if( debug ) fprintf( stderr, "%s: removing semaphores and buffer\n", proc_string ); remove_sems( pbuffer->semid ); if( shmctl( buffer_id, IPC_RMID, (struct shmid_ds *)0 ) == -1 ){ report_proc(); perror( "failed to remove shared memory buffer" ); } } void get_buffer() { int b; /* Grab the buffer space */ pbuffer = (struct buffer *)shmat( buffer_id, (char *)0, 0 ); if( pbuffer == NO_BUFFER ){ report_proc(); perror( "failed to attach shared memory" ); byee( -1 ); } /* Setup the data space pointers */ for( b = 0; b < blocks; b++ ) pbuffer->block[ b ].data = &pbuffer->data_space[ b * blocksize ]; } void start_reader_and_writer() { fflush( stdout ); fflush( stderr ); if( (writer_pid = fork()) == -1 ){ report_proc(); perror( "unable to fork" ); byee( -1 ); } else if( writer_pid == 0 ){ free_shm = 0; proc_string = "buffer (writer)"; reader_pid = getppid(); /* Never trust fork() to propogate signals - reset them */ set_handlers(); writer(); } else { proc_string = "buffer (reader)"; reader(); wait_for_writer_end(); } } /* Read from stdin into the buffer */ void reader() { if( debug ) fprintf( stderr, "R: Entering reader\n" ); get_buffer(); while( 1 ){ get_next_free_block(); if( ! fill_block() ) break; } if( debug ) fprintf( stderr, "R: Exiting reader\n" ); } void get_next_free_block() { test_writer(); /* Maybe wait till there is room in the buffer */ lock( pbuffer->semid, pbuffer->blocks_free_lock ); curr_block = &pbuffer->block[ pbuffer->next_block_in ]; pbuffer->next_block_in = INC( pbuffer->next_block_in ); } int fill_block() { int bytes; char *start; int toread; static char eof_reached = 0; if( eof_reached ){ curr_block->bytes = 0; unlock( pbuffer->semid, pbuffer->blocks_used_lock ); return 0; } start = curr_block->data; toread = blocksize; /* Fill the block with input. This reblocks the input. */ while( toread != 0 ){ bytes = read( fdin, start, toread ); if( bytes <= 0 ){ /* catch interrupted system calls for death * of children in pipeline */ if( bytes < 0 && errno == EINTR ) continue; break; } start += bytes; toread -= bytes; } if( bytes == 0 ) eof_reached = 1; if( bytes < 0 ){ report_proc(); perror( "failed to read input" ); byee( -1 ); } /* number of bytes available. Zero will be taken as eof */ if( !padblock || toread == blocksize ) curr_block->bytes = blocksize - toread; else { if( toread ) bzero( start, toread ); curr_block->bytes = blocksize; } if( debug > 1 ) fprintf( stderr, "R: got %d bytes\n", curr_block->bytes ); unlock( pbuffer->semid, pbuffer->blocks_used_lock ); return curr_block->bytes; } /* Write the buffer to stdout */ void writer() { int filled = 0; int maxfilled = (blocks * percent) / 100; int first_block; if( debug ) fprintf( stderr, "\tW: Entering writer\n blocks = %d\n maxfilled = %d\n", blocks, maxfilled ); get_buffer(); while( 1 ){ if( !filled ) first_block = pbuffer->next_block_out; get_next_filled_block(); if( !data_to_write() ) break; filled++; if( debug > 1 ) fprintf( stderr, "W: filled = %d\n", filled ); if( filled >= maxfilled ){ if( debug > 1 ) fprintf( stderr, "W: writing\n" ); write_blocks_to_stdout( filled, first_block ); filled = 0; } } write_blocks_to_stdout( filled, first_block ); if( showevery ){ pr_out(); fprintf( stderr, "\n" ); } if( print_total ){ fprintf( stderr, "Kilobytes Out %lu\n", outk ); } if( debug ) fprintf( stderr, "\tW: Exiting writer\n" ); } void get_next_filled_block() { /* Hang till some data is available */ lock( pbuffer->semid, pbuffer->blocks_used_lock ); curr_block = &pbuffer->block[ pbuffer->next_block_out ]; pbuffer->next_block_out = INC( pbuffer->next_block_out ); } int data_to_write() { return curr_block->bytes; } void write_blocks_to_stdout( filled, first_block ) int filled; int first_block; { pbuffer->next_block_out = first_block; while( filled-- ){ curr_block = &pbuffer->block[ pbuffer->next_block_out ]; pbuffer->next_block_out = INC( pbuffer->next_block_out ); write_block_to_stdout(); } } void write_block_to_stdout() { static unsigned long out = 0; static unsigned long last_gb = 0; static unsigned long next_k = 0; int written; if( next_k == 0 && showevery ){ if( debug > 3 ) fprintf( stderr, "W: next_k = %lu showevery = %d\n", next_k, showevery ); showevery = showevery / 1024; next_k = showevery; } if( (written = write( fdout, curr_block->data, curr_block->bytes )) != curr_block->bytes ){ report_proc(); perror( "write of data failed" ); fprintf( stderr, "bytes to write=%d, bytes written=%d, total written %10luK\n", curr_block->bytes, written, outk ); byee( -1 ); } if( write_pause ){ usleep( write_pause ); } out = curr_block->bytes / 1024; outk += out; last_gb += out; /* * on character special devices (tapes), do an lseek() every 1 Gb, * to overcome the 2Gb limit. This resets the file offset to * zero, but -- at least on exabyte SCSI drives -- does not perform * any actual action on the tape. */ if( Zflag && last_gb >= 1 K K ){ last_gb = 0; if( in_ISCHR ) (void) lseek( fdin, 0, SEEK_SET); if( out_ISCHR ) (void) lseek( fdout, 0, SEEK_SET); } if( showevery ){ if( debug > 3 ) fprintf( stderr, "W: outk = %lu, next_k = %lu\n", outk, next_k ); if( outk >= next_k ){ pr_out(); next_k += showevery; } } unlock( pbuffer->semid, pbuffer->blocks_free_lock ); } void byee( exit_val ) int exit_val; { if( writer_pid != 0 ){ if( exit_val != 0 ){ /* I am shutting down due to an error. * Shut the writer down or else it will try to access * the freed up locks */ end_writer(); } wait_for_writer_end(); } if( free_shm ){ buffer_remove(); } #ifdef SIGCHLD signal( SIGCHLD, SIG_IGN ); #else #ifdef SIGCLD signal( SIGCLD, SIG_IGN ); #endif #endif /* If the child died or was killed show this in the exit value */ if( writer_status ){ if( WEXITSTATUS( writer_status ) || WIFSIGNALED( writer_status ) ){ if( debug ) fprintf( stderr, "writer died badly: 0x%04x\n", writer_status ); exit( -2 ); } } exit( exit_val ); } /* Kill off the writer */ void end_writer() { if( writer_pid ) kill( writer_pid, SIGHUP ); } void wait_for_writer_end() { int deadpid; /* Now wait for the writer to finish */ while( writer_pid && ((deadpid = wait( &writer_status )) != writer_pid) && deadpid != -1 ) ; } void test_writer() { /* Has the writer gone unexpectedly? */ if( writer_pid == 0 ){ fprintf( stderr, "writer has died unexpectedly\n" ); byee( -1 ); } } /* Given a string of [] returns a num * suff = * m/M for 1meg * k/K for 1k * b/B for 512 */ int do_size( arg ) char *arg; { char format[ 20 ]; int ret; *format = '\0'; sscanf( arg, "%d%s", &ret, format ); switch( *format ){ case 'm': case 'M': ret = ret K K; break; case 'k': case 'K': ret = ret K; break; case 'b': case 'B': ret *= 512; break; } return ret; } void pr_out() { fprintf( stderr, " %10luK\r", outk ); } #ifdef SYS5 #include #ifndef __alpha bzero( b, l ) char *b; unsigned l; { memset( b, '\0', l ); } #endif /* __alpha */ usleep_back() { } void usleep( u ) unsigned u; { struct itimerval old, t; signal( SIGALRM, usleep_back ); t.it_interval.tv_sec = 0; t.it_interval.tv_usec = 0; t.it_value.tv_sec = u / 1000000; t.it_value.tv_usec = u % 1000000; setitimer( ITIMER_REAL, &t, &old ); pause(); setitimer( ITIMER_REAL, &old, NULL ); } #endif /* Called before error reports */ void report_proc() { fprintf( stderr, "%s: ", proc_string ); } buffer-1.19/sem.c0100644000175000001440000000601707263146526012202 0ustar mbusers/* Buffer. Very fast reblocking filter speedy writing of tapes. Copyright (C) 1990,1991 Lee McLoughlin This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. Lee McLoughlin. Dept of Computing, Imperial College, 180 Queens Gate, London, SW7 2BZ, UK. Email: L.McLoughlin@doc.ic.ac.uk */ /* This is a simple module to provide an easier to understand interface to * semaphores */ #include #include #include #include #include #include #include "sem.h" #if defined(SYS5) || defined(ultrix) || defined(_AIX) union semun { int val; struct semid_ds *buf; ushort *array; }; #endif /* IMPORTS */ /* Used to print error messages */ extern void report_proc(); /* Used to end the program - on error */ extern void byee(); /* Set a semaphore to a particular value - meant to be used before * first lock/unlock */ void sem_set( sem_id, semn, val ) int sem_id; int semn; int val; { union semun arg; extern int errno; arg.val = val; errno = 0; semctl( sem_id, semn, SETVAL, arg ); if( errno != 0 ){ report_proc(); perror( "internal error, sem_set" ); byee( -1 ); } } int new_sems( nsems ) int nsems; { int sem; int i; sem = semget( IPC_PRIVATE, nsems, IPC_CREAT|S_IREAD|S_IWRITE ); if( sem < 0 ){ report_proc(); perror( "internal error, couldn't create semaphore" ); byee( -1 ); } for( i = 0; i < nsems; i++ ){ sem_set( sem, i, 1 ); } return sem; } static do_sem( sem_id, pbuf, err ) int sem_id; struct sembuf *pbuf; char *err; { /* This just keeps us going in case of EINTR */ while( 1 ){ if( semop( sem_id, pbuf, 1 ) == -1 ){ if( errno == EINTR ){ continue; } report_proc(); fprintf( stderr, "internal error pid %d, lock id %d\n", getpid(), sem_id ); perror( err ); byee( -1 ); } return; } } void lock( sem_id, semn ) int sem_id; int semn; { struct sembuf sembuf; sembuf.sem_num = semn; sembuf.sem_op = -1; sembuf.sem_flg = 0; do_sem( sem_id, &sembuf, "lock error" ); } void unlock( sem_id, semn ) int sem_id; int semn; { struct sembuf sembuf; sembuf.sem_num = semn; sembuf.sem_op = 1; sembuf.sem_flg = 0; do_sem( sem_id, &sembuf, "unlock error" ); } void remove_sems( sem_id ) int sem_id; { if( sem_id == -1 ) return; if( semctl( sem_id, 0, IPC_RMID, NULL ) == -1 ){ report_proc(); perror( "internal error, failed to remove semaphore" ); } } buffer-1.19/sem.h0100644000175000001440000000224507263146526012206 0ustar mbusers/* Buffer. Very fast reblocking filter speedy writing of tapes. Copyright (C) 1990,1991 Lee McLoughlin This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. Lee McLoughlin. Dept of Computing, Imperial College, 180 Queens Gate, London, SW7 2BZ, UK. Email: L.McLoughlin@doc.ic.ac.uk */ /* This is a simple module to provide an easier to understand interface to * semaphores */ /* Allocate new semaphores */ int new_sems(); /* Perform actions on semaphores */ void sem_set(); void lock(); void unlock(); void remove_sems(); buffer-1.19/COPYING0100644000175000001440000004307607263146527012314 0ustar mbusers GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc. 675 Mass Ave, Cambridge, MA 02139, USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Library General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS Appendix: How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) 19yy This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) 19yy name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Library General Public License instead of this License.