pbzip2-1.1.9/AUTHORS0000644000000000000000000001026112322570763012135 0ustar 00000000000000Authors of pbzip2 ----------------- Jeff Gilchrist (http://gilchrist.ca/jeff/) - Using modified producer/consumer threading code from Andrae Muys - Uses libbzip2 by Julian Seward (http://www.bzip.org/) - Major contributions by Yavor Nikolov Contributions ------------- Bryan Stillwell - code cleanup, RPM spec, and prep work for inclusion in Fedora Extras Dru Lemley [http://lemley.net/smp.html] - help with large file support Kir Kolyshkin - autodetection for # of CPUs Joergen Ramskov - initial version of man page Peter Cordes - code cleanup Kurt Fitzner - port to Windows compilers and decompression throttling Oliver Falk - RPM spec update Jindrich Novy - code cleanup and bug fixes Benjamin Reed - autodetection for # of CPUs in OSX Chris Dearman - fixed pthreads race condition that led to pthread resources issues when processing large numbers of files and random segfaults Richard Russon - help fix decompression bug Paul Pluzhnikov - fixed minor memory leak Aníbal Monsalve Salazar - creates and maintains Debian packages Steve Christensen - creates and maintains Solaris packages (sunfreeware.com) Alessio Cervellin - creates and maintains Solaris packages (blastwave.org) Ying-Chieh Liao - created the FreeBSD port Andrew Pantyukhin - maintains the FreeBSD ports and willing to resolve any FreeBSD-related problems Roland Illig - creates and maintains NetBSD packages Matt Turner - code cleanup Álvaro Reguly - RPM spec update to support SUSE Linux Ivan Voras - support for stdin and pipes during compression and CPU detect changes John Dalton - code cleanup and bug fix for stdin support Rene Georgi - code and Makefile cleanup, support for direct decompress and bzcat René Rhéaume & Jeroen Roovers - patch to support uclibc's lack of a getloadavg function Reinhard Schiedermeier - support for tar --use-compress-prog=pbzip2 Elbert Pol - creates and maintains OS/2 packages Nico Vrouwe - support for CPU detection on Win32 Eduardo Terol - creates and maintains Windows 32bit package Nikita Zhuk - creates and maintains Mac OS X Automator action and workflow/service Jari Aalto - added long options to man page and -h output, added --loadavg, --read long options Scott Emery - ignore fwrite return and pass chown errors in writeFileMetaData if effective uid root Steven Chamberlain - code to support throttling compression to prevent memory exhaustion with slow output pipe Yavor Nikolov - multiple improvements and bugfixes David James - provided patch to fix deadlock due to unsynchronized broadcast (bug #876686) Gordon - provided patch for improving I/O error messages (bug #874605) Special thanks for suggestions and testing ------------------------------------------- Phillippe Welsh, James Terhune, Dru Lemley, Bryan Stillwell, George Chalissery, Kir Kolyshkin, Madhu Kangara, Mike Furr, Joergen Ramskov, Kurt Fitzner, Peter Cordes, Oliver Falk, Jindrich Novy, Benjamin Reed, Chris Dearman, Richard Russon, Aníbal Monsalve Salazar, Jim Leonard, Paul Pluzhniko, Robert Archard, Coran Fisher, Ken Takusagawa, David Pyke, Matt Turner, Damien Ancelin, Álvaro Reguly, Ivan Voras, John Dalton, Sami Liedes, Rene Georgi, René Rhéaume, Jeroen Roovers, Reinhard Schiedermeier, Kari Pahula, Elbert Pol, Nico Vrouwe, Eduardo Terol, Samuel Thibault, Michael Fuereder, Jari Aalto, Scott Emery, Steven Chamberlain, Yavor Nikolov, Nikita Zhuk, Joao Seabra, Conn Clark, Mark A. Haun, Tim Bielawa, Michal Gorny, Mikolaj Habdank, Christian Kujau, Marc-Christian Petersen, Piero Ottuzzi, Ephraim Ofir, Laszlo Ersek, Dima Tisnek, Tanguy Fautre. pbzip2-1.1.9/BZ2StreamScanner.cpp0000644000000000000000000003764112322570763014667 0ustar 00000000000000/* * File: BZ2StreamScanner.cpp * Author: Yavor Nikolov * * Created on March 6, 2010, 10:07 PM */ #include "pbzip2.h" #include "BZ2StreamScanner.h" #include #include #include #include #include #include using namespace std; namespace pbzip2 { const size_t BZ2StreamScanner::DEFAULT_OUT_BUFF_LIMIT; BZ2StreamScanner::BZ2StreamScanner( int hInFile, size_t inBuffCapacity ): _inBuff( NULL ), _inBuffCapacity( 0 ) { _outBuff.buf = NULL; _outBuff.bufSize = 0; init( hInFile, inBuffCapacity ); } /** * Initialize - position to beginning of input file and prepare for searching. * * @return 0 - on success; -1 on error. */ int BZ2StreamScanner::init( int hInFile, size_t inBuffCapacity ) { dispose(); CharType bz2header[] = "BZh91AY&SY"; // zero-terminated string CharType bz2ZeroHeader[] = { 'B', 'Z', 'h', '9', 0x17, 0x72, 0x45, 0x38, 0x50, 0x90, 0 }; _hInFile = hInFile; _eof = false; _bz2Header = bz2header; _bz2HeaderZero = bz2ZeroHeader; _bz2HeaderFound = false; _inBuffCapacity = 0; _errState = 0; _searchStatus = false; _outBuffCapacityHint = 0; _outBuffCapacityLimit = DEFAULT_OUT_BUFF_LIMIT; _outSequenceNumber = 0; _streamNumber = 0; // Prevent too small buffer if ( inBuffCapacity < 2 * _bz2Header.size() ) { inBuffCapacity = 2 * _bz2Header.size(); } // allocate memory to read in file _inBuff = new(std::nothrow) CharType[inBuffCapacity]; if ( _inBuff == NULL ) { _errState |= ERR_MEM_ALLOC_INBUFF; _inBuffEnd = NULL; handle_error( EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (FileData)! Aborting...\n" ); return -1; } _inBuffCapacity = inBuffCapacity; _inBuffCurrent = _inBuffSearchPtr = _inBuffEnd = _inBuff; return 0; } /** * dispose memory resources */ void BZ2StreamScanner::dispose() { disposeMemory( _outBuff.buf ); _outBuff.bufSize = 0; disposeMemory( _inBuff ); _inBuffCapacity = 0; // close( _hInFile ); } BZ2StreamScanner::~BZ2StreamScanner() { dispose(); } /** * Verify if there is enough space in output buffer. If not - then allocate. */ int BZ2StreamScanner::ensureOutBuffCapacity( size_t newSize ) { #ifdef PBZIP_DEBUG fprintf( stderr, " start ensureOutBuffCapacity/newSize=%"PRIuMAX": [", (uintmax_t) newSize ); printCurrentState(); fprintf( stderr, "\n" ); #endif if ( newSize <= _outBuffCapacity ) { return 0; // enough capacity already } if ( newSize > _outBuffCapacityHint ) { _outBuffCapacityHint = ( 11 * newSize ) / 10; if ( ( newSize <= getOutBuffCapacityLimit() ) && ( _outBuffCapacityHint > getOutBuffCapacityLimit() ) ) { _outBuffCapacityHint = getOutBuffCapacityLimit(); } } char * newBuff = new(std::nothrow) char[_outBuffCapacityHint]; if ( newBuff == NULL ) { handle_error( EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (ensureOutBuffCapacity/%u)!" "Aborting...\n", _outBuffCapacityHint ); _errState |= ERR_MEM_ALLOC_OUTBUFF; return -1; } if ( _outBuff.buf != NULL ) { memcpy( newBuff, _outBuff.buf, _outBuff.bufSize ); delete [] _outBuff.buf; } initOutBuff( newBuff, _outBuff.bufSize, _outBuffCapacityHint ); #ifdef PBZIP_DEBUG fprintf( stderr, " end ensureOutBuffCapacity/newSize=%"PRIuMAX": [", (uintmax_t) newSize ); printCurrentState(); fprintf( stderr, "\n" ); #endif return 0; } /** * Depending on wether we have already found bz2 header or not - either append * data to output buffer or discard it. * On append [current, end) is appended to output buffer. Output buffer is * extended if there is not enough existing space available in it. * * @return the number of bytes appended to output buff or skipped. -1 on error. */ int BZ2StreamScanner::appendOutBuffData(CharType * end) { int additionSize = end - getInBuffCurrent(); #ifdef PBZIP_DEBUG fprintf( stderr, " start appendOutBuffData/%d: [", additionSize ); printCurrentState(); fprintf( stderr, "\n" ); #endif if ( additionSize <= 0 ) { return 0; } if ( isBz2HeaderFound() ) { size_t newSize = _outBuff.bufSize + additionSize; if ( ensureOutBuffCapacity( newSize ) != 0 ) { return - 1; // failure encountered } memcpy( getOutBuffEnd(), getInBuffCurrent(), additionSize ); _outBuff.bufSize += additionSize; } // slide current position _inBuffCurrent = end; #ifdef PBZIP_DEBUG fprintf( stderr, " end appendOutBuffData/%d: [", additionSize ); printCurrentState(); fprintf( stderr, "\n" ); #endif return additionSize; } /** * Append available data from [current, search pos) to output buffer but * just up to fill current out buffer capacity */ int BZ2StreamScanner::appendOutBuffDataUpToLimit() { size_t maxCapacity = std::max( getOutBuffCapacityLimit(), _outBuffCapacity ); int maxAddition = maxCapacity - _outBuff.bufSize; if (maxAddition <= 0 ) { return 0; } CharType * end1; if ( eof() ) { end1 = getInBuffEnd(); } else { // subtract header size to keep the tail (since start of next header may be in it) end1 = std::min( getInBuffSearchPtr(), getInBuffEnd() - ( getHeaderSize() - 1 ) ); } CharType * end2 = getInBuffCurrent() + maxAddition; CharType * end = std::min( end1, end2 ); return appendOutBuffData( end ); } /** * Copy end section of input buffer to beginning just in case the BZIP2 header * is located between two buffer boundaries. Copy the other remaining * data into output buffer. */ int BZ2StreamScanner::rewindInBuff() { // temporarily mark tail beginning (not real header position) _inBuffSearchPtr = getInBuffEnd() - ( _bz2Header.size() - 1 ); int ret = appendOutBuffData(); if ( failed() || ( ret < 0 ) ) { return -1; } else if ( ret == 0 ) { // search pos <= current _inBuffSearchPtr = getInBuffCurrent(); } int tailSize = getInBuffEnd() - getInBuffSearchPtr(); #ifdef PBZIP_DEBUG fprintf( stderr, " rewindInBuff: tail len: %d; app ret=%d [", tailSize, ret ); printCurrentState(); fprintf( stderr, "\n" ); #endif // copy tail of input buffer to start and cut the rest std::copy( getInBuffSearchPtr(), getInBuffEnd(), getInBuffBegin() ); _inBuffEnd = getInBuffBegin() + tailSize; _inBuffCurrent = getInBuffBegin(); _inBuffSearchPtr = getInBuffBegin(); #ifdef PBZIP_DEBUG fprintf( stderr, " end rewindInBuff: tail len: %d; app ret=%d [", tailSize, ret ); printCurrentState(); fprintf( stderr, "\n" ); #endif return 0; } /** * Load data from file to input buffer. Read untill buffer is full or end of * file is reached or error is encountered. * * Enough additional capacity should be ensured otherwise may return 0 before * eof. * * @return Returns number of read bytes on success; 0 - on end of file; < 0 on error */ int BZ2StreamScanner::readData() { rewindInBuff(); if ( failed() ) { return -1; } if ( getInBuffSize() >= getInBuffCapacity() ) { handle_error( EF_EXIT, -1, "pbzip2: *ERROR: BZ2StreamScanner::readData not enough buffer free space!" " inBuffSize=%u, _inBuffCapacity=%u! Aborting...\n", getInBuffSize(), getInBuffCapacity() ); _errState |= ERR_IO_INSUFFICIENT_BUFF_CAPACITY; return -1; } int bytesRead = do_read( _hInFile, getInBuffEnd(), getInBuffCapacity() - getInBuffSize() ); #ifdef PBZIP_DEBUG fprintf( stderr, " readData: %d bytes read\n", bytesRead ); #endif if ( bytesRead > 0 ) { _inBuffEnd += bytesRead; } else if ( bytesRead < 0 ) { handle_error( EF_EXIT, -1, "pbzip2: *ERROR: Could not read from input file [err=%d]! Aborting...\n", bytesRead ); _errState |= ERR_IO_READ; return bytesRead; } else // ( bytesRead == 0 ) { _eof = true; } return bytesRead; } /** * Locate BZh header prefix in buffer. In case of first search - just check * the beginning of buffer and signal error if it doesn't match to headers. * * @return pointer to BZh header prefix if located. getInBuffEnd() if not. * failure() and getErrState() will indicate error if such occurred. */ BZ2StreamScanner::CharType * BZ2StreamScanner::locateHeaderPrefixInBuff() { size_t prefixLen = 3; #ifdef PBZIP_DEBUG fprintf( stderr, " start locateHeaderPrefixInBuff; " ); printCurrentState(); fprintf( stderr, "\n" ); #endif // first search if ( !isBz2HeaderFound() ) { if ( ( getInBuffSearchPtr() != getInBuffBegin() ) || ( getInBuffSize() < _bz2Header.size() ) ) { _errState |= ERR_INVALID_FILE_FORMAT; _inBuffSearchPtr = getInBuffEnd(); } else if ( _bz2Header.compare( 0, prefixLen, getInBuffSearchPtr(), prefixLen ) == 0 ) { // header prefix found } else { _errState |= ERR_INVALID_FILE_FORMAT; _inBuffSearchPtr = getInBuffEnd(); } } else { _inBuffSearchPtr = std::search( getInBuffSearchPtr(), getInBuffEnd(), _bz2Header.begin(), _bz2Header.begin() + prefixLen ); } #ifdef PBZIP_DEBUG if ( getInBuffSearchPtr() != getInBuffEnd() ) { fprintf( stderr, " end locateHeaderPrefixInBuff - header prefix found; " ); } else { fprintf( stderr, " end locateHeaderPrefixInBuff - header prefix not found; " ); } printCurrentState(); fprintf( stderr, "\n" ); #endif return getInBuffSearchPtr(); } /** * Search next bz2 header just in currently available input buffer. * (Doesn't read more data from file). * * @return pointer to header or getInBuffEnd() if such is not found. */ BZ2StreamScanner::CharType * BZ2StreamScanner::searchNextHeaderInBuff() { #ifdef PBZIP_DEBUG fprintf( stderr, " start searchNextHeaderInBuff; " ); printCurrentState(); fprintf( stderr, "\n" ); #endif _searchStatus = false; size_t prefixLen = 3; size_t hsp = prefixLen + 1; // header selection position locateHeaderPrefixInBuff(); while ( !failed() && ( getUnsearchedCount() >= getHeaderSize() ) ) { // _inBuffSearchPtr += prefixLen; basic_string * pHdr = NULL; if ( getInBuffSearchPtr()[hsp] == _bz2Header[hsp] ) { pHdr = &_bz2Header; #ifdef PBZIP_DEBUG fprintf( stderr, " searchNextHeaderInBuff - kind of NON-ZERO header\n" ); #endif } else if ( getInBuffSearchPtr()[hsp] == _bz2HeaderZero[hsp] ) { pHdr = &_bz2HeaderZero; #ifdef PBZIP_DEBUG fprintf( stderr, " searchNextHeaderInBuff - kind of ZERO header\n" ); #endif } if ( pHdr != NULL ) { CharType bwtSizeChar = getInBuffSearchPtr()[prefixLen]; if ( ( bwtSizeChar >= '1' ) && ( bwtSizeChar <= '9' ) ) { (*pHdr)[prefixLen] = bwtSizeChar; // compare the remaining part of magic header int cmpres = pHdr->compare( hsp, pHdr->size() - hsp, getInBuffSearchPtr() + hsp, pHdr->size() - hsp ); #ifdef PBZIP_DEBUG fprintf( stderr, " searchNextHeaderInBuff:cmpres=%d\n", cmpres ); #endif if ( cmpres == 0 ) { _searchStatus = true; #ifdef PBZIP_DEBUG fprintf( stderr, " end searchNextHeaderInBuff - found; " ); printCurrentState(); fprintf( stderr, "\n" ); #endif return _inBuffSearchPtr; } } } if ( !isBz2HeaderFound() ) { // not finding header on first search means failure _errState |= ERR_INVALID_FILE_FORMAT; break; } else { _inBuffSearchPtr += prefixLen; locateHeaderPrefixInBuff(); } } // no header has been found if we're here _inBuffSearchPtr = getInBuffEnd(); #ifdef PBZIP_DEBUG fprintf( stderr, " end searchNextHeaderInBuff; " ); printCurrentState(); fprintf( stderr, "\n" ); #endif return _inBuffSearchPtr; } #ifdef PBZIP_DEBUG void BZ2StreamScanner::printCurrentState() { fprintf( stderr, "current=%ld, search pos=%ld, end pos=%ld; s-c=%ld" "; out buf size=%d; out buf capacity=%d; header found=%d; search status=%d", (long)(getInBuffCurrent() - getInBuffBegin()), (long)(getInBuffSearchPtr() - getInBuffBegin()), (long)(getInBuffEnd() - getInBuffBegin()), (long)(getInBuffSearchPtr() - getInBuffCurrent()), (int)_outBuff.bufSize, (int)_outBuffCapacity, (int)isBz2HeaderFound(), (int)getSearchStatus() ); } #endif /** * Search next bz2 header. Read more data from file if needed. * * @return pointer to header is returned if found; * getInBuffEnd() - if not found (or error). * One should check failure() or _errorState for error details. */ BZ2StreamScanner::CharType * BZ2StreamScanner::searchNextHeader() { #ifdef PBZIP_DEBUG fprintf( stderr, " start searchNextHeader %u/%"PRIuMAX"... : ", (unsigned) (getInBuffSearchPtr() - getInBuffBegin()), (uintmax_t) getInBuffSize() ); printCurrentState(); fprintf( stderr, "\n" ); #endif if ( getUnsearchedCount() > 0 ) { searchNextHeaderInBuff(); } while ( !getSearchStatus() && !eof() && !failed() && !isOutBuffFullEnough() ) { readData(); if ( failed() ) { return getInBuffEnd(); } searchNextHeaderInBuff(); } if ( getSearchStatus() ) { _bz2HeaderFound = true; #ifdef PBZIP_DEBUG fprintf( stderr, " header found; " ); printCurrentState(); fprintf( stderr, "\n" ); #endif } if ( failed() ) { return _inBuffSearchPtr = getInBuffEnd(); } #ifdef PBZIP_DEBUG fprintf( stderr, " end searchNextHeader %u/%"PRIuMAX"... NOT FOUND: ", (unsigned) (getInBuffSearchPtr() - getInBuffBegin()), (uintmax_t) getInBuffSize() ); printCurrentState(); fprintf( stderr, "\n" ); #endif return _inBuffSearchPtr; } /** * Get next BZ2 stream from the input. * * @return output buffer initialized with bz2 stream. failure() should be checked * after calling this method - true would mean failure(). If failure() is false: * - outBuff.bufSize == 0 indicates end of file; */ outBuff * BZ2StreamScanner::getNextStream() { initOutBuff(); #ifdef PBZIP_DEBUG static OFF_T blockNum = 0; #endif outBuff * res = new(std::nothrow) outBuff; if ( res == NULL ) { handle_error( EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (getNextStream/%u)!" "Aborting...\n", (unsigned) sizeof( outBuff ) ); _errState |= ERR_MEM_ALLOC_OUTBUFF; return res; } res->buf = NULL; res->bufSize = std::numeric_limits::max(); // first search if ( !failed() && !isBz2HeaderFound() ) { #ifdef PBZIP_DEBUG blockNum = 0; fprintf( stderr, " First search start\n" ); #endif _searchStatus = false; searchNextHeader(); } if ( failed() ) { return res; } if ( ( getInBuffCurrent() == getInBuffEnd() ) && eof() ) { // end of file #ifdef PBZIP_DEBUG fprintf( stderr, " End of file\n" ); #endif res->bufSize = 0; return res; } if ( ( getInBuffCurrent() == getInBuffSearchPtr() ) || ( !getSearchStatus() && !eof() ) ) { // search for next header // Slide a bit to skip current header in order to search for next one. _inBuffSearchPtr = std::min( getInBuffSearchPtr() + _bz2Header.size(), getInBuffEnd() ); _searchStatus = false; #ifdef PBZIP_DEBUG fprintf( stderr, " getNextStream - Searching subsequent header... " ); printCurrentState(); fprintf( stderr, "\n" ); #endif searchNextHeader(); } if ( failed() ) { return res; } appendOutBuffDataUpToLimit(); if ( failed() ) { return res; } if ( _outSequenceNumber > 0 ) { // continuing an unterminated sequence ++_outSequenceNumber; } else if ( getInBuffCurrent() != getInBuffSearchPtr() ) { // start of long multi-part stream _outSequenceNumber = 1; } _outBuff.sequenceNumber = _outSequenceNumber; _outBuff.inSize = _outBuff.bufSize; _outBuff.blockNumber = _streamNumber; if ( getInBuffCurrent() == getInBuffSearchPtr() ) { // we're at end of stream (either single or multi-segment one) _outBuff.isLastInSequence = true; _outSequenceNumber = 0; ++_streamNumber; } else { _outBuff.isLastInSequence = false; } #ifdef PBZIP_DEBUG OFF_T startBlock = blockNum; blockNum += _outBuff.bufSize; fprintf( stderr, " end getNextStream/blockRange=[%"PRIuMAX", %"PRIuMAX"), stream no=%d; seq=%d: [", (uintmax_t) startBlock, (uintmax_t) blockNum, _outBuff.blockNumber, _outBuff.sequenceNumber ); printCurrentState(); fprintf( stderr, "\n" ); #endif *res = _outBuff; // clean-up pointers to returned data. initOutBuff(); return res; } void BZ2StreamScanner::initOutBuff( char * buf, size_t bufSize, size_t bufCapacity ) { _outBuff.buf = buf; _outBuff.bufSize = bufSize; _outBuffCapacity = bufCapacity; _outBuff.inSize = 0; } } // namespace pbzip2 pbzip2-1.1.9/BZ2StreamScanner.h0000644000000000000000000001040312322570763014317 0ustar 00000000000000/* * File: BZ2StreamScanner.h * Author: Yavor Nikolov * * Created on March 6, 2010, 10:07 PM */ #ifndef _BZ2STREAMSCANNER_H #define _BZ2STREAMSCANNER_H #include "pbzip2.h" #include #include using namespace std; namespace pbzip2 { class BZ2StreamScanner { public: typedef unsigned char CharType; static const size_t DEFAULT_IN_BUFF_CAPACITY = 1024 * 1024; // 1M static const size_t DEFAULT_OUT_BUFF_LIMIT = 1024 * 1024; enum BZ2SScannerErrorFlag { ERR_MEM_ALLOC_INBUFF = 1, ERR_MEM_ALLOC_OUTBUFF = 1 << 1, ERR_IO_READ = 1 << 2, ERR_IO_INSUFFICIENT_BUFF_CAPACITY = 1 << 3, ERR_INVALID_STATE = 1 << 4, ERR_INVALID_FILE_FORMAT = 1 << 5 }; BZ2StreamScanner( int hInFile, size_t inBuffCapacity = DEFAULT_IN_BUFF_CAPACITY ); int init( int hInFile, size_t inBuffCapacity = DEFAULT_IN_BUFF_CAPACITY ); virtual ~BZ2StreamScanner(); outBuff * getNextStream(); size_t getInBuffSize() const { return ( _inBuffEnd - _inBuff ); } size_t getInBuffCapacity() const { return _inBuffCapacity; } const basic_string & getHeader() const { return _bz2Header; } size_t getHeaderSize() const { return _bz2Header.size(); } int getErrState() const { return _errState; } bool failed() { return ( _errState != 0 ); } /** true if header has been found since last initialization */ bool isBz2HeaderFound() const { return _bz2HeaderFound; } /** status of last/current search only */ bool getSearchStatus() const { return _searchStatus; } // end of file bool eof() const { return _eof; } /** true if out buffer is full enough to produce output block */ bool isOutBuffFullEnough() const { return _outBuff.bufSize >= getOutBuffCapacityLimit(); } /** * dispose memory resources */ virtual void dispose(); #ifdef PBZIP_DEBUG void printCurrentState(); #endif private: /* disable copy c-tor */ BZ2StreamScanner( const BZ2StreamScanner& orig ) {} void initOutBuff( char * buf = NULL, size_t bufSize = 0, size_t bufCapacity = 0 ); int appendOutBuffData( CharType * end ); int appendOutBuffData() { return appendOutBuffData( getInBuffSearchPtr() ); } int appendOutBuffDataUpToLimit(); int ensureOutBuffCapacity( size_t newSize ); int readData(); CharType * getInBuffEnd() { return _inBuffEnd; } CharType * getInBuffBegin() { return _inBuff; } CharType * getInBuffCurrent() { return _inBuffCurrent; } CharType * getInBuffSearchPtr() { return _inBuffSearchPtr; } char * getOutBuffEnd() { return _outBuff.buf + _outBuff.bufSize; } size_t getUnsearchedCount() const { return _inBuffEnd - _inBuffSearchPtr; } /** * Search next bz2 header. Read more data from file if needed. * * @return pointer to header is returned if found; * getInBuffEnd() - if not found; NULL - on error. */ CharType * searchNextHeader(); /** * Search next bz2 header just in currently available input buffer. * (Doesn't read more data from file). * * @return pointer to header or getInBuffEnd() if such is not found. */ CharType * searchNextHeaderInBuff(); /** * Prepare for next read from file into input buffer. * Consumes remaining input data buffer and moves header tail to beginning. * */ int rewindInBuff(); /** * Locate BZh header prefix in buffer. In case of first search - just check * the beginning of buffer and signal error if it doesn't match to headers. * * @return pointer to BZh header prefix if located. getInBuffEnd() if not. * failure() and getErrState() will indicate error if such occurred. */ CharType * locateHeaderPrefixInBuff(); size_t getOutBuffCapacityLimit() const { return _outBuffCapacityLimit; } int _hInFile; // input file descriptor bool _eof; basic_string _bz2Header; basic_string _bz2HeaderZero; bool _bz2HeaderFound; bool _searchStatus; CharType * _inBuff; CharType * _inBuffEnd; // end of data read from file CharType * _inBuffCurrent; CharType * _inBuffSearchPtr; size_t _inBuffCapacity; // allocated memory capacity for in buffer outBuff _outBuff; size_t _outBuffCapacity; size_t _outBuffCapacityHint; // keep max used capacity size_t _outBuffCapacityLimit; unsigned int _errState; // 0 - ok; otherwise error int _outSequenceNumber; // output block sequence number in bz2 stream (>0 if segmented) int _streamNumber; }; } #endif /* _BZ2STREAMSCANNER_H */ pbzip2-1.1.9/COPYING0000644000000000000000000000341612322570763012124 0ustar 00000000000000This program, "pbzip2" is copyright (C) 2003-2011 Jeff Gilchrist. All rights reserved. The library "libbzip2" which pbzip2 uses, is copyright (C) 1996-2008 Julian R Seward. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required. 3. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. 4. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Jeff Gilchrist, Ottawa, Canada. pbzip2@compression.ca pbzip2 version 1.1.9 of Apr 13, 2014 pbzip2-1.1.9/ChangeLog0000644000000000000000000002504712322570763012647 0ustar 00000000000000Changes in 1.1.9 (Apr 13, 2014) - spec file refinement for rpm builds thanks to Ville Skytta - Makefile refinements - close redirected stdout on finish for better AFS/NFS support (bug #1300876) Thanks to Richard Brittain - Fix printf format vs actual type misalignments (#1236086) Changes in 1.1.8 (Jun 10, 2012) - Fix of metadata unpreserved on empty files compress (bug #1011021) Changes in 1.1.7 (Dec 11, 2011) - Fixed refusal to write to stdout on -dc from stdin (bug #886628) - Fixed occasional failure on decompress with --ignore-trailing-garbage=1 with multiple bad blocks in the archive (bug #886625) Changes in 1.1.6 (Oct 30, 2011) - Fixed bug - deadlock due to unsynchronized broadcasts (bug #876686) - Prevent deletion of input files on error (bug #874543) - Document how to compress/decompress from standard input (bug #820525) - Added more detailed kernel error messages (bug #874605) - Error-handling fixes for multi-archive scenarios (bug #883782) Changes in 1.1.5 (Jul 16, 2011) - Fixed excessive output permissions while compress/decompress is in progress (bug #807536) Changes in 1.1.4 (Apr 22, 2011) - Fixed hang on decompress with --ignore-trailing-garbage=1 when producer is interrupted on trailing garbage (bug #762464) - Added example to help for decompression piped to tar (bug #746806) - Fixed typo in Trailing Garbage printed message (bug #746799) Changes in 1.1.3 (Mar 27, 2011) - Print trailing garbage errors even when in quiet mode (bug #743635) - Default extension on decompress of .tbz2 changed to .tar for bzip2 compatibility (bug #743639) - Fixed hang on decompress with --ignore-trailing-garbage=1 and higher numCPU (e.g. > 2) (bug #740502) Changes in 1.1.2 (Feb 19, 2011) - Fix directdecompress segfault when destination file can't be opened (e.g. read-only) (bug #717852) - Implemented --ignore-trailing-garbage feature (bug #594868) - Fixed hang on decompress of some truncated archives (bug #590225) - Pulled an error check out of normal logic block for clarity - Debug print added after BZ2_bzDecompress to track it's return code. - A debug print fixed in queue::remove - Increased max memory usage limit from 1GB to 2GB - If no -m switch given on command line, default max memory limit will now automatically increase from 100 MB to minimum amount of memory required to support the number of CPUs requested - Improved performance when output buffer is full - Fixed bug which caused hang while decompressing prematurely truncated bzip2 stream - Consumer_decompress throttling modified to prevent potential deadlock/infinite loop in certain situations (Thanks to Laszlo Ersek for finding and helping track down the cause of this bug) - Fixed deadlock bug and performance issue when consumer working with long bzip2 sequences (Thanks to Tanguy Fautre for finding) - Fixed error message for block size range (max size was wrong) - Moved #include from pbzip2.cpp to pbzip2.h to fix OS/2 compiler issue Changes in 1.1.1 (Apr 17, 2010) - Modified decompression to use low-level libbz2 API to improve performance of long bzip2 streams of large single-stream bzip2 blocks - This release should now decompress files created with bzip2 at least as fast as bzip2 or slightly faster - Handle decompression of long bzip2 streams incrementally instead of loading whole stream in memory at once - Fixed issue in safe_cond_timed_wait which caused segmentation fault when compiled in DEBUG mode - Fixed issue with Sun Studio compiler - required explicit declaration of static const members in .cpp Changes in 1.1.0 (Mar 13, 2010) - Added support for multi-threaded decompression using STDIN/pipes - Added code to support throttling compression to prevent memory exhaustion with slow output pipe - Added -m switch to specify max memory usage amount before throttling starts (default 100MB) - Fixed bug that did not allow command line parameters to be used when compressing data from STDIN - Added long options to man page and -h output - Added --loadavg, --read long options - Added support for CPU detection on Win32 - Major improvements to protection of shared variables, error and signal handling, program termination - Added -S switch for thread stack size customization (needs USE_STACKSIZE_CUSTOMIZATION defined when compiling) - Fixed command line parsing bug for -b, -p, -m switches - Fixed infinite loop when fileWriter fails to create output file at start - Fixed bug that deleted input filename (with .bz2 extension for compression and without .bz2 extension for decompression) when a user interrupts process with CTRL-C while outputting to STDOUT - Fixed bug where 0 byte files were not processed properly when data input from STDIN - Ignores fwrite return and passes chown errors in writeFileMetaData if effective uid root - OutputBuffer usage redesigned as fixed-size circular buffer - Lots of minor bugs fixed (see AUTHORS or pbzip2.cpp for full details) - Special thanks to Yavor Nikolov for providing the majority of contributions to this release and a significant amount of testing Changes in 1.0.5 (Jan 08, 2009) - Now properly complains about trying to read or write compressed data to terminal, and exits - Further fixed CPU detection crash - Updated Makefile to force deletion when cleaning Changes in 1.0.4 (Dec 21, 2008) - Added support to use pbzip2 with tar (--use-compress-prog=pbzip2). - Added support for all remaining bzip2 command line options so pbzip2 can be used as a drop-in replacement for bzip2. - Fixed bug that would cause pbzip2 to crash if detection of the number of processors failed. - Now prevents uclibc from being exposed to its lack of a getloadavg function. Changes in 1.0.3 (Oct 31, 2008) - Added support for compression using stdin and pipes! Thanks to Ivan Voras for supplying the patch to enable this feature. - Added support for decompression using stdin and pipes but currently limited to only a single thread - Added support for testing bzip2 files using stdin and pipes - Added support to directly decompress files without using threads when files are small or the system only has 1 CPU. This mode is also used if the .bz2 file contains only 1 bzip2 stream. - Added support to compress 0 byte files like bzip2 - Added support for pbzcat symlink to automatically specify decompression mode to stdout, just like bzcat - Increased max supported path and filename from 255 to 2048 characters - Updated RPM spec to support SUSE Linux - Updated help display to show how many CPUs autodetect has found - Code cleanup Changes in 1.0.2 (Jul 25, 2007) - Added support to directly compress files without using threads when files are smaller than the specified block size or the system only has 1 CPU. This will speed things up considerably if you are compressing many small files. You can also force this behaviour by using -p1 - Added support for pbunzip2 symlink to automatically specify decompression mode - Changed pbzip2 exit code behaviour to match bzip2 for all error states (ie: trying to compress a file that already has a .bz2 extension) Changes in 1.0.1 (Mar 20, 2007) - Added #ifdef PBZIP_NO_LOADAVG to remove load average code for UNIX systems that do not support it such as HP-UX and OSF1 Changes in 1.0 (Mar 14, 2007) - Official non-beta release! - Fixed minor memory leak in queueDelete() - Added support for UNIX systems to modify max number of CPUs used based on load average Changes in 0.9.6 (Feb 05, 2006) - Fixed bug that caused blocks to be missed by decompression routine under certain conditions Changes in 0.9.5 (Dec 31, 2005) - Changed default output to silent like bzip2 and added -v switch to make verbose - Added support to autodetect number of CPUs on OSX - Added support to compile on Borland and other Windows compilers using pthreads-win32 open source library - Added decompression throttling in case too much backlog in filewriter - Fixed bug from patch in 0.9.4 that limited file block size to 900k - Fixed bug that caused file output to fail with some large files - Fixed pthreads race condition that could cause random segfaults - Fixed pthreads resource issue that prevented pbzip2 from compressing a large number of files at once Changes in 0.9.4 (Aug 30, 2005) - Fixed major bug that prevented pbzip2 from uncompressing files created with a BWT block size other than default 900k (oops!) - Fixed some bugs in the command line parser - Code cleanup - Updated RPM spec to match Fedora Extras policy Changes in 0.9.3 (Aug 22, 2005) - Added sanity check to ensure integers are the size we expect - Ported code to compile under MinGW in Windows - Small code cleanup - Added ChangeLog - Added man page - Added AUTHORS file - Renamed LICENCE.txt to COPYING - Renamed README.txt to README - Updated RPM spec file in preparation for inclusion into Fedora Extras - Moved ChangeLog info from pbzip2.cpp into this file - Removed extra spaces/tabs in pbzip2.cpp Changes in 0.9.2 (Mar 12, 2005) - Created workaround for a major pthreads problem (on Linux at least) that would cause segfaults and missed signals in the pthreads - Increased size of BZIP2 header match from 56 bits to 72 bits for more accurate detection - Modified some buffers to use the vector class for easier memory management and to eliminate one read pass of the input data for decompression so should be slightly faster - Now checks if input file exists before checking if output file already exists (when not using -f) - Now checks to ensure input is regular file and not a directory before processing - Now deletes input file only if output file exists and is greater than 0 bytes (-k will always preserve input file) - Now checks input file for bzip2 header before trying to decompress file - Minor code cleanup Changes in 0.9.1 (Jan 29, 2005) - Fixed bug: -c switch now implies -k so input file is not deleted when data is output to stdout Changes in 0.9 (Jan 24, 2005) - Added -c switch for data output to stdout - Added -t switch to test integrity of compressed data - Added autodetection for # of CPUs on supported platforms and made detected value default # CPUs to use - Fixed bug where pbzip2 processed 0 byte files - Tried to make program exit codes the same as bzip2 Changes in 0.8.3 (Jan 9, 2005) - Added support to maintain file metadata - Added signal handling to cleanup partial files - Added support to process multiple files Changes in 0.8.2 (Nov 30, 2004) - Added support for large (2GB+) files - Added version info - Made command line input more like bzip2 - Now checks if file is already .bz2 first Changes in 0.8.1 (Nov 27, 2004) - Fix pthread bug, now works on OSF1, code cleanup Changes in 0.8 (Sep 6, 2004) - Initial public release pbzip2-1.1.9/ErrorContext.cpp0000644000000000000000000000342512322570763014233 0ustar 00000000000000/* * File: ErrorContext.cpp * Author: Yavor Nikolov * * Created on 29 October 2011, 18:14 */ #include "pbzip2.h" #include "ErrorContext.h" #include #include namespace pbzip2 { ErrorContext * ErrorContext::_instance = 0; pthread_mutex_t ErrorContext::_err_ctx_mutex = PTHREAD_MUTEX_INITIALIZER; ErrorContext * ErrorContext::getInstance() { pthread_mutex_lock( &_err_ctx_mutex ); if ( ErrorContext::_instance == NULL ) { _instance = new(std::nothrow) ErrorContext; if ( _instance == NULL ) { fprintf( stderr, "pbzip2: *ERROR: Can't initialize error context - out of memory!\n" ); } } pthread_mutex_unlock( &_err_ctx_mutex ); return _instance; } void ErrorContext::printErrnoMsg( FILE * out, int err ) { if ( err != 0 ) { fprintf( out, "pbzip2: *ERROR: system call failed with errno=[%d: %s]!\n", err, std::strerror( err ) ); } } void ErrorContext::syncPrintErrnoMsg( FILE * out, int err ) { pthread_mutex_lock( &_err_ctx_mutex ); printErrnoMsg( out, err ); pthread_mutex_unlock( &_err_ctx_mutex ); } void ErrorContext::printErrorMessages( FILE * out ) { pthread_mutex_lock( &_err_ctx_mutex ); printErrnoMsg( out, _first_kernel_err_no ); printErrnoMsg( out, _last_kernel_err_no ); pthread_mutex_unlock( &_err_ctx_mutex ); } void ErrorContext::saveError() { int newerr = errno; pthread_mutex_lock( &_err_ctx_mutex ); if ( newerr != 0 ) { int & err_ref = ( _first_kernel_err_no == 0 ) ? _first_kernel_err_no : _last_kernel_err_no; err_ref = newerr; } _last_kernel_err_no = errno; pthread_mutex_unlock( &_err_ctx_mutex ); } void ErrorContext::reset() { pthread_mutex_lock( &_err_ctx_mutex ); _first_kernel_err_no = _last_kernel_err_no = 0; pthread_mutex_unlock( &_err_ctx_mutex ); } } // namespace pbzip2 pbzip2-1.1.9/ErrorContext.h0000644000000000000000000000145712322570763013703 0ustar 00000000000000/* * File: ErrorContext.h * Author: Yavor Nikolov * * Created on 29 October 2011, 18:14 */ #ifndef ERRORCONTEXT_H #define ERRORCONTEXT_H #include "pbzip2.h" namespace pbzip2 { class ErrorContext { private: static ErrorContext * _instance; static pthread_mutex_t _err_ctx_mutex; int _first_kernel_err_no; int _last_kernel_err_no; private: ErrorContext(): _first_kernel_err_no( 0 ), _last_kernel_err_no( 0 ) { } ErrorContext( ErrorContext const & s ); void operator=( ErrorContext const & s ); public: static ErrorContext * getInstance(); void printErrorMessages( FILE * out = stderr ); void saveError(); void reset(); static void printErrnoMsg( FILE * out, int err ); static void syncPrintErrnoMsg( FILE * out, int err ); }; } // namespace pbzip2 #endif /* ERRORCONTEXT_H */ pbzip2-1.1.9/Makefile0000644000000000000000000000474112322570763012533 0ustar 00000000000000# Make file for parallel BZIP2 SHELL = /bin/sh # Compiler to use CXX = g++ # Thread-related flags # On some compilers -pthreads CXXFLAGS_PTHREAD = -pthread # Comment out CXXFLAGS line below to disable pthread semantics in code CXXFLAGS_PTHREAD += -D_POSIX_PTHREAD_SEMANTICS LDLIBS_PTHREAD = -lpthread # Optimization flags CXXFLAGS = -O2 #CXXFLAGS += -g -Wall #CXXFLAGS += -ansi #CXXFLAGS += -pedantic #CXXFLAGS += -std=c++0x # Comment out CXXFLAGS line below for compatability mode for 32bit file sizes # (less than 2GB) and systems that have compilers that treat int as 64bit # natively (ie: modern AIX) CXXFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 # Uncomment CXXFLAGS line below if you want to compile pbzip2 without load # average support for systems that do not support it #CXXFLAGS += -DPBZIP_NO_LOADAVG # Uncomment CXXFLAGS line below to get debug output #CXXFLAGS += -DPBZIP_DEBUG # Comment out CXXFLAGS line below to disable Thread stack size customization CXXFLAGS += -DUSE_STACKSIZE_CUSTOMIZATION # Comment out CXXFLAGS line below to explicity set ignore trailing garbage # default behavior: 0 - disabled; 1 - enabled (ignore garbage by default) # If IGNORE_TRAILING_GARBAGE is not defined: behavior is automatically determined # by program name: bzip2, bunzip2, bzcat - ignore garbage; otherwise - not. #CXXFLAGS += -DIGNORE_TRAILING_GARBAGE=1 # Add thread-related flags CXXFLAGS += $(CXXFLAGS_PTHREAD) # Linker flags LDFLAGS = # External libraries LDLIBS = -lbz2 LDLIBS += $(LDLIBS_PTHREAD) # Where you want pbzip2 installed when you do 'make install' PREFIX = /usr all: pbzip2 # Standard pbzip2 compile pbzip2: pbzip2.cpp BZ2StreamScanner.cpp ErrorContext.cpp $(CXX) $(CXXFLAGS) $(LDFLAGS) $^ -o pbzip2 $(LDLIBS) # Choose this if you want to compile in a static version of the libbz2 library pbzip2-static: pbzip2.cpp BZ2StreamScanner.cpp ErrorContext.cpp libbz2.a $(CXX) $(CXXFLAGS) $(LDFLAGS) $^ -o pbzip2 -I. -L. $(LDLIBS) # Install the binary pbzip2 program and man page install: pbzip2 if ( test ! -d $(PREFIX)/bin ) ; then mkdir -p $(PREFIX)/bin ; fi if ( test ! -d $(PREFIX)/man ) ; then mkdir -p $(PREFIX)/man ; fi if ( test ! -d $(PREFIX)/man/man1 ) ; then mkdir -p $(PREFIX)/man/man1 ; fi cp -f pbzip2 $(PREFIX)/bin/pbzip2 chmod a+x $(PREFIX)/bin/pbzip2 ln -s -f $(PREFIX)/bin/pbzip2 $(PREFIX)/bin/pbunzip2 ln -s -f $(PREFIX)/bin/pbzip2 $(PREFIX)/bin/pbzcat cp -f pbzip2.1 $(PREFIX)/man/man1 chmod a+r $(PREFIX)/man/man1/pbzip2.1 clean: rm -f *.o pbzip2 pbzip2-1.1.9/Makefile.solaris.sunstudio0000644000000000000000000000357312322570763016244 0ustar 00000000000000# Make file for parallel BZIP2 (customized for Solaris 10 64-bit/SPARC IV+) SHELL = /bin/sh # Compiler to use CC=CC CFLAGS = -m64 CFLAGS += -fast -xO5 #CFLAGS += -Wall #CFLAGS += -g CFLAGS += -mt -lpthread # Comment out CFLAGS line below for compatability mode for 32bit file sizes # (less than 2GB) and systems that have compilers that treat int as 64bit # natively (ie: modern AIX) #CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 # Uncomment CFLAGS line below if you want to compile pbzip2 without load # average support for systems that do not support it #CFLAGS += -DPBZIP_NO_LOADAVG # Uncomment CFLAGS line below to get debug output #CFLAGS += -DPBZIP_DEBUG # Comment out CFLAGS line below to disable pthread semantics in code CFLAGS += -D_POSIX_PTHREAD_SEMANTICS # Comment out CFLAGS line below to disable Thread stack size customization CFLAGS += -DUSE_STACKSIZE_CUSTOMIZATION # Libraries LDFLAGS = -lbz2 LDFLAGS += -lpthread # Where you want pbzip2 installed when you do 'make install' PREFIX = /usr all: pbzip2 # Standard pbzip2 compile pbzip2: pbzip2.cpp BZ2StreamScanner.cpp ErrorContext.cpp $(CC) $(CFLAGS) $^ -o pbzip2 ${LDFLAGS} # Choose this if you want to compile in a static version of the libbz2 library pbzip2-static: pbzip2.cpp BZ2StreamScanner.cpp ErrorContext.cpp libbz2.a $(CC) $(CFLAGS) $^ -o pbzip2 -I. -L. $(LDFLAGS) # Install the binary pbzip2 program and man page install: pbzip2 if ( test ! -d $(PREFIX)/bin ) ; then mkdir -p $(PREFIX)/bin ; fi if ( test ! -d $(PREFIX)/man ) ; then mkdir -p $(PREFIX)/man ; fi if ( test ! -d $(PREFIX)/man/man1 ) ; then mkdir -p $(PREFIX)/man/man1 ; fi cp -f pbzip2 $(PREFIX)/bin/pbzip2 chmod a+x $(PREFIX)/bin/pbzip2 ln -s -f $(PREFIX)/bin/pbzip2 $(PREFIX)/bin/pbunzip2 ln -s -f $(PREFIX)/bin/pbzip2 $(PREFIX)/bin/pbzcat cp -f pbzip2.1 $(PREFIX)/man/man1 chmod a+r $(PREFIX)/man/man1/pbzip2.1 clean: rm -f *.o pbzip2 pbzip2-1.1.9/README0000644000000000000000000002063512322570763011753 0ustar 00000000000000Apr 13, 2014 Parallel BZIP2 v1.1.9 - by: Jeff Gilchrist Available at: http://compression.ca/ This is the README for pbzip2, a parallel implementation of the bzip2 block-sorting file compressor. The output of this version should be fully compatible with bzip2 v1.0.2 or newer (ie: anything compressed with pbzip2 can be decompressed with bzip2). pbzip2 is distributed under a BSD-style license. For details, see the file COPYING. 1. HOW TO BUILD -- UNIX Type `make'. This builds the pbzip2 program and dynamically links to the libbzip2 library. You should ensure that you have at least libbzip2 1.0.5 or newer installed as it contains some important security bug fixes. If you do not have libbzip2 installed on your system, you should first go to http://www.bzip.org/ and install it. Debian users need the package "libbz2-dev". If you want to install a pre-built package on Debian, run the following command: 'apt-get update; apt-get install pbzip2' If you would like to build pbzip2 with a statically linked libbzip2 library, download the bzip2 source from the above site, compile it, and copy the libbz2.a and bzlib.h files into the pbzip2 source directory. Then type `make pbzip2-static'. Note: This software has been tested on Linux (Intel, Alpha), Solaris (Sparc), HP-UX, Irix (SGI), and Tru64/OSF1 (Alpha). 2. HOW TO BUILD -- Windows On Windows, pbzip2 can be compiled using Cygwin. If you do not have libbzip2 installed on your system, you should first go to http://www.bzip.org/ and install it. Cygwin can be found at: http://www.cygwin.com/ From a Cygwin shell, go to the directory where the pbzip2 source files are located and type `make'. This builds the pbzip2 program and dynamically links to the libbzip2 library. If you would like to build pbzip2 with a statically linked libbzip2 library, download the bzip2 source from the above site, compile it, and copy the libbz2.a file into the pbzip2 source directory. Then type `make pbzip2-static'. 3. DISCLAIMER I TAKE NO RESPONSIBILITY FOR ANY LOSS OF DATA ARISING FROM THE USE OF THIS PROGRAM, HOWSOEVER CAUSED. DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. * Portions of this README were copied directly from the bzip2 README written by Julian Seward. 4. PBZIP2 DATA FORMAT You should be able to compress files larger than 4GB with pbzip2. Files that are compressed with pbzip2 are broken up into pieces and each individual piece is compressed. This is how pbzip2 runs faster on multiple CPUs since the pieces can be compressed simultaneously. The final .bz2 file may be slightly larger than if it was compressed with the regular bzip2 program due to this file splitting (usually less than 0.2% larger). Files that are compressed with pbzip2 will also gain considerable speedup when decompressed using pbzip2. Files that were compressed using bzip2 will not see speedup since bzip2 pacakages the data into a single chunk that cannot be split between processors. pbzip2 will still be able to decompress these files, but it will be slower than if the .bz2 file was created with pbzip2. A file compressed with bzip2 will contain one compressed stream of data that looks like this: [-----------------------------------------------------] Data compressed with pbzip2 is broken into multiple streams and each stream is bzip2 compressed looking like this: [-----|-----|-----|-----|-----|-----|-----|-----|-----] If you are writing software with libbzip2 to decompress data created with pbzip2, you must take into account that the data contains multiple bzip2 streams so you will encounter end-of-stream markers from libbzip2 after each stream and must look-ahead to see if there are any more streams to process before quitting. The bzip2 program itself will automatically handle this condition. 5. USAGE The pbzip2 program is a parallel version of bzip2 for use on shared memory machines. It provides near-linear speedup when used on true multi-processor machines and 5-10% speedup on Hyperthreaded machines. The output is fully compatible with the regular bzip2 data so any files created with pbzip2 can be uncompressed by bzip2 and vice-versa. The default settings for pbzip2 will work well in most cases. The only switch you will likely need to use is -d to decompress files and -p to set the # of processors for pbzip2 to use if autodetect is not supported on your system, or you want to use a specific # of CPUs. Note, that if you are using a large number of CPUs you may wish to lower your default stack size setting (with the -S switch or ulimit) to reduce the amount of memory each thread uses. Usage: pbzip2 [-1 .. -9] [-b#cdfhklm#p#qrS#tvVz] Switches: -b# Where # is block size in 100k steps (default 9 = 900k) -c, --stdout Output to standard out (stdout) -d,--decompress Decompress file -f,--force Force, overwrite existing output file -h,--help Print this help message -k,--keep Keep input file, do not delete -l,--loadavg Load average determines max number processors to use -m# Where # is max memory usage in 1MB steps (default 100 = 100MB) -p# Where # is the number of processors (default: autodetect) -q,--quiet Quiet mode (default) -r,--read Read entire input file into RAM and split between processors -S# Child thread stack size in 1KB steps (default stack size if unspecified) -t,--test Test compressed file integrity -v,--verbose Verbose mode -V Display version info for pbzip2 then exit -z,--compress Compress file (default) -1,--fast ... -9,--best Set BWT block size to 100k .. 900k (default 900k). --ignore-trailing-garbage=# Ignore trailing garbage flag (1 - ignored; 0 - forbidden) Example: pbzip2 myfile.tar This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with the default file block size of 900k and default BWT block size of 900k. Example: pbzip2 -b15vk myfile.tar This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with a file block size of 1500k and a BWT block size of 900k. Verbose mode will be enabled so progress and other messages will be output to the display and the file myfile.tar will not be deleted after compression is finished. Example: pbzip2 -p4 -r -5 myfile.tar second*.txt This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use 4 processors with a BWT block size of 500k. The file block size will be the size of "myfile.tar" divided by 4 (# of processors) so that the data will be split evenly among each processor. This requires you have enough RAM for pbzip2 to read the entire file into memory for compression. pbzip2 will then use the same options to compress all other files that match the wildcard "second*.txt" in that directory. Example: pbzip2 -l myfile.tar This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) if the 1 minute load average is less than 0.5, otherwise it will select the maximum # of processors so that only idle processors are used for pbzip2. If the system has 4 processors and the load average is 2.00, then pbzip2 will use 2 processors to compress the data. Example: tar cf myfile.tar.bz2 --use-compress-prog=pbzip2 dir_to_compress/ Example: tar -c directory_to_compress/ | pbzip2 -c > myfile.tar.bz2 These examples will compress the data being given to pbzip2 via pipe from TAR into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with the default file block size of 900k and default BWT block size of 900k. TAR is collecting all of the files from the "directory_to_compress/" directory and passing the data to pbzip2 as it works. Example: pbzip2 -d -m500 myfile.tar.bz2 This example will decompress the file "myfile.tar.bz2" into the decompressed file "myfile.tar". It will use the autodetected # of processors (or 2 processors if autodetect not supported). It will use a maximum of 500MB of memory for decompression. The switches -b, -r, -t, and -1..-9 are not valid for decompression. pbzip2-1.1.9/pbzip2-rpm-buildflags.patch0000644000000000000000000000052112322570763016220 0ustar 00000000000000--- Makefile~ 2012-07-07 12:52:15.000000000 +0300 +++ Makefile 2012-07-07 12:51:14.000000000 +0300 @@ -7,7 +7,7 @@ CXX = g++ # Optimization flags -CXXFLAGS = -O2 +CXXFLAGS ?= -O2 #CXXFLAGS += -g -Wall #CXXFLAGS += -ansi @@ -43,7 +43,7 @@ CXXFLAGS += -pthread # Linker flags -LDFLAGS = +LDFLAGS += # External libraries pbzip2-1.1.9/pbzip2.10000644000000000000000000001270012322570763012355 0ustar 00000000000000.TH pbzip2 1 .SH NAME pbzip2 \- parallel bzip2 file compressor, v1.1.9 .SH SYNOPSIS .B pbzip2 .RB [ " \-123456789 " ] .RB [ " \-b#cdfhklm#p#qrS#tvVz " ] [ .I "filenames \&..." ] .SH DESCRIPTION .I pbzip2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of this version is fully compatible with bzip2 v1.0.2 or newer (ie: anything compressed with .I pbzip2 can be decompressed with bzip2). .PP .I pbzip2 should work on any system that has a pthreads compatible C++ compiler (such as gcc). It has been tested on: Linux, Windows (cygwin), Solaris, Tru64/OSF1, HP-UX, and Irix. .PP The default settings for .I pbzip2 will work well in most cases. The only switch you will likely need to use is -d to decompress files and -p to set the # of processors for .I pbzip2 to use if autodetect is not supported on your system, or you want to use a specific # of CPUs. .SH OPTIONS .TP .B \-b# Where # is block size in 100k steps (default 9 = 900k) .TP .B \-c, \-\-stdout Output to standard out (stdout) .TP .B \-d,\-\-decompress Decompress file .TP .B \-f,\-\-force Force, overwrite existing output file .TP .B \-h,\-\-help Print this help message .TP .B \-k,\-\-keep Keep input file, do not delete .TP .B \-l,\-\-loadavg Load average determines max number processors to use .TP .B \-m# Where # is max memory usage in 1MB steps (default 100 = 100MB) .TP .B \-p# Where # is the number of processors (default: autodetect) .TP .B \-q,\-\-quiet Quiet mode (default) .TP .B \-r,\-\-read Read entire input file into RAM and split between processors .TP .B \-S# Child thread stack size in 1KB steps (default stack size if unspecified) .TP .B \-t,\-\-test Test compressed file integrity .TP .B \-v,\-\-verbose Verbose mode .TP .B \-V Display version info for .I pbzip2 then exit .TP .B \-z,\-\-compress Compress file (default) .TP .B \-1,\-\-fast ... \-9,\-\-best Set BWT block size to 100k .. 900k (default 900k). .TP .B \-\-ignore-trailing-garbage=# Ignore trailing garbage flag (1 - ignored; 0 - forbidden) .PP If no file names are given, pbzip2 compresses or decompresses from standard input to standard output. .SH FILE SIZES You should be able to compress files larger than 4GB with .I pbzip2. .PP Files that are compressed with .I pbzip2 are broken up into pieces and each individual piece is compressed. This is how .I pbzip2 runs faster on multiple CPUs since the pieces can be compressed simultaneously. The final .bz2 file may be slightly larger than if it was compressed with the regular bzip2 program due to this file splitting (usually less than 0.2% larger). Files that are compressed with .I pbzip2 will also gain considerable speedup when decompressed using .I pbzip2. .PP Files that were compressed using bzip2 will not see speedup since bzip2 packages the data into a single chunk that cannot be split between processors. .SH EXAMPLES Example 1: pbzip2 myfile.tar .PP This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with the default file block size of 900k and default BWT block size of 900k. .PP Example 2: pbzip2 -b15k myfile.tar .PP This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with a file block size of 1500k and a BWT block size of 900k. The file "myfile.tar" will not be deleted after compression is finished. .PP Example 3: pbzip2 -p4 -r -5 myfile.tar second*.txt .PP This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use 4 processors with a BWT block size of 500k. The file block size will be the size of "myfile.tar" divided by 4 (# of processors) so that the data will be split evenly among each processor. This requires you have enough RAM for pbzip2 to read the entire file into memory for compression. Pbzip2 will then use the same options to compress all other files that match the wildcard "second*.txt" in that directory. .PP Example 4: tar cf myfile.tar.bz2 --use-compress-prog=pbzip2 dir_to_compress/ .br Example 4: tar -c directory_to_compress/ | pbzip2 -c > myfile.tar.bz2 .PP These examples will compress the data being given to pbzip2 via pipe from TAR into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with the default file block size of 900k and default BWT block size of 900k. TAR is collecting all of the files from the "directory_to_compress/" directory and passing the data to pbzip2 as it works. .PP Example 5: pbzip2 -d -m500 myfile.tar.bz2 .PP This example will decompress the file "myfile.tar.bz2" into the decompressed file "myfile.tar". It will use the autodetected # of processors (or 2 processors if autodetect not supported). It will use a maximum of 500MB of memory for decompression. The switches -b, -r, and -1..-9 are not valid for decompression. .PP Example 6: pbzip2 -dc myfile.tar.bz2 | tar x .PP This example will decompress and untar the file "myfile.tar.bz2" piping the output of the decompressing pbzip2 to tar. .PP Example 7: pbzip2 -c < myfile.txt > myfile.txt.bz2 .PP This example will read myfile.txt from standard input compressing it to standard output which is redirected to to myfile.txt.bz2. .SH "SEE ALSO" bzip2(1) gzip(1) lzip(1) rzip(1) zip(1) .SH AUTHOR Jeff Gilchrist http://compression.ca pbzip2-1.1.9/pbzip2.cpp0000644000000000000000000035144612322570763013014 0ustar 00000000000000/* * File : pbzip2.cpp * * Title : Parallel BZIP2 (pbzip2) * * Author: Jeff Gilchrist (http://gilchrist.ca/jeff/) * - Modified producer/consumer threading code from * Andrae Muys * - uses libbzip2 by Julian Seward (http://sources.redhat.com/bzip2/) * - Major contributions by Yavor Nikolov * * Date : Apr 13, 2014 * * * This program, "pbzip2" is copyright (C) 2003-2011 Jeff Gilchrist. * All rights reserved. * * The library "libbzip2" which pbzip2 uses, is copyright * (C) 1996-2008 Julian R Seward. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. The origin of this software must not be misrepresented; you must * not claim that you wrote the original software. If you use this * software in a product, an acknowledgment in the product * documentation would be appreciated but is not required. * * 3. Altered source versions must be plainly marked as such, and must * not be misrepresented as being the original software. * * 4. The name of the author may not be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE * GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * Jeff Gilchrist, Ottawa, Canada. * pbzip2@compression.ca * pbzip2 version 1.1.9 of Apr 13, 2014 * */ #include "pbzip2.h" #include "BZ2StreamScanner.h" #include "ErrorContext.h" #include #include #include #include extern "C" { #include #include #include #include #include #include #include #include #include #include #include } // // GLOBALS // static int producerDone = 0; static int terminateFlag = 0; // Abnormal premature termination static int finishedFlag = 0; // Main thread work finished (about to exit) static int unfinishedWorkCleaned = 0; static int numCPU = 2; static int IgnoreTrailingGarbageFlag = 0; // ingnore trailing garbage on decompress flag static int QUEUESIZE = 2; static int SIG_HANDLER_QUIT_SIGNAL = SIGUSR1; // signal used to stop SignalHandlerThread #ifdef USE_STACKSIZE_CUSTOMIZATION static int ChildThreadStackSize = 0; // -1 - don't modify stacksize; 0 - use minimum; > 0 - use specified #ifndef PTHREAD_STACK_MIN #define PTHREAD_STACK_MIN 4096 #endif #endif // USE_STACKSIZE_CUSTOMIZATION static unsigned char Bz2HeaderZero[] = { 0x42, 0x5A, 0x68, 0x39, 0x17, 0x72, 0x45, 0x38, 0x50, 0x90, 0x00, 0x00, 0x00, 0x00 }; static OFF_T InFileSize; static OFF_T InBytesProduced = 0; static int NumBlocks = 0; static int NumBlocksEstimated = 0; static int NumBufferedBlocks = 0; static size_t NumBufferedTailBlocks = 0; static size_t NumBufferedBlocksMax = 0; static int NextBlockToWrite; static int LastGoodBlock; // set only to terminate write prematurely (ignoring garbage) static int MinErrorBlock; // lowest so far block number which has errors (on decompress; could be trailing garbage) static size_t OutBufferPosToWrite; // = 0; // position in output buffer static int Verbosity = 0; static int QuietMode = 1; static int OutputStdOut = 0; static int ForceOverwrite = 0; static int BWTblockSize = 9; static int FileListCount = 0; static std::vector OutputBuffer; static queue *FifoQueue; // fifo queue (global var used on termination cleanup) static pthread_mutex_t *OutMutex = NULL; static pthread_mutex_t *ProducerDoneMutex = NULL; static pthread_mutex_t ErrorHandlerMutex = PTHREAD_MUTEX_INITIALIZER; static pthread_mutex_t TerminateFlagMutex = PTHREAD_MUTEX_INITIALIZER; static pthread_mutex_t ProgressIndicatorsMutex = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t *notTooMuchNumBuffered; static pthread_cond_t TerminateCond = PTHREAD_COND_INITIALIZER; static pthread_cond_t OutBufferHeadNotEmpty = PTHREAD_COND_INITIALIZER; static pthread_cond_t ErrStateChangeCond = PTHREAD_COND_INITIALIZER; static pthread_attr_t ChildThreadAttributes; static struct stat fileMetaData; static const char *sigInFilename = NULL; static const char *sigOutFilename = NULL; static char BWTblockSizeChar = '9'; static sigset_t SignalMask; static pthread_t SignalHandlerThread; static pthread_t TerminatorThread; inline int syncGetProducerDone(); inline void syncSetProducerDone(int newValue); inline int syncGetTerminateFlag(); inline void syncSetTerminateFlag(int newValue); inline void syncSetFinishedFlag(int newValue); inline void syncSetLastGoodBlock(int newValue, int errBlock); inline int syncGetLastGoodBlock(); void cleanupUnfinishedWork(); void cleanupAndQuit(int exitCode); int initSignalMask(); int setupSignalHandling(); int setupTerminator(); inline void safe_mutex_lock(pthread_mutex_t *mutex); inline void safe_mutex_unlock(pthread_mutex_t *mutex); inline void safe_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex); inline void safe_cond_signal(pthread_cond_t *cond); int safe_cond_timed_wait(pthread_cond_t *cond, pthread_mutex_t *mutex, int seconds, const char *caller = "safe_cond_timed_wait"); template FI1 memstr(FI1 searchBuf, int searchBufSize, FI2 searchString, int searchStringSize); int producer_decompress(int, OFF_T, queue *); int directcompress(int, OFF_T, int, const char *); int directdecompress(const char *, const char *); int producer(int hInfile, int blockSize, queue *fifo); int mutexesInit(); void mutexesDelete(); queue *queueInit(int); void queueDelete(queue *); void outputBufferInit(size_t size); outBuff * outputBufferAdd(const outBuff & element, const char *caller); outBuff * outputBufferSeqAddNext(outBuff * preveElement, outBuff * newElement); inline size_t getOutputBufferPos(int blockNum); int getFileMetaData(const char *); int writeFileMetaData(const char *); int testBZ2ErrorHandling(int, BZFILE *, int); int testCompressedData(char *); ssize_t bufread(int hf, char *buf, size_t bsize); int detectCPUs(void); inline bool isIgnoredTrailingGarbage(); int waitForPreviousBlock(int blockNumToWait, int errBlockNumber); inline int getLastGoodBlockBeforeErr(int errBlockNumber, int outSequenceNumber); inline int issueDecompressError(int bzret, const outBuff * fileData, int outSequenceNumber, const bz_stream & strm, const char * errmsg, int exitCode); int decompressErrCheckSingle(int bzret, const outBuff * fileData, int outSequenceNumber, const bz_stream & strm, const char * errmsg, bool isTrailingGarbageErr); int decompressErrCheck(int bzret, const outBuff * fileData, int outSequenceNumber, const bz_stream & strm); inline bool hasTrailingGarbage(int bzret, const outBuff * fileData, const bz_stream & strm); int producerDecompressCheckInterrupt(int hInfile, outBuff *& fileData, int lastBlock); using pbzip2::ErrorContext; /* * Pointers to functions used by plain C pthreads API require C calling * conventions. */ extern "C" { void* signalHandlerProc(void* arg); void* terminatorThreadProc(void* arg); void *consumer_decompress(void *); void *fileWriter(void *); void *consumer(void *); } /* * Lock mutex or exit application immediately on error. */ inline void safe_mutex_lock(pthread_mutex_t *mutex) { int ret = pthread_mutex_lock(mutex); if (ret != 0) { fprintf(stderr, "pthread_mutex_lock error [%d]! Aborting immediately!\n", ret); cleanupAndQuit(-5); } } /* * Unlock mutex or exit application immediately on error. */ inline void safe_mutex_unlock(pthread_mutex_t *mutex) { int ret = pthread_mutex_unlock(mutex); if (ret != 0) { fprintf(stderr, "pthread_mutex_unlock error [%d]! Aborting immediately!\n", ret); cleanupAndQuit(-6); } } /* * Call pthread_cond_signal - check return code and exit application immediately * on error. */ inline void safe_cond_signal(pthread_cond_t *cond) { int ret = pthread_cond_signal(cond); if (ret != 0) { fprintf(stderr, "pthread_cond_signal error [%d]! Aborting immediately!\n", ret); cleanupAndQuit(-7); } } /* * Call pthread_cond_signal - check return code and exit application immediately * on error. */ inline void safe_cond_broadcast(pthread_cond_t *cond) { int ret = pthread_cond_broadcast(cond); if (ret != 0) { fprintf(stderr, "pthread_cond_broadcast error [%d]! Aborting immediately!\n", ret); cleanupAndQuit(-7); } } /* * Unlock mutex or exit application immediately on error. */ inline void safe_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex) { int ret = pthread_cond_wait(cond, mutex); if (ret != 0) { fprintf(stderr, "pthread_cond_wait error [%d]! Aborting immediately!\n", ret); pthread_mutex_unlock(mutex); cleanupAndQuit(-8); } } /* * Delegate to pthread_cond_timedwait. Check for errors and abort if * any encountered. Return 0 on success and non-zero code on error */ int safe_cond_timed_wait(pthread_cond_t *cond, pthread_mutex_t *mutex, int seconds, const char *caller) { struct timespec waitTimer; #ifndef WIN32 struct timeval tv; struct timezone tz; #else SYSTEMTIME systemtime; LARGE_INTEGER filetime; #endif #ifndef WIN32 gettimeofday(&tv, &tz); waitTimer.tv_sec = tv.tv_sec + seconds; waitTimer.tv_nsec = tv.tv_usec * 1000; #else GetSystemTime(&systemtime); SystemTimeToFileTime(&systemtime, (FILETIME *)&filetime); waitTimer.tv_sec = filetime.QuadPart / 10000000; waitTimer.tv_nsec = filetime.QuadPart - ((LONGLONG)waitTimer.tv_sec * 10000000) * 10; waitTimer.tv_sec += seconds; #endif #ifdef PBZIP_DEBUG fprintf(stderr, "%s: waitTimer.tv_sec: %"PRIiMAX" waitTimer.tv_nsec: %"PRIiMAX"\n", caller, (intmax_t)waitTimer.tv_sec, (intmax_t)waitTimer.tv_nsec); #endif int pret = pthread_cond_timedwait(cond, mutex, &waitTimer); // we are not using a compatible pthreads library so abort if ((pret != 0) && (pret != EINTR) && (pret != EBUSY) && (pret != ETIMEDOUT)) { ErrorContext::getInstance()->saveError(); pthread_mutex_unlock(mutex); handle_error(EF_EXIT, 1, "pbzip2: *ERROR: %s: pthread_cond_timedwait() call invalid [pret=%d]. This machine\n" " does not have compatible pthreads library. Aborting.\n", caller, pret); cleanupAndQuit(-9); } #ifdef PBZIP_DEBUG else if (pret != 0) { fprintf(stderr, "%s: pthread_cond_timedwait returned with non-fatal error [%d]\n", caller, pret); } #endif // PBZIP_DEBUG return 0; } /* * Delegate to write but keep writing until count bytes are written or * error is encountered (on success all count bytes would be written) */ ssize_t do_write(int fd, const void *buf, size_t count) { ssize_t bytesRemaining = count; ssize_t nbytes = 0; const char *pbuf = (const char *)buf; while ((bytesRemaining > 0) && ((nbytes = write(fd, pbuf, bytesRemaining)) > 0)) { bytesRemaining -= nbytes; pbuf += nbytes; } if (nbytes < 0) { ErrorContext::getInstance()->saveError(); return nbytes; } return (count - bytesRemaining); } /* * Delegate to read but keep writing until count bytes are read or * error is encountered (on success all count bytes would be read) */ ssize_t do_read(int fd, void *buf, size_t count) { ssize_t bytesRemaining = count; ssize_t nbytes = 0; char *pbuf = (char *)buf; while ((bytesRemaining > 0) && (nbytes = read(fd, pbuf, bytesRemaining)) > 0) { bytesRemaining -= nbytes; pbuf += nbytes; } if (nbytes < 0) { ErrorContext::getInstance()->saveError(); return nbytes; } return (count - bytesRemaining); } /* * Open output file with least required privileges */ int safe_open_output(const char *path) { int ret = open(path, O_WRONLY | O_CREAT | O_EXCL | O_BINARY, FILE_MODE); if (ret == -1) { ErrorContext::getInstance()->saveError(); } return ret; } /* * Based on bzip2.c code */ FILE *safe_fopen_output(const char *path, const char *mode) { int fh = safe_open_output(path); if (fh == -1) { return NULL; } FILE *fp = fdopen(fh, mode); if (fp == NULL) { ErrorContext::getInstance()->saveError(); close(fh); } return fp; } /** * Save the given file and save errno on failure. * * @param fd file to close * @return -1 on failure or 0 on success. */ inline int do_close(int fd) { int ret = close(fd); if (ret == -1) { ErrorContext::getInstance()->saveError(); } return ret; } inline int do_fclose(FILE *file) { int ret = fclose(file); if ( ret == EOF ) { ErrorContext::getInstance()->saveError(); } return ret; } inline int do_fflush(FILE *file) { int ret = fflush(file); if ( ret == EOF ) { ErrorContext::getInstance()->saveError(); } return ret; } /** * Close the given file. In case of error - save errno and print error message. * * @param file file to close * @param fileName name of file to print in case of failure * @return fclose return code */ inline int verbose_fclose(FILE *file, const char *fileName) { int ret; if ( (ret = fclose(file)) == EOF ) { ErrorContext::syncPrintErrnoMsg(stderr, errno); fprintf(stderr, "pbzip2: *ERROR: Failed to close file [%s]!\n", fileName); } return ret; } int do_remove(const char* pathname) { int ret = remove(pathname); if (ret == -1) { ErrorContext::getInstance()->saveError(); } return ret; } /** * Check if a given file exists. * * @return true if file exists and false if it doesn't */ bool check_file_exists( const char * filename ) { int hOutfile = open( filename, O_RDONLY | O_BINARY ); if ( hOutfile == -1 ) { ErrorContext::getInstance()->saveError(); return false; } else { close( hOutfile ); return true; } } /* ********************************************************* Atomically get producerDone value. */ inline int syncGetProducerDone() { int ret; safe_mutex_lock(ProducerDoneMutex); ret = producerDone; safe_mutex_unlock(ProducerDoneMutex); return ret; } /* ********************************************************* Atomically set producerDone value. */ inline void syncSetProducerDone(int newValue) { safe_mutex_lock(ProducerDoneMutex); producerDone = newValue; safe_mutex_unlock(ProducerDoneMutex); } /* * Atomic get terminateFlag */ inline int syncGetTerminateFlag() { int ret; safe_mutex_lock(&TerminateFlagMutex); ret = terminateFlag; safe_mutex_unlock(&TerminateFlagMutex); return ret; } /* * Atomically set termination flag and signal the related * condition. */ inline void syncSetTerminateFlag(int newValue) { safe_mutex_lock(&TerminateFlagMutex); terminateFlag = newValue; if (terminateFlag != 0) { // wake up terminator thread pthread_cond_signal(&TerminateCond); safe_mutex_unlock(&TerminateFlagMutex); // wake up all other possibly blocked on cond threads safe_mutex_lock(OutMutex); pthread_cond_broadcast(notTooMuchNumBuffered); safe_mutex_unlock(OutMutex); if (FifoQueue != NULL) { safe_mutex_lock(FifoQueue->mut); pthread_cond_broadcast(FifoQueue->notFull); pthread_cond_broadcast(FifoQueue->notEmpty); safe_mutex_unlock(FifoQueue->mut); } } else { safe_mutex_unlock(&TerminateFlagMutex); } } /* * Set finishedSucessFlag and signal the related condition. */ inline void syncSetFinishedFlag(int newValue) { safe_mutex_lock(&TerminateFlagMutex); finishedFlag = newValue; if (finishedFlag != 0) { pthread_cond_signal(&TerminateCond); } safe_mutex_unlock(&TerminateFlagMutex); } /** * Set last block which is maybe good (not guaranteed) and MinErrorBlock * (lowest block with errors encountered so far). * Only moving downwards has effect (attempts to raise the block numbers are ignored). * -1 means infinity. * * * @param newValue last block which is maybe good. -1 means +infinity. * @param errBlock block number which has errors */ inline void syncSetLastGoodBlock(int newValue, int errBlock) { bool changed = false; safe_mutex_lock(OutMutex); #ifdef PBZIP_DEBUG uintmax_t thid = (uintmax_t) pthread_self(); fprintf(stderr, "(%"PRIuMAX") syncSetLastGoodBlock: %d -> %d; MinErrorBlock: %d -> %d\n", thid, LastGoodBlock, newValue, MinErrorBlock, errBlock); #endif if ( (LastGoodBlock == -1) || (newValue < LastGoodBlock) ) { LastGoodBlock = newValue; changed = true; } if ( (MinErrorBlock == -1) || (errBlock < MinErrorBlock) ) { MinErrorBlock = errBlock; changed = true; } if ( changed ) { safe_cond_signal(&ErrStateChangeCond); safe_cond_signal(&OutBufferHeadNotEmpty); // wake up all other possibly blocked on cond threads pthread_cond_broadcast(notTooMuchNumBuffered); safe_mutex_unlock(OutMutex); if (FifoQueue != NULL) { safe_mutex_lock(FifoQueue->mut); pthread_cond_broadcast(FifoQueue->notFull); pthread_cond_broadcast(FifoQueue->notEmpty); safe_mutex_unlock(FifoQueue->mut); } } else { safe_mutex_unlock(OutMutex); } } inline int syncGetLastGoodBlock() { int ret; safe_mutex_lock(OutMutex); ret = LastGoodBlock; safe_mutex_unlock(OutMutex); return ret; } inline int syncGetMinErrorBlock() { int ret; safe_mutex_lock(OutMutex); ret = MinErrorBlock; safe_mutex_unlock(OutMutex); return ret; } inline bool isIgnoredTrailingGarbage() { return (IgnoreTrailingGarbageFlag != 0); } /* ********************************************************* Print error message and optionally exit or abort depending on exitFlag: 0 - don't quit; 1 - exit; 2 - abort. On exit - exitCode status is used. */ int handle_error(ExitFlag exitFlag, int exitCode, const char *fmt, ...) { va_list args; va_start(args, fmt); vfprintf(stderr, fmt, args); ErrorContext::getInstance()->printErrorMessages(stderr); fflush(stderr); va_end(args); if (exitFlag == EF_ABORT) { syncSetTerminateFlag(1); abort(); } if (exitFlag == EF_EXIT) { syncSetTerminateFlag(1); } return exitCode; } /** * * @return -1 - terminate flag set (error) * 0 - prev block is OK (i.e. we're on the first error here) * 2 - lower block number already in error state */ int waitForPreviousBlock(int blockNumToWait, int errBlockNumber) { #ifdef PBZIP_DEBUG uintmax_t thid = (uintmax_t) pthread_self(); safe_mutex_lock(OutMutex); fprintf( stderr, "(%"PRIuMAX") waitForPreviousBlock enter: LastGoodBlock=%d" "; blockNumToWait=%d; NextBlockToWrite=%d; MinErrorBlock=%d; errBlockNumber=%d\n", thid, LastGoodBlock, blockNumToWait, NextBlockToWrite, MinErrorBlock, errBlockNumber ); safe_mutex_unlock(OutMutex); #endif for (;;) { if (syncGetTerminateFlag() != 0) { #ifdef PBZIP_DEBUG fprintf(stderr, "(%"PRIuMAX") waitForPreviousBlock terminated [%d]: blockNumToWait=%d\n", thid, -1, blockNumToWait ); #endif return -1; } safe_mutex_lock(OutMutex); #ifdef PBZIP_DEBUG fprintf( stderr, "(%"PRIuMAX") waitForPreviousBlock before check: LastGoodBlock=%d; blockNumToWait=%d; NextBlockToWrite=%d; MinErrorBlock=%d\n", thid, LastGoodBlock, blockNumToWait, NextBlockToWrite, MinErrorBlock ); #endif // This check should (min error block) be before next one (next block to write) if ( (MinErrorBlock != -1) && (MinErrorBlock < errBlockNumber) ) { #ifdef PBZIP_DEBUG fprintf( stderr, "(%"PRIuMAX") waitForPreviousBlock exit [%d]: LastGoodBlock=%d; blockNumToWait=%d; NextBlockToWrite=%d; MinErrorBlock=%d\n", thid, 2, LastGoodBlock, blockNumToWait, NextBlockToWrite, MinErrorBlock ); #endif safe_mutex_unlock(OutMutex); return 2; } if (errBlockNumber <= NextBlockToWrite) { #ifdef PBZIP_DEBUG fprintf( stderr, "(%"PRIuMAX") waitForPreviousBlock exit [%d]: LastGoodBlock=%d; blockNumToWait=%d; NextBlockToWrite=%d; MinErrorBlock=%d\n", thid, 0, LastGoodBlock, blockNumToWait, NextBlockToWrite, MinErrorBlock ); #endif safe_mutex_unlock(OutMutex); return 0; } #ifdef PBZIP_DEBUG fprintf( stderr, "(%"PRIuMAX") waitForPreviousBlock to sleep: LastGoodBlock=%d; blockNumToWait=%d; NextBlockToWrite=%d; MinErrorBlock=%d\n", thid, LastGoodBlock, blockNumToWait, NextBlockToWrite, MinErrorBlock ); #endif safe_cond_timed_wait(&ErrStateChangeCond, OutMutex, 1, "waitForPreviousBlock"); safe_mutex_unlock(OutMutex); } } /** * * @param errBlockNumber block from input file * @param outSequenceNumber sequence in the output tail for the given block * @return Last input block not after the given which possibly (not guaranteed) * resulted in good out blocks. * -1 if such don't exist. */ inline int getLastGoodBlockBeforeErr(int errBlockNumber, int outSequenceNumber) { // if we got the error just in the beginning of a bzip2 stream if ( outSequenceNumber != -1 ) { return errBlockNumber; } else { return errBlockNumber - 1; } } /** * Helper function delegating to handle_error with the relevant error * message * * @param bzret * @param fileData * @param outSequenceNumber * @param strm * @param errmsg * @param exitCode * @return exitCode is returned */ inline int issueDecompressError(int bzret, const outBuff * fileData, int outSequenceNumber, const bz_stream & strm, const char * errmsg, int exitCode) { #ifdef PBZIP_DEBUG uintmax_t thid = (uintmax_t) pthread_self(); fprintf(stderr, "(%"PRIuMAX") enter issueDecompressError: msg=%s; ret=%d; block=%d; seq=%d; isLastInSeq=%d; avail_in=%d\n", thid, errmsg, bzret, fileData->blockNumber, outSequenceNumber, (int)fileData->isLastInSequence, strm.avail_in); #endif handle_error(EF_EXIT, exitCode, "pbzip2: %s: ret=%d; block=%d; seq=%d; isLastInSeq=%d; avail_in=%d\n", errmsg, bzret, fileData->blockNumber, outSequenceNumber, (int)fileData->isLastInSequence, strm.avail_in); return exitCode; } /** * Handle an error condition which is either trailing garbage-like one or not. * * * @param bzret * @param fileData block from input file * @param outSequenceNumber * @param strm * @param errmsg * @param isTrailingGarbageErr * @return ret < 0 - fatal error; * 0 - OK (no error at all); * 1 - first block of ignored trailing garbage; * 2 - error already signalled for earlier block */ int decompressErrCheckSingle(int bzret, const outBuff * fileData, int outSequenceNumber, const bz_stream & strm, const char * errmsg, bool isTrailingGarbageErr) { int lastGoodBlock = getLastGoodBlockBeforeErr(fileData->blockNumber, outSequenceNumber); #ifdef PBZIP_DEBUG uintmax_t thid = (uintmax_t) pthread_self(); fprintf(stderr, "(%"PRIuMAX") enter decompressErrCheckSingle: msg=%s; ret=%d; block=%d; seq=%d; isLastInSeq=%d; avail_in=%d; lastGoodBlock=%d\n", thid, errmsg, bzret, fileData->blockNumber, outSequenceNumber, (int)fileData->isLastInSequence, strm.avail_in, lastGoodBlock); #endif if ( (lastGoodBlock == -1) || !isIgnoredTrailingGarbage() ) { issueDecompressError(bzret, fileData, outSequenceNumber, strm, errmsg, -1); return -1; } else { // Cut off larger block numbers syncSetLastGoodBlock(lastGoodBlock, fileData->blockNumber); // wait until the state of previous block is known int prevState = waitForPreviousBlock(lastGoodBlock, fileData->blockNumber); if (prevState == 0) { // we're the first error block if (isTrailingGarbageErr) { // Trailing garbage detected and ignored - not a fatal warning fprintf(stderr, "pbzip2: *WARNING: Trailing garbage after EOF ignored!\n"); return 1; } else { // the first error is not kind of trailing garbage -> fatal one issueDecompressError(bzret, fileData, outSequenceNumber, strm, errmsg, -1); return -1; } } else if (prevState == 2) { // we're not the first error return 2; } else // (prevState == -1) { // fatal state encountered return -1; } } } /** * Check if trailing garbage has been identified during the last decompression * operation. * * @param bzret last bzip2 return code * @param fileData should be initialized before calling this * @param strm bzip2 library bz_stream * @return true if trailing garbage has been detected. false otherwise */ inline bool hasTrailingGarbage(int bzret, const outBuff * fileData, const bz_stream & strm) { return (bzret == BZ_STREAM_END) && ( (strm.avail_in != 0) || !fileData->isLastInSequence ); } /** * * @param bzret * @param fileData * @param outSequenceNumber * @param strm * @return ret < 0 - fatal error; * 0 - OK (no error at all); * 1 - first block of ignored trailing garbage; * 2 - error already signalled for earlier block */ int decompressErrCheck(int bzret, const outBuff * fileData, int outSequenceNumber, const bz_stream & strm) { if ( hasTrailingGarbage( bzret, fileData, strm ) ) { // Potential trailing garbage return decompressErrCheckSingle(bzret, fileData, outSequenceNumber, strm, "*ERROR during BZ2_bzDecompress - trailing garbage", true); } else if ( (bzret != BZ_STREAM_END) && (bzret != BZ_OK) ) { return decompressErrCheckSingle(bzret, fileData, outSequenceNumber, strm, "*ERROR during BZ2_bzDecompress - failure exit code", false); } else if ( strm.avail_in != 0 ) { return decompressErrCheckSingle(bzret, fileData, outSequenceNumber, strm, "*ERROR unconsumed in after BZ2_bzDecompress loop", false); } else if ( (bzret != BZ_STREAM_END) && fileData->isLastInSequence ) { return decompressErrCheckSingle(bzret, fileData, outSequenceNumber, strm, "*ERROR on decompress - last in segment reached before BZ_STREAM_END", false); } return 0; } /* * Initialize and set thread signal mask */ int initSignalMask() { int ret = 0; ret = sigemptyset(&SignalMask); ret = sigaddset(&SignalMask, SIGINT) | ret; ret = sigaddset(&SignalMask, SIGTERM) | ret; ret = sigaddset(&SignalMask, SIGABRT) | ret; ret = sigaddset(&SignalMask, SIG_HANDLER_QUIT_SIGNAL) | ret; #ifndef WIN32 ret = sigaddset(&SignalMask, SIGHUP) | ret; #endif if (ret == 0) { ret = pthread_sigmask(SIG_BLOCK, &SignalMask, NULL); } return ret; } /* * Initialize attributes for child threads. * */ int initChildThreadAttributes() { int ret = pthread_attr_init(&ChildThreadAttributes); if (ret < 0) { fprintf(stderr, "Can't initialize thread atrributes [err=%d]! Aborting...\n", ret); exit(-1); } #ifdef USE_STACKSIZE_CUSTOMIZATION if (ChildThreadStackSize > 0) { ret = pthread_attr_setstacksize(&ChildThreadAttributes, ChildThreadStackSize); if (ret != 0) { fprintf(stderr, "Can't set thread stacksize [err=%d]! Countinue with default one.\n", ret); } } #endif // USE_STACKSIZE_CUSTOMIZATION return ret; } /* * Setup and start signal handling. */ int setupSignalHandling() { int ret = initSignalMask(); if (ret == 0) { ret = pthread_create(&SignalHandlerThread, &ChildThreadAttributes, signalHandlerProc, NULL); } return ret; } /* * Setup and start signal handling. */ int setupTerminator() { return pthread_create(&TerminatorThread, &ChildThreadAttributes, terminatorThreadProc, NULL ); } /* ********************************************************* * Clean unfinished work (after error). * Deletes output file if such exists and if not using pipes. */ void cleanupUnfinishedWork() { if (unfinishedWorkCleaned != 0) { return; } struct stat statBuf; int ret = 0; #ifdef PBZIP_DEBUG fprintf(stderr, " Infile: %s Outfile: %s\n", sigInFilename, sigOutFilename); #endif // only cleanup files if we did something with them if ((sigInFilename == NULL) || (sigOutFilename == NULL) || (OutputStdOut == 1)) { unfinishedWorkCleaned = 1; return; } if (QuietMode != 1) { fprintf(stderr, "Cleanup unfinished work [Outfile: %s]...\n", sigOutFilename); } // check to see if input file still exists ret = stat(sigInFilename, &statBuf); if (ret == 0) { // only want to remove output file if input still exists if (QuietMode != 1) fprintf(stderr, "Deleting output file: %s, if it exists...\n", sigOutFilename); ret = remove(sigOutFilename); if (ret != 0) { ErrorContext::syncPrintErrnoMsg(stderr, errno); fprintf(stderr, "pbzip2: *WARNING: Deletion of output file (apparently) failed.\n"); } else { fprintf(stderr, "pbzip2: *INFO: Deletion of output file succeeded.\n"); sigOutFilename = NULL; } } else { fprintf(stderr, "pbzip2: *WARNING: Output file was not deleted since input file no longer exists.\n"); fprintf(stderr, "pbzip2: *WARNING: Output file: %s, may be incomplete!\n", sigOutFilename); } unfinishedWorkCleaned = 1; } /* ********************************************************* */ /* ********************************************************* * Terminator thread procedure: looking at terminateFlag * and exit application when it's set. */ void* terminatorThreadProc(void* arg) { int ret = pthread_mutex_lock(&TerminateFlagMutex); if (ret != 0) { ErrorContext::syncPrintErrnoMsg(stderr, errno); fprintf(stderr, "Terminator thread: pthread_mutex_lock error [%d]! Aborting...\n", ret); syncSetTerminateFlag(1); cleanupAndQuit(1); } while ((finishedFlag == 0) && (terminateFlag == 0)) { ret = pthread_cond_wait(&TerminateCond, &TerminateFlagMutex); } // Successfull end if (finishedFlag != 0) { ret = pthread_mutex_unlock(&TerminateFlagMutex); return NULL; } // Being here implies (terminateFlag != 0) ret = pthread_mutex_unlock(&TerminateFlagMutex); fprintf(stderr, "Terminator thread: premature exit requested - quitting...\n"); cleanupAndQuit(1); return NULL; // never reachable } /* ********************************************************* * Signal handler thread function to hook cleanup on * certain signals. */ void* signalHandlerProc(void* arg) { int signalCaught; // wait for specified in mask signal int ret = sigwait(&SignalMask, &signalCaught); if (ret != 0) { fprintf(stderr, "\n *signalHandlerProc - sigwait error: %d\n", ret); } else if (signalCaught == SIG_HANDLER_QUIT_SIGNAL) { return NULL; } else // ret == 0 { fprintf(stderr, "\n *Control-C or similar caught [sig=%d], quitting...\n", signalCaught); // Delegating cleanup and termination to Terminator Thread syncSetTerminateFlag(1); } return NULL; } /* * Cleanup unfinished work (output file) and exit with the given exit code. * To be used to quite on error with non-zero exitCode. */ void cleanupAndQuit(int exitCode) { // syncSetTerminateFlag(1); int ret = pthread_mutex_lock(&ErrorHandlerMutex); if (ret != 0) { fprintf(stderr, "Cleanup Handler: Failed to lock ErrorHandlerMutex! May double cleanup...\n"); } cleanupUnfinishedWork(); pthread_mutex_unlock(&ErrorHandlerMutex); exit(exitCode); } /* ********************************************************* This function will search the array pointed to by searchBuf[] for the string searchString[] and return a pointer to the start of the searchString[] if found otherwise return NULL if not found. */ template FI1 memstr(FI1 searchBuf, int searchBufSize, FI2 searchString, int searchStringSize) { FI1 searchBufEnd = searchBuf + searchBufSize; FI1 s = std::search(searchBuf, searchBufEnd, searchString, searchString + searchStringSize); return (s != searchBufEnd) ? s : NULL; } /** * Check for interrupt conditions - report if any and perform the relevant * cleanup * * @param hInfile * @param fileData * @param lastBlock * @return 0 - not interrupted; 1 - interrupted (terminate flag or other error encountered) */ int producerDecompressCheckInterrupt(int hInfile, outBuff *& fileData, int lastBlock) { bool isInterrupted = false; if (syncGetLastGoodBlock() != -1) { isInterrupted = true; #ifdef PBZIP_DEBUG fprintf (stderr, "producer_decompress: interrupt1 - LastGoodBlock set. " "Last produced=%d\n", lastBlock); #endif } if (syncGetTerminateFlag() != 0) { isInterrupted = true; #ifdef PBZIP_DEBUG fprintf (stderr, "producer_decompress: interrupt2 - TerminateFlag set. " "Last produced=%d\n", lastBlock); #endif } if (isInterrupted) { close(hInfile); disposeMemorySingle(fileData); return 1; } return 0; } /* ********************************************************* Function works in single pass. It's Splitting long streams into sequences of multiple segments. */ int producer_decompress(int hInfile, OFF_T fileSize, queue *fifo) { safe_mutex_lock(&ProgressIndicatorsMutex); InBytesProduced = 0; NumBlocks = 0; safe_mutex_unlock(&ProgressIndicatorsMutex); pbzip2::BZ2StreamScanner bz2StreamScanner(hInfile); // keep going until all the blocks are processed outBuff * fileData = bz2StreamScanner.getNextStream(); while (!bz2StreamScanner.failed() && (fileData->bufSize > 0)) { #ifdef PBZIP_DEBUG fprintf(stderr, " -> Bytes Read: %u bytes...\n", fileData->bufSize); #endif if (producerDecompressCheckInterrupt(hInfile, fileData, NumBlocks) != 0) { safe_mutex_lock(fifo->mut); safe_cond_broadcast(fifo->notEmpty); // just in case safe_mutex_unlock(fifo->mut); syncSetProducerDone(1); return 0; } if (QuietMode != 1) { // give warning to user if block is larger than 250 million bytes if (fileData->bufSize > 250000000) { fprintf(stderr, "pbzip2: *WARNING: Compressed block size is large [%"PRIuMAX" bytes].\n", (uintmax_t) fileData->bufSize); fprintf(stderr, " If program aborts, use regular BZIP2 to decompress.\n"); } } // add data to the decompression queue safe_mutex_lock(fifo->mut); while (fifo->full) { #ifdef PBZIP_DEBUG fprintf (stderr, "producer: queue FULL.\n"); #endif safe_cond_wait (fifo->notFull, fifo->mut); if (producerDecompressCheckInterrupt(hInfile, fileData, NumBlocks) != 0) { safe_cond_broadcast(fifo->notEmpty); // just in case syncSetProducerDone(1); safe_mutex_unlock(fifo->mut); return 0; } } #ifdef PBZIP_DEBUG fprintf(stderr, "producer: Buffer: %p Size: %"PRIuMAX" Block: %d\n", fileData->buf, (uintmax_t)fileData->bufSize, NumBlocks); #endif fifo->add(fileData); safe_cond_signal (fifo->notEmpty); safe_mutex_lock(&ProgressIndicatorsMutex); InBytesProduced += fileData->bufSize; NumBlocks = fileData->blockNumber + 1; safe_mutex_unlock(&ProgressIndicatorsMutex); safe_mutex_unlock(fifo->mut); fileData = bz2StreamScanner.getNextStream(); } // for close(hInfile); // last stream is always dummy one (either error or eof) delete fileData; if (bz2StreamScanner.failed()) { handle_error(EF_EXIT, 1, "pbzip2: producer_decompress: *ERROR: when reading bzip2 input stream\n"); return -1; } else if (!bz2StreamScanner.isBz2HeaderFound() || !bz2StreamScanner.eof()) { handle_error(EF_EXIT, 1, "pbzip2: producer_decompress: *ERROR: input file is not a valid bzip2 stream\n"); return -1; } syncSetProducerDone(1); safe_mutex_lock(fifo->mut); safe_cond_broadcast(fifo->notEmpty); // just in case safe_mutex_unlock(fifo->mut); #ifdef PBZIP_DEBUG fprintf(stderr, "producer: Done - exiting. Last Block: %d\n", NumBlocks); #endif return 0; } /** * Check for interrupt conditions - report if any and perform the relevant * cleanup * * @return 0 - not interrupted; 1 - interrupted (terminate flag or other error encountered) */ int consumerDecompressCheckInterrupt(const outBuff * lastElement) { bool isInterrupted = false; #ifdef PBZIP_DEBUG uintmax_t thid = (uintmax_t) pthread_self(); #endif if (syncGetTerminateFlag() != 0) { isInterrupted = true; #ifdef PBZIP_DEBUG fprintf (stderr, "(%"PRIuMAX") consumer_decompress: interrupt1 - TerminateFlag set.\n", thid); #endif } int minErrBlock = syncGetMinErrorBlock(); if ( (minErrBlock != -1) && ( (lastElement == NULL) || (lastElement->blockNumber >= minErrBlock) || lastElement->isLastInSequence ) ) { isInterrupted = true; #ifdef PBZIP_DEBUG fprintf (stderr, "(%"PRIuMAX") consumer_decompress: terminating1 - LastGoodBlock set [%d].\n", thid, syncGetLastGoodBlock()); #endif } if (isInterrupted) { return 1; } return 0; } /* ********************************************************* */ void *consumer_decompress(void *q) { queue *fifo = (queue *)q; outBuff *fileData = NULL; outBuff *lastFileData = NULL; char *DecompressedData = NULL; unsigned int outSize = 0; outBuff * prevOutBlockInSequence = NULL; int outSequenceNumber = -1; // sequence number in multi-part output blocks unsigned int processedIn = 0; int errState = 0; bz_stream strm; strm.bzalloc = NULL; strm.bzfree = NULL; strm.opaque = NULL; for (;;) { safe_mutex_lock(fifo->mut); for (;;) { if (consumerDecompressCheckInterrupt(fileData) != 0) { safe_mutex_unlock(fifo->mut); return (NULL); } if (!fifo->empty && (fifo->remove(fileData) == 1)) { // block retreived - break the loop and continue further break; } #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: queue EMPTY.\n"); #endif if (fifo->empty && ((syncGetProducerDone() == 1) || (syncGetTerminateFlag() != 0))) { // finished - either OK or terminated forcibly pthread_mutex_unlock(fifo->mut); // BZ2_bzDecompressEnd( &strm ); if ((syncGetTerminateFlag() == 0) && (outSequenceNumber != -1)) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR on decompress - " "premature end of archive stream (block=%d; seq=%d; outseq=%d)!\n", lastFileData->blockNumber, lastFileData->sequenceNumber, outSequenceNumber); } #ifdef PBZIP_DEBUG else { fprintf (stderr, "consumer: exiting2\n"); } #endif disposeMemorySingle( lastFileData ); return (NULL); } #ifdef PBZIP_DEBUG safe_cond_timed_wait(fifo->notEmpty, fifo->mut, 1, "consumer"); #else safe_cond_wait(fifo->notEmpty, fifo->mut); #endif } #ifdef PBZIP_DEBUG fprintf(stderr, "consumer: FileData: %p\n", fileData); fprintf(stderr, "consumer: Buffer: %p Size: %u Block: %d\n", fileData->buf, (unsigned)fileData->bufSize, fileData->blockNumber); #endif safe_cond_signal(fifo->notFull); safe_mutex_unlock(fifo->mut); if (lastFileData != NULL) { delete lastFileData; } lastFileData = fileData; #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: recieved %d.\n", fileData->blockNumber); #endif outSize = 900000; int bzret = BZ_OK; if (fileData->sequenceNumber < 2) { // start of new stream from in queue (0 -> single block; 1 - mutli) bzret = BZ2_bzDecompressInit(&strm, Verbosity, 0); if (bzret != BZ_OK) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR during BZ2_bzDecompressInit: %d\n", bzret); return (NULL); } } strm.avail_in = fileData->bufSize; strm.next_in = fileData->buf; while ((bzret == BZ_OK) && (strm.avail_in != 0)) { #ifdef PBZIP_DEBUG fprintf(stderr, "decompress: block=%d; seq=%d; prev=%llx; avail_in=%u; avail_out=%u\n", fileData->blockNumber, outSequenceNumber, (unsigned long long) prevOutBlockInSequence, strm.avail_in, strm.avail_out); #endif if (DecompressedData == NULL) { // allocate memory for decompressed data (start with default 900k block size) DecompressedData = new(std::nothrow) char[outSize]; // make sure memory was allocated properly if (DecompressedData == NULL) { handle_error(EF_EXIT, -1, " *ERROR: Could not allocate memory (DecompressedData)! Aborting...\n"); return (NULL); } processedIn = 0; strm.avail_out = outSize; strm.next_out = DecompressedData; } unsigned int availIn = strm.avail_in; bzret = BZ2_bzDecompress(&strm); processedIn += (availIn - strm.avail_in); #ifdef PBZIP_DEBUG fprintf(stderr, "decompress: BZ2_bzDecompress=%d; block=%d; seq=%d; prev=%llx; avail_in=%u; avail_out=%u\n", bzret, fileData->blockNumber, outSequenceNumber, (unsigned long long) prevOutBlockInSequence, strm.avail_in, strm.avail_out); #endif // issue out block if out buffer is full or stream end is detected if ( ((bzret == BZ_OK) && strm.avail_out == 0) || (bzret == BZ_STREAM_END) ) { outBuff * addret = NULL; unsigned int len = outSize - strm.avail_out; bool isLast = (bzret == BZ_STREAM_END); if ( hasTrailingGarbage( bzret, fileData, strm ) ) { // trailing garbage detected syncSetLastGoodBlock(fileData->blockNumber, fileData->blockNumber); } if (outSequenceNumber>0) { ++outSequenceNumber; outBuff * nextOutBlock = new(std::nothrow) outBuff( DecompressedData, len, fileData->blockNumber, outSequenceNumber, processedIn, isLast, NULL); if (nextOutBlock == NULL) { BZ2_bzDecompressEnd( &strm ); handle_error(EF_EXIT, -1, " *ERROR: Could not allocate memory (nextOutBlock)! Aborting...\n"); return (NULL); } addret = outputBufferSeqAddNext(prevOutBlockInSequence, nextOutBlock); #ifdef PBZIP_DEBUG fprintf(stderr, "decompress: outputBufferSeqAddNext->%llx; block=%d; seq=%d; prev=%llx\n", (unsigned long long)addret, fileData->blockNumber, outSequenceNumber, (unsigned long long) prevOutBlockInSequence); #endif } else // sequenceNumber = 0 { outSequenceNumber = (bzret == BZ_OK) ? 1 : 0; addret = outputBufferAdd(outBuff( DecompressedData, len, fileData->blockNumber, outSequenceNumber, processedIn, isLast, NULL), "consumer_decompress"); #ifdef PBZIP_DEBUG fprintf(stderr, "decompress: outputBufferAdd->%llx; block=%d; seq=%d; prev=%llx\n", (unsigned long long)addret, fileData->blockNumber, outSequenceNumber, (unsigned long long) prevOutBlockInSequence); #endif } if (addret == NULL) { // error encountered BZ2_bzDecompressEnd( &strm ); return (NULL); } prevOutBlockInSequence = addret; DecompressedData = NULL; } } /* * < 0 - fatal error; * 0 - OK (no error at all); * 1 - first block of ignored trailing garbage; * 2 - error already signalled for earlier block */ errState = decompressErrCheck(bzret, fileData, outSequenceNumber, strm); if (bzret == BZ_STREAM_END) { bzret = BZ2_bzDecompressEnd(&strm); if ( (bzret != BZ_OK) && ((errState == 0) || (errState == 1)) ) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR during BZ2_bzDecompressEnd: %d\n", bzret); return (NULL); } outSequenceNumber = -1; prevOutBlockInSequence = NULL; } #ifdef PBZIP_DEBUG fprintf(stderr, "\n Compressed Block Size: %u\n", (unsigned)fileData->bufSize); fprintf(stderr, " Original Block Size: %u\n", outSize); #endif disposeMemory(fileData->buf); #ifdef PBZIP_DEBUG fprintf(stderr, " OutputBuffer[%d].buf = %p\n", fileData->blockNumber, DecompressedData); fprintf(stderr, " OutputBuffer[%d].bufSize = %u\n", fileData->blockNumber, outSize); fflush(stderr); #endif if (errState != 0) { #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: exiting prematurely: errState=%d\n", errState); #endif return (NULL); } } // for #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: exiting\n"); #endif return (NULL); } /* ********************************************************* */ void *fileWriter(void *outname) { char *OutFilename; OFF_T CompressedSize = 0; int percentComplete = 0; int hOutfile = STDOUT_FILENO; // default to stdout int currBlock = 0; size_t outBufferPos = 0; int ret = -1; OFF_T bytesProcessed = 0; OutFilename = (char *) outname; outBuff * prevBlockInSequence = NULL; #ifdef PBZIP_DEBUG fprintf(stderr, "fileWriter function started\n"); #endif // write to file instead of stdout if (OutputStdOut == 0) { hOutfile = safe_open_output(OutFilename); // check to see if file creation was successful if (hOutfile == -1) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not create output file [%s]!\n", OutFilename); return (NULL); } } while (true) { #ifdef PBZIP_DEBUG int lastseq = 0; if (prevBlockInSequence != NULL) { lastseq = prevBlockInSequence->sequenceNumber; } #endif // Order is important. We don't need sync on NumBlocks when producer // is done. if ((syncGetProducerDone() == 1) && (currBlock >= NumBlocks) && (prevBlockInSequence == NULL)) { #ifdef PBZIP_DEBUG fprintf(stderr, "fileWriter [b:%d:%d]: done - quit loop.\n", currBlock, lastseq); #endif // We're done break; } if (syncGetTerminateFlag() != 0) { #ifdef PBZIP_DEBUG fprintf (stderr, "fileWriter [b:%d]: terminating1 - terminateFlag set\n", currBlock); #endif break; } safe_mutex_lock(OutMutex); #ifdef PBZIP_DEBUG outBuff * lastnext = (prevBlockInSequence != NULL) ? prevBlockInSequence->next : NULL; fprintf(stderr, "fileWriter: Block: %d Size: %"PRIuMAX" Next File Block: %d" ", outBufferPos: %"PRIuMAX", NumBlocks: %d, producerDone: %d, lastseq=%d" ", prev=%p, next=%p\n", currBlock, (uintmax_t)NumBufferedBlocksMax, NextBlockToWrite, (uintmax_t)outBufferPos, NumBlocks, syncGetProducerDone(), lastseq, prevBlockInSequence, lastnext); #endif if ( (LastGoodBlock != -1) && (NextBlockToWrite > LastGoodBlock) ) { #ifdef PBZIP_DEBUG fprintf (stderr, "fileWriter [b:%d]: quit - LastGoodBlock=%d\n", currBlock, LastGoodBlock); #endif safe_mutex_unlock(OutMutex); break; } if ((OutputBuffer[outBufferPos].buf == NULL) && ((prevBlockInSequence == NULL) || (prevBlockInSequence->next == NULL))) { safe_cond_timed_wait(&OutBufferHeadNotEmpty, OutMutex, 1, "fileWriter"); safe_mutex_unlock(OutMutex); // sleep a little so we don't go into a tight loop using up all the CPU // usleep(50000); continue; } else { safe_mutex_unlock(OutMutex); } outBuff * outBlock; if (prevBlockInSequence != NULL) { outBlock = prevBlockInSequence->next; } else { outBlock = &OutputBuffer[outBufferPos]; } #ifdef PBZIP_DEBUG fprintf(stderr, "fileWriter: Buffer: %p Size: %u Block: %d, Seq: %d, isLast: %d\n", OutputBuffer[outBufferPos].buf, OutputBuffer[outBufferPos].bufSize, currBlock, outBlock->sequenceNumber, (int)outBlock->isLastInSequence); #endif // write data to the output file ret = do_write(hOutfile, outBlock->buf, outBlock->bufSize); #ifdef PBZIP_DEBUG fprintf(stderr, "\n -> Total Bytes Written[%d:%d]: %d bytes...\n", currBlock, outBlock->sequenceNumber, ret); #endif if (ret < 0) { if (OutputStdOut == 0) close(hOutfile); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not write %d bytes to file [ret=%d]! Aborting...\n", outBlock->bufSize, ret); return (NULL); } CompressedSize += ret; bytesProcessed += outBlock->inSize; delete [] outBlock->buf; outBlock->buf = NULL; outBlock->bufSize = 0; if (outBlock->isLastInSequence) { if (++outBufferPos == NumBufferedBlocksMax) { outBufferPos = 0; } ++currBlock; } safe_mutex_lock(OutMutex); if (outBlock->isLastInSequence) { ++NextBlockToWrite; OutBufferPosToWrite = outBufferPos; } if (outBlock->sequenceNumber > 1) { --NumBufferedTailBlocks; } // --NumBufferedBlocks; // to be removed safe_cond_broadcast(notTooMuchNumBuffered); safe_cond_broadcast(&ErrStateChangeCond); safe_mutex_unlock(OutMutex); if (outBlock->sequenceNumber > 2) { delete prevBlockInSequence; } if (outBlock->isLastInSequence) { prevBlockInSequence = NULL; if (outBlock->sequenceNumber > 1) { delete outBlock; } } else { prevBlockInSequence = outBlock; } if (QuietMode != 1) { // print current completion status int percentCompleteOld = percentComplete; if (InFileSize > 0) { percentComplete = (100.0 * (double)bytesProcessed / (double)InFileSize); } #ifdef PBZIP_DEBUG fprintf(stderr, "Completed: %d%% NextBlockToWrite: %d/%"PRIuMAX" \r", percentComplete, NextBlockToWrite, (uintmax_t)NumBufferedBlocksMax); fflush(stderr); #else if (percentComplete != percentCompleteOld) { fprintf(stderr, "Completed: %d%% \r", percentComplete); fflush(stderr); } #endif } } // while if (currBlock == 0) { // zero-size file needs special handling ret = do_write(hOutfile, Bz2HeaderZero, sizeof(Bz2HeaderZero) ); if (ret < 0) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not write to file! Aborting...\n"); return (NULL); } } if (OutputStdOut == 0) { ret = close(hOutfile); if (ret == -1) { ErrorContext::getInstance()->saveError(); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not close output file! Aborting...\n"); return (NULL); } } if (QuietMode != 1) { fprintf(stderr, " Output Size: %"PRIuMAX" bytes\n", (uintmax_t)CompressedSize); } #ifdef PBZIP_DEBUG fprintf(stderr, "fileWriter exit\n"); fflush(stderr); #endif // wake up all other possibly blocked on cond threads if (FifoQueue != NULL) { safe_mutex_lock(FifoQueue->mut); safe_cond_broadcast(FifoQueue->notEmpty); // important safe_cond_broadcast(FifoQueue->notFull); // not really needed safe_mutex_unlock(FifoQueue->mut); } safe_mutex_lock(OutMutex); safe_cond_broadcast(notTooMuchNumBuffered); // not really needed safe_mutex_unlock(OutMutex); if (QuietMode != 1) { // print current completion status percentComplete = 100; #ifdef PBZIP_DEBUG fprintf(stderr, "Completed: %d%% NextBlockToWrite: %d/%"PRIuMAX" \r", percentComplete, NextBlockToWrite, (uintmax_t)NumBufferedBlocksMax); fflush(stderr); #else fprintf(stderr, "Completed: %d%% \r", percentComplete); fflush(stderr); #endif } return (NULL); } /* ********************************************************* */ int directcompress(int hInfile, OFF_T fileSize, int blockSize, const char *OutFilename) { char *FileData = NULL; char *CompressedData = NULL; OFF_T CompressedSize = 0; OFF_T bytesLeft = 0; OFF_T inSize = 0; unsigned int outSize = 0; int percentComplete = 0; int hOutfile = STDOUT_FILENO; // default to stdout int currBlock = 0; int rret = 0; int ret = 0; bytesLeft = fileSize; // write to file instead of stdout if (OutputStdOut == 0) { hOutfile = safe_open_output(OutFilename); // check to see if file creation was successful if (hOutfile == -1) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not create output file [%s]!\n", OutFilename); return -1; } } #ifdef WIN32 else { setmode(fileno(stdout), O_BINARY); } #endif // keep going until all the file is processed while (bytesLeft > 0) { if (syncGetTerminateFlag() != 0) { close(hInfile); if (OutputStdOut == 0) close(hOutfile); fprintf (stderr, "directcompress: terminating - terminateFlag set\n"); return -1; } // // READ DATA // // set buffer size if (bytesLeft > blockSize) inSize = blockSize; else inSize = bytesLeft; #ifdef PBZIP_DEBUG fprintf(stderr, " -> Bytes To Read: %"PRIuMAX" bytes...\n", (uintmax_t)inSize); #endif // allocate memory to read in file FileData = NULL; FileData = new(std::nothrow) char[inSize]; // make sure memory was allocated properly if (FileData == NULL) { close(hInfile); if (OutputStdOut == 0) close(hOutfile); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (FileData)! Aborting...\n"); return -1; } // read file data rret = do_read(hInfile, (char *) FileData, inSize); #ifdef PBZIP_DEBUG fprintf(stderr, " -> Total Bytes Read: %d bytes...\n\n", rret); #endif if (rret == 0) { if (FileData != NULL) delete [] FileData; break; } else if (rret < 0) { close(hInfile); if (FileData != NULL) delete [] FileData; if (OutputStdOut == 0) close(hOutfile); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not read from file! Aborting...\n"); return -1; } // set bytes left after read bytesLeft -= rret; // // COMPRESS DATA // outSize = (int) ((inSize*1.01)+600); // allocate memory for compressed data CompressedData = new(std::nothrow) char[outSize]; // make sure memory was allocated properly if (CompressedData == NULL) { close(hInfile); if (FileData != NULL) delete [] FileData; handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (CompressedData)! Aborting...\n"); return -1; } // compress the memory buffer (blocksize=9*100k, verbose=0, worklevel=30) ret = BZ2_bzBuffToBuffCompress(CompressedData, &outSize, FileData, inSize, BWTblockSize, Verbosity, 30); if (ret != BZ_OK) { close(hInfile); if (FileData != NULL) delete [] FileData; handle_error(EF_EXIT, -1, "pbzip2: *ERROR during compression: %d! Aborting...\n", ret); return -1; } #ifdef PBZIP_DEBUG fprintf(stderr, "\n Original Block Size: %"PRIuMAX"\n", (uintmax_t)inSize); fprintf(stderr, " Compressed Block Size: %u\n", outSize); #endif // // WRITE DATA // // write data to the output file ret = do_write(hOutfile, CompressedData, outSize); #ifdef PBZIP_DEBUG fprintf(stderr, "\n -> Total Bytes Written[%d]: %d bytes...\n", currBlock, ret); #endif if (ret <= 0) { close(hInfile); if (FileData != NULL) delete [] FileData; if (CompressedData != NULL) delete [] CompressedData; if (OutputStdOut == 0) close(hOutfile); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not write to file! Aborting...\n"); return -1; } CompressedSize += ret; currBlock++; // print current completion status int percentCompleteOld = percentComplete; percentComplete = 100 * currBlock / NumBlocksEstimated; if (QuietMode != 1) { if (percentComplete != percentCompleteOld) { fprintf(stderr, "Completed: %d%% \r", percentComplete); fflush(stderr); } } // clean up memory if (FileData != NULL) { delete [] FileData; FileData = NULL; } if (CompressedData != NULL) { delete [] CompressedData; CompressedData = NULL; } // check to make sure all the data we expected was read in if (rret != inSize) inSize = rret; } // while close(hInfile); if (OutputStdOut == 0) close(hOutfile); if (QuietMode != 1) { fprintf(stderr, " Output Size: %"PRIuMAX" bytes\n", (uintmax_t)CompressedSize); } syncSetProducerDone(1); // Not really needed for direct version return 0; } /* ********************************************************* */ int directdecompress(const char *InFilename, const char *OutFilename) { FILE *stream = NULL; FILE *zStream = NULL; BZFILE* bzf = NULL; unsigned char obuf[5000]; unsigned char unused[BZ_MAX_UNUSED]; unsigned char *unusedTmp; int bzerr, nread, streamNo; int nUnused; int ret = 0; int i; nUnused = 0; streamNo = 0; // see if we are using stdin or not if (strcmp(InFilename, "-") != 0) { // open the file for reading zStream = fopen(InFilename, "rb"); if (zStream == NULL) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not open input file [%s]! Aborting...\n", InFilename); return -1; } } else { #ifdef WIN32 setmode(fileno(stdin), O_BINARY); #endif zStream = stdin; } // check file stream for errors if (ferror(zStream)) { if (zStream != stdin) fclose(zStream); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem with input stream of file [%s]! Aborting...\n", InFilename); return -1; } // see if we are outputting to stdout if (OutputStdOut == 0) { stream = safe_fopen_output(OutFilename, "wb"); if (stream == NULL) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not open output file [%s]! Aborting...\n", OutFilename); return -1; } } else { #ifdef WIN32 setmode(fileno(stdout), O_BINARY); #endif stream = stdout; } // check file stream for errors if (ferror(stream)) { if (stream != stdout) fclose(stream); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem with output stream of file [%s]! Aborting...\n", InFilename); return -1; } // loop until end of file while(true) { if (syncGetTerminateFlag() != 0) { fprintf (stderr, "directdecompress: terminating1 - terminateFlag set\n"); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); return -1; } bzf = BZ2_bzReadOpen(&bzerr, zStream, Verbosity, 0, unused, nUnused); if (bzf == NULL || bzerr != BZ_OK) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); if (ret != 0) { syncSetTerminateFlag(1); } return ret; } streamNo++; while (bzerr == BZ_OK) { if (syncGetTerminateFlag() != 0) { fprintf (stderr, "directdecompress: terminating2 - terminateFlag set\n"); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); return -1; } nread = BZ2_bzRead(&bzerr, bzf, obuf, sizeof(obuf)); if (bzerr == BZ_DATA_ERROR_MAGIC) { // try alternate way of reading data if (ForceOverwrite == 1) { rewind(zStream); while (true) { int c = fgetc(zStream); if (c == EOF) break; ungetc(c,zStream); nread = fread(obuf, sizeof(unsigned char), sizeof(obuf), zStream ); if (ferror(zStream)) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); if (ret != 0) { syncSetTerminateFlag(1); } return ret; } if (nread > 0) (void) fwrite (obuf, sizeof(unsigned char), nread, stream); if (ferror(stream)) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); if (ret != 0) { syncSetTerminateFlag(1); } return ret; } } goto closeok; } } if ((bzerr == BZ_OK || bzerr == BZ_STREAM_END) && nread > 0) (void) fwrite(obuf, sizeof(unsigned char), nread, stream ); if (ferror(stream)) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); if (ret != 0) { syncSetTerminateFlag(1); } return ret; } } if (bzerr != BZ_STREAM_END) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); if (ret != 0) { syncSetTerminateFlag(1); } return ret; } BZ2_bzReadGetUnused(&bzerr, bzf, (void**)(&unusedTmp), &nUnused); if (bzerr != BZ_OK) { handle_error(EF_EXIT, 3, "pbzip2: *ERROR: Unexpected error [bzerr=%d]. Aborting!\n", bzerr); return 3; } for (i = 0; i < nUnused; i++) unused[i] = unusedTmp[i]; BZ2_bzReadClose(&bzerr, bzf); if (bzerr != BZ_OK) { handle_error(EF_EXIT, 3, "pbzip2: *ERROR: Unexpected error [bzerr=%d]. Aborting!\n", bzerr); return 3; } // check to see if we are at the end of the file if (nUnused == 0) { int c = fgetc(zStream); if (c == EOF) break; ungetc(c, zStream); } } closeok: // check file stream for errors if (ferror(zStream)) { if (zStream != stdin) fclose(zStream); if (stream != stdout) fclose(stream); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem with input stream of file [%s]! Skipping...\n", InFilename); return -1; } // close file ret = do_fclose(zStream); if (ret == EOF) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem closing file [%s]! Skipping...\n", InFilename); return -1; } // check file stream for errors if (ferror(stream)) { if (stream != stdout) fclose(stream); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem with output stream of file [%s]! Skipping...\n", InFilename); return -1; } ret = do_fflush(stream); if (ret != 0) { if (stream != stdout) fclose(stream); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem with output stream of file [%s]! Skipping...\n", InFilename); return -1; } if (stream != stdout) { ret = do_fclose(stream); if (ret == EOF) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Problem closing file [%s]! Skipping...\n", OutFilename); return -1; } } syncSetProducerDone(1); // Not really needed for direct version. return 0; } /* * Simulate an unconditional read(), reading in data to fill the * bsize-sized buffer if it can, even if it means calling read() multiple * times. This is needed since pipes and other "special" streams * sometimes don't allow reading of arbitrary sized buffers. */ ssize_t bufread(int hf, char *buf, size_t bsize) { size_t bufr = 0; int ret; int rsize = bsize; while (1) { ret = read(hf, buf, rsize); if (ret < 0) return ret; if (ret == 0) return bufr; bufr += ret; if (bufr == bsize) return bsize; rsize -= ret; buf += ret; } } /* ********************************************************* */ int producer(int hInfile, int blockSize, queue *fifo) { char *FileData = NULL; size_t inSize = 0; // int blockNum = 0; int ret = 0; // int pret = -1; // We will now totally ignore the fileSize and read the data as it // comes in. Aside from allowing us to process arbitrary streams, it's // also the *right thing to do* in unix environments where data may // be appended to the file as it's processed (e.g. log files). safe_mutex_lock(&ProgressIndicatorsMutex); NumBlocks = 0; InBytesProduced = 0; safe_mutex_unlock(&ProgressIndicatorsMutex); // keep going until all the file is processed while (1) { if (syncGetTerminateFlag() != 0) { close(hInfile); return -1; } // set buffer size inSize = blockSize; #ifdef PBZIP_DEBUG fprintf(stderr, " -> Bytes To Read: %"PRIuMAX" bytes...\n", (uintmax_t)inSize); #endif // allocate memory to read in file FileData = NULL; FileData = new(std::nothrow) char[inSize]; // make sure memory was allocated properly if (FileData == NULL) { close(hInfile); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (FileData)! Aborting...\n"); return -1; } // read file data ret = bufread(hInfile, (char *) FileData, inSize); #ifdef PBZIP_DEBUG fprintf(stderr, " -> Total Bytes Read: %d bytes...\n\n", ret); #endif if (ret == 0) { // finished reading. if (FileData != NULL) delete [] FileData; break; } else if (ret < 0) { close(hInfile); if (FileData != NULL) delete [] FileData; handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not read from file! Aborting...\n"); return -1; } // check to make sure all the data we expected was read in if ((size_t)ret != inSize) inSize = ret; #ifdef PBZIP_DEBUG fprintf(stderr, "producer: Going into fifo-mut lock (NumBlocks: %d)\n", NumBlocks); #endif // add data to the compression queue safe_mutex_lock(fifo->mut); while (fifo->full) { #ifdef PBZIP_DEBUG fprintf (stderr, "producer: queue FULL.\n"); #endif safe_cond_wait(fifo->notFull, fifo->mut); if (syncGetTerminateFlag() != 0) { pthread_mutex_unlock(fifo->mut); close(hInfile); return -1; } } #ifdef PBZIP_DEBUG fprintf(stderr, "producer: Buffer: %p Size: %"PRIuMAX" Block: %d\n", FileData, (uintmax_t)inSize, NumBlocks); #endif outBuff * queueElement = new(std::nothrow) outBuff(FileData, inSize, NumBlocks, 0); // make sure memory was allocated properly if (queueElement == NULL) { close(hInfile); handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (queueElement)! Aborting...\n"); return -1; } fifo->add(queueElement); safe_cond_signal(fifo->notEmpty); safe_mutex_lock(&ProgressIndicatorsMutex); ++NumBlocks; InBytesProduced += inSize; safe_mutex_unlock(&ProgressIndicatorsMutex); safe_mutex_unlock(fifo->mut); } // while close(hInfile); syncSetProducerDone(1); safe_mutex_lock(fifo->mut); safe_cond_broadcast(fifo->notEmpty); // just in case safe_mutex_unlock(fifo->mut); #ifdef PBZIP_DEBUG fprintf(stderr, "producer: Done - exiting. Num Blocks: %d\n", NumBlocks); #endif return 0; } /* ********************************************************* */ void *consumer (void *q) { queue *fifo; // char *FileData = NULL; outBuff *fileData; char *CompressedData = NULL; // unsigned int inSize = 0; unsigned int outSize = 0; // int blockNum = -1; int ret = -1; fifo = (queue *)q; for (;;) { if (syncGetTerminateFlag() != 0) { #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: terminating1 - terminateFlag set\n"); #endif return (NULL); } safe_mutex_lock(fifo->mut); for (;;) { if (!fifo->empty && (fifo->remove(fileData) == 1)) { // block retreived - break the loop and continue further break; } #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: queue EMPTY.\n"); #endif if (fifo->empty && ((syncGetProducerDone() == 1) || (syncGetTerminateFlag() != 0))) { safe_mutex_unlock(fifo->mut); #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: exiting2\n"); #endif return (NULL); } #ifdef PBZIP_DEBUG safe_cond_timed_wait(fifo->notEmpty, fifo->mut, 1, "consumer"); #else safe_cond_wait(fifo->notEmpty, fifo->mut); #endif } #ifdef PBZIP_DEBUG fprintf(stderr, "consumer: Buffer: %p Size: %u Block: %d\n", fileData->buf, (unsigned)fileData->bufSize, fileData->blockNumber); #endif safe_cond_signal(fifo->notFull); safe_mutex_unlock(fifo->mut); #ifdef PBZIP_DEBUG fprintf(stderr, "consumer: received %d.\n", fileData->blockNumber); #endif outSize = (unsigned int) (((fileData->bufSize)*1.01)+600); // allocate memory for compressed data CompressedData = new(std::nothrow) char[outSize]; // make sure memory was allocated properly if (CompressedData == NULL) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR: Could not allocate memory (CompressedData)! Aborting...\n"); return (NULL); } // compress the memory buffer (blocksize=9*100k, verbose=0, worklevel=30) ret = BZ2_bzBuffToBuffCompress(CompressedData, &outSize, fileData->buf, fileData->bufSize, BWTblockSize, Verbosity, 30); if (ret != BZ_OK) { handle_error(EF_EXIT, -1, "pbzip2: *ERROR during compression: %d! Aborting...\n", ret); return (NULL); } #ifdef PBZIP_DEBUG fprintf(stderr, "\n Original Block Size: %u\n", (unsigned)fileData->bufSize); fprintf(stderr, " Compressed Block Size: %u\n", outSize); #endif disposeMemory(fileData->buf); // store data to be written in output bin outBuff outBlock = outBuff(CompressedData, outSize, fileData->blockNumber, 0, fileData->bufSize); if (outputBufferAdd(outBlock, "consumer") == NULL) { return (NULL); } delete fileData; fileData = NULL; } // for #ifdef PBZIP_DEBUG fprintf (stderr, "consumer: exiting\n"); #endif return (NULL); } /* ********************************************************* */ int mutexesInit() { // initialize mutexes OutMutex = new(std::nothrow) pthread_mutex_t; // make sure memory was allocated properly if (OutMutex == NULL) { fprintf(stderr, "pbzip2: *ERROR: Could not allocate memory (OutMutex)! Aborting...\n"); return 1; } pthread_mutex_init(OutMutex, NULL); ProducerDoneMutex = new(std::nothrow) pthread_mutex_t; // make sure memory was allocated properly if (ProducerDoneMutex == NULL) { fprintf(stderr, "pbzip2: *ERROR: Could not allocate memory (ProducerDoneMutex)! Aborting...\n"); return 1; } pthread_mutex_init(ProducerDoneMutex, NULL); return 0; } /* ********************************************************* */ void mutexesDelete() { if (OutMutex != NULL) { pthread_mutex_destroy(OutMutex); delete OutMutex; OutMutex = NULL; } if (ProducerDoneMutex != NULL) { pthread_mutex_destroy(ProducerDoneMutex); delete ProducerDoneMutex; ProducerDoneMutex = NULL; } } /* ********************************************************* */ queue *queueInit(int queueSize) { queue *q; QUEUESIZE = queueSize; q = new(std::nothrow) queue; if (q == NULL) return NULL; q->qData = new(std::nothrow) queue::ElementTypePtr[queueSize]; if (q->qData == NULL) return NULL; q->size = queueSize; q->clear(); q->mut = NULL; q->mut = new(std::nothrow) pthread_mutex_t; if (q->mut == NULL) return NULL; pthread_mutex_init(q->mut, NULL); q->notFull = NULL; q->notFull = new(std::nothrow) pthread_cond_t; if (q->notFull == NULL) return NULL; pthread_cond_init(q->notFull, NULL); q->notEmpty = NULL; q->notEmpty = new(std::nothrow) pthread_cond_t; if (q->notEmpty == NULL) return NULL; pthread_cond_init(q->notEmpty, NULL); q->consumers = NULL; q->consumers = new(std::nothrow) pthread_t[queueSize]; if (q->consumers == NULL) return NULL; notTooMuchNumBuffered = NULL; notTooMuchNumBuffered = new(std::nothrow) pthread_cond_t; if (notTooMuchNumBuffered == NULL) return NULL; pthread_cond_init(notTooMuchNumBuffered, NULL); return (q); } /* ********************************************************* */ void queueDelete (queue *q) { if (q == NULL) return; if (q->mut != NULL) { pthread_mutex_destroy(q->mut); delete q->mut; q->mut = NULL; } if (q->notFull != NULL) { pthread_cond_destroy(q->notFull); delete q->notFull; q->notFull = NULL; } if (q->notEmpty != NULL) { pthread_cond_destroy(q->notEmpty); delete q->notEmpty; q->notEmpty = NULL; } delete [] q->consumers; delete [] q->qData; delete q; q = NULL; if (notTooMuchNumBuffered != NULL) { pthread_cond_destroy(notTooMuchNumBuffered); delete notTooMuchNumBuffered; notTooMuchNumBuffered = NULL; } return; } /** * Initialize output buffer contents with empty (NULL, 0) blocks * * @param size new size of buffer * */ void outputBufferInit(size_t size) { safe_mutex_lock(OutMutex); NextBlockToWrite = 0; OutBufferPosToWrite = 0; NumBufferedBlocks = 0; NumBufferedTailBlocks = 0; outBuff emptyElement; emptyElement.buf = NULL; emptyElement.bufSize = 0; // Resize and fill-in with empty elements OutputBuffer.assign(size, emptyElement); // unlikely to get here since more likely exception will be thrown if (OutputBuffer.size() != size) { fprintf(stderr, "pbzip2: *ERROR: Could not initialize (OutputBuffer); size=%"PRIuMAX"! Aborting...\n", (uintmax_t)size); safe_mutex_unlock(OutMutex); exit(1); } safe_mutex_unlock(OutMutex); } /** * Get output buffer index corresponding to the given absolute blockNumber * (buffer is used in circular mode) * * @param blockNum - absolute block number to translate * @return 0-based Output Buffer index where blockNum data should go */ inline size_t getOutputBufferPos(int blockNum) { // calculate output buffer position (used in circular mode) size_t outBuffPos = OutBufferPosToWrite + blockNum - NextBlockToWrite; if (outBuffPos >= NumBufferedBlocksMax) { outBuffPos -= NumBufferedBlocksMax; } return outBuffPos; } /** * Add next element to the given out buffer tail. * */ outBuff * outputBufferSeqAddNext(outBuff * preveElement, outBuff * newElement) { safe_mutex_lock(OutMutex); while ((NumBufferedTailBlocks >= NumBufferedBlocksMax) && (preveElement->buf != NULL)) { if (syncGetTerminateFlag() != 0) { #ifdef PBZIP_DEBUG fprintf (stderr, "%s: terminating2 - terminateFlag set\n", "consumer"); #endif pthread_mutex_unlock(OutMutex); return NULL; } if ( (LastGoodBlock != -1) && (LastGoodBlock < newElement->blockNumber) ) { #ifdef PBZIP_DEBUG fprintf (stderr, "%s: terminating3 - LastGoodBlock set\n", "consumer"); #endif pthread_mutex_unlock(OutMutex); return NULL; } #ifdef PBZIP_DEBUG fprintf (stderr, "%s/outputBufferSeqAddNext: Throttling from FileWriter backlog: %d\n", "consumer", NumBufferedBlocks); #endif safe_cond_wait(notTooMuchNumBuffered, OutMutex); } preveElement->next = newElement; ++NumBufferedTailBlocks; // size_t outBufPos = getOutputBufferPos(newElement->blockNumber); if (preveElement->buf == NULL) { // fileWriter has already consumed the previous block. Let it know // for that one early safe_cond_signal(&OutBufferHeadNotEmpty); } safe_mutex_unlock(OutMutex); return newElement; } /** * Store an item in OutputBuffer out bin. Synchronization is embedded to protect * from simultaneous access. * * @param elem - output buffer element to add * @param caller - used for debug purposes (caller function name) * * @return pointer to added element on success; NULL - on error */ outBuff * outputBufferAdd(const outBuff & element, const char *caller) { safe_mutex_lock(OutMutex); // wait while blockNum is out of range // [NextBlockToWrite, NextBlockToWrite + NumBufferedBlocksMax) int dist = element.blockNumber - NumBufferedBlocksMax; while (dist >= NextBlockToWrite) { if (syncGetTerminateFlag() != 0) { #ifdef PBZIP_DEBUG fprintf (stderr, "%s/outputBufferAdd: terminating2 - terminateFlag set\n", caller); #endif pthread_mutex_unlock(OutMutex); return NULL; } if ( (LastGoodBlock != -1) && (LastGoodBlock < element.blockNumber) ) { #ifdef PBZIP_DEBUG fprintf (stderr, "%s: terminating3 - LastGoodBlock set\n", "consumer"); #endif pthread_mutex_unlock(OutMutex); return NULL; } #ifdef PBZIP_DEBUG fprintf (stderr, "%s: Throttling from FileWriter backlog: %d\n", caller, NumBufferedBlocks); #endif safe_cond_wait(notTooMuchNumBuffered, OutMutex); } // calculate output buffer position (used in circular mode) size_t outBuffPos = getOutputBufferPos(element.blockNumber); OutputBuffer[outBuffPos] = element; ++NumBufferedBlocks; if (NextBlockToWrite == element.blockNumber) { safe_cond_signal(&OutBufferHeadNotEmpty); } safe_mutex_unlock(OutMutex); return &(OutputBuffer[outBuffPos]); } /* ********************************************************* Much of the code in this function is taken from bzip2.c */ int testBZ2ErrorHandling(int bzerr, BZFILE* bzf, int streamNo) { int bzerr_dummy; BZ2_bzReadClose(&bzerr_dummy, bzf); switch (bzerr) { case BZ_CONFIG_ERROR: fprintf(stderr, "pbzip2: *ERROR: Integers are not the right size for libbzip2. Aborting!\n"); exit(3); break; case BZ_IO_ERROR: fprintf(stderr, "pbzip2: *ERROR: Integers are not the right size for libbzip2. Aborting!\n"); return 1; break; case BZ_DATA_ERROR: fprintf(stderr, "pbzip2: *ERROR: Data integrity (CRC) error in data! Skipping...\n"); return -1; break; case BZ_MEM_ERROR: fprintf(stderr, "pbzip2: *ERROR: Could NOT allocate enough memory. Aborting!\n"); return 1; break; case BZ_UNEXPECTED_EOF: fprintf(stderr, "pbzip2: *ERROR: File ends unexpectedly! Skipping...\n"); return -1; break; case BZ_DATA_ERROR_MAGIC: if (streamNo == 1) { fprintf(stderr, "pbzip2: *ERROR: Bad magic number (file not created by bzip2)! Skipping...\n"); return -1; } else { fprintf(stderr, "pbzip2: *WARNING: Trailing garbage after EOF ignored!\n"); return 0; } default: fprintf(stderr, "pbzip2: *ERROR: Unexpected error. Aborting!\n"); exit(3); } return 0; } /* ********************************************************* Much of the code in this function is taken from bzip2.c */ int testCompressedData(char *fileName) { FILE *zStream = NULL; int ret = 0; BZFILE* bzf = NULL; unsigned char obuf[5000]; unsigned char unused[BZ_MAX_UNUSED]; unsigned char *unusedTmp; int bzerr, nread, streamNo; int nUnused; int i; nUnused = 0; streamNo = 0; // see if we are using stdin or not if (strcmp(fileName, "-") != 0) { // open the file for reading zStream = fopen(fileName, "rb"); if (zStream == NULL) { ErrorContext::getInstance()->saveError(); handle_error(EF_NOQUIT, -1, "pbzip2: *ERROR: Could not open input file [%s]! Skipping...\n", fileName); return -1; } } else { zStream = stdin; } // check file stream for errors if (ferror(zStream)) { handle_error(EF_NOQUIT, -1, "pbzip2: *ERROR: Problem with stream of file [%s]! Skipping...\n", fileName); if (zStream != stdin) verbose_fclose(zStream, fileName); return -1; } // loop until end of file while(true) { bzf = BZ2_bzReadOpen(&bzerr, zStream, Verbosity, 0, unused, nUnused); if (bzf == NULL || bzerr != BZ_OK) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) verbose_fclose(zStream, fileName); return ret; } streamNo++; while (bzerr == BZ_OK) { nread = BZ2_bzRead(&bzerr, bzf, obuf, sizeof(obuf)); if (bzerr == BZ_DATA_ERROR_MAGIC) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) verbose_fclose(zStream, fileName); return ret; } } if (bzerr != BZ_STREAM_END) { ret = testBZ2ErrorHandling(bzerr, bzf, streamNo); if (zStream != stdin) verbose_fclose(zStream, fileName); return ret; } BZ2_bzReadGetUnused(&bzerr, bzf, (void**)(&unusedTmp), &nUnused); if (bzerr != BZ_OK) { fprintf(stderr, "pbzip2: *ERROR: Unexpected error. Aborting!\n"); exit(3); } for (i = 0; i < nUnused; i++) unused[i] = unusedTmp[i]; BZ2_bzReadClose(&bzerr, bzf); if (bzerr != BZ_OK) { fprintf(stderr, "pbzip2: *ERROR: Unexpected error. Aborting!\n"); exit(3); } // check to see if we are at the end of the file if (nUnused == 0) { int c = fgetc(zStream); if (c == EOF) break; else ungetc(c, zStream); } } // check file stream for errors if (ferror(zStream)) { ErrorContext::getInstance()->saveError(); handle_error(EF_NOQUIT, -1, "pbzip2: *ERROR: Problem with stream of file [%s]! Skipping...\n", fileName); if (zStream != stdin) verbose_fclose(zStream, fileName); return -1; } // close file ret = verbose_fclose(zStream, fileName); if (ret == EOF) { fprintf(stderr, "pbzip2: *ERROR: Problem closing file [%s]! Skipping...\n", fileName); return -1; } return 0; } /* ********************************************************* */ int getFileMetaData(const char *fileName) { // get the file meta data and store it in the global structure return stat(fileName, &fileMetaData); } /* ********************************************************* */ int writeFileMetaData(const char *fileName) { int ret = 0; #ifndef WIN32 struct utimbuf uTimBuf; #else _utimbuf uTimBuf; #endif // store file times in structure uTimBuf.actime = fileMetaData.st_atime; uTimBuf.modtime = fileMetaData.st_mtime; // update file with stored file permissions ret = chmod(fileName, fileMetaData.st_mode); if (ret != 0) { ErrorContext::getInstance()->saveError(); return ret; } // update file with stored file access and modification times ret = utime(fileName, &uTimBuf); if (ret != 0) { ErrorContext::getInstance()->saveError(); return ret; } // update file with stored file ownership (if access allows) #ifndef WIN32 ret = chown(fileName, fileMetaData.st_uid, fileMetaData.st_gid); // following may happen on some Linux filesystems (i.e. NTFS) // extra error messages do no harm if (ret != 0) { ErrorContext::getInstance()->saveError(); if (geteuid() == 0) return ret; } #endif return 0; } /* ********************************************************* */ int detectCPUs() { int ncpu; // Set default to 1 in case there is no auto-detect ncpu = 1; // Autodetect the number of CPUs on a box, if available #if defined(__APPLE__) size_t len = sizeof(ncpu); int mib[2]; mib[0] = CTL_HW; mib[1] = HW_NCPU; if (sysctl(mib, 2, &ncpu, &len, 0, 0) < 0 || len != sizeof(ncpu)) ncpu = 1; #elif defined(_SC_NPROCESSORS_ONLN) ncpu = sysconf(_SC_NPROCESSORS_ONLN); #elif defined(WIN32) SYSTEM_INFO si; GetSystemInfo(&si); ncpu = si.dwNumberOfProcessors; #endif // Ensure we have at least one processor to use if (ncpu < 1) ncpu = 1; return ncpu; } /* ********************************************************* */ void banner() { fprintf(stderr, "Parallel BZIP2 v1.1.9 - by: Jeff Gilchrist [http://compression.ca]\n"); fprintf(stderr, "[Apr. 13, 2014] (uses libbzip2 by Julian Seward)\n"); fprintf(stderr, "Major contributions: Yavor Nikolov \n"); return; } /* ********************************************************* */ void usage(char* progname, const char *reason) { banner(); if (strncmp(reason, "HELP", 4) == 0) fprintf(stderr, "\n"); else fprintf(stderr, "\nInvalid command line: %s. Aborting...\n\n", reason); #ifndef PBZIP_NO_LOADAVG fprintf(stderr, "Usage: %s [-1 .. -9] [-b#cdfhklm#p#qrS#tVz] \n", progname); #else fprintf(stderr, "Usage: %s [-1 .. -9] [-b#cdfhkm#p#qrS#tVz] \n", progname); #endif // PBZIP_NO_LOADAVG fprintf(stderr, " -1 .. -9 set BWT block size to 100k .. 900k (default 900k)\n"); fprintf(stderr, " -b# Block size in 100k steps (default 9 = 900k)\n"); fprintf(stderr, " -c,--stdout Output to standard out (stdout)\n"); fprintf(stderr, " -d,--decompress Decompress file\n"); fprintf(stderr, " -f,--force Overwrite existing output file\n"); fprintf(stderr, " -h,--help Print this help message\n"); fprintf(stderr, " -k,--keep Keep input file, don't delete\n"); #ifndef PBZIP_NO_LOADAVG fprintf(stderr, " -l,--loadavg Load average determines max number processors to use\n"); #endif // PBZIP_NO_LOADAVG fprintf(stderr, " -m# Maximum memory usage in 1MB steps (default 100 = 100MB)\n"); fprintf(stderr, " -p# Number of processors to use (default"); #if defined(_SC_NPROCESSORS_ONLN) || defined(__APPLE__) fprintf(stderr, ": autodetect [%d])\n", detectCPUs()); #else fprintf(stderr, " 2)\n"); #endif // _SC_NPROCESSORS_ONLN || __APPLE__ fprintf(stderr, " -q,--quiet Quiet mode (default)\n"); fprintf(stderr, " -r,--read Read entire input file into RAM and split between processors\n"); #ifdef USE_STACKSIZE_CUSTOMIZATION fprintf(stderr, " -S# Child thread stack size in 1KB steps (default stack size if unspecified)\n"); #endif // USE_STACKSIZE_CUSTOMIZATION fprintf(stderr, " -t,--test Test compressed file integrity\n"); fprintf(stderr, " -v,--verbose Verbose mode\n"); fprintf(stderr, " -V,--version Display version info for pbzip2 then exit\n"); fprintf(stderr, " -z,--compress Compress file (default)\n"); fprintf(stderr, " --ignore-trailing-garbage=# Ignore trailing garbage flag (1 - ignored; 0 - forbidden)\n"); fprintf(stderr, "\n"); fprintf(stderr, "If no file names are given, pbzip2 compresses or decompresses from standard input to standard output.\n"); fprintf(stderr, "\n"); fprintf(stderr, "Example: pbzip2 -b15vk myfile.tar\n"); fprintf(stderr, "Example: pbzip2 -p4 -r -5 myfile.tar second*.txt\n"); fprintf(stderr, "Example: tar cf myfile.tar.bz2 --use-compress-prog=pbzip2 dir_to_compress/\n"); fprintf(stderr, "Example: pbzip2 -d -m500 myfile.tar.bz2\n"); fprintf(stderr, "Example: pbzip2 -dc myfile.tar.bz2 | tar x\n"); fprintf(stderr, "Example: pbzip2 -c < myfile.txt > myfile.txt.bz2 \n"); fprintf(stderr, "\n"); exit(-1); } /* ********************************************************* */ int main(int argc, char* argv[]) { queue *fifo; pthread_t output; char **FileList = NULL; char *InFilename = NULL; char *progName = NULL; char *progNamePos = NULL; char bz2Header[] = {"BZh91AY&SY"}; // using 900k block size std::string outFilename; // [2048]; char cmdLineTemp[2048]; unsigned char tmpBuff[50]; char stdinFile[2] = {"-"}; struct timeval tvStartTime; struct timeval tvStopTime; #ifndef WIN32 struct timezone tz; double loadAverage = 0.0; double loadAvgArray[3]; int useLoadAverage = 0; int numCPUtotal = 0; int numCPUidle = 0; #else SYSTEMTIME systemtime; LARGE_INTEGER filetime; LARGE_INTEGER fileSize_temp; HANDLE hInfile_temp; #endif double timeCalc = 0.0; double timeStart = 0.0; double timeStop = 0.0; int cmdLineTempCount = 0; int readEntireFile = 0; int zeroByteFile = 0; int hInfile = -1; int hOutfile = -1; int numBlocks = 0; int blockSize = 9*100000; int maxMemory = 100000000; int maxMemorySwitch = 0; int decompress = 0; int compress = 0; int testFile = 0; int errLevel = 0; int noThreads = 0; int keep = 0; int force = 0; int ret = 0; int fileLoop; size_t i, j, k; bool switchedMtToSt = false; // switched from multi- to single-thread // Initialize error context if (ErrorContext::getInstance() == NULL) { return 1; } // get current time for benchmark reference #ifndef WIN32 gettimeofday(&tvStartTime, &tz); #else GetSystemTime(&systemtime); SystemTimeToFileTime(&systemtime, (FILETIME *)&filetime); tvStartTime.tv_sec = filetime.QuadPart / 10000000; tvStartTime.tv_usec = (filetime.QuadPart - (LONGLONG)tvStartTime.tv_sec * 10000000) / 10; #endif // check to see if we are likely being called from TAR if (argc < 2) { OutputStdOut = 1; keep = 1; } // get program name to determine if decompress mode should be used progName = argv[0]; for (progNamePos = argv[0]; progNamePos[0] != '\0'; progNamePos++) { if (progNamePos[0] == PATH_SEP) progName = progNamePos + 1; } if ((strstr(progName, "unzip") != 0) || (strstr(progName, "UNZIP") != 0)) { decompress = 1; } if ((strstr(progName, "zcat") != 0) || (strstr(progName, "ZCAT") != 0)) { decompress = OutputStdOut = keep = 1; } #ifdef IGNORE_TRAILING_GARBAGE // default behavior is hard-coded (still dynamically changeable) IgnoreTrailingGarbageFlag = IGNORE_TRAILING_GARBAGE; #else // default depends on program name if ((strcmp(progName, "bzip2") == 0) || (strcmp(progName, "BZIP2") == 0) || (strcmp(progName, "bunzip2") == 0) || (strcmp(progName, "BUNZIP2") == 0) || (strcmp(progName, "bzcat") == 0) || (strcmp(progName, "BZCAT") == 0)) { // Favour traditional non-parallel bzip2 behavior IgnoreTrailingGarbageFlag = 1; } #endif // IGNORE_TRAILING_GARBAGE FileListCount = 0; FileList = new(std::nothrow) char *[argc]; if (FileList == NULL) { fprintf(stderr, "pbzip2: *ERROR: Not enough memory! Aborting...\n"); return 1; } // set default max memory usage to 100MB maxMemory = 100000000; NumBufferedBlocksMax = 0; numCPU = detectCPUs(); #ifndef WIN32 numCPUtotal = numCPU; #endif // parse command line switches for (i=1; (int)i < argc; i++) { if (argv[i][0] == '-') { if (argv[i][1] == '\0') { // support "-" as a filename FileList[FileListCount] = argv[i]; FileListCount++; continue; } else if (argv[i][1] == '-') { // get command line options with "--" if (strcmp(argv[i], "--best") == 0) { BWTblockSize = 9; } else if (strcmp(argv[i], "--decompress") == 0) { decompress = 1; } else if (strcmp(argv[i], "--compress") == 0) { compress = 1; } else if (strcmp(argv[i], "--fast") == 0) { BWTblockSize = 1; } else if (strcmp(argv[i], "--force") == 0) { force = 1; ForceOverwrite = 1; } else if (strcmp(argv[i], "--help") == 0) { usage(argv[0], "HELP"); } else if (strcmp(argv[i], "--keep") == 0) { keep = 1; } else if (strcmp(argv[i], "--license") == 0) { usage(argv[0], "HELP"); } #ifndef PBZIP_NO_LOADAVG else if (strcmp(argv[i], "--loadavg") == 0) { useLoadAverage = 1; } #endif else if (strcmp(argv[i], "--quiet") == 0) { QuietMode = 1; } else if (strcmp(argv[i], "--read") == 0) { readEntireFile = 1; } else if (strcmp(argv[i], "--stdout") == 0) { OutputStdOut = 1; keep = 1; } else if (strcmp(argv[i], "--test") == 0) { testFile = 1; } else if (strcmp(argv[i], "--verbose") == 0) { QuietMode = 0; } else if (strcmp(argv[i], "--version") == 0) { banner(); exit(0); } else if (strcmp(argv[i], "--ignore-trailing-garbage") == 0 ) { IgnoreTrailingGarbageFlag = 1; } else if (strcmp(argv[i], "--ignore-trailing-garbage=1") == 0 ) { IgnoreTrailingGarbageFlag = 1; } else if (strcmp(argv[i], "--ignore-trailing-garbage=0") == 0 ) { IgnoreTrailingGarbageFlag = 0; } continue; } #ifdef PBZIP_DEBUG fprintf(stderr, "argv[%u]: %s Len: %d\n", (unsigned)i, argv[i], (int)strlen(argv[i])); #endif // get command line options with single "-" // check for multiple switches grouped together for (j=1; argv[i][j] != '\0'; j++) { switch (argv[i][j]) { case 'p': k = j+1; cmdLineTempCount = 0; strcpy(cmdLineTemp, "2"); while (argv[i][k] != '\0' && k < sizeof(cmdLineTemp)) { // no more numbers, finish if ((argv[i][k] < '0') || (argv[i][k] > '9')) break; k++; cmdLineTempCount++; } if (cmdLineTempCount == 0) usage(argv[0], "Cannot parse -p argument"); strncpy(cmdLineTemp, argv[i]+j+1, cmdLineTempCount); cmdLineTemp[cmdLineTempCount] = '\0'; numCPU = atoi(cmdLineTemp); if (numCPU > 4096) { fprintf(stderr,"pbzip2: *ERROR: Maximal number of supported processors is 4096! Aborting...\n"); return 1; } else if (numCPU < 1) { fprintf(stderr,"pbzip2: *ERROR: Minimum number of supported processors is 1! Aborting...\n"); return 1; } j += cmdLineTempCount; #ifdef PBZIP_DEBUG fprintf(stderr, "-p%d\n", numCPU); #endif break; case 'b': k = j+1; cmdLineTempCount = 0; strcpy(cmdLineTemp, "9"); blockSize = 900000; while (argv[i][k] != '\0' && k < sizeof(cmdLineTemp)) { // no more numbers, finish if ((argv[i][k] < '0') || (argv[i][k] > '9')) break; k++; cmdLineTempCount++; } if (cmdLineTempCount == 0) usage(argv[0], "Cannot parse file block size"); strncpy(cmdLineTemp, argv[i]+j+1, cmdLineTempCount); cmdLineTemp[cmdLineTempCount] = '\0'; blockSize = atoi(cmdLineTemp)*100000; if ((blockSize < 100000) || (blockSize > 1000000000)) { fprintf(stderr,"pbzip2: *ERROR: File block size Min: 100k and Max: 1000000k! Aborting...\n"); return 1; } j += cmdLineTempCount; #ifdef PBZIP_DEBUG fprintf(stderr, "-b%d\n", blockSize); #endif break; case 'm': k = j+1; cmdLineTempCount = 0; strcpy(cmdLineTemp, "1"); maxMemory = 1000000; while (argv[i][k] != '\0' && k < sizeof(cmdLineTemp)) { // no more numbers, finish if ((argv[i][k] < '0') || (argv[i][k] > '9')) break; k++; cmdLineTempCount++; } if (cmdLineTempCount == 0) usage(argv[0], "Cannot parse -m argument"); strncpy(cmdLineTemp, argv[i]+j+1, cmdLineTempCount); cmdLineTemp[cmdLineTempCount] = '\0'; maxMemory = atoi(cmdLineTemp)*1000000; if ((maxMemory < 1000000) || (maxMemory > 2000000000)) { fprintf(stderr,"pbzip2: *ERROR: Memory usage size Min: 1MB and Max: 2000MB! Aborting...\n"); return 1; } maxMemorySwitch = 1; j += cmdLineTempCount; #ifdef PBZIP_DEBUG fprintf(stderr, "-m%d\n", maxMemory); #endif break; #ifdef USE_STACKSIZE_CUSTOMIZATION case 'S': k = j+1; cmdLineTempCount = 0; strcpy(cmdLineTemp, "0"); ChildThreadStackSize = -1; while (argv[i][k] != '\0' && k < sizeof(cmdLineTemp)) { // no more numbers, finish if ((argv[i][k] < '0') || (argv[i][k] > '9')) break; k++; cmdLineTempCount++; } if (cmdLineTempCount == 0) usage(argv[0], "Cannot parse -S argument"); strncpy(cmdLineTemp, argv[i]+j+1, cmdLineTempCount); cmdLineTemp[cmdLineTempCount] = '\0'; ChildThreadStackSize = atoi(cmdLineTemp)*1024; if (ChildThreadStackSize < 0) { fprintf(stderr,"pbzip2: *ERROR: Parsing -S: invalid stack size specified [%d]! Ignoring...\n", ChildThreadStackSize); } else if (ChildThreadStackSize < PTHREAD_STACK_MIN) { fprintf(stderr,"pbzip2: *WARNING: Stack size %d bytes less than minumum - adjusting to %d bytes.\n", ChildThreadStackSize, PTHREAD_STACK_MIN); ChildThreadStackSize = PTHREAD_STACK_MIN; } j += cmdLineTempCount; #ifdef PBZIP_DEBUG fprintf(stderr, "-S%d\n", ChildThreadStackSize); #endif break; #endif // USE_STACKSIZE_CUSTOMIZATION case 'd': decompress = 1; break; case 'c': OutputStdOut = 1; keep = 1; break; case 'f': force = 1; ForceOverwrite = 1; break; case 'h': usage(argv[0], "HELP"); break; case 'k': keep = 1; break; #ifndef PBZIP_NO_LOADAVG case 'l': useLoadAverage = 1; break; #endif case 'L': banner(); exit(0); break; case 'q': QuietMode = 1; break; case 'r': readEntireFile = 1; break; case 't': testFile = 1; break; case 'v': QuietMode = 0; break; case 'V': banner(); exit(0); break; case 'z': compress = 1; break; case '1': BWTblockSize = 1; break; case '2': BWTblockSize = 2; break; case '3': BWTblockSize = 3; break; case '4': BWTblockSize = 4; break; case '5': BWTblockSize = 5; break; case '6': BWTblockSize = 6; break; case '7': BWTblockSize = 7; break; case '8': BWTblockSize = 8; break; case '9': BWTblockSize = 9; break; } } } else { // add filename to list for processing FileListCount FileList[FileListCount] = argv[i]; FileListCount++; } } /* for */ Bz2HeaderZero[3] = '0' + BWTblockSize; bz2Header[3] = Bz2HeaderZero[3]; // check to make sure we are not trying to compress and decompress at same time if ((compress == 1) && (decompress == 1)) { fprintf(stderr,"pbzip2: *ERROR: Can't compress and uncompress data at same time. Aborting!\n"); fprintf(stderr,"pbzip2: For help type: %s -h\n", argv[0]); return 1; } if (FileListCount == 0) { if (testFile == 1) { #ifndef WIN32 if (isatty(fileno(stdin))) #else if (_isatty(_fileno(stdin))) #endif { fprintf(stderr,"pbzip2: *ERROR: Won't read compressed data from terminal. Aborting!\n"); fprintf(stderr,"pbzip2: For help type: %s -h\n", argv[0]); return 1; } // expecting data from stdin FileList[FileListCount] = stdinFile; FileListCount++; } else if (decompress == 1) { #ifndef WIN32 if (isatty(fileno(stdin))) #else if (_isatty(_fileno(stdin))) #endif { fprintf(stderr,"pbzip2: *ERROR: Won't read compressed data from terminal. Aborting!\n"); fprintf(stderr,"pbzip2: For help type: %s -h\n", argv[0]); return 1; } // expecting data from stdin via TAR OutputStdOut = 1; keep = 1; FileList[FileListCount] = stdinFile; FileListCount++; } else { if (OutputStdOut == 0) { // probably trying to input data from stdin if (QuietMode != 1) fprintf(stderr,"pbzip2: Assuming input data coming from stdin...\n\n"); OutputStdOut = 1; keep = 1; } #ifndef WIN32 if (isatty(fileno(stdout))) #else if (_isatty(_fileno(stdout))) #endif { fprintf(stderr,"pbzip2: *ERROR: Won't write compressed data to terminal. Aborting!\n"); fprintf(stderr,"pbzip2: For help type: %s -h\n", argv[0]); return 1; } // expecting data from stdin FileList[FileListCount] = stdinFile; FileListCount++; } } if (QuietMode != 1) { // display program banner banner(); // do sanity check to make sure integers are the size we expect #ifdef PBZIP_DEBUG fprintf(stderr, "off_t size: %u uint size: %u\n", (unsigned)sizeof(OFF_T), (unsigned)sizeof(unsigned int)); #endif if (sizeof(OFF_T) <= 4) { fprintf(stderr, "\npbzip2: *WARNING: off_t variable size only %u bits!\n", (unsigned)(sizeof(OFF_T)*CHAR_BIT)); if (decompress == 1) fprintf(stderr, " You will only able to uncompress files smaller than 2GB in size.\n\n"); else fprintf(stderr, " You will only able to compress files smaller than 2GB in size.\n\n"); } } // Calculate number of processors to use based on load average if requested #ifndef PBZIP_NO_LOADAVG if (useLoadAverage == 1) { // get current load average ret = getloadavg(loadAvgArray, 3); if (ret != 3) { loadAverage = 0.0; useLoadAverage = 0; if (QuietMode != 1) fprintf(stderr, "pbzip2: *WARNING: Could not get load average! Using requested processors...\n"); } else { #ifdef PBZIP_DEBUG fprintf(stderr, "Load Avg1: %f Avg5: %f Avg15: %f\n", loadAvgArray[0], loadAvgArray[1], loadAvgArray[2]); #endif // use 1 min load average to adjust number of processors used loadAverage = loadAvgArray[0]; // use [1] for 5 min average and [2] for 15 min average // total number processors minus load average rounded up numCPUidle = numCPUtotal - (int)(loadAverage + 0.5); // if user asked for a specific # processors and they are idle, use all requested // otherwise give them whatever idle processors are available if (numCPUidle < numCPU) numCPU = numCPUidle; if (numCPU < 1) numCPU = 1; } } #endif // Initialize child threads attributes initChildThreadAttributes(); // setup signal handling (should be before creating any child thread) sigInFilename = NULL; sigOutFilename = NULL; ret = setupSignalHandling(); if (ret != 0) { fprintf(stderr, "pbzip2: *ERROR: Can't setup signal handling [%d]. Aborting!\n", ret); return 1; } // Create and start terminator thread. ret = setupTerminator(); if (ret != 0) { fprintf(stderr, "pbzip2: *ERROR: Can't setup terminator thread [%d]. Aborting!\n", ret); return 1; } if (numCPU < 1) numCPU = 1; // display global settings if (QuietMode != 1) { if (testFile != 1) { fprintf(stderr, "\n # CPUs: %d\n", numCPU); #ifndef PBZIP_NO_LOADAVG if (useLoadAverage == 1) fprintf(stderr, " Load Average: %.2f\n", loadAverage); #endif if (decompress != 1) { fprintf(stderr, " BWT Block Size: %d00 KB\n", BWTblockSize); if (blockSize < 100000) fprintf(stderr, "File Block Size: %d bytes\n", blockSize); else fprintf(stderr, "File Block Size: %d KB\n", blockSize/1000); } fprintf(stderr, " Maximum Memory: %d MB\n", maxMemory/1000000); #ifdef USE_STACKSIZE_CUSTOMIZATION if (ChildThreadStackSize > 0) fprintf(stderr, " Stack Size: %d KB\n", ChildThreadStackSize/1024); #endif if (decompress == 1) { fprintf(stderr, " Ignore Trailing Garbage: %s\n", (IgnoreTrailingGarbageFlag == 1) ? "on" : "off" ); } } fprintf(stderr, "-------------------------------------------\n"); } int mutexesInitRet = mutexesInit(); if ( mutexesInitRet != 0 ) { return mutexesInitRet; } // create queue fifo = FifoQueue = queueInit(numCPU); if (fifo == NULL) { fprintf (stderr, "pbzip2: *ERROR: Queue Init failed. Aborting...\n"); return 1; } // process all files for (fileLoop=0; fileLoop < FileListCount; fileLoop++) { producerDone = 0; InFileSize = 0; NumBlocks = 0; switchedMtToSt = false; int errLevelCurrentFile = 0; ErrorContext::getInstance()->reset(); // set input filename InFilename = FileList[fileLoop]; // test file for errors if requested if (testFile != 0) { if (QuietMode != 1) { fprintf(stderr, " File #: %d of %d\n", fileLoop+1, FileListCount); if (strcmp(InFilename, "-") != 0) fprintf(stderr, " Testing: %s\n", InFilename); else fprintf(stderr, " Testing: \n"); } ret = testCompressedData(InFilename); if (ret > 0) return ret; else if (ret == 0) { if (QuietMode != 1) fprintf(stderr, " Test: OK\n"); } else errLevel = 2; if (QuietMode != 1) fprintf(stderr, "-------------------------------------------\n"); continue; } // set ouput filename outFilename = std::string(FileList[fileLoop]); if ((decompress == 1) && (strcmp(InFilename, "-") != 0)) { // check if input file is a valid .bz2 compressed file hInfile = open(InFilename, O_RDONLY | O_BINARY); // check to see if file exists before processing if (hInfile == -1) { ErrorContext::printErrnoMsg(stderr, errno); fprintf(stderr, "pbzip2: *ERROR: File [%s] NOT found! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } memset(tmpBuff, 0, sizeof(tmpBuff)); size_t size = do_read(hInfile, tmpBuff, strlen(bz2Header)+1); do_close(hInfile); if ((size == (size_t)(-1)) || (size < strlen(bz2Header)+1)) { ErrorContext::getInstance()->printErrorMessages(stderr); fprintf(stderr, "pbzip2: *ERROR: File [%s] is NOT a valid bzip2! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } else { // make sure start of file has valid bzip2 header if (memstr(tmpBuff, 4, bz2Header, 3) == NULL) { fprintf(stderr, "pbzip2: *ERROR: File [%s] is NOT a valid bzip2! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } // skip 4th char which differs depending on BWT block size used if (memstr(tmpBuff+4, size-4, bz2Header+4, strlen(bz2Header)-4) == NULL) { // check to see if this is a special 0 byte file if (memstr(tmpBuff+4, size-4, Bz2HeaderZero+4, strlen(bz2Header)-4) == NULL) { fprintf(stderr, "pbzip2: *ERROR: File [%s] is NOT a valid bzip2! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } #ifdef PBZIP_DEBUG fprintf(stderr, "** ZERO byte compressed file detected\n"); #endif } // set block size for decompression if ((tmpBuff[3] >= '1') && (tmpBuff[3] <= '9')) BWTblockSizeChar = tmpBuff[3]; else { fprintf(stderr, "pbzip2: *ERROR: File [%s] is NOT a valid bzip2! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } } // check if filename ends with .bz2 std::string bz2Tail(".bz2"); std::string tbz2Tail(".tbz2"); if ( ends_with_icase(outFilename, bz2Tail) ) { // remove .bz2 extension outFilename.resize( outFilename.size() - bz2Tail.size() ); } else if ( ends_with_icase(outFilename, tbz2Tail) ) { outFilename.resize( outFilename.size() - tbz2Tail.size() ); outFilename += ".tar"; } else { // add .out extension so we don't overwrite original file outFilename += ".out"; } } // decompress == 1 else { // check input file to make sure its not already a .bz2 file std::string bz2Tail(".bz2"); if ( ends_with_icase(std::string(InFilename), bz2Tail) ) { fprintf(stderr, "pbzip2: *ERROR: Input file [%s] already has a .bz2 extension! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } outFilename += bz2Tail; } // setup signal handling filenames safe_mutex_lock(&ErrorHandlerMutex); sigInFilename = InFilename; sigOutFilename = outFilename.c_str(); safe_mutex_unlock(&ErrorHandlerMutex); if (strcmp(InFilename, "-") != 0) { struct stat statbuf; // read file for compression hInfile = open(InFilename, O_RDONLY | O_BINARY); // check to see if file exists before processing if (hInfile == -1) { fprintf(stderr, "pbzip2: *ERROR: File [%s] NOT found! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } // get some information about the file fstat(hInfile, &statbuf); // check to make input is not a directory if (S_ISDIR(statbuf.st_mode)) { fprintf(stderr, "pbzip2: *ERROR: File [%s] is a directory! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } // check to make sure input is a regular file if (!S_ISREG(statbuf.st_mode)) { fprintf(stderr, "pbzip2: *ERROR: File [%s] is not a regular file! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } // get size of file #ifndef WIN32 InFileSize = statbuf.st_size; #else fileSize_temp.LowPart = GetFileSize((HANDLE)_get_osfhandle(hInfile), (unsigned long *)&fileSize_temp.HighPart); InFileSize = fileSize_temp.QuadPart; #endif // don't process a 0 byte file if (InFileSize == 0) { if (decompress == 1) { fprintf(stderr, "pbzip2: *ERROR: File is of size 0 [%s]! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } // make sure we handle zero byte files specially zeroByteFile = 1; } else { zeroByteFile = 0; } // get file meta data to write to output file if (getFileMetaData(InFilename) != 0) { fprintf(stderr, "pbzip2: *ERROR: Could not get file meta data from [%s]! Skipping...\n", InFilename); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } } else { hInfile = STDIN_FILENO; // stdin InFileSize = -1; // fake it } // check to see if output file exists if ((OutputStdOut == 0) && check_file_exists(outFilename.c_str())) { if (force != 1) { fprintf(stderr, "pbzip2: *ERROR: Output file [%s] already exists! Use -f to overwrite...\n", outFilename.c_str()); fprintf(stderr, "-------------------------------------------\n"); errLevel = 1; continue; } else { remove(outFilename.c_str()); } } if (readEntireFile == 1) { if (hInfile == STDIN_FILENO) { if (QuietMode != 1) fprintf(stderr, " *Warning: Ignoring -r switch since input is stdin.\n"); } else { // determine block size to try and spread data equally over # CPUs blockSize = InFileSize / numCPU; } } // display per file settings if (QuietMode != 1) { fprintf(stderr, " File #: %d of %d\n", fileLoop+1, FileListCount); fprintf(stderr, " Input Name: %s\n", hInfile != STDIN_FILENO ? InFilename : ""); if (OutputStdOut == 0) fprintf(stderr, " Output Name: %s\n\n", outFilename.c_str()); else fprintf(stderr, " Output Name: \n\n"); if (decompress == 1) fprintf(stderr, " BWT Block Size: %c00k\n", BWTblockSizeChar); if (strcmp(InFilename, "-") != 0) fprintf(stderr, " Input Size: %"PRIuMAX" bytes\n", (uintmax_t)InFileSize); } if (decompress == 1) { numBlocks = 0; // Do not use threads if we only have 1 CPU or small files if ((numCPU == 1) || (InFileSize < 1000000)) noThreads = 1; else noThreads = 0; // Enable threads method for uncompressing from stdin if ((numCPU > 1) && (strcmp(InFilename, "-") == 0)) noThreads = 0; } // if (decompress == 1) else { if (InFileSize > 0) { // calculate the # of blocks of data numBlocks = (InFileSize + blockSize - 1) / blockSize; // Do not use threads for small files where we only have 1 block to process // or if we only have 1 CPU if ((numBlocks == 1) || (numCPU == 1)) noThreads = 1; else noThreads = 0; } else { // Simulate a "big" number of buffers. Will need to resize it later numBlocks = 10000; } // write special compressed data for special 0 byte input file case if (zeroByteFile == 1) { hOutfile = STDOUT_FILENO; // write to file instead of stdout if (OutputStdOut == 0) { hOutfile = safe_open_output(outFilename.c_str()); // check to see if file creation was successful if (hOutfile == -1) { handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Could not create output file [%s]!\n", outFilename.c_str()); errLevelCurrentFile = errLevel = 1; break; } } // write data to the output file ret = do_write(hOutfile, Bz2HeaderZero, sizeof(Bz2HeaderZero)); int close_ret = 0; if (OutputStdOut == 0) { close_ret = do_close(hOutfile); // write store file meta data to output file if (writeFileMetaData(outFilename.c_str()) != 0) { handle_error(EF_NOQUIT, -1, "pbzip2: *ERROR: Could not write file meta data to [%s]!\n", outFilename.c_str()); } } if ( (ret != sizeof(Bz2HeaderZero)) || (close_ret == -1) ) { handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Could not write to file [%s]! Aborting...\n", outFilename.c_str()); fprintf(stderr, "-------------------------------------------\n"); errLevelCurrentFile = errLevel = 1; break; } if (QuietMode != 1) { fprintf(stderr, " Output Size: %u bytes\n", (unsigned)sizeof(Bz2HeaderZero)); fprintf(stderr, "-------------------------------------------\n"); } // remove input file unless requested not to by user or error occurred if ( (keep != 1) && (errLevelCurrentFile == 0) ) { struct stat statbuf; // only remove input file if output file exists bool removeFlag = (OutputStdOut != 0) || (stat(outFilename.c_str(), &statbuf) == 0); if (removeFlag) { if (do_remove(InFilename) == -1) { handle_error(EF_NOQUIT, 1, "Can't remove input file [%s]!", InFilename); } } } continue; } // if (zeroByteFile == 1) } // else (decompress == 1) #ifdef PBZIP_DEBUG fprintf(stderr, "# Blocks: %d\n", numBlocks); #endif // set global variable NumBlocksEstimated = numBlocks; // Calculate maximum number of buffered blocks to use NumBufferedBlocksMax = maxMemory / blockSize; // Subtract blocks for number of extra buffers in producer and fileWriter (~ numCPU for each) if ((int)NumBufferedBlocksMax - (numCPU * 2) < 1) NumBufferedBlocksMax = 1; else NumBufferedBlocksMax = NumBufferedBlocksMax - (numCPU * 2); #ifdef PBZIP_DEBUG fprintf(stderr, "pbzip2: maxMemory: %d blockSize: %d\n", maxMemory, blockSize); fprintf(stderr, "pbzip2: NumBufferedBlocksMax: %"PRIuMAX"\n", (uintmax_t)NumBufferedBlocksMax); #endif // check to see if our max buffered blocks is less than numCPU, if yes increase maxMemory // to support numCPU requested unless -m switch given by user if (NumBufferedBlocksMax < (size_t)numCPU) { if (maxMemorySwitch == 0) { NumBufferedBlocksMax = numCPU; if (QuietMode != 1) fprintf(stderr, "*Warning* Max memory limit increased to %"PRIuMAX" MB to support %d CPUs\n", (uintmax_t)((NumBufferedBlocksMax + (numCPU * 2)) * blockSize)/1000000, numCPU); } else { if (QuietMode != 1) fprintf(stderr, "*Warning* CPU usage and performance may be suboptimal due to max memory limit.\n"); } } LastGoodBlock = -1; MinErrorBlock = -1; // create output buffer outputBufferInit(NumBufferedBlocksMax); if (decompress == 1) { // use multi-threaded code if (noThreads == 0) { // do decompression if (QuietMode != 1) fprintf(stderr, "Decompressing data...\n"); for (i=0; (int)i < numCPU; i++) { ret = pthread_create(&fifo->consumers[i], &ChildThreadAttributes, consumer_decompress, fifo); if (ret != 0) { ErrorContext::getInstance()->saveError(); handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Not enough resources to create consumer thread #%u (code = %d) Aborting...\n", i, ret); ret = pthread_join(TerminatorThread, NULL); return 1; } } ret = pthread_create(&output, &ChildThreadAttributes, fileWriter, (void*)outFilename.c_str()); if (ret != 0) { handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Not enough resources to create fileWriter thread (code = %d) Aborting...\n", ret); ret = pthread_join(TerminatorThread, NULL); return 1; } // start reading in data for decompression ret = producer_decompress(hInfile, InFileSize, fifo); if (ret == -99) { // only 1 block detected, use single threaded code to decompress noThreads = 1; switchedMtToSt = true; // wait for fileWriter thread to exit if (pthread_join(output, NULL) != 0) { ErrorContext::getInstance()->saveError(); handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Error joining fileWriter thread (code = %d) Aborting...\n", ret); errLevelCurrentFile = errLevel = 1; ret = pthread_join(TerminatorThread, NULL); return 1; } } else if (ret != 0) { errLevelCurrentFile = errLevel = 1; } } // use single threaded code if ((noThreads == 1) && (errLevelCurrentFile == 0)) { if (QuietMode != 1) fprintf(stderr, "Decompressing data (no threads)...\n"); if (hInfile > 0) close(hInfile); ret = directdecompress(InFilename, outFilename.c_str()); if (ret != 0) { errLevelCurrentFile = errLevel = 1; } } } // if (decompress == 1) else { // do compression code // use multi-threaded code if (noThreads == 0) { if (QuietMode != 1) fprintf(stderr, "Compressing data...\n"); for (i=0; (int)i < numCPU; i++) { ret = pthread_create(&fifo->consumers[i], &ChildThreadAttributes, consumer, fifo); if (ret != 0) { ErrorContext::getInstance()->saveError(); handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Not enough resources to create consumer thread #%u (code = %d) Aborting...\n", i, ret); pthread_join(TerminatorThread, NULL); return 1; } } ret = pthread_create(&output, &ChildThreadAttributes, fileWriter, (void*)outFilename.c_str()); if (ret != 0) { handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Not enough resources to create fileWriter thread (code = %d) Aborting...\n", ret); pthread_join(TerminatorThread, NULL); return 1; } // start reading in data for compression ret = producer(hInfile, blockSize, fifo); if (ret != 0) errLevelCurrentFile = errLevel = 1; } else { // do not use threads for compression if (QuietMode != 1) fprintf(stderr, "Compressing data (no threads)...\n"); ret = directcompress(hInfile, InFileSize, blockSize, outFilename.c_str()); if (ret != 0) errLevelCurrentFile = errLevel = 1; } } // else if (noThreads == 0) { // wait for fileWriter thread to exit ret = pthread_join(output, NULL); if (ret != 0) { ErrorContext::printErrnoMsg(stderr, errno); errLevelCurrentFile = errLevel = 1; } } if ((noThreads == 0) || switchedMtToSt ) { // wait for consumer threads to exit for (i = 0; (int)i < numCPU; i++) { ret = pthread_join(fifo->consumers[i], NULL); if (ret != 0) { ErrorContext::printErrnoMsg(stderr, errno); errLevelCurrentFile = errLevel = 1; } } } if (syncGetTerminateFlag() != 0) { errLevelCurrentFile = errLevel = 1; } if (OutputStdOut == 0) { // write store file meta data to output file if (writeFileMetaData(outFilename.c_str()) != 0) { handle_error(EF_NOQUIT, -1, "pbzip2: *ERROR: Could not write file meta data to [%s]!\n", outFilename.c_str()); } } // remove input file unless requested not to by user or error occurred if ( (keep != 1) && (errLevelCurrentFile == 0) ) { struct stat statbuf; // only remove input file if output file exists bool removeFlag = (OutputStdOut != 0) || (stat(outFilename.c_str(), &statbuf) == 0); if (removeFlag) { if (do_remove(InFilename) == -1) { handle_error(EF_NOQUIT, 1, "Can't remove input file [%s]!", InFilename); } } } // reclaim memory OutputBuffer.clear(); fifo->clear(); if ( (errLevelCurrentFile == 0) && (syncGetTerminateFlag() == 0) ) { // finished processing file (mutex since accessed by cleanup procedure) safe_mutex_lock(&ErrorHandlerMutex); sigInFilename = NULL; sigOutFilename = NULL; safe_mutex_unlock(&ErrorHandlerMutex); } if (errLevelCurrentFile == 1) { syncSetTerminateFlag(1); break; } if (QuietMode != 1) fprintf(stderr, "-------------------------------------------\n"); } /* for */ // Explicit close on stdout if we've been writing there, after all input has been processed if (OutputStdOut == 1) { ret = close(STDOUT_FILENO); if (ret == -1) { ErrorContext::getInstance()->saveError(); handle_error(EF_EXIT, 1, "pbzip2: *ERROR: Failed to close STDOUT! Aborting...\n"); exit(1); } } // Terminate signal handler thread sending SIGQUIT signal ret = pthread_kill(SignalHandlerThread, SIG_HANDLER_QUIT_SIGNAL); if (ret != 0) { fprintf(stderr, "Couldn't signal signal QUIT to SignalHandlerThread [%d]. Quitting prematurely!\n", ret); exit(errLevel); } else { ret = pthread_join(SignalHandlerThread, NULL); if (ret != 0) { fprintf(stderr, "Error on join of SignalHandlerThread [%d]\n", ret); } } if (syncGetTerminateFlag() == 0) { syncSetFinishedFlag(1); } ret = pthread_join(TerminatorThread, NULL); if (ret != 0) { fprintf(stderr, "Error on join of TerminatorThread [%d]\n", ret); } // reclaim memory queueDelete(fifo); mutexesDelete(); disposeMemory(FileList); // get current time for end of benchmark #ifndef WIN32 gettimeofday(&tvStopTime, &tz); #else GetSystemTime(&systemtime); SystemTimeToFileTime(&systemtime, (FILETIME *)&filetime); tvStopTime.tv_sec = filetime.QuadPart / 10000000; tvStopTime.tv_usec = (filetime.QuadPart - (LONGLONG)tvStopTime.tv_sec * 10000000) / 10; #endif #ifdef PBZIP_DEBUG fprintf(stderr, "\n Start Time: %ld + %ld\n", tvStartTime.tv_sec, tvStartTime.tv_usec); fprintf(stderr, " Stop Time : %ld + %ld\n", tvStopTime.tv_sec, tvStopTime.tv_usec); #endif // convert time structure to real numbers timeStart = (double)tvStartTime.tv_sec + ((double)tvStartTime.tv_usec / 1000000); timeStop = (double)tvStopTime.tv_sec + ((double)tvStopTime.tv_usec / 1000000); timeCalc = timeStop - timeStart; if (QuietMode != 1) fprintf(stderr, "\n Wall Clock: %f seconds\n", timeCalc); exit(errLevel); } pbzip2-1.1.9/pbzip2.h0000644000000000000000000001624412322570763012453 0ustar 00000000000000/* * File: pbzip2.h * Author: Yavor Nikolov * * Created on March 6, 2010, 10:18 PM * * Change History: * 2010-05-20 - by Yavor Nikolov * - Transformed input queue as queue of queues to avoid deadlock and conten * tion issues. */ #ifndef _PBZIP2_H #define _PBZIP2_H #include #include #define FILE_MODE (S_IRUSR | S_IWUSR ) #ifndef WIN32 #define OFF_T off_t #else #define OFF_T __int64 #endif extern "C" { #ifndef WIN32 #include #include #include #else #include #include #define STDIN_FILENO 0 #define STDOUT_FILENO 1 #define STDERR_FILENO 2 #endif #ifdef __APPLE__ #include #endif #ifdef __sun #include #endif #ifndef __BORLANDC__ #define __STDC_FORMAT_MACROS #include #else #define PRIu64 "Lu" #define strncasecmp(x,y,z) strncmpi(x,y,z) #endif #ifdef __osf__ #define PRIu64 "llu" #endif #include #include #define __STDC_FORMAT_MACROS #include } // uncomment for debug output //#define PBZIP_DEBUG // uncomment to disable load average code (may be required for some platforms) //#define PBZIP_NO_LOADAVG // detect systems that are known not to support load average code #if defined (WIN32) || defined (__CYGWIN32__) || defined (__MINGW32__) || defined (__BORLANDC__) || defined (__hpux) || defined (__osf__) || defined(__UCLIBC__) #define PBZIP_NO_LOADAVG #endif #ifdef WIN32 #define PATH_SEP '\\' #define usleep(x) Sleep(x/1000) #define LOW_DWORD(x) ((*(LARGE_INTEGER *)&x).LowPart) #define HIGH_DWORD(x) ((*(LARGE_INTEGER *)&x).HighPart) #ifndef _TIMEVAL_DEFINED /* also in winsock[2].h */ #define _TIMEVAL_DEFINED struct timeval { long tv_sec; long tv_usec; }; #endif #else #define PATH_SEP '/' #endif #ifndef O_BINARY #define O_BINARY 0 #endif typedef struct outBuff { char *buf; unsigned int bufSize; int blockNumber; int sequenceNumber; unsigned int inSize; // original size before compression/decompressoin bool isLastInSequence; outBuff * next; // next in longer sequence of buffers for this block //outBuff * last; // last in sequence (here as quick hack since global one would be enough) outBuff( char * aBuf = NULL, unsigned int aBufSize = 0, int aBlockNumber = 0, int aSequenceNumber = 0, unsigned int aInSize = 0, bool isLast = true, outBuff * aNext = NULL): buf(aBuf), bufSize(aBufSize), blockNumber(aBlockNumber), sequenceNumber(aSequenceNumber), inSize(aInSize), isLastInSequence(isLast), next(aNext)//, //last(NULL) {} } outBuff; typedef enum ExitFlag { EF_NOQUIT = 0, EF_EXIT = 1, EF_ABORT = 2 } ExitFlag; typedef struct queue { typedef outBuff ElementType; typedef ElementType* ElementTypePtr; ElementTypePtr *qData; long size; long count; // actual element count, including tail-queue ones long head, tail; int full, empty; int topLevelFull, topLevelEmpty; pthread_mutex_t *mut; pthread_cond_t *notFull, *notEmpty; pthread_t *consumers; ElementTypePtr lastElement; // most recently added element queue(): count(0), lastElement(NULL) {} /** * Reset the queue putting it to initial empty state. */ void clear() { empty = 1; full = 0; head = 0; tail = 0; count = 0; lastElement = NULL; topLevelFull = 0; topLevelEmpty = 1; } void add(ElementTypePtr element) { #ifdef PBZIP_DEBUG fprintf (stderr, "queue::add: elem=%llx\n", (unsigned long long)element); if (element != NULL) { fprintf (stderr, " queue::add: seq=%d; blk=%d; islast=%d\n", element->sequenceNumber, element->blockNumber, (int)element->isLastInSequence); } #endif if ( element->sequenceNumber > 1 ) { // multi-part sequence: append to same one lastElement->next = element; } else { // primary element (either first in sequence; or a standalone one) qData[tail] = element; ++tail; if (tail == size) tail = 0; if (tail == head) topLevelFull = 1; topLevelEmpty = 0; } lastElement = element; ++count; if (count == size) full = 1; empty = 0; } /** * Remove the head returning it into element. If the given element is * tail of multi-segment sequence - just moves to next segment. * * @param element - removed element is copied here * @return 1 on success; 0 - on denied request; -1 - on error */ int remove(ElementTypePtr & element) { ElementTypePtr & headElem = qData[head]; #ifdef PBZIP_DEBUG fprintf (stderr, "queue::remove: head=%llx; elem=%llx; count=%ld\n", (unsigned long long)headElem, (unsigned long long)element, count); if (headElem != NULL) { fprintf (stderr, " queue::remove: head: seq=%d; blk=%d; islast=%d\n", headElem->sequenceNumber, headElem->blockNumber, (int)headElem->isLastInSequence); } if (element != NULL) { fprintf (stderr, " queue::remove: element: seq=%d; blk=%d; islast=%d\n", element->sequenceNumber, element->blockNumber, (int)element->isLastInSequence); } #endif if ( (element != NULL) && !element->isLastInSequence ) { if (element->next != NULL) { element = element->next; } else { // 2+ part of long-sequence BZ2 stream. Next // segment is not ready yet. return 0; } } else if (topLevelEmpty) { return 0; } else { element = headElem; ++head; if (head == size) head = 0; if (head == tail) topLevelEmpty = 1; topLevelFull = 0; } --count; if (count == 0) empty = 1; full = 0; return 1; } } queue; /* ********************************************************* Print error message and optionally exit or abort depending on exitFlag: 0 - don't quit; 1 - exit; 2 - abort. On exit - exitCode status is used. */ int handle_error(ExitFlag exitFlag, int exitCode, const char *fmt, ...); /* * Delegate to read but keep writing until count bytes are read or * error is encountered (on success all count bytes would be read) */ ssize_t do_read(int fd, void *buf, size_t count); /* * Delegate to write but keep writing until count bytes are written or * error is encountered (on success all count bytes would be written) */ ssize_t do_write(int fd, const void *buf, size_t count); /** * Dispose the given buffer memory if not NULL and make it NULL. Provided * buffer should be allocated with new[]. */ template inline void disposeMemory(C *& pBuff) { if (pBuff != NULL) { delete [] pBuff; pBuff = NULL; } } /** * Dispose the given buffer memory if not NULL and make it NULL. Provided * buffer should be allocated with new. */ template inline void disposeMemorySingle(C *& pBuff) { if (pBuff != NULL) { delete pBuff; pBuff = NULL; } } /** * Check if a given string ends with a given suffix ignoring case difference. * * @return true if str ends with suffix; false - otherwise */ template inline bool ends_with_icase( const std::basic_string & str, const std::basic_string & suffix ) { int ti = str.size() - suffix.size(); if ( ti < 0 ) { return false; } size_t si = 0; while ( si < suffix.size() ) { if ( ::tolower( str[ti] ) != ::tolower( suffix[si] ) ) { return false; } ++si; ++ti; } return true; } #endif /* _PBZIP2_H */ pbzip2-1.1.9/pbzip2.spec0000644000000000000000000001060212322570763013146 0ustar 00000000000000Name: pbzip2 Version: 1.1.9 Release: 1%{?dist} Summary: Parallel implementation of bzip2 URL: http://www.compression.ca/pbzip2/ License: BSD Group: Applications/File BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) %if %{?suse_version:1}0 BuildRequires: bzip2 %else BuildRequires: bzip2-devel %endif Source0: http://www.compression.ca/pbzip2/%{name}-%{version}.tar.gz Patch0: %{name}-rpm-buildflags.patch %description PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of this version is fully compatible with bzip2 v1.0.2 or newer (ie: anything compressed with pbzip2 can be decompressed with bzip2). %prep %setup -q %patch0 -p1 %build CXXFLAGS="%{optflags}" CFLAGS="%{optflags}" LDFLAGS="%{?__global_ldflags}" make %{?_smp_mflags} %install rm -rf %{buildroot} install -D -m755 %{name} %{buildroot}%{_bindir}/%{name} install -D -m644 %{name}.1 %{buildroot}%{_mandir}/man1/%{name}.1 ln -sf ./%{name} %{buildroot}%{_bindir}/pbunzip2 ln -sf ./%{name} %{buildroot}%{_bindir}/pbzcat %clean rm -rf %{buildroot} %files %defattr(-,root,root) %doc AUTHORS ChangeLog COPYING README %{_bindir}/%{name} %{_bindir}/pbunzip2 %{_bindir}/pbzcat %{_mandir}/man1/* %changelog * Sun Apr 13 2014 Jeff Gilchrist - 1.1.9-1 - Release 1.1.9 - spec file refinement for rpm builds thanks to to Ville Skytta - Makefile refinements for * Sun Jun 10 2012 Jeff Gilchrist - 1.1.8-1 - Release 1.1.8 * Sun Dec 11 2011 Jeff Gilchrist - 1.1.7-1 - Release 1.1.7 * Sun Oct 30 2011 Jeff Gilchrist - 1.1.6-1 - Release 1.1.6 * Sat Jul 16 2011 Jeff Gilchrist - 1.1.5-1 - Release 1.1.5 * Sat Apr 22 2011 Jeff Gilchrist - 1.1.4-1 - Release 1.1.4 * Sun Mar 27 2011 Jeff Gilchrist - 1.1.3-1 - Release 1.1.3 * Sat Feb 19 2011 Jeff Gilchrist - 1.1.2-1 - Release 1.1.2 * Sat Apr 17 2010 Jeff Gilchrist - 1.1.1-1 - Release 1.1.1 * Sat Mar 13 2010 Jeff Gilchrist - 1.1.0-1 - Release 1.1.0 * Fri Jan 8 2009 Jeff Gilchrist - 1.0.5-1 - Release 1.0.5 * Fri Dec 21 2008 Jeff Gilchrist - 1.0.4-1 - Release 1.0.4 * Tue Oct 31 2008 Jeff Gilchrist - 1.0.3-1 - Release 1.0.3 - Added support for SUSE RPM build - Added symlink for pbzcat * Thu Jul 26 2007 Jeff Gilchrist - 1.0.2-2 - Fixed symbolic link for pbunzip2 file * Tue Jul 25 2007 Jeff Gilchrist - 1.0.2-1 - Release 1.0.2 * Tue Mar 20 2007 Jeff Gilchrist - 1.0.1-1 - Release 1.0.1 * Wed Mar 14 2007 Jeff Gilchrist - 1.0-1 - Release 1.0 * Tue Sep 12 2006 Jeff Gilchrist - 0.9.6-4 - Rebuild for Fedora Extras 6 * Tue May 23 2006 Jeff Gilchrist - 0.9.6-3 - Added support for $RPM_OPT_FLAGS thanks to Ville Skytta * Tue Feb 28 2006 Jeff Gilchrist - 0.9.6-2 - Rebuild for Fedora Extras 5 * Sun Feb 5 2006 Jeff Gilchrist - 0.9.6-1 - Release 0.9.6 * Sat Dec 31 2005 Jeff Gilchrist - 0.9.5-1 - Release 0.9.5 * Tue Aug 30 2005 Jeff Gilchrist - 0.9.4-1 - Updated RPM spec with suggestions from Oliver Falk * Fri Jul 29 2005 Bryan Stillwell - 0.9.3-1 - Release 0.9.3 - Removed non-packaging changelog info - Added dist macro to release field - Clean buildroot at the beginning of the install section - Modified buildroot tag to match with Fedora PackagingGuidelines - Shortened Requires and BuildRequires list - Changed description to match with the Debian package * Sat Mar 12 2005 Jeff Gilchrist - 0.9.2-1 - Release 0.9.2 * Sat Jan 29 2005 Jeff Gilchrist - 0.9.1-1 - Release 0.9.1 * Sun Jan 24 2005 Jeff Gilchrist - 0.9-1 - Release 0.9 * Sun Jan 9 2005 Jeff Gilchrist - 0.8.3-1 - Release 0.8.3 * Mon Nov 30 2004 Jeff Gilchrist - 0.8.2-1 - Release 0.8.2 * Sat Nov 27 2004 Jeff Gilchrist - 0.8.1-1 - Release 0.8.1 * Thu Oct 28 2004 Bryan Stillwell - 0.8-1 - Initial packaging