pax_global_header00006660000000000000000000000064122175542300014513gustar00rootroot0000000000000052 comment=48c0d7eb763f539108c570155bfdddf12ef7af7e lbzip2-2.3/000077500000000000000000000000001221755423000125615ustar00rootroot00000000000000lbzip2-2.3/ALGORITHM000066400000000000000000000060141221755423000140330ustar00rootroot00000000000000lbzip2 implements many different algorithms. Some of them were designed specially for use in lbzip2, but most of them were based on publicly available descriptions. The following is the list of publications describing many algorithms behind lbzip2, most of them are easily available online. Title: A Block-sorting Lossless Data Compression Algorithm Author: Burrows, Michael Author: Wheeler, David John URL: ftp://gatekeeper.dec.com/pub/DEC/SRC/research-reports/SRC-124.pdf Title: In-Place Calculation of Minimum-Redundancy Codes Author: Moffat, Alistair Author: Katajainen, Jyrki URL: http://www.diku.dk/~jyrki/Paper/WADS95.pdf Title: A Fast Algorithm for Optimal Length-Limited Huffman Codes Author: Larmore, Lawrence Author: Hirschberg, Daniel URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.157.4015&rep=rep1&type=pdf Title: The Art of Computer Programming, volume 3: Sorting and Searching Author: Knuth, Donald Title: On the implementation of minimum-redundancy prefix codes Author: Moffat, Alistair Author: Turpin, Andrew URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.121.6586&rep=rep1&type=pdf Title: Decoding prefix codes Author: Liddell, Mike Author: Moffat, Alistair URL: http://www.ece.iit.edu/~biitcomm/research/Variable-Length%20Codes/prefix%20codes%20decoding/Decoding%20prefix%20codes.pdf Title: Fast lightweight suffix array construction and checking Author: Burkhardt, Stefan Author: K"arkk"ainen, Juha URL: http://www.cs.ucr.edu/~stelo/cpm/cpm03/Juha.pdf Title: An Efficient Method for in Memory Construction of Suffix Arrays Author: Itoh, Hideo Author: Tanaka, Hozumi URL: http://web.iiit.ac.in/~abhishek_shukla/suffix/In%20Memory%20Suffix%20Array%20Construction.pdf Title: Space-efficient linear time construction of suffix arrays Author: Ko, Pang Author: Aluru, Srinivas URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.9781&rep=rep1&type=pdf Title: Faster suffix sorting Author: Larsson, Jesper Author: Sadakane, Kunihiko URL: http://www.larsson.dogma.net/ssrev-tr.pdf Title: MSufSort Author: Maniscalco, Michael URL: http://www.michael-maniscalco.com/msufsort.htm Title: Short description of improved two-stage suffix sorting algorithm Author: Mori, Yuta URL: http://homepage3.nifty.com/wpage/software/itssort.txt Former versions of lbzip2 also used to benefit from: Title: An Incomplex Algorithm for Fast Suffix Array Construction Author: Schurmann, Klaus-Bernd Author: Stoye, Jens URL: http://techfak.uni-bielefeld.de/~stoye/cpublications/alenex2005final.pdf Title: Engineering a Lightweight Suffix Array Construction Algorithm Author: Manzini, Giovanni Author: Ferragina, Paolo URL: http://people.unipmn.it/manzini/papers/esa02.pdf Title: Engineering a Sort Function Author: Bentley, Jon Louis Author: McIlroy, Douglas Malcolm URL: http://www.comp.nus.edu/~tanhuiyi/cs1102/2007-2008SEM2/spe862jb.pdf Title: Fast Algorithms for Sorting and Searching Strings Author: Sedgewick, Robert Author: Bentley, Jon Louis URL: http://www.cs.tufts.edu/~nr/comp150fp/archive/bob-sedgewick/fast-strings.pdf lbzip2-2.3/AUTHORS000066400000000000000000000001111221755423000136220ustar00rootroot00000000000000Laszlo Ersek Mikolaj Izdebski lbzip2-2.3/BOOTSTRAP000066400000000000000000000015301221755423000140600ustar00rootroot00000000000000The code repository doesn't contain any auto-generated files or files imported from Gnulib, which are needed to build lbzip2 and which can be found in distribution tarballs. In order to build lbzip2 from repository these files must be generated first (the process is called bootstrapping). To bootstrap lbzip2 you need: * perl * Gnulib * Autoconf 2.63 or newer * Automake 1.11 or newer To bootstrap lbzip2 simply run `./build-aux/autogen.sh' from top source directory. gnulib-tool as well as perl, autoconf, autoheader, automake, and aclocal must be in PATH. To reverse the bootstrapping process (i.e. to remove auto-generated files and files imported from Gnulib) you can run `./build-aux/autogen.sh -r'. Be aware that this command may remove whole subdirectories if they are meant to hold only auto-generated files (lib, m4, build-aux/snippet). lbzip2-2.3/COPYING000066400000000000000000001045131221755423000136200ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . lbzip2-2.3/ChangeLog000066400000000000000000000555731221755423000143520ustar00rootroot000000000000002013-09-22 Mikolaj Izdebski Release version 2.3 Complete ChangeLog Add Serhii Khilinich to THANKS Add website link to --version report Add website link to manual page Add website link to README 2013-07-21 Mikolaj Izdebski Fix assertion * src/encode.c (package_merge): Fix assertion about remaining number of non-singleton trees uppon Package Merge completion. Small comment improvements 2013-06-30 Mikolaj Izdebski Fix typos in prefix decoder code 2013-06-29 Mikolaj Izdebski Remove unused retriever FSM state * src/decode.c: Remove S_TRAILER enum. 2013-06-26 Mikolaj Izdebski Optimize Huffman algorithm * src/encode.c (build_tree): Rewrite loop body to minimize number of comparisons and branches. Extract transmission cost computation as a separate function * src/encode.c (transmission_cost): New function. 2013-06-23 Mikolaj Izdebski Fix a typo in ALGORITHM Avoid compiler warnings in Package Merge mode * src/encode.c (make_code_lengths): Define V only when it's used. Add prefix coder sanity check * src/encode.c (make_code_lengths): Verify if Kraft's inequality is satisfied. Reimplement Package Merge algorithm * src/encode.c (package_merge): Rewrite from scratch. 2013-06-22 Mikolaj Izdebski During cleanup remove all files genared by autotools build-aux/autogen.sh: Remove test-driver during cleanup. 2013-03-30 Mikołaj Izdebski Fix incorrect usage of foreach() macro src/signals.c (pending): Call foreach() macro with second argument as array, not pointer to array. 2013-03-28 Mikołaj Izdebski Improve comments in decode.c 2013-03-27 Mikołaj Izdebski Don't hardcode MAX_CODE_LENGTH * src/deocde.c (make_tree): Use constant MAX_CODE_LENGTH instead of hardcoding its value. 2013-03-02 Mikolaj Izdebski Add release note for Debian bug 700680 Fix Debian bug 700680 src/main.c (input_init): Allow input files not to be regular files when outputting to stdout or to the bit bucket. 2012-10-28 Mikolaj Izdebski Document copying with -cdf Copy files only with --stdout process.c (work): Allow copying files only when output fd is stdout. Add NEWS entry for v2.3 2012-10-21 Mikolaj Izdebski Assert that the first group is always coded by the first tree Fix a deadlock and enable file copying with -d -f src/process.c (copy_on_input_avail): Decrement out_slots semaphore. (copy_on_write_complete): Increment out_slots semaphore. (copy_terminate): Rename from never; raise SIGUSR2 if terminal condition is met. 2012-10-20 Mikolaj Izdebski Avoid shadowing auto variable Fix assertion failure, closes #8 src/encode.c (generate_initial_trees): Rewrite from scratch. 2012-09-14 Mikolaj Izdebski Minor comment improvements and whitespace changes 2012-09-11 Mikolaj Izdebski Fix dead initialization bug 2012-08-06 Mikolaj Izdebski Ignore --small src/main.c (main): Reset small to zero. Improve initial tree generation * src/encode.c (generate_initial_trees): During initial set paritioning round set cardinalities to the nearest instead of rounding up. This decreases standard deviation of the size of trees and thus improves compression. Add THANKS Document bugs fixed Explicitly license auxiliary programs under GPLv3+ * tests/minbzcat.c: Relicense under GPLv3+. * tests/timeout.c: Add explicit license terms. Rebase to the newest version of yambi * ALGORITHM: Fix a typo. * Makefile.am: Remove yambi subdirectory. * build-aux/autogen.sh: Implement new directory structure layout. * build-aux/make-crctab.pl: Likewise. * build-aux/style-check.pl: Likewise. * configure.ac: Lower autoconf requirement. Optimize runtime by adding gl_ASSERT_NO_GNULIB_TESTS and gl_ASSERT_NO_GNULIB_POSIXCHECK. Add new compiler warnings. Implement new option "--enable-tracing". Implement new directory structure layout. * src/Makefile.am: Import from newer version of yambi. * src/common.h: Likewise. * src/compress.c: Likewise. * src/decode.c: Likewise. * src/decode.h: Likewise. * src/divbwt.c: Likewise. * src/encode.c: Likewise. * src/encode.h: Likewise. * src/expand.c: Likewise. * src/main.c: Likewise. * src/main.h: Likewise. * src/parse.c: Likewise. * src/process.c: Likewise. * src/process.h: Likewise. * src/signals.c: Likewise. * src/signals.h: Likewise. * tests/Makefile.am: Likewise. * tests/cve2.bz2: Likewise. * src/lbunzip2.c: Remove file. * src/lbunzip2.h: Likewise. * src/lbzip2.c: Likewise. * src/lbzip2.h: Likewise. * src/pqueue.c: Likewise. * src/pqueue.h: Likewise. * yambi/Makefile.am: Likewise. * yambi/collect.c: Likewise. * yambi/decode.c: Likewise. * yambi/decode.h: Likewise. * yambi/divsufsort.c: Likewise. * yambi/emit.c: Likewise. * yambi/encode.c: Likewise. * yambi/encode.h: Likewise. * yambi/prefix.c: Likewise. * yambi/private.h: Likewise. * yambi/retrieve.c: Likewise. * yambi/transmit.c: Likewise. * yambi/yambi.h: Likewise. 2012-06-23 Mikolaj Izdebski Update ChangeLog Bump version to 2.2 Simplify AUTHORS file Add workaround for Debian bug #673378 * yambi/retrieve.c (YBdec_retrieve): Return YB_UNDERFLOW if in_avail is 0. This Works around dDebian bug #673378. Revert "Implement double-ended queue." * src/deque.c: Delete file. * src/deque.h: Likewise. 2012-02-22 Mikołaj Izdebski Implement double-ended queue. * src/deque.c: New file. * src/deque.h: New file. * src/Makefile.am: Add deque.[ch]. Make pqueue_pop() return the popped element * src/pqueue.c (pqueue_pop): Return popped element. * src/pqueue.h (pqueue_pop): Change function prototype. Don't print EPIPE and EFBIG fatal errors * src/main.c (DEF): Skip printing fatal error messages if error number is EPIPE or EFGIG. Cleanup yambi code * yambi/divsufsort.h: Delete header file. * yambi/Makefile.am: Remove reference to divsufsort.h. * yambi/divsufsort.c: Include encode.h instead of divsufsort.h. Change cyclic_divbwt() function name to YBpriv_cyclic_divbwt(). * yambi/encode.h (YBpriv_block_sort, YBpriv_sais): Remove prototypes of previously removed functions. * yambi/encode.c (YBenc_work): Replace reference to cyclic_divbwt() with YBpriv_cyclic_divbwt(). * yambi/private.h (SLong): Remove unused type definition. 2012-02-21 Mikołaj Izdebski Implement new error handling Fix a bug in tandem repeat sort code * yambi/divsufsort.c (trsort): Fix a bug introduced in 336008f by altering the main trsort loop invariant. Adding a pre-condition of `-n < *SA' seems to fix the issue. 2012-02-16 Mikołaj Izdebski Remove obsolete shallow factor * yambi/collect.c, yambi/encode.h: Get rid of shallow_factor. * yambi/yambi.h: Remove YB_DEFAULT_SHALLOW #definition. * src/lbzip2.c (work_compr): Remove YB_DEFAULT_SHALLOW 2012-02-02 Mikołaj Izdebski Update copyright dates Disable load0 test Add version 2.2 entry in NEWS Update references in ALGORITHM Deprecate `--exponential' * build-aux/pretty-usage.pl, man/lbzip2.1: Update `--exponential' info. * src/main.c, src/lbzip2.c, src/lbzip2.h: Drop support for exponential. 2012-01-09 Mikołaj Izdebski Fix undefined behavior in divsufsort * yambi/divsufsort.c (trsort): Add bounds checks before moving pointer beyond allocated space. Extend bugfix from ca8e703c * yambi/divsufsort.c: Apply patch provided by Yuta Mori. Fix a bug with sorting periodic strings yambi/divsufsort.c (tr_introsort): Add depth condition. Integrate divsufsort with yambi * yambi/collect.c, yambi/encode.h: Let mtfv[] and SA[] share space and block[] be a separate array. * yambi/encode.c: Use divsufsort instead of old blocksort code. Adjust divsufsort to yambi * yambi/divsufsort.c: Remove include guardians. Define sauchar_t, saint_t and saidx_t in terms of . Remove __inline. (MAX): Remove unused macro. (construct_SA): Remove unused function. (construct_BWT): Remove unused parameter `m'. (cyclic_divsufsort): Remove unused function. (cyclic_divbwt): Write resulting BWT transform to SA[] instead of separate output array. Replace malloc() with xmalloc(). Optimize code for case with no B* suffixes. * yambi/divsufsort.h: Remove C++ code. Remove unused prototype. Import code from cyclic-divsufsort-lite yambi/divsufsort.c, yambi/divsufsort.h: New files. Remove existing block-sorting code yambi/blocksort.c, yambi/sais.c: Remove files. 2012-01-07 Mikołaj Izdebski Integrate new block-sorting code with yambi yambi/collect.c: Share block[] and mtfv[] space for better cache perfoemance and lower memory usage. yambi/encode.c: Read BWT characters from bwt[] and not block[]. yambi/encode.h: Add YBpriv_sais() prototype. Add bwt field to YBenc_s. yambi/private.h: Implement peekb() and pokeb() macros. build-aux/style-check.pl, yambi/Makefile.am: Mention yambi/sais.c. Refactor block-sorting code yambi/blocksort.c: Refactor quicksort and bucket sort code for better readability. Remove BPR code. Remove code copyrighted by Juliad Seward and the corresponding license block. Import cyclic-sais yambi/sais.c: Import from cyclic-sais 1.0 by Yuta Mori. 2011-12-25 Mikolaj Izdebski Remove unused constants * src/lbunzip2.c: Remove magic_mask and magic_hdr definitions. Remove unused constant definition * yambi/yambi.h: Remove YB_CANCELED macro definition. 2011-12-23 Mikolaj Izdebski Import new test case -- load0 * tests/load0.bz2: Initial version. * tests/Makefile.am: Add load0.bz2 as a new test. Limit time and memory usage during tests * tests/timeout.c: Initial version. * tests/Makefile.am: Build timeout from timeout.c. * tests/Tester: Try limiting virtual memory usage with `ulimit -v'. Run decompresion subprocesses with alarm set to 1 minute. Make minbzcat be more compatible with bzip2 * tests/minbzcat.c (main): Count block size digit as part of stream header. Fixes #3. Add check for fatal signals in tester * tests/Tester: Add checks for subprocess exit codes >= 128. All tests in which either lbzip2 or minbzcat exited because of unhandled signals are treated as failed. 2011-12-04 Mikołaj Izdebski Add two more test cases. * tests/incomp: Initial version. * tests/incomp-1.bz2, tests/incomp-2.bz2: Generate using tests/incomp. * tests/README: Update to mention incomp-[12].bz2. * tests/Makefile.am: Add the new test cases. Fix bug #5 * src/lbunzip2.c (work_decompr): Assign w2w_blk->bs100k to w2m_blk->bs100k only if YB_OK == ybret. Fixes: #5. This bug caused files with input blocks larger than 1 MiB to be rejected by lbunzip2. * tests/ch255.bz2: Add as a regression test for bug #5. * tests/Makefile.am: Re-enable crc1.bz2, add ch255.bz2. Workaround bug #5 * src/lbunzip2.c (mux): Disable stream CRC check. Fixes #5. * tests/Makefile.am: Exclude crc1.bz2 from test cases. 2011-11-24 Mikołaj Izdebski Get rid of AM_MAINTAINER_MODE * configure.ac: Remove AM_MAINTAINER_MODE, which was included by accident. 2011-11-23 Mikołaj Izdebski Bump version to 2.1 Fix a use-after-free vulnerability in lbunzip2 * src/lbunzip2.c (work_retrieve.c): Move access to s2w_blk before work_release() call. Fixes #4. 2011-11-11 Mikołaj Izdebski Fix build error on OpenBSD 5.0 * src/main.c (opts_setup): Use _SC_THREAD_THREADS_MAX conditionally. 2011-11-05 Mikołaj Izdebski Update autogen.sh to remove gitlog-to-changelog * build-aux/autogen.sh: Remove gitlog-to-changelog with -r option. 2011-11-02 Mikołaj Izdebski Update and improve documentation * README: Add Laszlo Ersek's copyright notice. * yambi/blocksort.c: Add example Manber-Myers algorithm implementation. 2011-10-31 Mikołaj Izdebski Include ChangeLog.old in tarball * Makefile.am: Add ChangeLog.old to EXTRA_DIST. Create changelogs * build-aux/autogen.sh: Add gitlog-to-changelog Gnulib module. * ChangeLog.old: Import from v0.23. * ChangeLog: Generate with build-aux/gitlog-to-changelog. 2011-10-30 Mikołaj Izdebski Minor documentation cleanup Add BOOTSTRAP Add filenames and short desc to license blocks Run automated spelling check Make tests/Tester more portable 2011-10-29 Mikołaj Izdebski Make install and uninstall targets work Add list of subdirs to README Extend AUTHORS Import ALGORITHM Add a minimal README Cleanup manpage 2011-10-28 Mikołaj Izdebski Add scantab.h to noinst_HEADERS 2011-10-27 Mikołaj Izdebski Minor code cleanups to avoid gcc warnings * Add extra braces in generated scantab.h * Add __attribute__((printf, ...)) to log_fatal() * Add extra switch case in lbunzip2 parser Make file sizes unsigned integers (uintmax_t) Remove SETX from lbunzip2 2011-10-23 Mikołaj Izdebski Fix Tester to support VPATH builds Display progress info during decompression Improve error messages in lbunzip2 muxer Move progress info printing code to main.c Fix a bug that prevented VPATH build from working Reimplement lbunzip2 magic pattern scanner as DFA 2011-10-22 Mikołaj Izdebski Cleanup and improve lbunzip2 header parser Improvements implemented: * block overruns are detected (fixes `overrun.bz2') * stream CRCs are computed and checked (fixes `crc1.bz2') * initial stream headers are parsed (fixes `trash.bz2' and `void.bz2') * files with gaps between streams are rejected (fixes `gap.bz2') * minor speed enchancements NEWS were updated to reflect recent improvements. 2011-10-22 Mikołaj Izdebski Implement initial header parser for lbunzip2 Partly fixes `gap.bz2' bug. For now the parser isn't detecting parse errors and parsing results (CRC and bs100k) are unused. Reorder scanner code in lbunzip2.c Pad lbunzip2 input blocks to multiple of 32 bits 2011-10-20 Mikołaj Izdebski Add more files to EXTRA_DIST Fix a typo in style-check.pl naming. 2011-10-18 Mikołaj Izdebski Fix a minor bug in autogen.sh 2011-10-17 Mikołaj Izdebski Reimplement BPR algorithm 2011-10-13 Mikołaj Izdebski Improve comments in blocksort.c, prefix.c and retrieve.c 2011-10-08 Laszlo Ersek trivial typo fix in man page 2011-10-07 Mikołaj Izdebski Fix -Wno-unknown-warning-option Narrow usage text width to 79 characters Add missing _Noreturn attributes Add -Wno-unknown-warning-option for clang Remove Trace() macros Turn off warnings about unreachable code Link lbunzip2 and lbzcat manpages to lbzip2 Use uintmax_t for block identifiers Import a newer version of minbzcat 2011-10-04 Mikołaj Izdebski Unset LBZIP2 et al. in tests/Tester Import and extend lbzip2 manpage 2011-09-27 Mikołaj Izdebski Print version and usage info on stdout, not stderr Add build-aux/pretty-usage.pl to help maintaining usage message. 2011-09-23 Mikołaj Izdebski autogen.sh: set IFS to space-tab-newline Add make-crctab.pl Style fixups based on style-check.pl report Add build-aux/style-check.pl 2011-09-22 Mikołaj Izdebski Move autogen.sh to build-aux/ Fix a bug with displaying an error message 2011-09-20 Mikołaj Izdebski Fix a typo in tests/cve.c A typo in tests/cve.c caused tests/cve.bz2 to be invalid. Update both. 2011-09-19 Mikołaj Izdebski minbzcat: add checks for Kraft's inequality Fix a typo in yambi/Makefile.am Write initial NEWS Import basic test cases Add noinst_HEADERS in src and yambi subdirs 2011-09-14 Mikołaj Izdebski Replace AC_SYS_LARGEFILE with largefile Gnulib module Downgrade to autoconf 2.67 to ease development on Debian stable. Import realloc-gnu module. Remove redundant checks from configure.ac. 2011-09-12 Mikołaj Izdebski Rename the project back to lbzip2 2011-09-11 Mikołaj Izdebski In lbunzip2 initialise `scan' with all ones, not all zeros This decreases probability of finding false-positives when looking for the magic bit pattern. Remove split_chkstart() Upcoming stream parser obsoletes split_chkstart(). Prioritize decompressing earlier blocks Decompress retrieved blocks basing on their position within the stream, not FCFS. Replace lacos_rbtree with pqueue Fix a bug related to htonl() usage Drop fupport for LBZIP2_TRACE_ALLOC Replace home-made xalloc() with xmalloc() provided by Gnulib. Disable compiler warnings about aggregate return 2011-09-10 Mikołaj Izdebski Reduce memory usage during decompression 2011-09-09 Mikołaj Izdebski Remove emacs file variables Add missing copyright notices Implement smarter scanner in lbunzip2 Redesign yambi's retrieve pass. Remove IBS structure and related functions from yambi API. Perform yambi's retrieve pass directly during lbunzip2's scan pass and not during decompress pass. 2011-09-07 Mikołaj Izdebski Read 32 bits at time in lbunzip2 Add #include for size_t Include sys/types.h instead of stddef.h for size_t. Guard config.h inclusions Surround #include with #ifdef HAVE_CONFIG_H. Make use of SIZE_MAX Replace (size_t)-1 with SIZE_MAX. Improve configure.ac Added -r option to autogen.sh Delete corr-perf.sh, malloc_trace.pl and test.sh Optimize lbunzip2 state machine 2011-08-28 Mikołaj Izdebski Restore atime and mtime with nanosec accuracy Convert gettimeofday() calls to gettime() Don't display file name with progress info 2011-08-26 Mikołaj Izdebski Add a configure switch to enable warnings 2011-08-25 Mikołaj Izdebski Rename to bzimp2 Update externally visible names to bzimp2. Remove man page. Split usage and version printing options. Fix GCC compatibility bugs 2011-08-23 Mikołaj Izdebski Remove .cvsignore Add more checks to configure.ac 2011-08-20 Mikołaj Izdebski Include in all source files Replace utime() with fdutimens() Create autogen.sh Migrate to GNU build system Move lbzip2 source files to "src" subdirectory. Remove Makefiles. Create initial "configure.ac" and "Makefile.am" files. 2011-08-19 Mikołaj Izdebski Display progress info during compression Display compression ratio in verbose mode 2011-08-16 Mikołaj Izdebski Remove redundant includes 2011-08-15 Mikołaj Izdebski Fix a compatibility bug 2011-08-14 Mikołaj Izdebski Add support for decompressing randomized blocks 2011-08-13 Mikołaj Izdebski Minor comments and code cleanup Add more coments on prefix code decoding. Remove a potential incompatibility with platforms that don't use U2 encoding for signed integers. Remove unused auto variables. Replace loops zeroing bigger arrays with calls to bzero(). 2011-08-12 Mikołaj Izdebski Minor formatting cleanup Eliminate SIZES() macro Get rid of SIZES() to simplify the code. Introduce xread() and xwrite() Generalize existing I/O code and create xread() and xwrite() to ease maintaintenance. Pack "fd", "sep" and "fmt" into struct filespec to emphasise their close relationship and simplify the code. 2011-08-11 Mikołaj Izdebski Clean up the code Minor code cleanups: * move nested function declaration to global scope, * convert K&R function definitions to ANSI, * move common macros to header file, * remove unused auto variables, * make pointer casts not strip the const qualifier, * change some variable types between signed and unsigned, * add missing copyright notices, * correct intentation in lbzip2.c, * remove white spaces from end of lines, * replace tabs by spaces (except from Makefiles). Improve compression performance The main block sorting algorithm is reworked to behave better in average case while remaining fast enough in the worst case. 2011-08-09 Mikołaj Izdebski Make decompression error messages more datailed Implement Yberr_detail() -- a function returning a textual representation of decompression error codes. Make use of Yberr_detail() in lbunzip2.c. 2011-07-30 Mikołaj Izdebski Fix a buffer overflow in yambi decompression code Fix a security vulnerability. A specially crafted compressed file could potentially cause a buffer overflow in YBibs_retrieve. Some minor optimizations and cleanups: * utilize htonl() and nothl() for (hopefully) faster memory access * remove outdated code that wasn't used anyways * change default values of some yambi tuning parameters * reorder some struct members; move bigger arrays to the end * remove some automatic variables to reduce register pollution 2011-07-30 Mikołaj Izdebski Fix a bug in sorting algorithm implementation Fix a bug in main sorting algorithm implementation. Fix two different bugs Fix a bug in decompression code that was caused by a typo. Fix a compatibility issue in yambi/private.h. 2011-07-29 Mikołaj Izdebski Fix a bug in lbzip2 compresion code This commit fixes a bug that was introduced in the previous commit. Minor enchancements: * implement --exponential command line switch with the same semantics as bzip2's equivalent * reduce average-case memory consumption in compression mode by about 3.5 MB per worker Minor cleanups: * remove unused code * remove whitespaces from end of source code lines * replace some tabs with spaces * correct line indentation * remove bogus __builtin_prefetch'es * remove GPLv2 as now the code in under GPLv3+ only * remove README as it doesn't reflect current code * remove ChangeLog as it was obsoleted by migration to git 2011-07-28 Mikołaj Izdebski Import a newer version of yambi Import a newer version of yambi, which contains compression code. Remove libbz2 compatibility layer from yambi. Modify lbzip2 and lbunzip2 so that they use yambi's API directly, without the compatibility layer. Update Makefiles. Remove dependency on libbz2. Remove lbunzip2_single Remove lbunzip2_single, as upcoming sequential decompressor is obsoleting it. 2011-07-26 Laszlo Ersek base decompression on newly imported yambi Import yambi. Adapt Makefiles and decompression modules. import lbzip2-0.23 lbzip2-2.3/ChangeLog.old000066400000000000000000000300401221755423000151050ustar00rootroot00000000000000Version: lbzip2-0.23 Focus: Minor feature enhancements Date: 03-Mar-2010 Changes: In this release, if lbzip2 intends to exit with status 1 due to any fatal error, but any SIGPIPE or SIGXFSZ with inherited SIG_DFL action was generated for lbzip2 previously, then lbzip2 terminates by way of one of said signals, after cleaning up any interrupted output file. This should improve compatibility with GNU tar, when it spawns lbzip2 as a filter, and closes the pipe between them early, before it receives an EOF from lbzip2. Version: lbzip2-0.22 Focus: Minor bugfixes Date: 18-Feb-2010 Changes: Building lbzip2 on Debian unstable discovered that the "lfs.sh" build script, due to a typo, did not invoke the "getconf" utility in a SUSv2-conformant way. This bug has been corrected. Version: lbzip2-0.21 Focus: Minor bugfixes Date: 17-Feb-2010 Changes: Code examination revealed that lbzip2-0.18 introduced a race between the following two code paths: (1) the muxer thread displays an error message when it encounters a write error, (2) the main thread, in preparation to terminate the process, frees the output file name after an INT or TERM signal is delivered to it. This bug had negligible chance to occur, but it was fixed nonetheless. Version: lbzip2-0.20 Focus: Code cleanup Date: 28-Dec-2009 Changes: This release cleans up two theoretical portability problems. The input file's sanity check in the multiple-workers decompressor relied on character constants being encoded in ASCII; such character constants were replaced by their ASCII octet values. "Flexible" array members were replaced by unnamed memory regions following the affected structures, since the "struct hack" idiom entails undefined behavior in C89. Version: lbzip2-0.19 Focus: Minor bugfixes Date: 01-Dec-2009 Changes: This release works around a GNU/kFreeBSD standards-compliance problem. GNU/kFreeBSD does not define some STREAMS-related errno macros mandated by SUSv2. Consequently, lbzip2-0.18 cannot be built on GNU/kFreeBSD. This version checks if those (and some other) macros are defined before relying them. Version: lbzip2-0.18 Focus: Major feature enhancements Date: 29-Nov-2009 Changes: After adding sanity checks to both decompressors, the following features were implemented: removal of input FILE operands; options --keep and --force; copying of owner, group, permission bits, access time, modification time to regular output files. Logging was cleaned up and internally categorized into INFO, WARNING and FATAL levels; a separate exit status was introduced for the case when a warning message was printed. The decompressor robustness tests were re-executed. The author has finally replaced bzip2 with lbzip2 on his system. Version: lbzip2-0.17 Focus: Code cleanup Date: 29-Oct-2009 Changes: Uninitialized fields were accessed on the stack in a structure assignment expression. Theoretically, this might have entailed read accesses to trap representations. Since those fields weren't initialized because they weren't used at all in the first place, this portability bug was fixed by introducing a dedicated structure without those fields. Version: lbzip2-0.16 Focus: Minor bugfixes Date: 25-Oct-2009 Changes: Non-regular input files passed on the command line are skipped (symbolic links are followed, because lbzip2 doesn't remove input files). Instead of exiting, lbzip2 skips input files passed on the command line which it cannot open, or for which it cannot create the corresponding output files. Standard output is closed also with --stdout and file operands, not only in filter mode. Signal handling has been cleaned up; lbzip2 can now be interrupted during skipping many files in a row. Writing an error message to a broken pipe doesn't result in an incomplete output file anymore. Version: lbzip2-0.16rc1 Focus: Major feature enhancements Date: 21-Oct-2009 Changes: With this release candidate, implementing features like settable compression block size and working with file operands, lbzip2 can be considered a multi-threaded, command-line compatible replacement for bzip2 in most situations. Lbzip2 never deletes or overwrites files, however. The workaround for an earlier GNU getconf bug was extended to EGLIBC in the large file support build script. The development status was degraded to beta. Version: lbzip2-0.15 Focus: Minor bugfixes Date: 05-Apr-2009 Changes: A bug has been fixed where the single-worker decompressor could theoretically omit to decompress the last part of the compressed input and wrongfully signify premature EOF instead. In this release, logically independent buffer sizes are detached from each other. Several buffer sizes should be more IO-friendly now. The splitter block size of the multiple-workers decompressor was recalculated; an effort was made to deduce it formally in comments. The "malloc_trace.pl" script tracks peak memory usage. The documentation was refined. Version: lbzip2-0.14 Focus: Major bugfixes Date: 15-Mar-2009 Changes: The single-worker decompressor's multiplexer, while passing back all accrued released input slots to the splitter, wrote out at most one decompressed sub-block per one iteration of its outermost loop. In case of a slowly progressing muxer, this could lead to skyrocketing memory allocation on part of the worker. Now the SWD's muxer writes out all accrued decompressed sub-blocks per iteration. The documentation has been updated. The build system has received a minuscule portability improvement. Version: lbzip2-0.13 Focus: Code cleanup Date: 28-Jan-2009 Changes: Besides some code cleanup, various portability and other workarounds were introduced to cope with bugs in C libraries, shells, and upstream bzip2. Version: lbzip2-0.12 Focus: Minor feature enhancements Date: 26-Jan-2009 Changes: In this release the shell script test.sh tests the installed instance of lbzip2, which can be different from the currently built one. pbzip2 is not a requirement for testing anymore, but test.sh will try to measure its performance if it's available. The same applies to the bzip2 module of 7za (from p7zip). Some GNU/Linux specific hints on enabling SUSv2 conformance (which is requried for building and testing) were added to the README. Version: lbzip2-0.11 Focus: Minor feature enhancements Date: 21-Jan-2009 Changes: This release adds support for testing by end-users, in a portable way. Version: lbzip2-0.10 Focus: Minor bugfixes Date: 18-Jan-2009 Changes: Testing on a 128-core HP SuperDome by a volunteering user showed a known bottleneck in the multiple-workers decompressor to be significant on many-core machines: whenever there were less input blocks than cores, the work was distributed unevenly. Hence, the splitter-to-workers queue of "scan and decompress" tasks was replaced with two queues, a low priority, splitter-to-workers one of "scan" tasks, and a high priority, workers-to-workers one of "decompress" tasks. Alas, this also increased the number of context switches. The new worker broadcast conditions were formally proven in the comments. Version: lbzip2-0.09 Focus: Minor feature enhancements Date: 06-Jan-2009 Changes: Now the Makefiles, with the help of the standard getconf utility, select a programming environment, if there is one, in which large files are supported. Version: lbzip2-0.08 Focus: Code cleanup Date: 04-Dec-2008 Changes: In the multiple-workers decompressor, the tail pointer of the splitter-to-workers queue proved to be private to the splitter. Accordingly, said pointer was eliminated as a shared resource, simplifying the code. Version: lbzip2-0.07 Focus: Documentation Date: 29-Nov-2008 Changes: A manual page using the man macro package was added. The README file describes the compressed output and why there is only one exit status for all types of errors. A more portable Makefile was created. A little code was cleaned up. Version: lbzip2-0.06a Focus: Documentation Date: 29-Sep-2008 Changes: A class of valid bz2 files the multiple-workers decompressor (MWD) possibly refuses has been described in the Bugs section of the README. By concatenating empty bzip2 streams (each having a length of 14 bytes) and optionally inserting such a sequence before, after or between non-empty bzip2 streams, the size of the input block that unavoidably contains an entire bzip2 block header can be increased without bound. This invalidates the assumption the MWD is based on. However, neither bzip2 nor lbzip2 creates such files, and bz2 file sets that do defeat the MWD when catenated should be rare. Version: lbzip2-0.06 Focus: Minor feature enhancements Date: 16-Sep-2008 Changes: When decompressing with a single worker thread, lbzip2 was previously 45% slower than standard bzip2. The new, dedicated single-worker decompressor is only 3% slower, and provides input and output buffering, which is useful in pipelines and on network file systems. Hence using lbzip2 incurs virtually no performance penalty over bzip2 even on a single-core machine. A script was added to help automated testing. Some thread notification conditions have been cleaned up. In this release, lbzip2 compresses an empty file to a valid bzip2 stream instead of an empty file. Version: lbzip2-0.05 Focus: Major feature enhancements Date: 10-Sep-2008 Changes: The decompressor was redesigned: all CPU-bound operations were moved into the worker threads, so that now, besides the muxer, the splitter is purely I/O-bound too. Lbzip2 supports tracing its memory allocation with the new "-t" option. Both the compressor and decompressor were retested. Version: lbzip2-0.04 Focus: Major feature enhancements Date: 01-Sep-2008 Changes: Decompression was extracted from the split-work-multiplex skeleton into a separate module. Compression was added. The project has been renamed to lbzip2. The reordering of processed sub-blocks happens entirely in the multiplexer now, changing the time complexity from O(log n) to O(1) inside the critical section between workers and the muxer. The command line conforms to utility syntax guidelines. Lbzip2 queries the number of online processors if sysconf() supports it. Block serial numbers have fixed 64 bit width. The README file was updated. The development status has been advanced to Beta. Version: lbunzip2-0.03 Focus: Major bugfixes Date: 20-Aug-2008 Changes: After running lbunzip2 on the bz2 test material of CERT-FI 20469, a bug was fixed where a worker (decompressor) thread could get into an infinite loop, spinning until finally outrunning the multiplexer thread, then consuming all available memory and exiting. Version: lbunzip2-0.02 Focus: Major bugfixes Date: 18-Aug-2008 Changes: Version 0.01 didn't throttle the decompressor threads when the multiplexer thread was blocked on the write() system call, thus memory consumption could grow indefinitely. This is fixed now. Some performance testing was done on five multicore machines (Alpha, Athlon, Itanium, Sparc, and Xeon). The README file was rewritten. Version: lbunzip2-0.01 Focus: Initial freshmeat announcement Date: 13-Aug-2008 Changes: Initial freshmeat announcement. lbzip2-2.3/Makefile.am000066400000000000000000000015751221755423000146250ustar00rootroot00000000000000#- # Copyright (C) 2011, 2012 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # SUBDIRS = lib src man tests ACLOCAL_AMFLAGS = -Im4 EXTRA_DIST = build-aux/autogen.sh build-aux/make-scantab.pl ChangeLog.old \ build-aux/pretty-usage.pl build-aux/style-check.pl build-aux/make-crctab.pl lbzip2-2.3/NEWS000066400000000000000000000071121221755423000132610ustar00rootroot00000000000000NEWS for lbzip2 -*- outline -*- * What's new in lbzip2 2.3 ** Bug fixes Version 2.3 fixes an assertion failure bug that could cause core dumps when compressing some kind of data. A bug which prevented symbolic links to be opened in some situations (when compressing or decompressing files to stdout or when discarding output) was fixed. Version 2.3 fixes a bug that caused SIGXFSZ to be handled incorrectly on 32-bit systems. ** New features When decompressing with both --force and --stdout options given lbzip2 2.3 will simply copy files that are not in bzip2 format (instead of reporting invalid file format), but will still reject files that appear to be in bzip2 format but are damaged or malformed somehow. This mimics behavior of gzip. * What's new in lbzip2 2.2 ** Bug fixes In version 2.2 decompressor was made more bzip2-compatible. All valid bzip2 files should now be properly decompressed by lbzip2. Memory allocation was reduced significantly. Previous versions could allocate large amounts of memory during decompression and testing of highly compressed files. Version 2.2 fixes a bug that caused some legitimate bz2 files to be rejected during decompression with a CRC error message. ** New features Compression performance is increased significantly by inclusion of new block-sorting code using divsufsort algorithm. ** Abandoned features The new block-sorting algorithm made `--exponential' command-line switch irrelevant. The switch is retained for compatibility with bzip2 and older versions of lbzip2, but otherwise has no effect. File write errors caused by broken pipes or exceeded file size limits are no longer reported to standard error. * What's new in lbzip2 2.1 ** Bug fixes Version 2.1 fixes a use-after-free security vulnerability in decompressor code responsible for displaying progress information. * What's new in lbzip2 2.0 ** Bug fixes Now lbzip2 creates one compressed stream per bzip2 file instead of multiple concatenated streams. lbzip2 now doesn't decompress streams embedded within trailing garbage after initial sequence of streams. Now lbzip2 detects and rejects more kinds of invalid bzip2 files (files with invalid stream CRCs, files with blocks larger than stated in stream headers and some others). ** New features Both compression and decompression speed is increased significantly. When invoked with `-v' or `--verbose' options lbzip2 now displays information about progress of compression or decompression, provided that stderr is connected to a terminal. When invoked with `-v' or `--verbose' options lbzip2 now displays compression ratio and space savings for each compressed or decompressed file. lbzip2 now supports `--exponential' option with the same semantics as bzip2's one. More information in the manual. Now lbzip2 displays more detailed messages on decompression failure. ** Changed features Help and version information is now printed to standard output instead of standard error stream. ** Abandoned features Support for tracing memory allocation by setting the environmental variable `LBZIP2_TRACE_ALLOC' was dropped. ** Other lbzip2 is now licensed under the terms of GPL v3.0 or any later version. Former versions used to be released under GPL v2+. lbzip2 doesn't depend on libbz2 any longer. Now it uses its own implementation of bzip2 compression and decompression algorithms. Now lbzip2's build process is managed by the GNU build system. lbzip2-2.3/README000066400000000000000000000027171221755423000134500ustar00rootroot00000000000000lbzip2, parallel bzip2 compression utility http://lbzip2.org/ Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski Copyright (C) 2008, 2009, 2010 Laszlo Ersek This README file is part of lbzip2 version 2.3. lbzip2 is a parallel, SMP-based, bzip2-compatible compression utility. lbzip2 compresses and decompresses files using a variation of BWT compression stack. More information on this topic can be found in the ALGORITHM file. lbzip2 uses Gnulib and its building and testing process is managed by the GNU build system. For more information on building and installing lbzip2 see the INSTALL file. For usage see the manual pages. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. A copy of the GNU General Public License is contained in the COPYING file. The following is a list of subdirectories with explanation of their purpose: build-aux - different auxiliary build scripts lib - Gnulib C files m4 - M4 macros for Autoconf (part of Gnulib) man - manual pages src - C source code of higher-level part of lbzip2 tests - regression tests lbzip2-2.3/THANKS000066400000000000000000000031561221755423000135010ustar00rootroot00000000000000The authors of lbzip2 would like to thank (in no particular order): - Adam Maulis at ELTE IIG, for fruitful discussions; for inspiring them to parallelize the bit-shifting part of the decompressor, thus rendering the splitter fully IO-bound; for inspiring them to create the single-worker decompressor; and also for letting them test on "opteron"; - Julian Seward for creating bzip and bzip2; - Paul Sladen for his Wikipedia contributions on bzip2, eg. http://en.wikipedia.org/wiki/Bzip2#File_format, and for his pyflate test data (http://www.paul.sladen.org/projects/pyflate/); - Michael Thomas from Caltech HEP for allowing them to test the earlier lbunzip2 on their Itanium 2 machines; - Bryan Stillwell for testing and retesting lbzip2 on "superdome"; his results inspired them to remove the bottleneck in the multiple-workers decompressor that proved significant on many-core machines; - Zsolt Bartos-Elekes and Imre Csatlos for sending test reports, see http://lacos.hu/; - Gabor Kovesdan for creating the FreeBSD port; - Paul Wise for reporting a possible (but mostly harmless) "read access to a trap representation" bug (http://lists.debian.org/debian-mentors/2009/10/msg00470.html); - Paolo Bonzini for discussing SIGPIPE/EPIPE behavior; - Yuta Mori for creating divsufsort and porting it to support cyclic strings; - Serhii Khilinich for helping with creation of lbzip2 logo and motivating me to create lbzip2 web page; They would also like to thank the Department of Electrical and Information Engineering at the University of Oulu, for making available their invaluable PROTOS Genome Test Suite c10-archive under the GPL. lbzip2-2.3/build-aux/000077500000000000000000000000001221755423000144535ustar00rootroot00000000000000lbzip2-2.3/build-aux/autogen.sh000077500000000000000000000031261221755423000164560ustar00rootroot00000000000000#!/bin/sh #- # Copyright (C) 2011 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # set -e IFS=' '' '' ' if ! test -r build-aux/autogen.sh || ! test -r src/main.c; then echo autogen.sh: need to be called from top source directory >&2 exit 1 fi # Option -r removes autogenerated files. if test x"$1" = x-r; then rm -Rf lib m4 build-aux/snippet autom4te.cache for f in config.guess config.sub depcomp install-sh missing \ gitlog-to-changelog test-driver .gitignore do rm -f build-aux/$f; done rm -f configure aclocal.m4 INSTALL for dir in . src man tests; do rm -f $dir/Makefile.in; done rm -f src/crctab.c src/scantab.h exit fi set -x perl ./build-aux/make-crctab.pl perl ./build-aux/make-scantab.pl gnulib-tool --avoid=xalloc-die --add-import pthread utimens warnings \ timespec-add timespec-sub dtotimespec stat-time lstat malloc-gnu \ fprintf-posix inttypes xalloc largefile gitlog-to-changelog # flockfile? aclocal -Im4 autoconf autoheader automake --add-missing --copy lbzip2-2.3/build-aux/make-crctab.pl000077500000000000000000000021071221755423000171640ustar00rootroot00000000000000#!/usr/bin/perl -w #- # Copyright (C) 2011 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # use Text::Wrap; open F, ">src/crctab.c" or die; printf F q(/* This file was generated automatically by make-crctab.pl. For comments refer to the generator script -- make-crctab.pl. */ #include "common.h" #include "encode.h" uint32_t crc_table[256] = { %s }; ), wrap ' ',' ',map { $c=$_*2**24; for (1..8) { $c = 2*$c&2**32-1 ^ 0x04C11DB7 & -($c>>31); } sprintf "0x%08lX,",$c } 0..255 lbzip2-2.3/build-aux/make-scantab.pl000077500000000000000000000062551221755423000173510ustar00rootroot00000000000000#!/usr/bin/perl -w #- # Copyright (C) 2011 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # #============================================================================# # # # Generate transition tables for deterministic finite automatons (DFAs) # # used to detect the 48-bit magic bit sequence present at the beginning # # of every bzip2 compressed block. These DFAs are already minimalistic. # # # # Each DFA has 49 states, numbered from 0 to 48. Starting with the initial # # state 0, the DFA inputs one byte (big_dfa) or one bit (mini_dfa) at time # # and advances to a next state. As soon as the bit pattern is detected, # # DFA moves to the accepting state (number 48) and remains there. # # # #============================================================================# # The bit patern used in bzip2. @x = split //, sprintf "%024b%024b", 0x314159, 0x265359; # Use Knuth–Morris–Pratt algorithm to create the mini-DFA, # which inputs only one bit at time. $i = 0; $j = $p[0] = -1; while ($i < @x-1) { while ($j > -1 && $x[$i] != $x[$j]) { $j = $p[$j]; } $i++; $j++; if ($x[$i] == $x[$j]) { $p[$i] = $p[$j]; } else { $p[$i] = $j; } } # Create non-accepting states of the mini-DFA. for $s (0..47) { $d[$s][0] = ($x[$s]?$p[$s]:$s)+1; $d[$s][1] = ($x[$s]?$s:$p[$s])+1; } # Create the big, 8-bit DFA, which inputs 8 bits at time. for $s0 (0..47) { for $c (0..255) { $s = $s0; for $k (0..7) { $b = (($c << $k) & 0x80) >> 7; $s = $d[$s][$b]; last if $s == 48; } $t[$s0][$c] = $s; } } for $c (0..255) { $t[48][$c] = 48; } # Dump the mini DFA. @mini=(); for $s (0..47) { @vec = (); for $c (0..1) { $y = $d[$s][$c]; push @vec,"$y,"; } push @mini, '{' . join(' ',@vec) . '},' } # Dump the big DFA. @big=(); for $s (0..48) { @vec = (); for $c (0..255) { $y = $t[$s][$c]; push @vec,"$y,"; } push @big, '{' . join(' ',@vec) . '},' } use Text::Wrap; $Text::Wrap::columns = 80; open F, ">src/scantab.h" or die; printf F q(/* This file was generated automatically by make-scantab.pl. For comments refer to the generator script -- make-scantab.pl. */ static const unsigned char ACCEPT = 48; static const unsigned char mini_dfa[48][2] = { %s }; static const unsigned char big_dfa[49][256] = { %s }; ), wrap(' ',' ',join(' ',@mini)), wrap(' ',' ',join(' ',@big)); lbzip2-2.3/build-aux/pretty-usage.pl000077500000000000000000000074331221755423000174530ustar00rootroot00000000000000#!/usr/bin/perl #- # Copyright (C) 2011, 2012 Mikolaj Izdebski # Copyright (C) 2008, 2009, 2010 Laszlo Ersek # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # use Text::Wrap; $il = 2; # indentation level $lw = 19; # label width $sw = 80; # screen width $ind = ' 'x$il; undef $Text::Wrap::unexpand; while () { if (!/^@(.*)/) { $s.=$_; next } $label=$1; @d=(); while () { last unless /./; chop; push @d,$_; } $desc = join " ", @d; $Text::Wrap::columns = $lw+$il; $s .= wrap $ind,$ind,$label; $s =~ s/\n([^\n]*)$/\n/; $t=$1; $t .= ' 'x($il+$lw-length$t); $Text::Wrap::columns = $sw; $s .= wrap "$t: ",' 'x($il+$lw+2),$desc; $s .= "\n"; } @c = map {s/\n/\\n/g; $_} grep {/./} split /([^\0]{509})/, $s; unshift @c,'%s'x@c; $_ = join '", "', @c; $_ = "#define USAGE_STRING \"$_\""; $s = ""; while (1) { if (length($_) <= 77) { $s.="$_\n"; last; } /^(.{77}[^\\]?)(.*)$/; $s.="$1\\\n"; $_="$2"; } open F, "src/main.c" or die; undef $/; $_=; s/#define USAGE_STRING.*(.*\\\n)*.*[^\\]\n/$s\n/ or die; open F, ">src/main.c" or die; print F; __END__ Usage: 1. PROG [-n WTHRS] [-k|-c|-t] [-d|-z] [-1 .. -9] [-f] [-v] [-S] {FILE} 2. PROG -h|-V Recognized PROG names: @bunzip2, lbunzip2 Decompress. Forceable with `-d'. @bzcat, lbzcat Decompress to stdout. Forceable with `-cd'. @ Compress. Forceable with `-z'. Environment variables: @LBZIP2, BZIP2, BZIP Insert arguments between PROG and the rest of the command line. Tokens are separated by spaces and tabs; no escaping. Options: @-n WTHRS Set the number of (de)compressor threads to WTHRS, where WTHRS is a positive integer. @-k, --keep Don't remove FILE operands. Open regular input files with more than one link. @-c, --stdout Write output to stdout even with FILE operands. Implies `-k'. Incompatible with `-t'. @-t, --test Test decompression; discard output instead of writing it to files or stdout. Implies `-k'. Incompatible with `-c'. @-d, --decompress Force decompression over the selection by PROG. @-z, --compress Force compression over the selection by PROG. @-1 .. -9 Set the compression block size to 100K .. 900K. @--fast Alias for `-1'. @--best Alias for `-9'. This is the default. @-f, --force Open non-regular input files. Open input files with more than one link. Try to remove each output file before opening it. With `-cd' copy files not in bzip2 format. @-v, --verbose Log each (de)compression start to stderr. Display compression ratio and space savings. Display progress information if stderr is connected to a terminal. @-S Print condition variable statistics to stderr. @-s, --small, -q, --quiet, --repetitive-fast, --repetitive-best, --exponential Accepted for compatibility, otherwise ignored. @-h, --help Print this help to stdout and exit. @-L, --license, -V, --version Print version information to stdout and exit. Operands: @FILE Specify files to compress or decompress. If no FILE is given, work as a filter. FILEs with `.bz2', `.tbz', `.tbz2' and `.tz2' name suffixes will be skipped when compressing. When decompressing, `.bz2' suffixes will be removed in output filenames; `.tbz', `.tbz2' and `.tz2' suffixes will be replaced by `.tar'; other filenames will be suffixed with `.out'. lbzip2-2.3/build-aux/style-check.pl000077500000000000000000000033671221755423000172370ustar00rootroot00000000000000#!/usr/bin/perl -w #- # Copyright (C) 2011, 2012 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # Hardcoded files to consider. @ARGV=qw( src/common.h src/signals.c src/encode.h src/decode.c src/encode.c src/process.c src/compress.c src/expand.c src/parse.c src/signals.h src/main.c src/divbwt.c src/decode.h src/process.h src/main.h ) if !@ARGV; # The user knows better. sub msg { print "$f: @_\n"; ++$cnt } for $f (@ARGV) { open F, $f or msg "file doesn't exist" and next; undef $/; $_=; ++$nf; # ASCII chars, whitespaces, line length. /([^\x20-\x7e\n])/ and msg "contains prohibited chars"; /[^\n]\n$/ or msg "doesn't end with a single NL"; / \n/ and msg "has trailing whitespace before NL"; /\n{4}/ and msg "has more than 2 consec blank lines"; /\n[^\n]{80}/ and msg "has line longer than 79 chars"; # C specific stuff. m{^/\*-([^*]|\*[^/])*Copyright[^*]+2012([^*]|\*[^/])*\*/} or msg "has missing or outdated copyright notice"; $f ne "src/common.h" xor /\n *# *include *\n/ or msg "missing or excessive #include "; } $nf='No' if !$nf; $cnt='no' if !$cnt; print "$nf file(s) checked, $cnt warning(s).\n"; lbzip2-2.3/configure.ac000066400000000000000000000105201221755423000150450ustar00rootroot00000000000000#- # Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . dnl TODO: check for GCC builtins: expect, prefetch, ctz, ctzl, ctzll AC_PREREQ([2.63]) AC_INIT([lbzip2], [2.3]) AC_CONFIG_SRCDIR([src/main.c]) AC_CONFIG_AUX_DIR([build-aux]) AC_CONFIG_HEADERS([lib/config.h]) AM_INIT_AUTOMAKE([1.11 color-tests]) AM_SILENT_RULES([yes]) AC_PROG_LN_S AC_PROG_RANLIB AC_PROG_CC_C89 gl_ASSERT_NO_GNULIB_TESTS gl_ASSERT_NO_GNULIB_POSIXCHECK gl_EARLY AC_ARG_ENABLE([tracing], [AS_HELP_STRING([--enable-tracing], [enable tracing])], [AS_VAR_IF([enableval], [yes], [ AC_DEFINE([ENABLE_TRACING], [1], [Define to enable tracing]) ]) ]) AC_ARG_ENABLE([warnings], [AS_HELP_STRING([--enable-warnings], [try to enable many C compiler warnings])], [AS_VAR_IF([enableval], [yes], [ m4_foreach([warn], [attributes, bad-function-cast, builtin-macro-redefined, cast-align, cast-qual, coverage-mismatch, declaration-after-statement, disabled-optimization, extra, extra-tokens, float-equal, format-nonliteral, format-security, format-y2k, init-self, inline, invalid-pch, logical-op, long-long, missing-declarations, missing-format-attribute, missing-include-dirs, missing-noreturn, missing-prototypes, mudflap, multichar, nested-externs, old-style-definition, overlength-strings, packed, packed-bitfield-compat, pointer-arith, redundant-decls, shadow, stack-protector, strict-aliasing, strict-prototypes, switch-default, switch-enum, sync-nand, unknown-pragmas, unused, unused-local-typedefs, unused-macros, vla, volatile-register-var, write-strings], [m4_set_add([kjn_warn_set], warn)]) AC_TRY_COMPILE([#ifndef __clang__ not clang #endif], [], [gl_WARN_ADD([-Wno-unknown-warning-option])]) for kjn_warn in '' all extra m4_set_dump([kjn_warn_set], [ ]) do : gl_WARN_ADD([-W$kjn_warn]) done AC_SUBST([WARN_CFLAGS]) ]) ]) gl_INIT AC_CONFIG_FILES([Makefile src/Makefile lib/Makefile man/Makefile tests/Makefile]) AC_OUTPUT AC_COPYRIGHT([Copyright (C) 2011, 2012 Mikolaj Izdebski This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . ]) lbzip2-2.3/man/000077500000000000000000000000001221755423000133345ustar00rootroot00000000000000lbzip2-2.3/man/Makefile.am000066400000000000000000000013571221755423000153760ustar00rootroot00000000000000#- # Copyright (C) 2011, 2012 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # man_MANS = lbzip2.1 lbunzip2.1 lbzcat.1 EXTRA_DIST = $(man_MANS) lbzip2-2.3/man/lbunzip2.1000066400000000000000000000000221221755423000151550ustar00rootroot00000000000000.so man1/lbzip2.1 lbzip2-2.3/man/lbzcat.1000066400000000000000000000000221221755423000146670ustar00rootroot00000000000000.so man1/lbzip2.1 lbzip2-2.3/man/lbzip2.1000066400000000000000000000332721221755423000146270ustar00rootroot00000000000000.ig Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski Copyright (C) 2008, 2009, 2010 Laszlo Ersek Copyright (C) 1996 Julian Seward This manual page is part of lbzip2, version 2.3. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . .. .TH LBZIP2 1 "22 September 2013" "lbzip2-2.3" "User commands" .SH NAME lbzip2 \- parallel bzip2 utility .SH SYNOPSIS .BR lbzip2 "|" bzip2 " [" \-n .IR WTHRS ] .RB [ \-k "|" \-c "|" \-t "] [" \-d "] [" \-1 " .. " \-9 "] [" \-f "] [" \-v ] .RB [ \-S "] [" .IR "FILE ... " ] .BR lbunzip2 "|" bunzip2 " [" \-n .IR WTHRS ] .RB [ \-k "|" \-c "|" \-t "] [" \-z "] [" \-f "] [" \-v ] .RB [ \-S "] [" .IR "FILE ... " ] .BR lbzcat "|" bzcat " [" \-n .IR WTHRS ] .RB [ \-z "] [" \-f "] [" \-v ] .RB [ \-S "] [" .IR "FILE ... " ] .BR lbzip2 "|" bzip2 "|" lbunzip2 "|" bunzip2 "|" lbzcat "|" bzcat " " \-h .SH DESCRIPTION Compress or decompress .I FILE operands or standard input to regular files or standard output using the Burrows-Wheeler block-sorting text compression algorithm. The .B lbzip2 utility employs multiple threads and an input-bound splitter even when decompressing .B .bz2 files created by standard bzip2. Compression is generally considerably better than that achieved by more conventional LZ77/LZ78-based compressors, and competitive with all but the best of the PPM family of statistical compressors. Compression is always performed, even if the compressed file is slightly larger than the original. The worst case expansion is for files of zero length, which expand to fourteen bytes. Random data (including the output of most file compressors) is coded with asymptotic expansion of around 0.5%. The command-line options are deliberately very similar to those of .BR bzip2 " and " gzip , but they are not identical. .SH INVOCATION The default mode of operation is compression. If the utility is invoked as .BR lbunzip2 " or " bunzip2 , the mode is switched to decompression. Calling the utility as .BR lbzcat " or " bzcat selects decompression, with the decompressed byte-stream written to standard output. .SH OPTIONS .TP .BI "\-n " WTHRS Set the number of (de)compressor threads to .IR "WTHRS" . If this option is not specified, .B lbzip2 tries to query the system for the number of online processors (if both the compilation environment and the execution environment support that), or exits with an error (if it's unable to determine the number of processors online). .TP .BR \-k ", " \-\-keep Don't remove .I FILE operands after successful (de)compression. Open regular input files with more than one link. .TP .BR \-c ", " \-\-stdout Write output to standard output, even when .I FILE operands are present. Implies .BR \-k " and excludes " \-t . .TP .BR \-t ", " \-\-test Test decompression; discard output instead of writing it to files or standard output. Implies .BR \-k " and excludes " \-c . Roughly equivalent to passing .B \-c and redirecting standard output to the bit bucket. .TP .BR \-d ", " \-\-decompress Force decompression over the mode of operation selected by the invocation name. .TP .BR \-z ", " \-\-compress Force compression over the mode of operation selected by the invocation name. .TP .BR \-1 " .. " \-9 Set the compression block size to 100K .. 900K, in 100K increments. Ignored during decompression. See also the BLOCK SIZE section below. .TP .B \-\-fast Alias for .BR \-1 . .TP .B \-\-best Alias for .BR \-9 . This is the default. .TP .BR \-f ", " \-\-force Open non-regular input files. Open input files with more than one link, breaking links when .B \-k isn't specified in addition. Try to remove each output file before opening it. By default .B lbzip2 will not overwrite existing files; if you want this to happen, you should specify .BR \-f . If .B \-c and .B \-d are also given don't reject files not in bzip2 format, just copy them without change; without .B \-f lbzip2 would stop after reaching a file that is not in bzip2 format. .TP .BR \-v ", " \-\-verbose Be more verbose. Print more detailed information about (de)compression progress to standard error: before processing each file, print a message stating the names of input and output files; during (de)compression, print a rough percentage of completeness and estimated time of arrival (only if standard error is connected to a terminal); after processing each file print a message showing compression ratio, space savings, total compression time (wall time) and average (de)compression speed (bytes of plain data processed per second). .TP .B \-S Print condition variable statistics to standard error for each completed (de)compression operation. Useful in profiling. .TP .BR \-s ", " \-\-small ", " \-q ", " \-\-quiet ", " \-\-repetitive\-fast ", " \ \-\-repetitive\-best ", " \-\-exponential Accepted for compatibility with .BR bzip2 , otherwise ignored. .TP .BR \-h ", " \-\-help Print help on command-line usage on standard output and exit successfully. .TP .BR \-L ", " \-\-license ", " \-V ", " \-\-version Print license and version information on standard output and exit successfully. .SH ENVIRONMENT .TP .IR LBZIP2 ", " BZIP2 ", " BZIP Before parsing the command line, lbzip2 inserts the contents of these variables, in the order specified, between the invocation name and the rest of the command line. Tokens are separated by spaces and tabs, which cannot be escaped. .SH OPERANDS .TP .I FILE Specify files to compress or decompress. .IR FILE s with .BR .bz2 ", " .tbz ", " .tbz2 " and " .tz2 name suffixes will be skipped when compressing. When decompressing, .B .bz2 suffixes will be removed in output filenames; .BR .tbz ", " .tbz2 " and " .tz2 suffixes will be replaced by .BR .tar ; other filenames will be suffixed with .BR .out ". If an " INT " or " TERM " signal is delivered to " lbzip2 , then it removes the regular output file currently open before exiting. If no FILE is given, lbzip2 works as a filter, processing standard input to standard output. In this case, .B lbzip2 will decline to write compressed output to a terminal (or read compressed input from a terminal), as this would be entirely incomprehensible and therefore pointless. .SH "EXIT STATUS" .TP .B 0 if .B lbzip2 finishes successfully. This presumes that whenever it tries, .B lbzip2 never fails to write to standard error. .TP .B 1 if .B lbzip2 encounters a fatal error. .TP .B 4 if .B lbzip2 issues warnings without encountering a fatal error. This presumes that whenever it tries, .B lbzip2 never fails to write to standard error. .TP .BR SIGPIPE ", " SIGXFSZ .RB "if " lbzip2 " intends to exit with status " 1 " due to any fatal error," .RB "but any such signal with inherited " SIG_DFL " action was generated for" .BR lbzip2 " previously, then " lbzip2 " terminates by way of one of said" signals, after cleaning up any interrupted output file. .TP .B SIGABRT if a runtime assertion fails (i.e. .B lbzip2 detects a bug in itself). Hopefully whoever compiled your binary wasn't bold enough to .BR "#define NDEBUG" . .TP .BR SIGINT ", " SIGTERM .B lbzip2 catches these signals so that it can remove an interrupted output file. In such cases, .B lbzip2 exits by re-raising (one of) the received signal(s). .SH "BLOCK SIZE" .B lbzip2 compresses large files in blocks. It can operate at various block sizes, ranging from 100k to 900k in 100k steps, and it allocates only as much memory as it needs to. The block size affects both the compression ratio achieved, and the amount of memory needed both for compression and decompression. Compression and decompression speed is virtually unaffected by block size, provided that the file being processed is large enough to be split among all worker threads. The flags .BR \-1 " through " \-9 specify the block size to be 100,000 bytes through 900,000 bytes (the default) respectively. At decompression-time, the block size used for compression is read from the compressed file -- the flags .BR \-1 " to " \-9 are irrelevant to and so ignored during decompression. Larger block sizes give rapidly diminishing marginal returns; most of the compression comes from the first two or three hundred k of block size, a fact worth bearing in mind when using .B lbzip2 on small machines. It is also important to appreciate that the decompression memory requirement is set at compression-time by the choice of block size. In general you should try and use the largest block size memory constraints allow. Another significant point applies to small files. By design, only one of .BR lbzip2 's worker threads can work on a single block. This means that if the number of blocks in the compressed file is less than the number of processors online, then some of worker threads will remain idle for the entire time. Compressing small files with smaller block sizes can therefore significantly increase both compression and decompression speed. The speed difference is more noticeable as the number of CPU cores grows. .SH "ERROR HANDLING" Dealing with error conditions is the least satisfactory aspect of .BR lbzip2 . The policy is to try and leave the filesystem in a consistent state, then quit, even if it means not processing some of the files mentioned in the command line. `A consistent state' means that a file exists either in its compressed or uncompressed form, but not both. This boils down to the rule `delete the output file if an error condition occurs, leaving the input intact'. Input files are only deleted when we can be pretty sure the output file has been written and closed successfully. .SH "RESOURCE ALLOCATION" .B lbzip2 needs various kinds of system resources to operate. Those include memory, threads, mutexes and condition variables. The policy is to simply give up if a resource allocation failure occurs. Resource consumption grows linearly with number of worker threads. If .B lbzip2 fails because of lack of some resources, decreasing number of worker threads may help. It would be possible for .B lbzip2 to try to reduce number of worker threads (and hence the resource consumption), or to move on to subsequent files in the hope that some might need less resources, but the complications for doing this seem more trouble than they are worth. .SH "DAMAGED FILES" .B lbzip2 attempts to compress data by performing several non-trivial transformations on it. Every compression of a file implies an assumption that the compressed file can be decompressed to reproduce the original. Great efforts in design, coding and testing have been made to ensure that this program works correctly. However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it very difficult to rule out the possibility of bugs remaining in the program. That is not to say this program is inherently unreliable. Indeed, I very much hope the opposite is true -- .B lbzip2 has been carefully constructed and extensively tested. As a self-check for your protection, .B lbzip2 uses 32-bit CRCs to make sure that the decompressed version of a file is identical to the original. This guards against corruption of the compressed data, and against undiscovered bugs in .B lbzip2 (hopefully unlikely). The chances of data corruption going undetected is microscopic, about one chance in four billion for each file processed. Be aware, though, that the check occurs upon decompression, so it can only tell you that that something is wrong. CRCs can only detect corrupted files, they can't help you recover the original, uncompressed data. However, because of the block nature of the compression algorithm, it may be possible to recover some parts of the damaged file, even if some blocks are destroyed. .SH BUGS Separate input files don't share worker threads; at most one input file is worked on at any moment. .SH AUTHORS .B lbzip2 was originally written by Laszlo Ersek , http://lacos.hu/. Versions 2.0 and later were written by Mikolaj Izdebski. .SH COPYRIGHT Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski .br Copyright (C) 2008, 2009, 2010 Laszlo Ersek .br Copyright (C) 1996 Julian Seward This manual page is part of lbzip2, version 2.3. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . .SH THANKS Adam Maulis at ELTE IIG; Julian Seward; Paul Sladen; Michael Thomas from Caltech HEP; Bryan Stillwell; Zsolt Bartos-Elekes; Imre Csatlos; Gabor Kovesdan; Paul Wise; Paolo Bonzini; Department of Electrical and Information Engineering at the University of Oulu; Yuta Mori. .SH "SEE ALSO" .TP .BR lbzip2 " home page" http://lbzip2.org/ .TP .BR bzip2 (1) http://www.bzip.org/ .TP .BR pbzip2 (1) http://compression.ca/pbzip2/ .TP .BR bzip2smp (1) http://bzip2smp.sourceforge.net/ .TP .BR smpbzip2 (1) http://home.student.utwente.nl/n.werensteijn/smpbzip2/ .TP .BR dbzip2 (1) http://www.mediawiki.org/wiki/Dbzip2 .TP .BR p7zip (1) http://p7zip.sourceforge.net/ lbzip2-2.3/src/000077500000000000000000000000001221755423000133505ustar00rootroot00000000000000lbzip2-2.3/src/Makefile.am000066400000000000000000000027251221755423000154120ustar00rootroot00000000000000#- # Copyright (C) 2011, 2012 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # AM_CPPFLAGS = -I$(top_srcdir)/lib AM_CFLAGS = $(WARN_CFLAGS) $(WERROR_CFLAGS) bin_PROGRAMS = lbzip2 noinst_HEADERS = \ common.h \ decode.h \ encode.h \ main.h \ process.h \ scantab.h \ signals.h lbzip2_SOURCES = \ compress.c \ crctab.c \ decode.c \ divbwt.c \ encode.c \ expand.c \ main.c \ parse.c \ process.c \ signals.c lbzip2_LDADD = $(top_builddir)/lib/libgnu.a $(LIB_CLOCK_GETTIME) $(LIB_PTHREAD) install-exec-hook: @cd '$(DESTDIR)$(bindir)' && for prog in lbunzip2 lbzcat; do \ rm -f $$prog$(EXEEXT) && $(LN_S) lbzip2$(EXEEXT) $$prog$(EXEEXT); done uninstall-hook: @cd '$(DESTDIR)$(bindir)' && for prog in lbunzip2 lbzcat; do \ rm -f $$prog$(EXEEXT); done lbzip2-2.3/src/common.h000066400000000000000000000072071221755423000150170ustar00rootroot00000000000000/*- common.h -- common declarations Copyright (C) 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #ifdef HAVE_CONFIG_H #include #endif #include /* assert() */ #include /* errno */ #include /* uint32_t */ #include /* bool */ #include /* size_t */ #include /* abort() */ #include "xalloc.h" /* XMALLOC() */ /* Tracing, useful in debugging, but not officially supported. */ #ifdef ENABLE_TRACING #define Trace(x) info x #else #define Trace(x) #endif /* Minimal and maximal alphabet size used in prefix coding. We always have 2 RLE symbols, from 0 to 255 MTF values and 1 EOF symbol. */ #define MIN_ALPHA_SIZE (2+0+1) #define MAX_ALPHA_SIZE (2+255+1) #define MIN_TREES 2 #define MAX_TREES 6 #define GROUP_SIZE 50 #define MIN_CODE_LENGTH 1 /* implied by MIN_ALPHA_SIZE > 1u */ #define MAX_CODE_LENGTH 20 #define MAX_BLOCK_SIZE 900000 #define MAX_GROUPS ((MAX_BLOCK_SIZE + GROUP_SIZE - 1) / GROUP_SIZE) #define MAX_SELECTORS 32767 enum error { OK, /* no error */ MORE, FINISH, ERR_MAGIC, /* bad stream header magic */ ERR_HEADER, /* bad block header magic */ ERR_BITMAP, /* empty source alphabet */ ERR_TREES, /* bad number of trees */ ERR_GROUPS, /* no coding groups */ ERR_SELECTOR, /* invalid selector */ ERR_DELTA, /* invalid delta code */ ERR_PREFIX, /* invalid prefix code */ ERR_INCOMPLT, /* incomplete prefix code */ ERR_EMPTY, /* empty block */ ERR_UNTERM, /* unterminated block */ ERR_RUNLEN, /* missing run length */ ERR_BLKCRC, /* block CRC mismatch */ ERR_STRMCRC, /* stream CRC mismatch */ ERR_OVERFLOW, /* block overflow */ ERR_BWTIDX, /* primary index too large */ ERR_EOF, /* unexpected end of file */ }; /* Minimum and maximum. It's important to keep the same condition in both macros because then some compilers on some architectures (like gcc on x86) will generate better code. */ #define min(x,y) ((x) < (y) ? (x) : (y)) #define max(x,y) ((x) < (y) ? (y) : (x)) /* Check GCC version. This not only works for GNU C, but also Clang and possibly others. If a particular compiler defines __GNUC__ but it's not GCC compatible then it's that compilers problem. */ #define GNUC_VERSION (10000 * (__GNUC__ + 0) + 100 * (__GNUC_MINOR__ + 0) + \ (__GNUC_PATCHLEVEL__ + 0)) /* Explicit static branch prediction to help compiler generating faster code. */ #if GNUC_VERSION >= 30004 # define likely(x) __builtin_expect((x), 1) # define unlikely(x) __builtin_expect((x), 0) #else # define likely(x) (x) # define unlikely(x) (x) #endif #ifndef __GNUC__ #define __attribute__(x) #endif lbzip2-2.3/src/compress.c000066400000000000000000000131731221755423000153540ustar00rootroot00000000000000/*- compress.c -- high-level compression routines Copyright (C) 2011, 2012 Mikolaj Izdebski Copyright (C) 2008, 2009, 2010 Laszlo Ersek This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include "main.h" /* bs100k */ #include "encode.h" /* encode() */ #include "process.h" /* struct process */ /* transmit threshold */ #define TRANSM_THRESH 2 struct in_blk { struct position pos; void *buffer; size_t size; const char unsigned *next; size_t left; }; struct work_blk { struct position pos; struct encoder_state *enc; size_t size; uint32_t crc; int last; size_t weight; }; struct out_blk { struct position pos; void *buffer; size_t size; uint32_t crc; int last; size_t weight; }; static struct pqueue(struct in_blk *) coll_q; static struct pqueue(struct work_blk *) trans_q; static struct pqueue(struct out_blk *) reord_q; static struct position reord_pos; static uintmax_t next_id; /* next free input block sequence number */ static uint32_t combined_crc; static bool can_collect(void) { return !empty(coll_q) && work_units > 0; } static void do_collect(void) { struct in_blk *iblk; struct work_blk *wblk; iblk = dequeue(coll_q); --work_units; sched_unlock(); wblk = XMALLOC(struct work_blk); wblk->pos = iblk->pos; /* Allocate an encoder with given block size and default parameters. */ wblk->enc = encoder_init(bs100k * 100000u, CLUSTER_FACTOR); /* Collect as much data as we can. */ wblk->weight = iblk->left; collect(wblk->enc, iblk->next, &iblk->left); wblk->weight -= iblk->left; iblk->next += wblk->weight; if (0u < iblk->left) { wblk->last = 0; ++iblk->pos.minor; sched_lock(); enqueue(coll_q, iblk); sched_unlock(); } else { wblk->last = 1; source_release_buffer(iblk->buffer); free(iblk); } /* Do the hard work. */ wblk->size = encode(wblk->enc, &wblk->crc); sched_lock(); enqueue(trans_q, wblk); } static bool can_transmit(void) { return !empty(trans_q) && (out_slots > TRANSM_THRESH || (out_slots > 0 && pos_eq(peek(trans_q)->pos, reord_pos))); } static void do_transmit(void) { struct work_blk *wblk; struct out_blk *oblk; wblk = dequeue(trans_q); --out_slots; sched_unlock(); oblk = XMALLOC(struct out_blk); oblk->pos = wblk->pos; oblk->crc = wblk->crc; oblk->last = wblk->last; oblk->size = wblk->size; oblk->weight = wblk->weight; /* Allocate the output buffer and transmit the block into it. */ oblk->buffer = XNMALLOC((oblk->size + 3) / 4, uint32_t); transmit(wblk->enc, oblk->buffer); free(wblk); sched_lock(); ++work_units; enqueue(reord_q, oblk); } static bool can_reorder(void) { return !empty(reord_q) && pos_eq(peek(reord_q)->pos, reord_pos); } static void do_reorder(void) { struct out_blk *oblk; int last; oblk = dequeue(reord_q); sink_write_buffer(oblk->buffer, oblk->size, oblk->weight); combined_crc = combine_crc(combined_crc, oblk->crc); last = oblk->last; free(oblk); if (last) { ++reord_pos.major; reord_pos.minor = 0u; } else { ++reord_pos.minor; } } static bool can_terminate(void) { return eof && empty(coll_q) && work_units == num_worker && out_slots == total_out_slots; } static void on_input_avail(void *buffer, size_t size) { struct in_blk *iblk = XMALLOC(struct in_blk); iblk->pos.major = next_id++; iblk->pos.minor = 0u; iblk->buffer = buffer; iblk->size = size; iblk->next = buffer; iblk->left = size; sched_lock(); enqueue(coll_q, iblk); sched_unlock(); } static void on_write_complete(void *buffer) { free(buffer); sched_lock(); ++out_slots; sched_unlock(); } static void write_header(void) { uint8_t buffer[HEADER_SIZE]; buffer[0] = 0x42; buffer[1] = 0x5A; buffer[2] = 0x68; buffer[3] = 0x30 + bs100k; xwrite(buffer, HEADER_SIZE); } static void write_trailer(void) { uint8_t buffer[TRAILER_SIZE]; buffer[0] = 0x17; buffer[1] = 0x72; buffer[2] = 0x45; buffer[3] = 0x38; buffer[4] = 0x50; buffer[5] = 0x90; buffer[6] = combined_crc >> 24; buffer[7] = (combined_crc >> 16) & 0xFF; buffer[8] = (combined_crc >> 8) & 0xFF; buffer[9] = combined_crc & 0xFF; xwrite(buffer, TRAILER_SIZE); } static void init(void) { pqueue_init(coll_q, in_slots); pqueue_init(trans_q, work_units); pqueue_init(reord_q, out_slots); reord_pos.major = 0; reord_pos.minor = 0; next_id = 0; assert(1 <= bs100k && bs100k <= 9); combined_crc = 0; write_header(); } static void uninit(void) { write_trailer(); pqueue_uninit(coll_q); pqueue_uninit(trans_q); pqueue_uninit(reord_q); } static const struct task task_list[] = { { "reorder", can_reorder, do_reorder }, { "transmit", can_transmit, do_transmit }, { "collect", can_collect, do_collect }, { NULL, NULL, NULL }, }; const struct process compression = { task_list, init, uninit, can_terminate, on_input_avail, on_write_complete, }; lbzip2-2.3/src/decode.c000066400000000000000000001070351221755423000147450ustar00rootroot00000000000000/*- decode.c -- low-level decompressor Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include /* ntohl() */ #include /* memcpy() */ #include "main.h" #include "decode.h" /* Prefix code decoding is performed using a multilevel table lookup. The fastest way to decode is to simply build a lookup table whose size is determined by the longest code. However, the time it takes to build this table can also be a factor if the data being decoded is not very long. The most common codes are necessarily the shortest codes, so those codes dominate the decoding time, and hence the speed. The idea is you can have a shorter table that decodes the shorter, more probable codes, and then point to subsidiary tables for the longer codes. The time it costs to decode the longer codes is then traded against the time it takes to make longer tables. This result of this trade are in the constant HUFF_START_WIDTH below. HUFF_START_WIDTH is the number of bits the first level table can decode in one step. Subsequent tables always decode one bit at time. The current value of HUFF_START_WIDTH was determined with a series of benchmarks. The optimum value may differ though from machine to machine, and possibly even between compilers. Your mileage may vary. */ #define HUFF_START_WIDTH 10 /* Notes on prefix code decoding: 1) Width of a tree node is defined as 2^-d, where d is depth of that node. A prefix tree is said to be complete iff all leaf widths sum to 1. If this sum is less (greater) than 1, we say the tree is incomplete (oversubscribed). See also: Kraft's inequality. In this implementation, malformed trees (oversubscribed or incomplete) aren't rejected directly at creation (that's the moment when both bad cases are detected). Instead, invalid trees cause decode error only when they are actually used to decode a group. This is nonconforming behavior -- the original bzip2, which serves as a reference implementation, accepts malformed trees as long as nonexistent codes don't appear in compressed stream. Neither bzip2 nor any alternative implementation I know produces such trees, so this behavior seems sane. 2) When held in variables, codes are usually in left-justified form, meaning that they occupy consecutive most significant bits of the variable they are stored in, while less significant bits of variable are padded with zeroes. Such form allows for easy lexicographical comparison of codes using unsigned arithmetic comparison operators, without the need for normalization. */ /* Structure used for quick decoding of prefix codes. */ struct tree { uint16_t start[1 << HUFF_START_WIDTH]; uint64_t base[MAX_CODE_LENGTH + 2]; /* 2 sentinels (first and last pos) */ unsigned count[MAX_CODE_LENGTH + 1]; /* 1 sentinel (first pos) */ uint16_t perm[MAX_ALPHA_SIZE]; }; /* start[] - decoding start point. `k = start[c] & 0x1F' is code length. If k <= HUFF_START_WIDTH then `s = start[c] >> 5' is the immediate symbol value. If k > HUFF_START_WIDTH then s is undefined, but code starting with c is guaranteed to be at least k bits long. base[] - base codes. For k in 1..20, base[k] is either the first code of length k or it is equal to base[k+1] if there are no codes of length k. The other 2 elements are sentinels: base[0] is always zero, base[21] is plus infinity (represented as UINT64_MAX). count[] - cumulative code length counts. For k in 1..20, count[k] is the number of symbols which codes are shorter than k bits; count[0] is a sentinel (always zero). perm[] - sorting permutation. The rules of canonical prefix coding require that the source alphabet is sorted stably by ascending code length (the order of symbols of the same code length is preserved). The perm table holds the sorting permutation. */ #define ROW_WIDTH 16u #define SLIDE_LENGTH 8192u #define NUM_ROWS (256u / ROW_WIDTH) #define CMAP_BASE (SLIDE_LENGTH - 256) struct retriever_internal_state { unsigned state; /* current state of retriever FSA */ uint8_t selector[MAX_SELECTORS]; /* coding tree selectors */ unsigned num_trees; /* number of prefix trees used */ unsigned num_selectors; /* number of tree selectors present */ unsigned alpha_size; /* number of distinct prefix codes */ uint8_t code_len[MAX_ALPHA_SIZE]; unsigned mtf[MAX_TREES]; /* current state of inverse MTF FSA */ struct tree tree[MAX_TREES]; /* coding trees */ uint16_t big; /* big descriptor of the bitmap */ uint16_t small; /* small descriptor of the bitmap */ unsigned j; /* general purpose index */ unsigned t; /* current tree number */ unsigned g; /* current group number */ uint8_t *imtf_row[NUM_ROWS]; uint8_t imtf_slide[SLIDE_LENGTH]; unsigned runChar; unsigned run; unsigned shift; }; /* FSM states from which retriever can be started or resumed. */ enum { S_INIT, S_BWT_IDX, S_BITMAP_BIG, S_BITMAP_SMALL, S_SELECTOR_MTF, S_DELTA_TAG, S_PREFIX, }; /* Internal symbol values differ from that used in bzip2! 257 - RUN-A 258 - RUN-B 1-255 - MTFV 0 - EOB */ #define RUN_A (256+1) #define RUN_B (256+2) #define EOB 0 #define RUN(s) ((s) - 256) #define IS_RUN(s) ((s) >= 256) #define IS_EOB(s) ((s) == EOB) /* Given a list of code lengths, make a set of tables to decode that set of codes. Return value is passed in mtf array of the decoder state. On success value from zero to five is passed (the tables are built only in this case), but also error codes ERR_INCOMPLT or ERR_PREFIX may be returned, which means that given code set is incomplete or (respectively) the code is invalid (an oversubscribed set of lengths). Because the alphabet size is always less or equal to 258 (2 RUN symbols, at most 255 MFV values and 1 EOB symbol) the average code length is strictly less than 9. Hence the probability of decoding code longer than 10 bits is quite small (usually < 0.2). lbzip2 utilises this fact by implementing a hybrid algorithm for prefix decoding. For codes of length <= 10 lbzip2 maintains a LUT (look-up table) that maps codes directly to corresponding symbol values. Codes longer than 10 bits are not mapped by the LUT are decoded using cannonical prefix decoding algorithm. The above value of 10 bits was determined using a series of benchmarks. It's not hardcoded but instead it is defined as a constant HUFF_START_WIDTH (see the comment above). If on some system a different value works better, it can be adjusted freely. */ static void make_tree(struct retriever_internal_state *rs) { unsigned n; /* alphabet size */ const uint8_t *L; /* code lengths */ uint32_t *C; /* code length count; C[0] is a sentinel */ uint64_t *B; /* left-justified base */ uint16_t *P; /* symbols sorted by code length */ uint16_t *S; /* lookup table */ unsigned k; /* current code length */ unsigned s; /* current symbol */ unsigned cum; unsigned code; uint64_t sofar; uint64_t next; uint64_t inc; uint64_t v; /* Initialize constants. */ n = rs->alpha_size; L = rs->code_len; C = rs->tree[rs->t].count; B = rs->tree[rs->t].base; P = rs->tree[rs->t].perm; S = rs->tree[rs->t].start; /* Count symbol lengths. */ for (k = 0; k <= MAX_CODE_LENGTH; k++) C[k] = 0; for (s = 0; s < n; s++) { k = L[s]; C[k]++; } /* Make sure there are no zero-length codes. */ assert(C[0] == 0); /* Check if Kraft's inequality is satisfied. */ sofar = 0; for (k = MIN_CODE_LENGTH; k <= MAX_CODE_LENGTH; k++) sofar += (uint64_t)C[k] << (MAX_CODE_LENGTH - k); if (sofar != (1 << MAX_CODE_LENGTH)) { rs->mtf[rs->t] = sofar < (1 << MAX_CODE_LENGTH) ? ERR_INCOMPLT : ERR_PREFIX; return; } /* Create left-justified base table. */ sofar = 0; for (k = MIN_CODE_LENGTH; k <= MAX_CODE_LENGTH; k++) { next = sofar + ((uint64_t)C[k] << (64 - k)); assert(next == 0 || next >= sofar); B[k] = sofar; sofar = next; } /* Ensure that "sofar" has overflowed to zero. */ assert(sofar == 0); /* The last few entries of lj-base may have overflowed to zero, so replace all trailing zeros with the greatest possible 64-bit value (which is greater than the greatest possible left-justified base). */ assert(k == MAX_CODE_LENGTH + 1); k = MAX_CODE_LENGTH; while (C[k] == 0) { assert(k > MIN_CODE_LENGTH); assert(B[k] == 0); B[k--] = -1; } /* Transform counts into indices (cumulative counts). */ cum = 0; for (k = MIN_CODE_LENGTH; k <= MAX_CODE_LENGTH; k++) { uint32_t t1 = C[k]; C[k] = cum; cum += t1; } assert(cum == n); /* Perform counting sort. */ P[C[L[0]]++] = RUN_A; P[C[L[1]]++] = RUN_B; for (s = 2; s < n - 1; s++) P[C[L[s]]++] = s - 1; P[C[L[n - 1]]++] = EOB; /* Create first, complete start entries. */ code = 0; inc = 1 << (HUFF_START_WIDTH - 1); for (k = 1; k <= HUFF_START_WIDTH; k++) { for (s = C[k - 1]; s < C[k]; s++) { uint16_t x = (P[s] << 5) | k; v = code; code += inc; while (v < code) S[v++] = x; } inc >>= 1; } /* Fill remaining, incomplete start entries. */ assert(k == HUFF_START_WIDTH + 1); sofar = (uint64_t)code << (64 - HUFF_START_WIDTH); while (code < (1 << HUFF_START_WIDTH)) { while (sofar >= B[k + 1]) k++; S[code] = k; code++; sofar += (uint64_t)1 << (64 - HUFF_START_WIDTH); } assert(sofar == 0); /* Restore cumulative counts as they were destroyed by the sorting phase. The sentinel wasn't touched, so no need to restore it. */ for (k = MAX_CODE_LENGTH; k > 0; k--) { C[k] = C[k - 1]; } assert(C[0] == 0); /* Valid tables were created successfully. */ rs->mtf[rs->t] = rs->t; } /* The following is a lookup table for determining the position of the first zero bit (starting at the most significant bit) in a 6-bit integer. 0xxxxx... -> 1 10xxxx... -> 2 110xxx... -> 3 1110xx... -> 4 11110x... -> 5 111110... -> 6 111111... -> no zeros (marked as 7) */ static const uint8_t table[64] = { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 6, 7, }; /* Pattern L[] R[] 0xxxxx 1 0 100xxx 3 +1 10100x 5 +2 101010 6 +3 101011 6 +1 10110x 5 0 101110 6 +1 101111 6 -1 110xxx 3 -1 11100x 5 0 111010 6 +1 111011 6 -1 11110x 5 -2 111110 6 -1 111111 6 -3 The actual R[] entries are biased (3 is added). */ static const uint8_t L[64] = { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 3, 3, 5, 5, 6, 6, 5, 5, 6, 6, 3, 3, 3, 3, 3, 3, 3, 3, 5, 5, 6, 6, 5, 5, 6, 6, }; static const uint8_t R[64] = { 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 6, 4, 3, 3, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 4, 2, 1, 1, 2, 0, }; #define DECLARE unsigned w; uint64_t v; const uint32_t *next, *limit, \ *tt_limit; uint32_t *tt #define SAVE() (bs->buff = v, bs->live = w, bs->data = next, \ ds->block_size = tt - ds->tt) #define RESTORE() (v = bs->buff, w = bs->live, next = bs->data, \ limit = bs->limit, tt = ds->tt + ds->block_size, \ tt_limit = ds->tt + MAX_BLOCK_SIZE) /* Make sure that bit buffer v holds at least 32 bits, but no more than 63. If there buffer contains from 0 to 31 bits then an attempt to append next 32 bits is made. If there is not enough input available then current state is saved (including FSM state, which is saved as s) and the function returns. Note that it would be wrong to put more than 63 bits (i.e. 64 bits) in v as a potential value of UINT64_MAX could be misrepresented as plus infinity. */ #define NEED(s) \ { \ if (w < 32u) { \ if (unlikely(next == limit)) { \ SAVE(); \ if (bs->eof) \ return ERR_EOF; \ rs->state = (s); \ return MORE; \ case (s): \ if (unlikely(bs->data == bs->limit)) { \ assert(bs->eof); \ return ERR_EOF; \ } \ RESTORE(); \ assert (w < 32u); \ } \ v |= (uint64_t)ntohl(*next) << (64u - (w += 32u)); \ next++; \ } \ } /* Same as NEED(), but assumes that there is at least one 32-bit word of input available. */ #define NEED_FAST() \ { \ if (w < 32u) { \ v |= (uint64_t)ntohl (*next) << (64u - (w += 32u)); \ next++; \ } \ } /* Return k most significant bits of bit buffer v. */ #define PEEK(k) (v >> (64u - (k))) /* Remove k most significant bits of bit buffer v. */ #define DUMP(k) (v <<= (k), w -= (k), (void)0) /* Remove and return k most significant bits of bit buffer v. */ #define TAKE(x,k) ((x) = PEEK(k), DUMP(k)) /* Implementation of Sliding Lists algorithm for doing Inverse Move-To-Front (IMTF) transformation in O(n) space and amortized O(sqrt(n)) time. The naive IMTF algorithm does the same in both O(n) space and time. IMTF could be done in O(log(n)) time using algorithms based on (quite) complex data structures such as self-balancing binary search trees, but these algorithms have quite big constant factor which makes them impractical for MTF of 256 items. */ static uint8_t mtf_one(uint8_t **imtf_row, uint8_t *imtf_slide, uint8_t c) { uint8_t *pp; /* We expect the index to be small, so we have a special case for that. */ if (likely(c < ROW_WIDTH)) { unsigned nn = c; pp = imtf_row[0]; c = pp[nn]; /* Forgive me the ugliness of this code, but mtf_one() is executed frequently and needs to be fast. An equivalent (simpler and slower) version is given in #else clause. */ #if ROW_WIDTH == 16 switch (nn) { default: abort(); #define R(n) case n: pp[n] = pp[n-1] R(15); R(14); R(13); R(12); R(11); R(10); R(9); R(8); R(7); R(6); R(5); R(4); R(3); R(2); R(1); #undef R } #else while (nn > 0) { pp[nn] = pp[nn - 1]; nn--; } #endif } else { /* A general case for indices >= ROW_WIDTH. */ /* If the sliding list already reached the bottom of memory pool allocated for it, we need to rebuild it. */ if (unlikely(imtf_row[0] == imtf_slide)) { uint8_t *kk = imtf_slide + SLIDE_LENGTH; uint8_t **rr = imtf_row + NUM_ROWS; while (rr > imtf_row) { uint8_t *bg = *--rr; uint8_t *bb = bg + ROW_WIDTH; assert(bg >= imtf_slide && bb <= imtf_slide + SLIDE_LENGTH); while (bb > bg) *--kk = *--bb; *rr = kk; } } { uint8_t **lno = imtf_row + c / ROW_WIDTH; uint8_t *bb = *lno; pp = bb + c % ROW_WIDTH; c = *pp; while (pp > bb) { uint8_t *tt = pp--; *tt = *pp; } while (lno > imtf_row) { uint8_t **lno1 = lno; pp = --(*--lno); **lno1 = pp[ROW_WIDTH]; } } } *pp = c; return c; } int retrieve(struct decoder_state *restrict ds, struct bitstream *bs) { struct retriever_internal_state *restrict rs = ds->internal_state; DECLARE; RESTORE(); switch (rs->state) { case S_INIT: NEED(S_BWT_IDX); TAKE(ds->rand, 1u); TAKE(ds->bwt_idx, 24u); /* Retrieve bitmap. */ NEED(S_BITMAP_BIG); TAKE(rs->big, 16u); rs->small = 0; rs->alpha_size = 0u; rs->j = 0; do { if (rs->big & 0x8000) { TAKE(rs->small, 16u); NEED(S_BITMAP_SMALL); } do { rs->imtf_slide[CMAP_BASE + rs->alpha_size] = rs->j++; rs->alpha_size += rs->small >> 15; rs->small <<= 1; } while (rs->j & 0xF); rs->big <<= 1; } while (rs->j < 256u); if (rs->alpha_size == 0) return ERR_BITMAP; rs->alpha_size += 2u; /* -1 MTFV, +2 RUN, +1 EOB */ TAKE(rs->num_trees, 3u); if (rs->num_trees < MIN_TREES || rs->num_trees > MAX_TREES) return ERR_TREES; TAKE(rs->num_selectors, 15u); if (rs->num_selectors == 0) return ERR_GROUPS; /* Retrieve selector MTF values. */ for (rs->j = 0; rs->j < rs->num_selectors; rs->j++) { unsigned k = table[PEEK(6u)]; if (unlikely(k > rs->num_trees)) return ERR_SELECTOR; rs->selector[rs->j] = k - 1u; DUMP(k); NEED(S_SELECTOR_MTF); } /* Retrieve decoding tables. */ for (rs->t = 0; rs->t < rs->num_trees; rs->t++) { rs->j = 0u; TAKE(rs->code_len[0u], 5); while (rs->j < rs->alpha_size) { unsigned k = PEEK(6u); rs->code_len[rs->j] += R[k]; if (unlikely(rs->code_len[rs->j] < 3 + MIN_CODE_LENGTH || rs->code_len[rs->j] > 3 + MAX_CODE_LENGTH)) return ERR_DELTA; rs->code_len[rs->j] -= 3; k = L[k]; if (k != 6u) { rs->j++; if (rs->j < rs->alpha_size) rs->code_len[rs->j] = rs->code_len[rs->j - 1u]; } DUMP(k); NEED(S_DELTA_TAG); } make_tree(rs); } /* Initialize IMTF decoding structure. */ { unsigned i; for (i = 0; i < NUM_ROWS; i++) rs->imtf_row[i] = rs->imtf_slide + CMAP_BASE + i * ROW_WIDTH; } rs->runChar = rs->imtf_row[0][0]; rs->run = 0; rs->shift = 0; /* Initialize IBWT frequency table. */ memset(ds->ftab, 0, sizeof(ds->ftab)); /* Retrieve block MTF values. Block MTF values (MTFV) are prefix-encoded with varying trees. MTFVs are divided into max. 18000 groups, each group contains 50 MTFVs (except the last one, which can contain from 1 to 50 MTFVs). Each group has assigned a prefix-free codebook. As there are up to 6 codebooks, the group's codebook number (called selector) is a value from 0 to 5. A selector of 6 or 7 means oversubscribed or incomplete codebook. If such selector is encountered, decoding is aborted. */ /* Bound selectors at 18001. */ if (rs->num_selectors > 18001) rs->num_selectors = 18001; for (rs->g = 0; rs->g < rs->num_selectors; rs->g++) { unsigned s, x, k, i; /* Select the tree coding this group. */ i = rs->selector[rs->g]; rs->t = rs->mtf[i]; if (unlikely(rs->t >= MAX_TREES)) return rs->t; /* Update IMTF table. */ for (; i > 0; i--) rs->mtf[i] = rs->mtf[i - 1]; rs->mtf[0] = rs->t; /* In one coding group we can have at most 50 codes, 20 bits each, so the largest possible group size is 1000 bits. If there are at least 1000 bits of input available then we can safely assume that the whole group can be decoded without asking for more input. There are two code paths. The first one is executed when there is at least 1024 bits of input available (i.e. 32 words, 32 bits each). In this case we can apply several optimizations, most notably we are allowed to keep state in local variables and we can use NEED_FAST() - faster version of NEED(). The second code path is executed when there is not enough input to for fast decoding. */ if (likely((limit - next) >= 32)) { struct tree *T = &rs->tree[rs->t]; unsigned j; unsigned run = rs->run; unsigned runChar = rs->runChar; unsigned shift = rs->shift; for (j = 0; j < GROUP_SIZE; j++) { NEED_FAST(); x = T->start[PEEK(HUFF_START_WIDTH)]; k = x & 0x1F; if (likely(k <= HUFF_START_WIDTH)) { s = x >> 5; } else { while (v >= T->base[k + 1]) k++; s = T->perm[T->count[k] + ((v - T->base[k]) >> (64 - k))]; } DUMP(k); if (unlikely(IS_EOB(s))) { rs->run = run; rs->runChar = runChar; goto eob; } if (likely(IS_RUN(s) && run <= MAX_BLOCK_SIZE)) { run += RUN(s) << shift++; continue; } if (unlikely(run > tt_limit - tt)) { return ERR_OVERFLOW; } ds->ftab[runChar] += run; while (run-- > 0) { *tt++ = runChar; } runChar = mtf_one(rs->imtf_row, rs->imtf_slide, s); shift = 0; run = 1; } rs->run = run; rs->runChar = runChar; rs->shift = shift; } else { /* There are up to GROUP_SIZE codes in any group. */ for (rs->j = 0; rs->j < GROUP_SIZE; rs->j++) { struct tree *T; NEED(S_PREFIX); T = &rs->tree[rs->t]; x = T->start[PEEK(HUFF_START_WIDTH)]; k = x & 0x1F; if (likely(k <= HUFF_START_WIDTH)) { /* Use look-up table in average case. */ s = x >> 5; } else { /* Code length exceeds HUFF_START_WIDTH, use canonical prefix decoding algorithm instead of look-up table. */ while (v >= T->base[k + 1]) k++; s = T->perm[T->count[k] + ((v - T->base[k]) >> (64 - k))]; } DUMP(k); if (unlikely(IS_EOB(s))) { eob: if (unlikely(rs->run > tt_limit - tt)) return ERR_OVERFLOW; ds->ftab[rs->runChar] += rs->run; while (rs->run--) { *tt++ = rs->runChar; } SAVE(); /* Sanity-check the BWT primary index. */ if (ds->block_size == 0) return ERR_EMPTY; if (ds->bwt_idx >= ds->block_size) return ERR_BWTIDX; free(ds->internal_state); ds->internal_state = NULL; return OK; } /* If we decoded a RLE symbol, increase run length and keep going. However, we need to stop accepting RLE symbols if the run gets too long. Note that rejecting further RLE symbols after the run has reached the length of 900k bytes is perfectly correct because runs longer than 900k bytes will cause block overflow anyways and hence stop decoding with an error. */ if (likely(IS_RUN(s) && rs->run <= MAX_BLOCK_SIZE)) { rs->run += RUN(s) << rs->shift++; continue; } /* At this point we most likely have a run of one or more bytes. Zero-length run is possible only at the beginning, once per block, so any optimization involving zero-length runs are pointless. */ if (unlikely(rs->run > tt_limit - tt)) { return ERR_OVERFLOW; } /* Dump the run. */ ds->ftab[rs->runChar] += rs->run; while (rs->run-- > 0) { *tt++ = rs->runChar; } rs->runChar = mtf_one(rs->imtf_row, rs->imtf_slide, s); rs->shift = 0; rs->run = 1; } } } return ERR_UNTERM; default: abort(); } } /*== IBWT / IMTF ==*/ /* Block size threshold above which block randomization has any effect. Randomizing blocks of size <= RAND_THRESH is a no-op. */ #define RAND_THRESH 617u /* A table filled with arbitrary numbers, in range 50-999, used for derandomizing randomized blocks. These numbers are strictly related to the bzip2 file format and they are not subject to change. */ static const uint16_t rand_table[512] = { 619, 720, 127, 481, 931, 816, 813, 233, 566, 247, 985, 724, 205, 454, 863, 491, 741, 242, 949, 214, 733, 859, 335, 708, 621, 574, +73, 654, 730, 472, 419, 436, 278, 496, 867, 210, 399, 680, 480, +51, 878, 465, 811, 169, 869, 675, 611, 697, 867, 561, 862, 687, 507, 283, 482, 129, 807, 591, 733, 623, 150, 238, +59, 379, 684, 877, 625, 169, 643, 105, 170, 607, 520, 932, 727, 476, 693, 425, 174, 647, +73, 122, 335, 530, 442, 853, 695, 249, 445, 515, 909, 545, 703, 919, 874, 474, 882, 500, 594, 612, 641, 801, 220, 162, 819, 984, 589, 513, 495, 799, 161, 604, 958, 533, 221, 400, 386, 867, 600, 782, 382, 596, 414, 171, 516, 375, 682, 485, 911, 276, +98, 553, 163, 354, 666, 933, 424, 341, 533, 870, 227, 730, 475, 186, 263, 647, 537, 686, 600, 224, 469, +68, 770, 919, 190, 373, 294, 822, 808, 206, 184, 943, 795, 384, 383, 461, 404, 758, 839, 887, 715, +67, 618, 276, 204, 918, 873, 777, 604, 560, 951, 160, 578, 722, +79, 804, +96, 409, 713, 940, 652, 934, 970, 447, 318, 353, 859, 672, 112, 785, 645, 863, 803, 350, 139, +93, 354, +99, 820, 908, 609, 772, 154, 274, 580, 184, +79, 626, 630, 742, 653, 282, 762, 623, 680, +81, 927, 626, 789, 125, 411, 521, 938, 300, 821, +78, 343, 175, 128, 250, 170, 774, 972, 275, 999, 639, 495, +78, 352, 126, 857, 956, 358, 619, 580, 124, 737, 594, 701, 612, 669, 112, 134, 694, 363, 992, 809, 743, 168, 974, 944, 375, 748, +52, 600, 747, 642, 182, 862, +81, 344, 805, 988, 739, 511, 655, 814, 334, 249, 515, 897, 955, 664, 981, 649, 113, 974, 459, 893, 228, 433, 837, 553, 268, 926, 240, 102, 654, 459, +51, 686, 754, 806, 760, 493, 403, 415, 394, 687, 700, 946, 670, 656, 610, 738, 392, 760, 799, 887, 653, 978, 321, 576, 617, 626, 502, 894, 679, 243, 440, 680, 879, 194, 572, 640, 724, 926, +56, 204, 700, 707, 151, 457, 449, 797, 195, 791, 558, 945, 679, 297, +59, +87, 824, 713, 663, 412, 693, 342, 606, 134, 108, 571, 364, 631, 212, 174, 643, 304, 329, 343, +97, 430, 751, 497, 314, 983, 374, 822, 928, 140, 206, +73, 263, 980, 736, 876, 478, 430, 305, 170, 514, 364, 692, 829, +82, 855, 953, 676, 246, 369, 970, 294, 750, 807, 827, 150, 790, 288, 923, 804, 378, 215, 828, 592, 281, 565, 555, 710, +82, 896, 831, 547, 261, 524, 462, 293, 465, 502, +56, 661, 821, 976, 991, 658, 869, 905, 758, 745, 193, 768, 550, 608, 933, 378, 286, 215, 979, 792, 961, +61, 688, 793, 644, 986, 403, 106, 366, 905, 644, 372, 567, 466, 434, 645, 210, 389, 550, 919, 135, 780, 773, 635, 389, 707, 100, 626, 958, 165, 504, 920, 176, 193, 713, 857, 265, 203, +50, 668, 108, 645, 990, 626, 197, 510, 357, 358, 850, 858, 364, 936, 638, }; void decode(struct decoder_state *ds) { uint32_t i, j = 0; uint32_t cum; uint8_t uc; uint32_t *tt = ds->tt; /* Transform counts into indices (cumulative counts). */ cum = 0; for (i = 0; i < 256; i++) ds->ftab[i] = (cum += ds->ftab[i]) - ds->ftab[i]; assert(cum == ds->block_size); /* Construct the IBWT singly-linked cyclic list. Traversing that list starting at primary index produces the source string. Each list node consists of a pointer to the next node and a character of the source string. Those 2 values are packed into a single 32bit integer. The character is kept in bits 0-7 and the pointer in stored in bits 8-27. Bits 28-31 are unused (always clear). Note: Iff the source string consists of a string repeated k times (eg. ABABAB - the string AB is repeated k=3 times) then this algorithm will construct k independent (not connected), isomorphic lists. */ for (i = 0u; i < ds->block_size; i++) { uc = tt[i]; tt[ds->ftab[uc]] += (i << 8); ds->ftab[uc]++; } assert(ds->ftab[255] == ds->block_size); /* Derandomize the block if necessary. The derandomization algorithm is implemented inefficiently, but the assumption is that randomized blocks are unlikely to be encountered. Most of bzip2 implementations try to avoid randomizing blocks because it usually leads to decreased compression ratio. */ if (unlikely(ds->rand)) { uint8_t *block; /* Allocate a temporary array to hold the block. */ block = XNMALLOC(ds->block_size, uint8_t); /* Copy the IBWT linked list into the temporary array. */ j = tt[ds->bwt_idx]; for (i = 0; i < ds->block_size; i++) { j = tt[j >> 8]; block[i] = j; } /* Derandomize the block. */ i = 0, j = RAND_THRESH; while (j < ds->block_size) { block[j] ^= 1; i = (i + 1) & 0x1FF; j += rand_table[i]; } /* Reform a linked list from the array. */ for (i = 0; i < ds->block_size; i++) tt[i] = ((i + 1) << 8) + block[i]; /* Release the temporary array. */ free(block); } ds->rle_state = 0; ds->rle_crc = -1; ds->rle_index = ds->rand ? 0 : ds->tt[ds->bwt_idx]; ds->rle_avail = ds->block_size; ds->rle_prev = 0; ds->rle_char = 0; } #define M1 0xFFFFFFFFu /* Emit decoded block into buffer buf of size *buf_sz. Buffer size is updated to reflect the remaining space left in the buffer. Returns OK if the block was completely emitted, MORE if more output space is needed to fully emit the block or ERR_RUNLEN if data error was detected (missing run length). */ int emit(struct decoder_state *ds, void *buf, size_t *buf_sz) { uint32_t p; /* IBWT linked list pointer */ uint32_t a; /* available input bytes */ uint32_t s; /* CRC checksum */ uint8_t c; /* current character */ uint8_t d; /* next character */ const uint32_t *t; /* IBWT linked list base address */ uint8_t *b; /* next free byte in output buffer */ uint32_t m; /* number of free output bytes available */ assert(ds); assert(buf); assert(buf_sz && *buf_sz > 0); t = ds->tt; b = buf; m = *buf_sz; s = ds->rle_crc; p = ds->rle_index; a = ds->rle_avail; c = ds->rle_char; d = ds->rle_prev; /*=== UNRLE FINITE STATE AUTOMATON ===*/ /* There are 6 states, numbered from 0 to 5. */ /* Excuse me, but the following is a write-only code. It wasn't written for readability or maintainability, but rather for high efficiency. */ switch (ds->rle_state) { default: abort(); case 1: if (unlikely(!m--)) break; s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (c != d) break; if (unlikely(!a--)) break; c = p = t[p >> 8]; case 2: if (unlikely(!m--)) { ds->rle_state = 2; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (c != d) break; if (unlikely(!a--)) break; c = p = t[p >> 8]; case 3: if (unlikely(!m--)) { ds->rle_state = 3; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (c != d) break; if (unlikely(!a--)) return ERR_RUNLEN; c = p = t[p >> 8]; case 4: if (unlikely(m < c)) { c -= m; while (m--) s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = d)]; ds->rle_state = 4; break; } m -= c; while (c--) s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = d)]; case 0: if (unlikely(!a--)) break; c = p = t[p >> 8]; case 5: if (unlikely(!m--)) { ds->rle_state = 5; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; } if (likely(a != M1 && m != M1)) { for (;;) { if (unlikely(!a--)) break; d = c; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 1; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (likely(c != d)) { if (unlikely(!a--)) break; d = c; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 1; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (likely(c != d)) { if (unlikely(!a--)) break; d = c; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 1; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (likely(c != d)) { if (unlikely(!a--)) break; d = c; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 1; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (c != d) continue; } } } if (unlikely(!a--)) break; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 2; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (c != d) continue; if (unlikely(!a--)) break; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 3; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; if (c != d) continue; if (unlikely(!a--)) return ERR_RUNLEN; if (m < (c = p = t[p >> 8])) { c -= m; while (m--) s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = d)]; ds->rle_state = 4; break; } m -= c; while (c--) s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = d)]; if (unlikely(!a--)) break; c = p = t[p >> 8]; if (unlikely(!m--)) { ds->rle_state = 5; break; } s = (s << 8) ^ crc_table[(s >> 24) ^ (*b++ = c)]; } } /* Exactly one of `a' and `m' is equal to M1. */ assert((a == M1) != (m == M1)); ds->rle_avail = a; if (m == M1) { assert(a != M1); ds->rle_index = p; ds->rle_char = c; ds->rle_prev = d; ds->rle_crc = s; *buf_sz = 0; return MORE; } assert(a == M1); ds->crc = s ^ M1; *buf_sz = m; return OK; } void decoder_init(struct decoder_state *ds) { ds->internal_state = XMALLOC(struct retriever_internal_state); ds->internal_state->state = S_INIT; ds->tt = XNMALLOC(MAX_BLOCK_SIZE, uint32_t); ds->block_size = 0; } void decoder_free(struct decoder_state *ds) { free(ds->tt); free(ds->internal_state); } lbzip2-2.3/src/decode.h000066400000000000000000000044631221755423000147530ustar00rootroot00000000000000/*- decode.h -- low-level decompressor header Copyright (C) 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ struct header { int bs100k; uint32_t crc; }; struct parser_state { int state; int bs100k; int prev_bs100k; uint32_t stored_crc; uint32_t computed_crc; }; struct in_blk; struct bitstream { unsigned live; uint64_t buff; struct in_blk *block; const uint32_t *data; const uint32_t *limit; bool eof; }; struct decoder_state { struct retriever_internal_state *internal_state; bool rand; /* block randomized */ unsigned bwt_idx; /* BWT primary index */ unsigned block_size; /* compressed block size */ uint32_t crc; /* expected block CRC */ uint32_t ftab[256]; /* frequency table used in counting sort */ uint32_t *tt; int rle_state; /* FSA state */ uint32_t rle_crc; /* CRC checksum */ uint32_t rle_index; /* IBWT linked list pointer */ uint32_t rle_avail; /* available input bytes */ uint8_t rle_char; /* current character */ uint8_t rle_prev; /* prevoius character */ }; struct source; extern uint32_t crc_table[256]; void parser_init(struct parser_state *ps, int bs100k); int parse(struct parser_state *ps, struct header *hd, struct bitstream *bs, unsigned *garbage); int scan(struct bitstream *bs, unsigned skip); void decoder_init(struct decoder_state *ds); void decoder_free(struct decoder_state *ds); int retrieve(struct decoder_state *ds, struct bitstream *bs); void decode(struct decoder_state *ds); int verify(struct decoder_state *ds); int emit(struct decoder_state *ds, void *buf, size_t *buf_sz); lbzip2-2.3/src/divbwt.c000066400000000000000000001451561221755423000150270ustar00rootroot00000000000000/*- divbwt.c -- Burrows-Wheeler transformation Copyright (C) 2012 Mikolaj Izdebski Copyright (c) 2012 Yuta Mori This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include "encode.h" /*- Settings -*/ #define SS_INSERTIONSORT_THRESHOLD 8 #define SS_BLOCKSIZE 1024 #define ALPHABET_SIZE 256 /*- Datatypes -*/ typedef uint8_t sauchar_t; typedef int32_t saidx_t; typedef int_fast32_t saint_t; /*- Constants -*/ #define INLINE /* for divsufsort.c */ #define BUCKET_A_SIZE (ALPHABET_SIZE) #define BUCKET_B_SIZE (ALPHABET_SIZE * ALPHABET_SIZE) /* for sssort.c */ #if defined(SS_INSERTIONSORT_THRESHOLD) # if SS_INSERTIONSORT_THRESHOLD < 1 # undef SS_INSERTIONSORT_THRESHOLD # define SS_INSERTIONSORT_THRESHOLD (1) # endif #else # define SS_INSERTIONSORT_THRESHOLD (8) #endif #if defined(SS_BLOCKSIZE) # if SS_BLOCKSIZE < 0 # undef SS_BLOCKSIZE # define SS_BLOCKSIZE (0) # elif 32768 <= SS_BLOCKSIZE # undef SS_BLOCKSIZE # define SS_BLOCKSIZE (32767) # endif #else # define SS_BLOCKSIZE (1024) #endif /* minstacksize = log(SS_BLOCKSIZE) / log(3) * 2 */ #if SS_BLOCKSIZE == 0 # if defined(BUILD_DIVSUFSORT64) # define SS_MISORT_STACKSIZE (96) # else # define SS_MISORT_STACKSIZE (64) # endif #elif SS_BLOCKSIZE <= 4096 # define SS_MISORT_STACKSIZE (16) #else # define SS_MISORT_STACKSIZE (24) #endif #if defined(BUILD_DIVSUFSORT64) # define SS_SMERGE_STACKSIZE (64) #else # define SS_SMERGE_STACKSIZE (32) #endif /* for trsort.c */ #define TR_INSERTIONSORT_THRESHOLD (8) #if defined(BUILD_DIVSUFSORT64) # define TR_STACKSIZE (96) #else # define TR_STACKSIZE (64) #endif /*- Macros -*/ #ifndef SWAP # define SWAP(_a, _b) do { t = (_a); (_a) = (_b); (_b) = t; } while(0) #endif /* SWAP */ #ifndef MIN # define MIN(_a, _b) (((_a) < (_b)) ? (_a) : (_b)) #endif /* MIN */ #define STACK_PUSH(_a, _b, _c, _d)\ do {\ assert(ssize < STACK_SIZE);\ stack[ssize].a = (_a), stack[ssize].b = (_b),\ stack[ssize].c = (_c), stack[ssize++].d = (_d);\ } while(0) #define STACK_PUSH5(_a, _b, _c, _d, _e)\ do {\ assert(ssize < STACK_SIZE);\ stack[ssize].a = (_a), stack[ssize].b = (_b),\ stack[ssize].c = (_c), stack[ssize].d = (_d), stack[ssize++].e = (_e);\ } while(0) #define STACK_POP(_a, _b, _c, _d)\ do {\ assert(0 <= ssize);\ if(ssize == 0) { return; }\ (_a) = stack[--ssize].a, (_b) = stack[ssize].b,\ (_c) = stack[ssize].c, (_d) = stack[ssize].d;\ } while(0) #define STACK_POP5(_a, _b, _c, _d, _e)\ do {\ assert(0 <= ssize);\ if(ssize == 0) { return; }\ (_a) = stack[--ssize].a, (_b) = stack[ssize].b,\ (_c) = stack[ssize].c, (_d) = stack[ssize].d, (_e) = stack[ssize].e;\ } while(0) /* for divsufsort.c */ #define BUCKET_A(_c0) bucket_A[(_c0)] #if ALPHABET_SIZE == 256 #define BUCKET_B(_c0, _c1) (bucket_B[((_c1) << 8) | (_c0)]) #define BUCKET_BSTAR(_c0, _c1) (bucket_B[((_c0) << 8) | (_c1)]) #else #define BUCKET_B(_c0, _c1) (bucket_B[(_c1) * ALPHABET_SIZE + (_c0)]) #define BUCKET_BSTAR(_c0, _c1) (bucket_B[(_c0) * ALPHABET_SIZE + (_c1)]) #endif /* for trsort.c */ #define TR_GETC(_p) (((_p) < (ISAn - ISAd)) ? ISAd[(_p)] : ISA[(ISAd - ISA + (_p)) % (ISAn - ISA)]) /* for sssort.c and trsort.c */ static const saint_t lg_table[256]= { -1,0,1,1,2,2,2,2,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4, 5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7 }; /*---- sssort ----*/ /*- Private Functions -*/ #if (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) static INLINE saint_t ss_ilg(saidx_t n) { #if SS_BLOCKSIZE == 0 # if defined(BUILD_DIVSUFSORT64) return (n >> 32) ? ((n >> 48) ? ((n >> 56) ? 56 + lg_table[(n >> 56) & 0xff] : 48 + lg_table[(n >> 48) & 0xff]) : ((n >> 40) ? 40 + lg_table[(n >> 40) & 0xff] : 32 + lg_table[(n >> 32) & 0xff])) : ((n & 0xffff0000) ? ((n & 0xff000000) ? 24 + lg_table[(n >> 24) & 0xff] : 16 + lg_table[(n >> 16) & 0xff]) : ((n & 0x0000ff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff])); # else return (n & 0xffff0000) ? ((n & 0xff000000) ? 24 + lg_table[(n >> 24) & 0xff] : 16 + lg_table[(n >> 16) & 0xff]) : ((n & 0x0000ff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff]); # endif #elif SS_BLOCKSIZE < 256 return lg_table[n]; #else return (n & 0xff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff]; #endif } #endif /* (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) */ #if SS_BLOCKSIZE != 0 static const saint_t sqq_table[256] = { 0, 16, 22, 27, 32, 35, 39, 42, 45, 48, 50, 53, 55, 57, 59, 61, 64, 65, 67, 69, 71, 73, 75, 76, 78, 80, 81, 83, 84, 86, 87, 89, 90, 91, 93, 94, 96, 97, 98, 99, 101, 102, 103, 104, 106, 107, 108, 109, 110, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 128, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 144, 145, 146, 147, 148, 149, 150, 150, 151, 152, 153, 154, 155, 155, 156, 157, 158, 159, 160, 160, 161, 162, 163, 163, 164, 165, 166, 167, 167, 168, 169, 170, 170, 171, 172, 173, 173, 174, 175, 176, 176, 177, 178, 178, 179, 180, 181, 181, 182, 183, 183, 184, 185, 185, 186, 187, 187, 188, 189, 189, 190, 191, 192, 192, 193, 193, 194, 195, 195, 196, 197, 197, 198, 199, 199, 200, 201, 201, 202, 203, 203, 204, 204, 205, 206, 206, 207, 208, 208, 209, 209, 210, 211, 211, 212, 212, 213, 214, 214, 215, 215, 216, 217, 217, 218, 218, 219, 219, 220, 221, 221, 222, 222, 223, 224, 224, 225, 225, 226, 226, 227, 227, 228, 229, 229, 230, 230, 231, 231, 232, 232, 233, 234, 234, 235, 235, 236, 236, 237, 237, 238, 238, 239, 240, 240, 241, 241, 242, 242, 243, 243, 244, 244, 245, 245, 246, 246, 247, 247, 248, 248, 249, 249, 250, 250, 251, 251, 252, 252, 253, 253, 254, 254, 255 }; static INLINE saidx_t ss_isqrt(saidx_t x) { saidx_t y, e; if(x >= (SS_BLOCKSIZE * SS_BLOCKSIZE)) { return SS_BLOCKSIZE; } e = (x & 0xffff0000) ? ((x & 0xff000000) ? 24 + lg_table[(x >> 24) & 0xff] : 16 + lg_table[(x >> 16) & 0xff]) : ((x & 0x0000ff00) ? 8 + lg_table[(x >> 8) & 0xff] : 0 + lg_table[(x >> 0) & 0xff]); if(e >= 16) { y = sqq_table[x >> ((e - 6) - (e & 1))] << ((e >> 1) - 7); if(e >= 24) { y = (y + 1 + x / y) >> 1; } y = (y + 1 + x / y) >> 1; } else if(e >= 8) { y = (sqq_table[x >> ((e - 6) - (e & 1))] >> (7 - (e >> 1))) + 1; } else { return sqq_table[x] >> 4; } return (x < (y * y)) ? y - 1 : y; } #endif /* SS_BLOCKSIZE != 0 */ /*---------------------------------------------------------------------------*/ /* Compares two suffixes. */ static INLINE saint_t ss_compare(const sauchar_t *T, const saidx_t *p1, const saidx_t *p2, saidx_t depth) { const sauchar_t *U1, *U2, *U1n, *U2n; for(U1 = T + depth + *p1, U2 = T + depth + *p2, U1n = T + *(p1 + 1) + 2, U2n = T + *(p2 + 1) + 2; (U1 < U1n) && (U2 < U2n) && (*U1 == *U2); ++U1, ++U2) { } return U1 < U1n ? (U2 < U2n ? *U1 - *U2 : 1) : (U2 < U2n ? -1 : 0); } static INLINE saint_t ss_compare_last(const sauchar_t *T, const saidx_t *PA, const saidx_t *p1, const saidx_t *p2, saidx_t depth, saidx_t size) { const sauchar_t *U1, *U2, *U1n, *U2n; for(U1 = T + depth + *p1, U2 = T + depth + *p2, U1n = T + size, U2n = T + *(p2 + 1) + 2; (U1 < U1n) && (U2 < U2n) && (*U1 == *U2); ++U1, ++U2) { } if(U1 < U1n) { return (U2 < U2n) ? *U1 - *U2 : 1; } else if(U2 == U2n) { return 1; } for(U1 = T + (U1 - T) % size, U1n = T + PA[0] + 2; (U1 < U1n) && (U2 < U2n) && (*U1 == *U2); ++U1, ++U2) { } return U1 < U1n ? (U2 < U2n ? *U1 - *U2 : 1) : (U2 < U2n ? -1 : 0); } /*---------------------------------------------------------------------------*/ #if (SS_BLOCKSIZE != 1) && (SS_INSERTIONSORT_THRESHOLD != 1) /* Insertionsort for small size groups */ static void ss_insertionsort(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *last, saidx_t depth) { saidx_t *i, *j; saidx_t t; saint_t r; for(i = last - 2; first <= i; --i) { for(t = *i, j = i + 1; 0 < (r = ss_compare(T, PA + t, PA + *j, depth));) { do { *(j - 1) = *j; } while((++j < last) && (*j < 0)); if(last <= j) { break; } } if(r == 0) { *j = ~*j; } *(j - 1) = t; } } #endif /* (SS_BLOCKSIZE != 1) && (SS_INSERTIONSORT_THRESHOLD != 1) */ /*---------------------------------------------------------------------------*/ #if (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) static INLINE void ss_fixdown(const sauchar_t *Td, const saidx_t *PA, saidx_t *SA, saidx_t i, saidx_t size) { saidx_t j, k; saidx_t v; saint_t c, d, e; for(v = SA[i], c = Td[PA[v]]; (j = 2 * i + 1) < size; SA[i] = SA[k], i = k) { d = Td[PA[SA[k = j++]]]; if(d < (e = Td[PA[SA[j]]])) { k = j; d = e; } if(d <= c) { break; } } SA[i] = v; } /* Simple top-down heapsort. */ static void ss_heapsort(const sauchar_t *Td, const saidx_t *PA, saidx_t *SA, saidx_t size) { saidx_t i, m; saidx_t t; m = size; if((size % 2) == 0) { m--; if(Td[PA[SA[m / 2]]] < Td[PA[SA[m]]]) { SWAP(SA[m], SA[m / 2]); } } for(i = m / 2 - 1; 0 <= i; --i) { ss_fixdown(Td, PA, SA, i, m); } if((size % 2) == 0) { SWAP(SA[0], SA[m]); ss_fixdown(Td, PA, SA, 0, m); } for(i = m - 1; 0 < i; --i) { t = SA[0], SA[0] = SA[i]; ss_fixdown(Td, PA, SA, 0, i); SA[i] = t; } } /*---------------------------------------------------------------------------*/ /* Returns the median of three elements. */ static INLINE saidx_t * ss_median3(const sauchar_t *Td, const saidx_t *PA, saidx_t *v1, saidx_t *v2, saidx_t *v3) { saidx_t *t; if(Td[PA[*v1]] > Td[PA[*v2]]) { SWAP(v1, v2); } if(Td[PA[*v2]] > Td[PA[*v3]]) { if(Td[PA[*v1]] > Td[PA[*v3]]) { return v1; } else { return v3; } } return v2; } /* Returns the median of five elements. */ static INLINE saidx_t * ss_median5(const sauchar_t *Td, const saidx_t *PA, saidx_t *v1, saidx_t *v2, saidx_t *v3, saidx_t *v4, saidx_t *v5) { saidx_t *t; if(Td[PA[*v2]] > Td[PA[*v3]]) { SWAP(v2, v3); } if(Td[PA[*v4]] > Td[PA[*v5]]) { SWAP(v4, v5); } if(Td[PA[*v2]] > Td[PA[*v4]]) { SWAP(v2, v4); SWAP(v3, v5); } if(Td[PA[*v1]] > Td[PA[*v3]]) { SWAP(v1, v3); } if(Td[PA[*v1]] > Td[PA[*v4]]) { SWAP(v1, v4); SWAP(v3, v5); } if(Td[PA[*v3]] > Td[PA[*v4]]) { return v4; } return v3; } /* Returns the pivot element. */ static INLINE saidx_t * ss_pivot(const sauchar_t *Td, const saidx_t *PA, saidx_t *first, saidx_t *last) { saidx_t *middle; saidx_t t; t = last - first; middle = first + t / 2; if(t <= 512) { if(t <= 32) { return ss_median3(Td, PA, first, middle, last - 1); } else { t >>= 2; return ss_median5(Td, PA, first, first + t, middle, last - 1 - t, last - 1); } } t >>= 3; first = ss_median3(Td, PA, first, first + t, first + (t << 1)); middle = ss_median3(Td, PA, middle - t, middle, middle + t); last = ss_median3(Td, PA, last - 1 - (t << 1), last - 1 - t, last - 1); return ss_median3(Td, PA, first, middle, last); } /*---------------------------------------------------------------------------*/ /* Binary partition for substrings. */ static INLINE saidx_t * ss_partition(const saidx_t *PA, saidx_t *first, saidx_t *last, saidx_t depth) { saidx_t *a, *b; saidx_t t; for(a = first - 1, b = last;;) { for(; (++a < b) && ((PA[*a] + depth) >= (PA[*a + 1] + 1));) { *a = ~*a; } for(; (a < --b) && ((PA[*b] + depth) < (PA[*b + 1] + 1));) { } if(b <= a) { break; } t = ~*b; *b = *a; *a = t; } if(first < a) { *first = ~*first; } return a; } /* Multikey introsort for medium size groups. */ static void ss_mintrosort(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *last, saidx_t depth) { #define STACK_SIZE SS_MISORT_STACKSIZE struct { saidx_t *a, *b, c; saint_t d; } stack[STACK_SIZE]; const sauchar_t *Td; saidx_t *a, *b, *c, *d, *e, *f; saidx_t s, t; saint_t ssize; saint_t limit; saint_t v, x = 0; for(ssize = 0, limit = ss_ilg(last - first);;) { if((last - first) <= SS_INSERTIONSORT_THRESHOLD) { #if 1 < SS_INSERTIONSORT_THRESHOLD if(1 < (last - first)) { ss_insertionsort(T, PA, first, last, depth); } #endif STACK_POP(first, last, depth, limit); continue; } Td = T + depth; if(limit-- == 0) { ss_heapsort(Td, PA, first, last - first); } if(limit < 0) { for(a = first + 1, v = Td[PA[*first]]; a < last; ++a) { if((x = Td[PA[*a]]) != v) { if(1 < (a - first)) { break; } v = x; first = a; } } if(Td[PA[*first] - 1] < v) { first = ss_partition(PA, first, a, depth); } if((a - first) <= (last - a)) { if(1 < (a - first)) { STACK_PUSH(a, last, depth, -1); last = a, depth += 1, limit = ss_ilg(a - first); } else { first = a, limit = -1; } } else { if(1 < (last - a)) { STACK_PUSH(first, a, depth + 1, ss_ilg(a - first)); first = a, limit = -1; } else { last = a, depth += 1, limit = ss_ilg(a - first); } } continue; } /* choose pivot */ a = ss_pivot(Td, PA, first, last); v = Td[PA[*a]]; SWAP(*first, *a); /* partition */ for(b = first; (++b < last) && ((x = Td[PA[*b]]) == v);) { } if(((a = b) < last) && (x < v)) { for(; (++b < last) && ((x = Td[PA[*b]]) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } } for(c = last; (b < --c) && ((x = Td[PA[*c]]) == v);) { } if((b < (d = c)) && (x > v)) { for(; (b < --c) && ((x = Td[PA[*c]]) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } for(; b < c;) { SWAP(*b, *c); for(; (++b < c) && ((x = Td[PA[*b]]) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } for(; (b < --c) && ((x = Td[PA[*c]]) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } if(a <= d) { c = b - 1; if((s = a - first) > (t = b - a)) { s = t; } for(e = first, f = b - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } if((s = d - c) > (t = last - d - 1)) { s = t; } for(e = b, f = last - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } a = first + (b - a), c = last - (d - c); b = (v <= Td[PA[*a] - 1]) ? a : ss_partition(PA, a, c, depth); if((a - first) <= (last - c)) { if((last - c) <= (c - b)) { STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); STACK_PUSH(c, last, depth, limit); last = a; } else if((a - first) <= (c - b)) { STACK_PUSH(c, last, depth, limit); STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); last = a; } else { STACK_PUSH(c, last, depth, limit); STACK_PUSH(first, a, depth, limit); first = b, last = c, depth += 1, limit = ss_ilg(c - b); } } else { if((a - first) <= (c - b)) { STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); STACK_PUSH(first, a, depth, limit); first = c; } else if((last - c) <= (c - b)) { STACK_PUSH(first, a, depth, limit); STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); first = c; } else { STACK_PUSH(first, a, depth, limit); STACK_PUSH(c, last, depth, limit); first = b, last = c, depth += 1, limit = ss_ilg(c - b); } } } else { limit += 1; if(Td[PA[*first] - 1] < v) { first = ss_partition(PA, first, last, depth); limit = ss_ilg(last - first); } depth += 1; } } #undef STACK_SIZE } #endif /* (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) */ /*---------------------------------------------------------------------------*/ #if SS_BLOCKSIZE != 0 static INLINE void ss_blockswap(saidx_t *a, saidx_t *b, saidx_t n) { saidx_t t; for(; 0 < n; --n, ++a, ++b) { t = *a, *a = *b, *b = t; } } static INLINE void ss_rotate(saidx_t *first, saidx_t *middle, saidx_t *last) { saidx_t *a, *b, t; saidx_t l, r; l = middle - first, r = last - middle; for(; (0 < l) && (0 < r);) { if(l == r) { ss_blockswap(first, middle, l); break; } if(l < r) { a = last - 1, b = middle - 1; t = *a; do { *a-- = *b, *b-- = *a; if(b < first) { *a = t; last = a; if((r -= l + 1) <= l) { break; } a -= 1, b = middle - 1; t = *a; } } while(1); } else { a = first, b = middle; t = *a; do { *a++ = *b, *b++ = *a; if(last <= b) { *a = t; first = a + 1; if((l -= r + 1) <= r) { break; } a += 1, b = middle; t = *a; } } while(1); } } } /*---------------------------------------------------------------------------*/ static void ss_inplacemerge(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *middle, saidx_t *last, saidx_t depth) { const saidx_t *p; saidx_t *a, *b; saidx_t len, half; saint_t q, r; saint_t x; for(;;) { if(*(last - 1) < 0) { x = 1; p = PA + ~*(last - 1); } else { x = 0; p = PA + *(last - 1); } for(a = first, len = middle - first, half = len >> 1, r = -1; 0 < len; len = half, half >>= 1) { b = a + half; q = ss_compare(T, PA + ((0 <= *b) ? *b : ~*b), p, depth); if(q < 0) { a = b + 1; half -= (len & 1) ^ 1; } else { r = q; } } if(a < middle) { if(r == 0) { *a = ~*a; } ss_rotate(a, middle, last); last -= middle - a; middle = a; if(first == middle) { break; } } --last; if(x != 0) { while(*--last < 0) { } } if(middle == last) { break; } } } /*---------------------------------------------------------------------------*/ /* Merge-forward with internal buffer. */ static void ss_mergeforward(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *middle, saidx_t *last, saidx_t *buf, saidx_t depth) { saidx_t *a, *b, *c, *bufend; saidx_t t; saint_t r; bufend = buf + (middle - first) - 1; ss_blockswap(buf, first, middle - first); for(t = *(a = first), b = buf, c = middle;;) { r = ss_compare(T, PA + *b, PA + *c, depth); if(r < 0) { do { *a++ = *b; if(bufend <= b) { *bufend = t; return; } *b++ = *a; } while(*b < 0); } else if(r > 0) { do { *a++ = *c, *c++ = *a; if(last <= c) { while(b < bufend) { *a++ = *b, *b++ = *a; } *a = *b, *b = t; return; } } while(*c < 0); } else { *c = ~*c; do { *a++ = *b; if(bufend <= b) { *bufend = t; return; } *b++ = *a; } while(*b < 0); do { *a++ = *c, *c++ = *a; if(last <= c) { while(b < bufend) { *a++ = *b, *b++ = *a; } *a = *b, *b = t; return; } } while(*c < 0); } } } /* Merge-backward with internal buffer. */ static void ss_mergebackward(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *middle, saidx_t *last, saidx_t *buf, saidx_t depth) { const saidx_t *p1, *p2; saidx_t *a, *b, *c, *bufend; saidx_t t; saint_t r; saint_t x; bufend = buf + (last - middle) - 1; ss_blockswap(buf, middle, last - middle); x = 0; if(*bufend < 0) { p1 = PA + ~*bufend; x |= 1; } else { p1 = PA + *bufend; } if(*(middle - 1) < 0) { p2 = PA + ~*(middle - 1); x |= 2; } else { p2 = PA + *(middle - 1); } for(t = *(a = last - 1), b = bufend, c = middle - 1;;) { r = ss_compare(T, p1, p2, depth); if(0 < r) { if(x & 1) { do { *a-- = *b, *b-- = *a; } while(*b < 0); x ^= 1; } *a-- = *b; if(b <= buf) { *buf = t; break; } *b-- = *a; if(*b < 0) { p1 = PA + ~*b; x |= 1; } else { p1 = PA + *b; } } else if(r < 0) { if(x & 2) { do { *a-- = *c, *c-- = *a; } while(*c < 0); x ^= 2; } *a-- = *c, *c-- = *a; if(c < first) { while(buf < b) { *a-- = *b, *b-- = *a; } *a = *b, *b = t; break; } if(*c < 0) { p2 = PA + ~*c; x |= 2; } else { p2 = PA + *c; } } else { if(x & 1) { do { *a-- = *b, *b-- = *a; } while(*b < 0); x ^= 1; } *a-- = ~*b; if(b <= buf) { *buf = t; break; } *b-- = *a; if(x & 2) { do { *a-- = *c, *c-- = *a; } while(*c < 0); x ^= 2; } *a-- = *c, *c-- = *a; if(c < first) { while(buf < b) { *a-- = *b, *b-- = *a; } *a = *b, *b = t; break; } if(*b < 0) { p1 = PA + ~*b; x |= 1; } else { p1 = PA + *b; } if(*c < 0) { p2 = PA + ~*c; x |= 2; } else { p2 = PA + *c; } } } } /* D&C based merge. */ static void ss_swapmerge(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *middle, saidx_t *last, saidx_t *buf, saidx_t bufsize, saidx_t depth) { #define STACK_SIZE SS_SMERGE_STACKSIZE #define GETIDX(a) ((0 <= (a)) ? (a) : (~(a))) #define MERGE_CHECK(a, b, c)\ do {\ if(((c) & 1) ||\ (((c) & 2) && (ss_compare(T, PA + GETIDX(*((a) - 1)), PA + *(a), depth) == 0))) {\ *(a) = ~*(a);\ }\ if(((c) & 4) && ((ss_compare(T, PA + GETIDX(*((b) - 1)), PA + *(b), depth) == 0))) {\ *(b) = ~*(b);\ }\ } while(0) struct { saidx_t *a, *b, *c; saint_t d; } stack[STACK_SIZE]; saidx_t *l, *r, *lm, *rm; saidx_t m, len, half; saint_t ssize; saint_t check, next; for(check = 0, ssize = 0;;) { if((last - middle) <= bufsize) { if((first < middle) && (middle < last)) { ss_mergebackward(T, PA, first, middle, last, buf, depth); } MERGE_CHECK(first, last, check); STACK_POP(first, middle, last, check); continue; } if((middle - first) <= bufsize) { if(first < middle) { ss_mergeforward(T, PA, first, middle, last, buf, depth); } MERGE_CHECK(first, last, check); STACK_POP(first, middle, last, check); continue; } for(m = 0, len = MIN(middle - first, last - middle), half = len >> 1; 0 < len; len = half, half >>= 1) { if(ss_compare(T, PA + GETIDX(*(middle + m + half)), PA + GETIDX(*(middle - m - half - 1)), depth) < 0) { m += half + 1; half -= (len & 1) ^ 1; } } if(0 < m) { lm = middle - m, rm = middle + m; ss_blockswap(lm, middle, m); l = r = middle, next = 0; if(rm < last) { if(*rm < 0) { *rm = ~*rm; if(first < lm) { for(; *--l < 0;) { } next |= 4; } next |= 1; } else if(first < lm) { for(; *r < 0; ++r) { } next |= 2; } } if((l - first) <= (last - r)) { STACK_PUSH(r, rm, last, (next & 3) | (check & 4)); middle = lm, last = l, check = (check & 3) | (next & 4); } else { if((next & 2) && (r == middle)) { next ^= 6; } STACK_PUSH(first, lm, l, (check & 3) | (next & 4)); first = r, middle = rm, check = (next & 3) | (check & 4); } } else { if(ss_compare(T, PA + GETIDX(*(middle - 1)), PA + *middle, depth) == 0) { *middle = ~*middle; } MERGE_CHECK(first, last, check); STACK_POP(first, middle, last, check); } } #undef STACK_SIZE } #endif /* SS_BLOCKSIZE != 0 */ /*---------------------------------------------------------------------------*/ /*- Function -*/ /* Substring sort */ static void sssort(const sauchar_t *T, const saidx_t *PA, saidx_t *first, saidx_t *last, saidx_t *buf, saidx_t bufsize, saidx_t depth, saidx_t n, saint_t lastsuffix) { saidx_t *a; #if SS_BLOCKSIZE != 0 saidx_t *b, *middle, *curbuf; saidx_t j, k, curbufsize, limit; #endif saidx_t i; if(lastsuffix != 0) { ++first; } #if SS_BLOCKSIZE == 0 ss_mintrosort(T, PA, first, last, depth); #else if((bufsize < SS_BLOCKSIZE) && (bufsize < (last - first)) && (bufsize < (limit = ss_isqrt(last - first)))) { if(SS_BLOCKSIZE < limit) { limit = SS_BLOCKSIZE; } buf = middle = last - limit, bufsize = limit; } else { middle = last, limit = 0; } for(a = first, i = 0; SS_BLOCKSIZE < (middle - a); a += SS_BLOCKSIZE, ++i) { #if SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE ss_mintrosort(T, PA, a, a + SS_BLOCKSIZE, depth); #elif 1 < SS_BLOCKSIZE ss_insertionsort(T, PA, a, a + SS_BLOCKSIZE, depth); #endif curbufsize = last - (a + SS_BLOCKSIZE); curbuf = a + SS_BLOCKSIZE; if(curbufsize <= bufsize) { curbufsize = bufsize, curbuf = buf; } for(b = a, k = SS_BLOCKSIZE, j = i; j & 1; b -= k, k <<= 1, j >>= 1) { ss_swapmerge(T, PA, b - k, b, b + k, curbuf, curbufsize, depth); } } #if SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE ss_mintrosort(T, PA, a, middle, depth); #elif 1 < SS_BLOCKSIZE ss_insertionsort(T, PA, a, middle, depth); #endif for(k = SS_BLOCKSIZE; i != 0; k <<= 1, i >>= 1) { if(i & 1) { ss_swapmerge(T, PA, a - k, a, middle, buf, bufsize, depth); a -= k; } } if(limit != 0) { #if SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE ss_mintrosort(T, PA, middle, last, depth); #elif 1 < SS_BLOCKSIZE ss_insertionsort(T, PA, middle, last, depth); #endif ss_inplacemerge(T, PA, first, middle, last, depth); } #endif if(lastsuffix != 0) { /* Insert last type B* suffix. */ saint_t r; for(a = first, i = *(first - 1), r = 1; (a < last) && ((*a < 0) || (0 < (r = ss_compare_last(T, PA, PA + i, PA + *a, depth, n)))); ++a) { *(a - 1) = *a; } if(r == 0) { *a = ~*a; } *(a - 1) = i; } } /*---- trsort ----*/ /*- Private Functions -*/ static INLINE saint_t tr_ilg(saidx_t n) { #if defined(BUILD_DIVSUFSORT64) return (n >> 32) ? ((n >> 48) ? ((n >> 56) ? 56 + lg_table[(n >> 56) & 0xff] : 48 + lg_table[(n >> 48) & 0xff]) : ((n >> 40) ? 40 + lg_table[(n >> 40) & 0xff] : 32 + lg_table[(n >> 32) & 0xff])) : ((n & 0xffff0000) ? ((n & 0xff000000) ? 24 + lg_table[(n >> 24) & 0xff] : 16 + lg_table[(n >> 16) & 0xff]) : ((n & 0x0000ff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff])); #else return (n & 0xffff0000) ? ((n & 0xff000000) ? 24 + lg_table[(n >> 24) & 0xff] : 16 + lg_table[(n >> 16) & 0xff]) : ((n & 0x0000ff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff]); #endif } /*---------------------------------------------------------------------------*/ /* Simple insertionsort for small size groups. */ static void tr_insertionsort(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *first, saidx_t *last) { saidx_t *a, *b; saidx_t t, r; for(a = first + 1; a < last; ++a) { for(t = *a, b = a - 1; 0 > (r = TR_GETC(t) - TR_GETC(*b));) { do { *(b + 1) = *b; } while((first <= --b) && (*b < 0)); if(b < first) { break; } } if(r == 0) { *b = ~*b; } *(b + 1) = t; } } /*---------------------------------------------------------------------------*/ static INLINE void tr_fixdown(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *SA, saidx_t i, saidx_t size) { saidx_t j, k; saidx_t v; saidx_t c, d, e; for(v = SA[i], c = TR_GETC(v); (j = 2 * i + 1) < size; SA[i] = SA[k], i = k) { k = j++; d = TR_GETC(SA[k]); if(d < (e = TR_GETC(SA[j]))) { k = j; d = e; } if(d <= c) { break; } } SA[i] = v; } /* Simple top-down heapsort. */ static void tr_heapsort(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *SA, saidx_t size) { saidx_t i, m; saidx_t t; m = size; if((size % 2) == 0) { m--; if(TR_GETC(SA[m / 2]) < TR_GETC(SA[m])) { SWAP(SA[m], SA[m / 2]); } } for(i = m / 2 - 1; 0 <= i; --i) { tr_fixdown(ISA, ISAd, ISAn, SA, i, m); } if((size % 2) == 0) { SWAP(SA[0], SA[m]); tr_fixdown(ISA, ISAd, ISAn, SA, 0, m); } for(i = m - 1; 0 < i; --i) { t = SA[0], SA[0] = SA[i]; tr_fixdown(ISA, ISAd, ISAn, SA, 0, i); SA[i] = t; } } /*---------------------------------------------------------------------------*/ /* Returns the median of three elements. */ static INLINE saidx_t * tr_median3(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *v1, saidx_t *v2, saidx_t *v3) { saidx_t *t; if(TR_GETC(*v1) > TR_GETC(*v2)) { SWAP(v1, v2); } if(TR_GETC(*v2) > TR_GETC(*v3)) { if(TR_GETC(*v1) > TR_GETC(*v3)) { return v1; } else { return v3; } } return v2; } /* Returns the median of five elements. */ static INLINE saidx_t * tr_median5(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *v1, saidx_t *v2, saidx_t *v3, saidx_t *v4, saidx_t *v5) { saidx_t *t; if(TR_GETC(*v2) > TR_GETC(*v3)) { SWAP(v2, v3); } if(TR_GETC(*v4) > TR_GETC(*v5)) { SWAP(v4, v5); } if(TR_GETC(*v2) > TR_GETC(*v4)) { SWAP(v2, v4); SWAP(v3, v5); } if(TR_GETC(*v1) > TR_GETC(*v3)) { SWAP(v1, v3); } if(TR_GETC(*v1) > TR_GETC(*v4)) { SWAP(v1, v4); SWAP(v3, v5); } if(TR_GETC(*v3) > TR_GETC(*v4)) { return v4; } return v3; } /* Returns the pivot element. */ static INLINE saidx_t * tr_pivot(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *first, saidx_t *last) { saidx_t *middle; saidx_t t; t = last - first; middle = first + t / 2; if(t <= 512) { if(t <= 32) { return tr_median3(ISA, ISAd, ISAn, first, middle, last - 1); } else { t >>= 2; return tr_median5(ISA, ISAd, ISAn, first, first + t, middle, last - 1 - t, last - 1); } } t >>= 3; first = tr_median3(ISA, ISAd, ISAn, first, first + t, first + (t << 1)); middle = tr_median3(ISA, ISAd, ISAn, middle - t, middle, middle + t); last = tr_median3(ISA, ISAd, ISAn, last - 1 - (t << 1), last - 1 - t, last - 1); return tr_median3(ISA, ISAd, ISAn, first, middle, last); } /*---------------------------------------------------------------------------*/ typedef struct _trbudget_t trbudget_t; struct _trbudget_t { saidx_t chance; saidx_t remain; saidx_t incval; saidx_t count; }; static INLINE void trbudget_init(trbudget_t *budget, saidx_t chance, saidx_t incval) { budget->chance = chance; budget->remain = budget->incval = incval; } static INLINE saint_t trbudget_check(trbudget_t *budget, saidx_t size) { if(size <= budget->remain) { budget->remain -= size; return 1; } if(budget->chance == 0) { budget->count += size; return 0; } budget->remain += budget->incval - size; budget->chance -= 1; return 1; } /*---------------------------------------------------------------------------*/ static INLINE void tr_partition(const saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *first, saidx_t *middle, saidx_t *last, saidx_t **pa, saidx_t **pb, saidx_t v) { saidx_t *a, *b, *c, *d, *e, *f; saidx_t t, s; saidx_t x = 0; for(b = middle - 1; (++b < last) && ((x = TR_GETC(*b)) == v);) { } if(((a = b) < last) && (x < v)) { for(; (++b < last) && ((x = TR_GETC(*b)) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } } for(c = last; (b < --c) && ((x = TR_GETC(*c)) == v);) { } if((b < (d = c)) && (x > v)) { for(; (b < --c) && ((x = TR_GETC(*c)) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } for(; b < c;) { SWAP(*b, *c); for(; (++b < c) && ((x = TR_GETC(*b)) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } for(; (b < --c) && ((x = TR_GETC(*c)) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } if(a <= d) { c = b - 1; if((s = a - first) > (t = b - a)) { s = t; } for(e = first, f = b - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } if((s = d - c) > (t = last - d - 1)) { s = t; } for(e = b, f = last - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } first += (b - a), last -= (d - c); } *pa = first, *pb = last; } static void tr_copy(saidx_t *ISA, const saidx_t *ISAn, const saidx_t *SA, saidx_t *first, saidx_t *a, saidx_t *b, saidx_t *last, saidx_t depth) { /* sort suffixes of middle partition by using sorted order of suffixes of left and right partition. */ saidx_t *c, *d, *e; saidx_t s, v; v = b - SA - 1; for(c = first, d = a - 1; c <= d; ++c) { if((s = *c - depth) < 0) { s += ISAn - ISA; } if(ISA[s] == v) { *++d = s; ISA[s] = d - SA; } } for(c = last - 1, e = d + 1, d = b; e < d; --c) { if((s = *c - depth) < 0) { s += ISAn - ISA; } if(ISA[s] == v) { *--d = s; ISA[s] = d - SA; } } } static void tr_partialcopy(saidx_t *ISA, const saidx_t *ISAn, const saidx_t *SA, saidx_t *first, saidx_t *a, saidx_t *b, saidx_t *last, saidx_t depth) { saidx_t *c, *d, *e; saidx_t s, v, t; saidx_t rank, lastrank, newrank = -1; v = b - SA - 1; lastrank = -1; for(c = first, d = a - 1; c <= d; ++c) { t = *c; if((s = t - depth) < 0) { s += ISAn - ISA; } if(ISA[s] == v) { *++d = s; rank = ISA[t]; if(lastrank != rank) { lastrank = rank; newrank = d - SA; } ISA[s] = newrank; } } lastrank = -1; for(e = d; first <= e; --e) { rank = ISA[*e]; if(lastrank != rank) { lastrank = rank; newrank = e - SA; } if(newrank != rank) { ISA[*e] = newrank; } } lastrank = -1; for(c = last - 1, e = d + 1, d = b; e < d; --c) { t = *c; if((s = t - depth) < 0) { s += ISAn - ISA; } if(ISA[s] == v) { *--d = s; rank = ISA[t]; if(lastrank != rank) { lastrank = rank; newrank = d - SA; } ISA[s] = newrank; } } } static void tr_introsort(saidx_t *ISA, const saidx_t *ISAd, const saidx_t *ISAn, saidx_t *SA, saidx_t *first, saidx_t *last, trbudget_t *budget) { #define STACK_SIZE TR_STACKSIZE struct { const saidx_t *a; saidx_t *b, *c; saint_t d, e; }stack[STACK_SIZE]; saidx_t *a, *b, *c; saidx_t t; saidx_t v, x = 0; saidx_t incr = ISAd - ISA; saint_t limit, next; saint_t ssize, trlink = -1; for(ssize = 0, limit = tr_ilg(last - first);;) { assert((ISAd < ISAn) || (limit == -3)); if(limit < 0) { if(limit == -1) { /* tandem repeat partition */ tr_partition(ISA, ISAd - incr, ISAn, first, first, last, &a, &b, last - SA - 1); /* update ranks */ if(a < last) { for(c = first, v = a - SA - 1; c < a; ++c) { ISA[*c] = v; } } if(b < last) { for(c = a, v = b - SA - 1; c < b; ++c) { ISA[*c] = v; } } /* push */ if(1 < (b - a)) { STACK_PUSH5(NULL, a, b, 0, 0); STACK_PUSH5(ISAd - incr, first, last, -2, trlink); trlink = ssize - 2; } if((a - first) <= (last - b)) { if(1 < (a - first)) { STACK_PUSH5(ISAd, b, last, tr_ilg(last - b), trlink); last = a, limit = tr_ilg(a - first); } else if(1 < (last - b)) { first = b, limit = tr_ilg(last - b); } else { STACK_POP5(ISAd, first, last, limit, trlink); } } else { if(1 < (last - b)) { STACK_PUSH5(ISAd, first, a, tr_ilg(a - first), trlink); first = b, limit = tr_ilg(last - b); } else if(1 < (a - first)) { last = a, limit = tr_ilg(a - first); } else { STACK_POP5(ISAd, first, last, limit, trlink); } } } else if(limit == -2) { /* tandem repeat copy */ a = stack[--ssize].b, b = stack[ssize].c; if(stack[ssize].d == 0) { tr_copy(ISA, ISAn, SA, first, a, b, last, ISAd - ISA); } else { if(0 <= trlink) { stack[trlink].d = -1; } tr_partialcopy(ISA, ISAn, SA, first, a, b, last, ISAd - ISA); } STACK_POP5(ISAd, first, last, limit, trlink); } else { /* sorted partition */ if(0 <= *first) { a = first; do { ISA[*a] = a - SA; } while((++a < last) && (0 <= *a)); first = a; } if(first < last) { a = first; do { *a = ~*a; } while(*++a < 0); next = (incr < (ISAn - ISAd)) ? ((ISA[*a] != TR_GETC(*a)) ? tr_ilg(a - first + 1) : -1) : -3; if(++a < last) { for(b = first, v = a - SA - 1; b < a; ++b) { ISA[*b] = v; } } /* push */ if(trbudget_check(budget, a - first)) { if((a - first) <= (last - a)) { STACK_PUSH5(ISAd, a, last, -3, trlink); ISAd += incr, last = a, limit = next; } else { if(1 < (last - a)) { STACK_PUSH5(ISAd + incr, first, a, next, trlink); first = a, limit = -3; } else { ISAd += incr, last = a, limit = next; } } } else { if(0 <= trlink) { stack[trlink].d = -1; } if(1 < (last - a)) { first = a, limit = -3; } else { STACK_POP5(ISAd, first, last, limit, trlink); } } } else { STACK_POP5(ISAd, first, last, limit, trlink); } } continue; } if((last - first) <= TR_INSERTIONSORT_THRESHOLD) { tr_insertionsort(ISA, ISAd, ISAn, first, last); limit = -3; continue; } if(limit-- == 0) { tr_heapsort(ISA, ISAd, ISAn, first, last - first); for(a = last - 1; first < a; a = b) { for(x = TR_GETC(*a), b = a - 1; (first <= b) && (TR_GETC(*b) == x); --b) { *b = ~*b; } } limit = -3; continue; } /* choose pivot */ a = tr_pivot(ISA, ISAd, ISAn, first, last); SWAP(*first, *a); v = TR_GETC(*first); /* partition */ tr_partition(ISA, ISAd, ISAn, first, first + 1, last, &a, &b, v); if((last - first) != (b - a)) { next = (incr < (ISAn - ISAd)) ? ((ISA[*a] != v) ? tr_ilg(b - a) : -1) : -3; /* update ranks */ for(c = first, v = a - SA - 1; c < a; ++c) { ISA[*c] = v; } if(b < last) { for(c = a, v = b - SA - 1; c < b; ++c) { ISA[*c] = v; } } /* push */ if((1 < (b - a)) && (trbudget_check(budget, b - a))) { if((a - first) <= (last - b)) { if((last - b) <= (b - a)) { if(1 < (a - first)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); STACK_PUSH5(ISAd, b, last, limit, trlink); last = a; } else if(1 < (last - b)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); first = b; } else { ISAd += incr, first = a, last = b, limit = next; } } else if((a - first) <= (b - a)) { if(1 < (a - first)) { STACK_PUSH5(ISAd, b, last, limit, trlink); STACK_PUSH5(ISAd + incr, a, b, next, trlink); last = a; } else { STACK_PUSH5(ISAd, b, last, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } else { STACK_PUSH5(ISAd, b, last, limit, trlink); STACK_PUSH5(ISAd, first, a, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } else { if((a - first) <= (b - a)) { if(1 < (last - b)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); STACK_PUSH5(ISAd, first, a, limit, trlink); first = b; } else if(1 < (a - first)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); last = a; } else { ISAd += incr, first = a, last = b, limit = next; } } else if((last - b) <= (b - a)) { if(1 < (last - b)) { STACK_PUSH5(ISAd, first, a, limit, trlink); STACK_PUSH5(ISAd + incr, a, b, next, trlink); first = b; } else { STACK_PUSH5(ISAd, first, a, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } else { STACK_PUSH5(ISAd, first, a, limit, trlink); STACK_PUSH5(ISAd, b, last, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } } else { if((1 < (b - a)) && (0 <= trlink)) { stack[trlink].d = -1; } if((a - first) <= (last - b)) { if(1 < (a - first)) { STACK_PUSH5(ISAd, b, last, limit, trlink); last = a; } else if(1 < (last - b)) { first = b; } else { STACK_POP5(ISAd, first, last, limit, trlink); } } else { if(1 < (last - b)) { STACK_PUSH5(ISAd, first, a, limit, trlink); first = b; } else if(1 < (a - first)) { last = a; } else { STACK_POP5(ISAd, first, last, limit, trlink); } } } } else { if(trbudget_check(budget, last - first)) { limit = (incr < (ISAn - ISAd)) ? ((ISA[*first] != TR_GETC(*first)) ? (limit + 1) : -1) : -3; ISAd += incr; } else { if(0 <= trlink) { stack[trlink].d = -1; } STACK_POP5(ISAd, first, last, limit, trlink); } } } #undef STACK_SIZE } /*---------------------------------------------------------------------------*/ /*- Function -*/ /* Tandem repeat sort */ static void trsort(saidx_t *ISA, saidx_t *SA, saidx_t n, saidx_t depth) { saidx_t *ISAd; saidx_t *first, *last, *a; trbudget_t budget; saidx_t t, skip, unsorted; if (-n >= *SA) { return; } trbudget_init(&budget, tr_ilg(n) * 2 / 3, n); for(ISAd = ISA + depth;; ISAd += ISAd - ISA) { assert(n > ISAd - ISA); first = SA; skip = 0; unsorted = 0; do { if((t = *first) < 0) { first -= t; skip += t; } else { if(skip != 0) { *(first + skip) = skip; skip = 0; } last = SA + ISA[t] + 1; if(1 < (last - first)) { budget.count = 0; tr_introsort(ISA, ISAd, ISA + n, SA, first, last, &budget); if(budget.count != 0) { unsorted += budget.count; } else { skip = first - last; } } else if((last - first) == 1) { skip = -1; } first = last; } } while(first < (SA + n)); if(skip != 0) { *(first + skip) = skip; } if(unsorted == 0 || -n >= *SA) { break; } if(n <= (ISAd - ISA) * 2) { do { if((t = *first) < 0) { first -= t; } else { last = SA + ISA[t] + 1; for(a = first; a < last; ++a) { ISA[*a] = a - SA; } first = last; } } while(first < (SA + n)); break; } } } /*---- divsufsort ----*/ /*- Private Functions -*/ /* Sorts suffixes of type B*. */ static saidx_t sort_typeBstar(const sauchar_t *T, saidx_t *SA, saidx_t *bucket_A, saidx_t *bucket_B, saidx_t n) { saidx_t *PAb, *ISAb, *buf; saidx_t i, j, k, t, m, bufsize; saint_t c0, c1; int flag; /* Initialize bucket arrays. */ for(i = 0; i < BUCKET_A_SIZE; ++i) { bucket_A[i] = 0; } for(i = 0; i < BUCKET_B_SIZE; ++i) { bucket_B[i] = 0; } /* Count the number of occurrences of the first one or two characters of each type A, B and B* suffix. Moreover, store the beginning position of all type B* suffixes into the array SA. */ for(i = 1, flag = 1; i < n; ++i) { if(T[i - 1] != T[i]) { if(T[i - 1] > T[i]) { flag = 0; } break; } } i = n - 1, m = n, c0 = T[n - 1], c1 = T[0]; if((c0 < c1) || ((c0 == c1) && (flag != 0))) { if(flag == 0) { ++BUCKET_BSTAR(c0, c1); SA[--m] = i; } else { ++BUCKET_B(c0, c1); } for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) <= c1); --i, c1 = c0) { ++BUCKET_B(c0, c1); } } for(; 0 <= i;) { /* type A suffix. */ do { ++BUCKET_A(c1 = c0); } while((0 <= --i) && ((c0 = T[i]) >= c1)); if(0 <= i) { /* type B* suffix. */ ++BUCKET_BSTAR(c0, c1); SA[--m] = i; /* type B suffix. */ for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) <= c1); --i, c1 = c0) { ++BUCKET_B(c0, c1); } } } m = n - m; assert(m <= n/2); if(m == 0) { return 0; } /* note: A type B* suffix is lexicographically smaller than a type B suffix that begins with the same first two characters. */ /* Calculate the index of start/end point of each bucket. */ for(c0 = 0, i = 0, j = 0; c0 < ALPHABET_SIZE; ++c0) { t = i + BUCKET_A(c0); BUCKET_A(c0) = i + j; /* start point */ i = t + BUCKET_B(c0, c0); for(c1 = c0 + 1; c1 < ALPHABET_SIZE; ++c1) { j += BUCKET_BSTAR(c0, c1); BUCKET_BSTAR(c0, c1) = j; /* end point */ i += BUCKET_B(c0, c1); } } /* Sort the type B* suffixes by their first two characters. */ PAb = SA + n - m; ISAb = SA + m; for(i = m - 2; 0 <= i; --i) { t = PAb[i], c0 = T[t], c1 = T[t + 1]; SA[--BUCKET_BSTAR(c0, c1)] = i; } t = PAb[m - 1], c0 = T[t], c1 = T[t + 1]; SA[--BUCKET_BSTAR(c0, c1)] = m - 1; /* Sort the type B* substrings using sssort. */ buf = SA + m, bufsize = n - (2 * m); for(c0 = ALPHABET_SIZE - 2, j = m; 0 < j; --c0) { for(c1 = ALPHABET_SIZE - 1; c0 < c1; j = i, --c1) { i = BUCKET_BSTAR(c0, c1); if(1 < (j - i)) { sssort(T, PAb, SA + i, SA + j, buf, bufsize, 2, n, *(SA + i) == (m - 1)); } } } /* Compute ranks of type B* substrings. */ for(i = m - 1; 0 <= i; --i) { if(0 <= SA[i]) { j = i; do { ISAb[SA[i]] = i; } while((0 <= --i) && (0 <= SA[i])); SA[i + 1] = i - j; if(i <= 0) { break; } } j = i; do { ISAb[SA[i] = ~SA[i]] = j; } while(SA[--i] < 0); ISAb[SA[i]] = j; } /* Construct the inverse suffix array of type B* suffixes using trsort. */ trsort(ISAb, SA, m, 1); /* Set the sorted order of type B* suffixes. */ i = n - 1, j = m, c0 = T[n - 1], c1 = T[0]; if((c0 < c1) || ((c0 == c1) && (flag != 0))) { t = i; for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) <= c1); --i, c1 = c0) { } if(flag == 0) { SA[ISAb[--j]] = ((1 < (t - i))) ? t : ~t; } } for(; 0 <= i;) { for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) >= c1); --i, c1 = c0) { } if(0 <= i) { t = i; for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) <= c1); --i, c1 = c0) { } SA[ISAb[--j]] = ((1 < (t - i))) ? t : ~t; } } if(SA[ISAb[0]] == ~((saidx_t)0)) { /* check last type */ if(T[n - 1] <= T[0]) { /* is type B? */ SA[ISAb[0]] = 0; } } #ifdef DEBUG for(i = m; i < n; ++i) { SA[i] = ~n; } #endif /* Calculate the index of start/end point of each bucket. */ BUCKET_B(ALPHABET_SIZE - 1, ALPHABET_SIZE - 1) = n; /* end point */ for(c0 = ALPHABET_SIZE - 2, k = m - 1; 0 <= c0; --c0) { i = BUCKET_A(c0 + 1) - 1; for(c1 = ALPHABET_SIZE - 1; c0 < c1; --c1) { t = i - BUCKET_B(c0, c1); BUCKET_B(c0, c1) = i; /* end point */ /* Move all type B* suffixes to the correct position. */ for(i = t, j = BUCKET_BSTAR(c0, c1); j <= k; --i, --k) { SA[i] = SA[k]; } } BUCKET_BSTAR(c0, c0 + 1) = i - BUCKET_B(c0, c0) + 1; /* start point */ BUCKET_B(c0, c0) = i; /* end point */ } return m; } static saidx_t construct_BWT(const sauchar_t *T, saidx_t *SA, saidx_t *bucket_A, saidx_t *bucket_B, saidx_t n) { saidx_t *i, *j, *k; saidx_t s, t, orig = -10; saint_t c0, c1, c2; /* Construct the sorted order of type B suffixes by using the sorted order of type B* suffixes. */ for(c1 = ALPHABET_SIZE - 2; 0 <= c1; --c1) { /* Scan the suffix array from right to left. */ for(i = SA + BUCKET_BSTAR(c1, c1 + 1), j = SA + BUCKET_A(c1 + 1) - 1, k = NULL, c2 = -1; i <= j; --j) { if(0 <= (s = *j)) { assert(s < n); assert(T[s] == c1); assert(T[s] <= T[((s + 1) < n) ? (s + 1) : (0)]); if(s != 0) { t = s - 1; } else { t = n - 1; orig = j - SA; } assert(T[t] <= T[s]); c0 = T[t]; *j = ~((saidx_t)c0); if(c0 != c2) { if(0 <= c2) { BUCKET_B(c2, c1) = k - SA; } k = SA + BUCKET_B(c2 = c0, c1); } assert(k < j); *k-- = (((t != 0) ? T[t - 1] : T[n - 1]) > c2) ? ~t : t; } else { *j = ~s; assert(~s < n); } } } /* Construct the BWTed string by using the sorted order of type B suffixes. */ k = SA + BUCKET_A(c2 = 0); /* Scan the suffix array from left to right. */ for(i = SA, j = SA + n; i < j; ++i) { if(0 <= (s = *i)) { if(s != 0) { t = s - 1; } else { t = n - 1; orig = i - SA; } assert(T[t] >= T[s]); c0 = T[t]; *i = c0; if(c0 != c2) { BUCKET_A(c2) = k - SA; k = SA + BUCKET_A(c2 = c0); } if(t != 0) { c1 = T[t - 1]; } else { c1 = T[n - 1]; orig = k - SA; } assert(i <= k); *k++ = (c1 < c2) ? ~((saidx_t)c1) : t; } else { *i = ~s; } } assert(orig != -10); assert((0 <= orig) && (orig < n)); return orig; } /*---------------------------------------------------------------------------*/ /*- Function -*/ saidx_t divbwt(sauchar_t *T, saidx_t *SA, saidx_t n) { saidx_t *bucket_A, *bucket_B; saidx_t m, pidx, i; /* Check arguments. */ assert(n > 0); if(n == 1) { SA[0] = T[0]; return 0; } T[n] = T[0]; bucket_A = XCALLOC(BUCKET_A_SIZE, saidx_t); bucket_B = XCALLOC(BUCKET_B_SIZE, saidx_t); /* Burrows-Wheeler Transform. */ m = sort_typeBstar(T, SA, bucket_A, bucket_B, n); if(0 < m) { pidx = construct_BWT(T, SA, bucket_A, bucket_B, n); } else { pidx = 0; for(i = 0; i < n; ++i) { SA[i] = T[0]; } } free(bucket_B); free(bucket_A); return pidx; } lbzip2-2.3/src/encode.c000066400000000000000000000723451221755423000147640ustar00rootroot00000000000000/*- encode.c -- low-level compressor Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include "encode.h" #include /* htonl() */ #include /* memset() */ #include /* bzero() */ /* PREFIX CODING (also called Huffman coding) bzip2 file format uses cannonical, prefix-free codes in the last stage of coding process. bzip2 predescor -- bzip -- used arithmenic coding instead. Any cannonical, prefix-free codes can be used in bzip2 file. One could use Shannon or Shannon-Fano codes, but they are usually suboptimal. In bzip2 file format the maximal code length is limited to 20, meaning that no code longer than 20 bits can exist. For generating such liength-limited prefix code bzip2 uses an algorithm based on the original Huffman algorithm, but it has several disadvantages. It may require several iterations to converge and the generated codes can be suboptimal in some cases. One of the best known algorithms for generating optimal length-limited prefix code is Package-Merge algorithm. Unfortunatelly it is significantly slower and uses more memory than simple algorithms like Huffman algorithm. lbzip2 implements a hybrid algorithm. First a lightweight in-place algorithm based on Huffman algorithm is used to create optimal prefix codes. Then the maximal code length is computed and if it found to exceed the maximal allowed length (wich is 20), then these results are discarded and the Package-Merge algorithm is used to solve the problem from scratch. */ struct encoder_state { bool cmap[256]; uint32_t block_crc; uint32_t bwt_idx; uint32_t out_expect_len; uint32_t nmtf; uint32_t nblock; uint32_t alpha_size; uint32_t max_block_size; uint32_t cluster_factor; uint8_t *block; void *mtfv; uint8_t *selector; uint8_t *selectorMTF; uint32_t num_selectors; uint32_t num_trees; uint32_t count[MAX_TREES][32]; /* There is a sentinel symbol added at the end of each alphabet, hence the +1s below. */ uint8_t length[MAX_TREES][MAX_ALPHA_SIZE + 1]; uint32_t lookup[MAX_TREES][MAX_ALPHA_SIZE + 1]; uint32_t rfreq[MAX_TREES][MAX_ALPHA_SIZE + 1]; unsigned tmap_old2new[MAX_TREES]; unsigned tmap_new2old[MAX_TREES]; }; extern uint32_t crc_table[256]; #define CRC(x) crc = (crc << 8) ^ crc_table[(crc >> 24) ^ (x)] #define MAX_RUN_LENGTH (4+255) struct encoder_state * encoder_init(unsigned long max_block_size, unsigned cluster_factor) { struct encoder_state *s = XMALLOC(struct encoder_state); assert(s != 0); assert(max_block_size > 0 && max_block_size <= MAX_BLOCK_SIZE); assert(cluster_factor > 0 && cluster_factor <= 65535); s->max_block_size = max_block_size; s->cluster_factor = cluster_factor; s->selector = XNMALLOC(18000 + 1 + 1, uint8_t); s->selectorMTF = XNMALLOC(18000 + 1 + 7, uint8_t); s->block = XNMALLOC(max_block_size + 1, uint8_t); bzero(s->cmap, 256u * sizeof(bool)); return s; } void collect(struct encoder_state *s, const uint8_t *inbuf, size_t *buf_sz) { /* Cache some often used member variables for faster access. */ size_t avail = *buf_sz; const uint8_t *p = inbuf; const uint8_t *pLim = p + avail; uint8_t *q = s->block; uint8_t *qMax = s->block + s->max_block_size - 1; unsigned ch, last; uint32_t run; uint32_t save_crc; uint32_t crc = -1; state0: /*=== STATE 0 ===*/ if (unlikely(q > qMax || p == pLim)) goto done; ch = *p++; CRC(ch); #define S1 \ s->cmap[ch] = true; \ *q++ = ch; \ if (unlikely (q > qMax || p == pLim)) \ goto done; \ last = ch; \ ch = *p++; \ CRC (ch); \ if (unlikely (ch == last)) \ goto state2 state1: /*=== STATE 1 ===*/ S1; S1; S1; S1; goto state1; state2: /*=== STATE 2 ===*/ *q++ = ch; if (unlikely(q > qMax || p == pLim)) goto done; ch = *p++; CRC(ch); if (ch != last) goto state1; /*=== STATE 3 ===*/ *q++ = ch; if (unlikely((q >= qMax && (q > qMax || (p < pLim && *p == last))) || p == pLim)) goto done; ch = *p++; CRC(ch); if (ch != last) goto state1; /*=== STATE 4+ ===*/ assert(q < qMax); *q++ = ch; /* While the run is shorter than MAX_RUN_LENGTH characters, keep trying to append more characters to it. */ for (run = 4; run < MAX_RUN_LENGTH; run++) { /* Check for end of input buffer. */ if (unlikely(p == pLim)) { *q++ = run - 4; s->cmap[run - 4] = true; goto done; } /* Fetch the next character. */ ch = *p++; save_crc = crc; CRC(ch); /* If the character does not match, terminate the current run and start a fresh one. */ if (ch != last) { *q++ = run - 4; s->cmap[run - 4] = true; if (likely(q <= qMax)) goto state1; /* There is no space left to begin a new run. Unget the last character and finish. */ p--; crc = save_crc; goto done; } } /* The run has reached maximal length, so it must be ended prematurely. */ *q++ = MAX_RUN_LENGTH - 4; s->cmap[MAX_RUN_LENGTH - 4] = true; goto state0; done: s->nblock = q - s->block; s->block_crc = crc; *buf_sz -= p - inbuf; } /* return ninuse */ static unsigned make_map_e(uint8_t *cmap, const bool *inuse) { unsigned i, j; j = 0; for (i = 0; i < 256; i++) { int k = inuse[i]; cmap[i] = j; j += k; } return j; } /*---------------------------------------------------*/ /* returns nmtf */ static uint32_t do_mtf(uint16_t *mtfv, uint32_t *mtffreq, uint8_t *cmap, int32_t nblock, int32_t EOB) { uint8_t order[255]; int32_t i; int32_t k; int32_t t; uint8_t c; uint8_t u; uint32_t *bwt = (void *)mtfv; const uint16_t *mtfv0 = mtfv; for (i = 0; i <= EOB; i++) mtffreq[i] = 0; k = 0; u = 0; for (i = 0; i < 255; i++) order[i] = i + 1; #define RUN() \ if (unlikely(k)) \ do { \ mtffreq[*mtfv++ = --k & 1]++; \ k >>= 1; \ } while (k) \ #define MTF() \ { \ uint8_t *p = order; \ t = *p; \ *p = u; \ for (;;) \ { \ if (c == t) { u = t; break; } \ u = *++p; \ *p = t; \ if (c == u) break; \ t = *++p; \ *p = u; \ } \ t = p - order + 2; \ *mtfv++ = t; \ mtffreq[t]++; \ } for (i = 0; i < nblock; i++) { if ((c = cmap[*bwt++]) == u) { k++; continue; } RUN(); MTF(); } RUN(); *mtfv++ = EOB; mtffreq[EOB]++; return mtfv - mtfv0; #undef RUN #undef MTF } size_t encode(struct encoder_state *s, uint32_t *crc) { uint32_t cost; uint32_t pk; uint32_t i; const uint8_t *sp; /* selector pointer */ uint8_t *smp; /* selector MTFV pointer */ uint8_t c; /* value before MTF */ uint8_t j; /* value after MTF */ uint32_t p; /* MTF state */ uint32_t EOB; uint8_t cmap[256]; EOB = make_map_e(cmap, s->cmap) + 1; assert(EOB >= 2); assert(EOB < 258); /* Sort block. */ assert(s->nblock > 0); s->mtfv = XNMALLOC(s->nblock + GROUP_SIZE, uint32_t); s->bwt_idx = divbwt(s->block, s->mtfv, s->nblock); free(s->block); s->nmtf = do_mtf(s->mtfv, s->lookup[0], cmap, s->nblock, EOB); cost = 48 /* header */ + 32 /* crc */ + 1 /* rand bit */ + 24 /* bwt index */ + 00 /* {cmap} */ + 3 /* nGroups */ + 15 /* nSelectors */ + 00 /* {sel} */ + 00 /* {tree} */ + 00; /* {mtfv} */ cost += generate_prefix_code(s); sp = s->selector; smp = s->selectorMTF; /* A trick that allows to do MTF without branching, using arithmetical and logical operations only. The whole MTF state packed into one 32-bit integer. */ /* Set up initial MTF state. */ p = 0x543210; assert(*sp < MAX_TREES); assert(s->tmap_old2new[*sp] == 0); while ((c = *sp) != MAX_TREES) { uint32_t v, z, l, h; c = s->tmap_old2new[c]; assert(c < s->num_trees); assert((size_t)(sp - s->selector) < s->num_selectors); v = p ^ (0x111111 * c); z = (v + 0xEEEEEF) & 0x888888; l = z ^ (z - 1); h = ~l; p = (p | l) & ((p << 4) | h | c); #if GNUC_VERSION >= 30406 j = (__builtin_ctz(h) >> 2) - 1; #else h &= -h; j = !!(h & 0x01010100); h |= h >> 4; j |= h >> 11; j |= h >> 18; j &= 7; #endif sp++; *smp++ = j; cost += j + 1; } /* Add zero to seven dummy selectors in order to make block size multiply of 8 bits. */ j = cost & 0x7; j = (8 - j) & 0x7; s->num_selectors += j; cost += j; while (j--) *smp++ = 0; assert(cost % 8 == 0); /* Calculate the cost of transmitting character map. */ for (i = 0; i < 16; i++) { pk = 0; for (j = 0; j < 16; j++) pk |= s->cmap[16 * i + j]; cost += pk << 4; } cost += 16; /* Big bucket costs 16 bits on its own. */ /* Convert cost from bits to bytes. */ assert(cost % 8 == 0); cost >>= 3; s->out_expect_len = cost; *crc = s->block_crc; return cost; } /* Sort source alphabet by descending fequency. Use plain simple insertion sort because (1) the alphabet is small enough and (2) we expect symbols to be already nearly sorted on common data. */ static void sort_alphabet(uint64_t *first, uint64_t *last) { uint64_t t, *a, *b, *b1; for (a = first + 1; a < last; ++a) { t = *(b1 = a); for (b = b1 - 1; *b < t; --b) { *b1 = *b; if ((b1 = b) == first) break; } *b1 = t; } } #ifndef PACKAGE_MERGE /* Build a prefix-free tree. Because the source alphabet is already sorted, we need not to maintain a priority queue -- two normal FIFO queues (one for leaves and one for internal nodes) will suffice. */ static void build_tree(uint32_t *restrict T, uint64_t *restrict P, int32_t n) { unsigned r; /* index of the next tree in the queue */ unsigned s; /* index of the next singleton leaf */ unsigned t; /**/ uint64_t w1, w2; r = n; s = n; /* Start with the last singleton tree. */ for (t = n-1; t > 0; t--) { if (s < 1 || (r > t+2 && P[r-2] < P[s-1])) { /* Select two internal nodes. */ T[r-1] = t; T[r-2] = t; w1 = P[r-1]; w2 = P[r-2]; r -= 2; } else if (r < t+2 || (s > 1 && P[s-2] <= P[r-1])) { /* Select two singleton leaf nodes. */ w1 = P[s-1]; w2 = P[s-2]; s -= 2; } else { /* Select one internal node and one singleton leaf node. */ T[r-1] = t; w1 = P[r-1]; w2 = P[s-1]; s--; r--; } P[t] = (P[t] & 0xFFFF) + ((w1 + w2) & ~(uint64_t)0xFF00FFFF) + max(w1 & 0xFF000000, w2 & 0xFF000000) + 0x01000000; } assert(r == 2); assert(s == 0); assert(t == 0); } /* Compute counts from given Huffman tree. The tree itself is clobbered. */ static void compute_depths(uint32_t *restrict C, uint32_t *restrict T, uint32_t n) { uint32_t a; /* total number of nodes at current level */ uint32_t u; /* number of internal nodes */ uint32_t t; /* current tree node */ uint32_t d; /* current node depth */ T[1] = 0; /* The root always has depth of 0. */ C[0] = 0; /* There are no zero-length codes in bzip2. */ t = 2; /* The root is done, advance to the next node (of index 2). */ d = 1; /* The root was the last node at depth 0, go deeper. */ a = 2; /* At depth of 1 there are always exactly 2 nodes. */ /* Repeat while we have more nodes. */ while (d < 32) { u = 0; /* So far we haven't seen any internal nodes at this level. */ while (t < n && T[T[t]] + 1 == d) { assert(a > u); u++; T[t++] = d; /* Overwrite parent pointer with node depth. */ } C[d] = a - u; d++; a = u << 1; } assert(a == 0); } #endif /*!PACKAGE_MERGE */ /* The following is an implementation of the Package-Merge algorithm for finding an optimal length-limited prefix-free codeset. */ static void package_merge(uint32_t *restrict C, const uint64_t *restrict Pr, uint32_t n) { #define BITS_PER_SYMBOL 9 /* ceil(log2(MAX_ALPHA_SIZE)) */ #define SYMBOLS_PER_WORD 7 /* floor(64 / BITS_PER_SYMBOL) */ #define VECTOR_SIZE 3 /* ceil(MAX_CODE_LENGTH / SYMBOLS_PER_WORD) */ #define QUEUE_SIZE (MAX_ALPHA_SIZE - 1) /* It can be easily proved by induction that number of elemnts stored in the queue is always stricly less elements than alphabet size. */ uint64_t W[2*QUEUE_SIZE]; /* internal node weights */ uint64_t P[2*QUEUE_SIZE][VECTOR_SIZE]; unsigned x; /* number of unprocessed singleton nodes */ unsigned i; /* general purpose index */ unsigned k; /* vector index */ unsigned d; /* current node depth */ unsigned jP; /* current index in queue P */ unsigned jL; /* current index in queue L */ unsigned szP; /* current size of queue P */ unsigned iP; /* current offset of head of queue P */ uint64_t dw; /* symbol weight at current depth */ iP = MAX_CODE_LENGTH % 2 * (MAX_ALPHA_SIZE - 1); szP = 0; d = VECTOR_SIZE - 1; dw = (uint64_t)1 << (MAX_CODE_LENGTH % SYMBOLS_PER_WORD * BITS_PER_SYMBOL); for (;;) { dw >>= BITS_PER_SYMBOL; d += -!dw; if (d+1 == 0) break; dw += -!dw & ((uint64_t)1 << ((SYMBOLS_PER_WORD - 1) * BITS_PER_SYMBOL)); x = n; jP = iP; iP ^= MAX_ALPHA_SIZE - 1; jL = iP; for (jL = iP; x + szP > 1; jL++) { if (szP == 0 || (x > 1 && Pr[x-2] < W[jP])) { W[jL] = Pr[x-1] + Pr[x-2]; for (k = 0; k < VECTOR_SIZE; k++) P[jL][k] = 0; P[jL][d] += 2 * dw; x -= 2; } else if (x == 0 || (szP > 1 && W[jP+1] <= Pr[x-1])) { W[jL] = W[jP] + W[jP+1]; for (k = 0; k < VECTOR_SIZE; k++) P[jL][k] = P[jP][k] + P[jP+1][k]; jP += 2; szP -= 2; } else { W[jL] = W[jP] + Pr[x-1]; for (k = 0; k < VECTOR_SIZE; k++) P[jL][k] = P[jP][k]; P[jL][d] += dw; jP++; x--; szP--; } } szP = jL - iP; assert(szP >= n/2); assert(szP < n); } assert(iP == 0); assert(szP == n-1); k = VECTOR_SIZE * SYMBOLS_PER_WORD; for (i = VECTOR_SIZE; i--;) { dw = P[szP-1][i]; for (d = SYMBOLS_PER_WORD * BITS_PER_SYMBOL; d;) C[k--] = (dw >> (d -= BITS_PER_SYMBOL)) & 0x1ff; } C[0] = 0; } static void make_code_lengths(uint32_t C[], uint8_t L[], uint32_t P0[], uint32_t n) { uint32_t i; int32_t k; int32_t d; int32_t c; uint64_t P[MAX_ALPHA_SIZE]; assert(n >= MIN_ALPHA_SIZE); assert(n <= MAX_ALPHA_SIZE); /* Label weights with sequence numbers. Labelling has two main purposes: firstly it allows to sort pairs of weight and sequence number more easily; secondly: the package-merge algorithm requires weights to be strictly monotonous and putting unique values in lower bits assures that. */ for (i = 0; i < n; i++) { /* FFFFFFFF00000000 - symbol frequency 00000000FF000000 - node depth 0000000000FF0000 - initially one 000000000000FFFF - symbol */ if (P0[i] == 0) P[i] = ((uint64_t)1 << 32) | 0x10000 | (MAX_ALPHA_SIZE - i); else P[i] = ((uint64_t)P0[i] << 32) | 0x10000 | (MAX_ALPHA_SIZE - i); } /* Sort weights and sequence numbers together. */ sort_alphabet(P, P + n); { #ifndef PACKAGE_MERGE uint32_t V[MAX_ALPHA_SIZE]; build_tree(V, P, n); compute_depths(C, V, n); } k = 0; for (d = MAX_CODE_LENGTH + 1; d < 32; d++) k += C[d]; /* If any code exceeds length limit, fallback to package-merge algorithm. */ if (k != 0) { for (i = 0; i < n; i++) { if (P0[MAX_ALPHA_SIZE - (P[i] & 0xFFFF)] == 0) P[i] = ((uint64_t)1 << 32) | 0x10000 | (P[i] & 0xFFFF); else P[i] = ((uint64_t)P0[MAX_ALPHA_SIZE - (P[i] & 0xFFFF)] << 32) | 0x10000 | (P[i] & 0xFFFF); } #endif package_merge(C, P, n); } /* Generate code lengths and transform counts into base codes. */ i = 0; c = 0; for (d = 0; d <= MAX_CODE_LENGTH; d++) { k = C[d]; C[d] = c; c = (c + k) << 1; while (k != 0) { assert(i < n); L[MAX_ALPHA_SIZE - (P[i] & 0xFFFF)] = d; i++; k--; } } assert(c == (1UL << (MAX_CODE_LENGTH + 1))); assert(i == n); } static void assign_codes(uint32_t C[], uint32_t L[], uint8_t B[], uint32_t n) { /* Assign prefix-free codes. */ uint32_t i; for (i = 0; i < n; i++) L[i] = C[B[i]]++; } /* Create initial mapping of symbols to trees. The goal is to divide all as symbols [0,as) into nt equivalence classes (EC) [0,nt) such that standard deviation of symbol frequencies in classes is minimal. We use a kind of a heuristic to achieve that. There might exist a better way to achieve that, but this one seems to be good (and fast) enough. If the symbol v belongs to the equivalence class t then set s->length[t][v] to zero. Otherwise set it to 1. */ static void generate_initial_trees(struct encoder_state *s, unsigned nm, unsigned nt) { unsigned a, b; /* range [a,b) of symbols forming current EC */ unsigned freq; /* symbol frequency */ unsigned cum; /* cumulative frequency */ unsigned as; /* effective alphabet size (alphabet size minus number of symbols with frequency equal to zero) */ unsigned t; /* current tree */ /* Equivalence classes are initially empty. */ memset(s->length, 1, sizeof(s->length)); /* Determine effective alphabet size. */ as = 0; for (a = 0, cum = 0; cum < nm; a++) { freq = s->lookup[0][a]; cum += freq; as += min(freq, 1); } assert(cum == nm); /* Bound number of EC by number of symbols. Each EC is non-empty, so number of symbol EC must be <= number of symbols. */ nt = min(nt, as); /* For each equivalence class: */ a = 0; for (t = 0; nt > 0; t++, nt--) { assert(nm > 0); assert(as >= nt); /* Find a range of symbols which total count is roughly proportional to one nt-th of all values. */ freq = s->lookup[0][a]; cum = freq; as -= min(freq, 1); b = a+1; while (as > nt-1 && cum * nt < nm) { freq = s->lookup[0][b]; cum += freq; as -= min(freq, 1); b++; } if (cum > freq && (2*cum - freq) * nt > 2*nm) { cum -= freq; as += min(freq, 1); b--; } assert(a < b); assert(cum > 0); assert(cum <= nm); assert(as >= nt-1); Trace(("Tree %u: EC=[%3u,%3u), |EC|=%3u, cum=%6u", t, a, b, b-a, cum)); /* Now [a,b) is our range -- assign it to equivalence class t. */ bzero(&s->length[t][a], b - a); a = b; nm -= cum; } assert(as == 0); assert(nm == 0); } /* Find the tree which takes the least number of bits to encode current group. Return number from 0 to nt-1 identifying the selected tree. */ static int find_best_tree(const uint16_t *gs, unsigned nt, const uint64_t *len) { unsigned c, bc; /* code length, best code length */ unsigned t, bt; /* tree, best tree */ uint64_t cp; /* cost packed */ unsigned i; /* Compute how many bits it takes to encode current group by each of trees. Utilize vector operations for best performance. Let's hope the compiler unrolls the loop for us. */ cp = 0; for (i = 0; i < GROUP_SIZE; i++) cp += len[gs[i]]; /* At the beginning assume the first tree is the best. */ bc = cp & 0x3ff; bt = 0; /* Iterate over other trees (starting from second one) to see which one is the best to encode current group. */ for (t = 1; t < nt; t++) { cp >>= 10; c = cp & 0x3ff; if (c < bc) bc = c, bt = t; } /* Return our favorite. */ return bt; } /* Compute bit cost of transmitting a single tree and all symbols it codes. */ static uint32_t transmission_cost(const uint8_t *length, const uint32_t *rfreq, uint32_t as) { unsigned v; unsigned p; uint32_t cost; /* Compute cost of transmiting RUNA symbols. */ cost = 6; /* fixed code length (5bits) and delta code (1bit) */ p = length[0]; cost += rfreq[0] * p; /* cost of transmitting prefix codes */ /* Compute cost of transmiting all other symbols, from RUNB to EOB. */ for (v = 1; v < as; v++) { int c, d; c = length[v]; assert(c >= 1 && c <= 20); d = p - c; if (d < 0) d = -d; cost += 1 + 2 * d; /* cost of transmitting delta code ... */ p = c; cost += rfreq[v] * length[v]; /* ... and prefix codes */ } return cost; } /* The main function generating prefix code for the whole block. Input: MTF values Output: trees and selectors What this function does: 1) decides how many trees to generate 2) divides groups into equivalence classes (using Expectation-Maximization algorithm, which is a heuristic usually giving suboptimal results) 3) generates an optimal prefix tree for each class (with a hubrid algorithm consisting of Huffman algorithm and Package-Merge algorithm) 4) generates selectors 5) sorts trees by their first occurence in selector sequence 6) computes and returns cost (in bits) of transmitting trees and codes */ unsigned generate_prefix_code(struct encoder_state *s) { uint32_t as; uint32_t nt; uint32_t iter, i; uint32_t cost; uint16_t *mtfv = s->mtfv; uint32_t nm = s->nmtf; as = mtfv[nm - 1] + 1; /* the last mtfv is EOB */ s->num_selectors = (nm + GROUP_SIZE - 1) / GROUP_SIZE; /* Decide how many prefix-free trees to use for current block. The best for compression ratio would be to always use the maximal number of trees. However, the space it takes to transmit these trees can also be a factor, especially if the data being encoded is not very long. If we use less trees for smaller block then the space needed to transmit additional trees is traded against the space saved by using more trees. */ assert(nm >= 2); nt = (nm > 2400 ? 6 : nm > 1200 ? 5 : nm > 600 ? 4 : nm > 300 ? 3 : nm > 150 ? 2 : 1); /* Complete the last group with dummy symbols. */ for (i = nm; i < s->num_selectors * GROUP_SIZE; i++) mtfv[i] = as; /* Grow up an initial forest. */ generate_initial_trees(s, nm, nt); /* Perform a few iterations of the Expectation-Maximization algorithm to improve trees. */ iter = s->cluster_factor; while (iter-- > 0) { uint64_t len_pack[MAX_ALPHA_SIZE + 1]; uint16_t *gs; uint32_t v, t; uint8_t *sp; /* Pack code lengths of all trees into 64-bit integers in order to take advantage of 64-bit vector arithmetic. Each group holds at most 50 codes, each code is at most 20 bit long, so each group is coded by at most 1000 bits. We can store that in 10 bits. */ for (v = 0; v < as; v++) len_pack[v] = (((uint64_t)s->length[0][v] ) + ((uint64_t)s->length[1][v] << 10) + ((uint64_t)s->length[2][v] << 20) + ((uint64_t)s->length[3][v] << 30) + ((uint64_t)s->length[4][v] << 40) + ((uint64_t)s->length[5][v] << 50)); len_pack[as] = 0; sp = s->selector; /* (E): Expectation step -- estimate likehood. */ bzero(s->rfreq, nt * sizeof(*s->rfreq)); for (gs = mtfv; gs < mtfv + nm; gs += GROUP_SIZE) { /* Check out which prefix-free tree is the best to encode current group. Then increment symbol frequencies for the chosen tree and remember the choice in the selector array. */ t = find_best_tree(gs, nt, len_pack); assert(t < nt); *sp++ = t; for (i = 0; i < GROUP_SIZE; i++) s->rfreq[t][gs[i]]++; } assert((size_t)(sp - s->selector) == s->num_selectors); *sp = MAX_TREES; /* sentinel */ /* (M): Maximization step -- maximize expectations. */ for (t = 0; t < nt; t++) make_code_lengths(s->count[t], s->length[t], s->rfreq[t], as); } cost = 0; /* Reorder trees. This also removes unused trees. */ { /* Only lowest nt bits are used, from 0 to nt-1. If i-th bit is set then i-th tree exists but hasn't been seen yet. */ unsigned not_seen = (1 << nt) - 1; unsigned t, v; uint8_t *sp = s->selector; nt = 0; while (not_seen > 0 && (t = *sp++) < MAX_TREES) { if (not_seen & (1 << t)) { not_seen -= 1 << t; s->tmap_old2new[t] = nt; s->tmap_new2old[nt] = t; nt++; /* Create lookup tables for this tree. These tables are used by the transmiter to quickly send codes for MTF values. */ assign_codes(s->count[t], s->lookup[t], s->length[t], as); s->lookup[t][as] = 0; s->length[t][as] = 0; /* Compute cost of transmiting the tree and all symbols it codes. */ cost += transmission_cost(s->length[t], s->rfreq[t], as); } } /* If there is only one prefix tree in current block, we need to create a second dummy tree. This increases the cost of transmitting the block, but unfortunately bzip2 doesn't allow blocks with a single tree. */ assert(nt >= 1); if (nt == 1) { nt = 2; t = s->tmap_new2old[0] ^ 1; s->tmap_old2new[t] = 1; s->tmap_new2old[1] = t; for (v = 0; v < MAX_ALPHA_SIZE; v++) s->length[t][v] = MAX_CODE_LENGTH; cost += as + 5; } } s->num_trees = nt; return cost; } #define SEND(n,v) \ b = (b << (n)) | (v); \ if ((k += (n)) >= 32) { \ uint32_t w = (uint32_t)(b >> (k -= 32)); \ *p++ = htonl(w); \ } void transmit(struct encoder_state *s, void *buf) { uint64_t b; unsigned k; uint8_t *sp; unsigned t; unsigned v; uint16_t *mtfv; uint32_t *p; unsigned ns; unsigned as; uint32_t gr; /* Initialize bit buffer. */ b = 0; k = 0; p = buf; /* Transmit block metadata. */ SEND(24, 0x314159); SEND(24, 0x265359); SEND(32, s->block_crc ^ 0xFFFFFFFF); SEND(1, 0); /* non-rand */ SEND(24, s->bwt_idx); /* bwt primary index */ /* Transmit character map. */ { unsigned pack[16]; unsigned pk; unsigned i; unsigned j; unsigned big = 0; for (i = 0; i < 16; i++) { pk = 0; for (j = 0; j < 16; j++) pk = (pk << 1) + s->cmap[16 * i + j]; pack[i] = pk; big = (big << 1) + !!pk; } SEND(16, big); for (i = 0; i < 16; i++) if (pack[i]) { SEND(16, pack[i]); } } /* Transmit selectors. */ assert(s->num_trees >= MIN_TREES && s->num_trees <= MAX_TREES); SEND(3, s->num_trees); ns = s->num_selectors; SEND(15, ns); sp = s->selectorMTF; while (ns--) { v = 1 + *sp++; SEND(v, (1 << v) - 2); } mtfv = s->mtfv; as = mtfv[s->nmtf - 1] + 1; ns = (s->nmtf + GROUP_SIZE - 1) / GROUP_SIZE; /* Transmit prefix trees. */ for (t = 0; t < s->num_trees; t++) { int32_t a, c; uint8_t *len = s->length[s->tmap_new2old[t]]; a = len[0]; SEND(6, a << 1); for (v = 1; v < as; v++) { c = len[v]; while (a < c) { SEND(2, 2); a++; } while (a > c) { SEND(2, 3); a--; } SEND(1, 0); } } /* Transmit prefix codes. */ for (gr = 0; gr < ns; gr++) { unsigned i; /* symbol index in group */ const uint32_t *L; /* symbol-to-code lookup table */ const uint8_t *B; /* code lengths */ unsigned mv; /* symbol (MTF value) */ t = s->selector[gr]; L = s->lookup[t]; B = s->length[t]; for (i = 0; i < GROUP_SIZE; i++) { mv = *mtfv++; SEND(B[mv], L[mv]); } } /* Flush */ assert(k % 8 == 0); assert(k / 8 == s->out_expect_len % 4); assert(p == (uint32_t *)buf + s->out_expect_len / 4); SEND(31, 0); free(s->selector); free(s->selectorMTF); free(s->mtfv); free(s); } lbzip2-2.3/src/encode.h000066400000000000000000000023641221755423000147630ustar00rootroot00000000000000/*- encode.h -- low-level compressor header Copyright (C) 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #define CLUSTER_FACTOR 8u #define HEADER_SIZE 4u #define TRAILER_SIZE 10u struct encoder_state; struct encoder_state *encoder_init(unsigned long mbs, unsigned cf); void collect(struct encoder_state *e, const uint8_t *buf, size_t *buf_sz); size_t encode(struct encoder_state *e, uint32_t *crc); void transmit(struct encoder_state *e, void *buf); unsigned generate_prefix_code(struct encoder_state *s); int32_t divbwt(uint8_t *T, int32_t *SA, int32_t n); #define combine_crc(cc,c) (((cc) << 1) ^ ((cc) >> 31) ^ (c) ^ -1) lbzip2-2.3/src/expand.c000066400000000000000000000526511221755423000150040ustar00rootroot00000000000000/*- expand.c -- high-level decompression routines Copyright (C) 2011, 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include "decode.h" /* decode() */ #include "main.h" /* bs100k */ #include "process.h" /* struct process */ #include /* bzero() */ /* MEMORY ALLOCATION A single 26-byte bzip2 stream can decompress to as much as 47 MB, meaning compression ratio of about 1.8 million to one. (See ch255.bz2 in ../tests.) This case can appear in practice (for example when compressing a disk image with long runs of zeroes), but also someone could craft such bzip2 file in hope of crashing victim's system causing denial of service. For this reason proper resource allocation seems to be important. One of the simplest algorithms of allocating memory is FCFS (First-Come, First-Served), in which the memory is allocated to the first thread that asks for it. Unfortunatelly this algorithm is deadlock-prone. The only situaltion in which dceadlock can occur is when the block that is the next to be outputed is stalled waiting for memory. To solve this problem it's enough to make sure that this block can always allocate enogh resources to proceed. In lbzip2 memory is allocated to all threads as needed, according to FCFS policy. However, to avoid deadlock, the last remaining unit of memory can be allocated only to the block which is the next expected on the output. */ /* XXX these macros are buggy, they don't even compute correct numbers... */ #ifdef ENABLE_TRACING #define nbsx2(p) ((unsigned)((p).major*in_granul*8+((p).minor>>27))) #define nbs2(p) nbsx2(*((const struct position *)(p))) #define nbsd2(p) (empty(p) ? 0u : nbs2(dq_get(p,0))) #define nbsp2(p) (empty(p) ? 0u : nbs2(peek(p))) #endif #define SCAN_THRESH 1u #define EMIT_THRESH 2u #define UNORD_THRESH (SCAN_THRESH + EMIT_THRESH) static const char * err2str(int err) { static const char *table[] = { "bad stream header magic", "bad block header magic", "empty source alphabet", "bad number of trees", "no coding groups", "invalid selector", "invalid delta code", "invalid prefix code", "incomplete prefix code", "empty block", "unterminated block", "missing run length", "block CRC mismatch", "stream CRC mismatch", "block overflow", "primary index too large", "unexpected end of file", }; assert(err >= ERR_MAGIC && err <= ERR_EOF); return table[err - ERR_MAGIC]; } struct in_blk { void *buffer; size_t size; unsigned ref_count; uintmax_t offset; }; struct detached_bitstream { struct position pos; uintmax_t offset; unsigned live; uint64_t buff; bool eof; }; struct unord_blk { struct position base; struct detached_bitstream end_pos; bool complete; bool legitimate; }; struct retr_blk { struct detached_bitstream curr_pos; struct position base; struct decoder_state ds; struct unord_blk *unord_link; }; struct emit_blk { struct position base; struct decoder_state ds; int status; uintmax_t end_offset; }; struct head_blk { struct position base; struct header hdr; }; struct out_blk { struct position base; size_t size; uint32_t crc; uint32_t blk_sz; int status; uintmax_t end_offset; }; static unsigned eof_missing; static struct deque(struct in_blk *) input_q; static uintmax_t head_offs; static uintmax_t tail_offs; static struct pqueue(struct retr_blk *) retr_q; static struct pqueue(struct emit_blk *) emit_q; static struct pqueue(struct out_blk *) reord_q; static struct deque(struct head_blk) order_q; static struct pqueue(struct unord_blk *) unord_q; static bool parse_token; static bool parsing_done; static struct pqueue(struct detached_bitstream *) scan_q; static uintmax_t reord_offs; static struct detached_bitstream parser_bs; static struct parser_state par; #if 1 #define check_invariants() #else static void check_invariants(void) { unsigned i; /* Obvious stuff. */ assert(head_offs <= tail_offs); assert(work_units <= num_worker); assert(out_slots <= total_out_slots); for (i = 0; i < size(retr_q); i++) { assert(retr_q.root[i]->curr_pos.offset >= head_offs); assert(retr_q.root[i]->curr_pos.offset >= head_offs); } for (i = 0; i < size(scan_q); i++) { assert(scan_q.root[i]->pos.major >= parser_bs.pos.major); } } #endif #if 0 static void dump_info(void) { Trace(("STATUS:" "==============================" "\n\t" "eof: %2d" "\n\t" "parsing_done: %2d" "\n\t" "parse_token: %2d" "\n\t" "work_units: %2u/%2u" "\n\t" "out_slots: %2u/%2u" "\n\t" "input_head: {%6u}" "\n\t" "input_tail: {%6u}" "\n\t" "size(input_q): %2u" "\n\t" "size(scan_q): %2u {%6u}" "\n\t" "size(retr_q): %2u {%6u}" "\n\t" "size(emit_q): %2u {%6u}" "\n\t" "size(reord_q): %2u {%6u}" "\n\t" "size(order_q): %2u {%6u}" "\n\t" "size(unord_q): %2u {%6u}", eof, parsing_done, parse_token, work_units, num_worker, out_slots, total_out_slots, (unsigned)head_offs * 32u, (unsigned)tail_offs * 32u, size(input_q), size(scan_q), nbsp2(scan_q), size(retr_q), nbsp2(retr_q), size(emit_q), nbsp2(emit_q), size(reord_q), nbsp2(reord_q), size(order_q), nbsd2(order_q), size(unord_q), nbsp2(unord_q))); } #endif static struct detached_bitstream bits_init(uintmax_t offset) { struct detached_bitstream bs; bs.live = 0u; bs.buff = 0u; bs.eof = false; bs.offset = offset; bs.pos.major = offset / (in_granul / 4); bs.pos.minor = (offset % (in_granul / 4)) << 32; return bs; } static bool can_attach(struct detached_bitstream bs) { /* No one should try to attach a block that has already been released. */ assert(bs.offset >= head_offs); return bs.offset < tail_offs || (eof && bs.offset == tail_offs); } static struct bitstream attach(struct detached_bitstream dbs) { struct in_blk *blk; size_t id; uintmax_t limit; const uint32_t *base; struct bitstream bs; assert(dbs.offset >= head_offs); assert(dbs.offset <= tail_offs); bs.live = dbs.live; bs.buff = dbs.buff; bs.eof = dbs.eof; if (dbs.offset == tail_offs) { assert(eof); bs.block = NULL; bs.data = NULL; bs.limit = NULL; assert(!bs.eof || bs.live < 32); bs.eof = (bs.live < 32u); } else { /* TODO: binary search */ id = 0u; limit = head_offs; do { blk = dq_get(input_q, id); limit += blk->size; id++; } while (dbs.offset >= limit); blk->ref_count++; bs.block = blk; base = blk->buffer; assert(dbs.offset >= blk->offset); assert(dbs.offset - blk->offset < blk->size); bs.data = base + (dbs.offset - blk->offset); bs.limit = base + blk->size; } check_invariants(); sched_unlock(); return bs; } static struct detached_bitstream detach(struct bitstream bs) { struct detached_bitstream dbs; struct in_blk *blk; unsigned bit_offset; uintmax_t offset; ptrdiff_t remains; sched_lock(); remains = bs.limit - bs.data; assert(remains >= 0); blk = bs.block; offset = blk != NULL ? blk->offset + blk->size - remains : tail_offs; offset = min(offset, tail_offs); dbs.live = bs.live; dbs.buff = bs.buff; dbs.eof = bs.eof; dbs.offset = offset; bit_offset = -bs.live % 32u; offset -= (bs.live + 31u) / 32u; assert(bit_offset < 32u); assert(32u * dbs.offset - dbs.live == 32u * offset + bit_offset); dbs.pos.major = offset / (in_granul / 4); dbs.pos.minor = (((offset % (in_granul / 4)) << 32) + (bit_offset << 27)); if (blk && --blk->ref_count == 0) { source_release_buffer(blk->buffer); free(blk); } return dbs; } /* Release any input blocks that are behind current base position. */ static void advance(struct detached_bitstream bs) { parser_bs = bs; /* Release input blocks. */ while (!empty(input_q)) { struct in_blk *blk = dq_get(input_q, 0u); if (blk->offset + blk->size > bs.offset) break; head_offs += blk->size; (void)shift(input_q); if (--blk->ref_count == 0) { source_release_buffer(blk->buffer); free(blk); } } /* Release retrieve jobs. */ while (!empty(retr_q) && peek(retr_q)->curr_pos.offset < head_offs) { struct retr_blk *rb = dequeue(retr_q); Trace(("Advanced over miss-recognized bit pattern at {%u}", nbsx2(rb->base))); decoder_free(&rb->ds); free(rb); work_units++; } /* Release scan jobs. */ while (!empty(scan_q) && peek(scan_q)->offset < head_offs) { free(dequeue(scan_q)); } } static bool can_parse(void) { return (!parsing_done && parse_token && work_units > 0u && can_attach(parser_bs)); } static void do_parse(void) { int rv; struct head_blk head_blk; struct bitstream true_bitstream; unsigned garbage; Trace(("Parser running at {%lu}", 32ul + 32ul * parser_bs.offset - parser_bs.live)); parse_token = 0; --work_units; true_bitstream = attach(parser_bs); rv = parse(&par, &head_blk.hdr, &true_bitstream, &garbage); advance(detach(true_bitstream)); check_invariants(); Trace(("Parser advancved to {%lu}", 32ul + 32ul * parser_bs.offset - parser_bs.live)); if (rv == MORE) { parse_token = true; work_units++; check_invariants(); return; } if (rv == FINISH) { source_close(); parse_token = true; parsing_done = true; assert(garbage <= 32); assert(parser_bs.live < 32); assert(parser_bs.offset <= tail_offs); parser_bs.live += garbage; if (parser_bs.live >= 32) { parser_bs.live -= 32; parser_bs.offset--; } if (parser_bs.offset == tail_offs && parser_bs.live < 8 * eof_missing) { failf(&ispec, "compressed data error: %s", err2str(ERR_EOF)); } Trace(("Parser encountered End-Of-File at {%lu}", 32ul + 32ul * parser_bs.offset - parser_bs.live)); /* Release input blocks. */ while (!empty(input_q)) { struct in_blk *blk = shift(input_q); head_offs += blk->size; if (--blk->ref_count == 0) { source_release_buffer(blk->buffer); free(blk); } } /* Release retrieve jobs. */ while (!empty(retr_q)) { struct retr_blk *rb = dequeue(retr_q); Trace(("Parser discovered a bit pattern beyond EOF at {%u}", nbsx2(rb->base))); decoder_free(&rb->ds); free(rb); work_units++; } /* Release scan jobs. */ while (!empty(scan_q)) { free(dequeue(scan_q)); } /* Release unord blocks. */ while (!empty(unord_q)) { struct unord_blk *ublk = dequeue(unord_q); if (ublk->complete) free(ublk); else { ublk->complete = true; ublk->legitimate = false; } } /* Release work unit. */ work_units++; check_invariants(); return; } if (rv != OK) { Trace(("Parser found a parse error at {%lu}", 32ul + 32ul * parser_bs.offset - parser_bs.live)); failf(&ispec, "compressed data error: %s", err2str(rv)); } head_blk.base = parser_bs.pos; push(order_q, head_blk); while (!empty(unord_q) && pos_lt(peek(unord_q)->base, parser_bs.pos)) { struct unord_blk *ublk = dequeue(unord_q); Trace(("Parser discovered a mis-recognized bit pattern at {%u}", nbsx2(ublk->base))); if (ublk->complete) { free(ublk); } else { ublk->complete = true; ublk->legitimate = false; } } if (!empty(unord_q) && pos_eq(peek(unord_q)->base, parser_bs.pos)) { struct unord_blk *ublk = dequeue(unord_q); Trace(("Parser took advantage of pattern found by scanner at {%u}", nbsx2(ublk->base))); advance(ublk->end_pos); if (ublk->complete) { parse_token = true; free(ublk); } else { ublk->complete = true; ublk->legitimate = true; } /* We didn't create retrieve job after all, so release work unit reserved for it. */ work_units++; } else { struct retr_blk *rb = XMALLOC(struct retr_blk); rb->unord_link = NULL; decoder_init(&rb->ds); rb->curr_pos = parser_bs; rb->base = parser_bs.pos; enqueue(retr_q, rb); Trace(("Parser found a unique block at {%u}", nbsx2(rb->base))); } check_invariants(); } static bool can_retrieve(void) { return !empty(retr_q) && can_attach(peek(retr_q)->curr_pos); } static void do_retrieve(void) { struct retr_blk *rb; struct emit_blk *eb; struct bitstream true_bitstream; int rv; assert(!parsing_done); rb = dequeue(retr_q); true_bitstream = attach(rb->curr_pos); rv = retrieve(&rb->ds, &true_bitstream); rb->curr_pos = detach(true_bitstream); if (parsing_done) { decoder_free(&rb->ds); free(rb); work_units++; check_invariants(); return; } if (rb->unord_link != NULL && rb->unord_link->complete && !rb->unord_link->legitimate) { /* We are not yet finished retrieving, but were proven not to be legitimate. Continuing would be pointless, so release resources and abort this retrieve job. */ Trace(("Retriever found himself redundand")); work_units++; decoder_free(&rb->ds); free(rb); check_invariants(); return; } if (rb->unord_link == NULL || rb->unord_link->complete) { advance(rb->curr_pos); Trace(("Retriever advanced parser to {%lu}", 32ul + 32ul * parser_bs.offset - parser_bs.live)); } else { rb->unord_link->end_pos = rb->curr_pos; } if (rv == MORE) { Trace(("Retriever blocked waiting for input")); enqueue(retr_q, rb); check_invariants(); return; } if (rb->unord_link != NULL && !rb->unord_link->complete) { /* Parser doesn't know about us yet, so we must be ahead of master. */ rb->unord_link->complete = true; rb->unord_link->end_pos = rb->curr_pos; } else { /* Parser already knows about us. We must be legitimate because non-legititimate blocks would have already been aborted. */ assert(rb->unord_link == NULL || (rb->unord_link->complete && rb->unord_link->legitimate)); /* Parser cannot be running now because we are the master so far. */ assert(!parse_token); /* We are done with the bitstream, pass mastership to the parser to let him continue. */ parse_token = 1; /* Parser knows about us, unord block is no longer needed. */ free(rb->unord_link); } check_invariants(); sched_unlock(); if (rv == OK) decode(&rb->ds); eb = XMALLOC(struct emit_blk); eb->ds = rb->ds; eb->base = rb->base; eb->end_offset = rb->curr_pos.offset; free(rb); eb->status = rv; sched_lock(); enqueue(emit_q, eb); check_invariants(); } static bool can_emit(void) { return (!empty(emit_q) && (out_slots > EMIT_THRESH || (out_slots > 0 && !empty(order_q) && pos_eq(peek(emit_q)->base, dq_get(order_q, 0).base)))); } static void do_emit(void) { struct emit_blk *eb; struct out_blk *oblk; int rv; out_slots--; eb = dequeue(emit_q); check_invariants(); sched_unlock(); oblk = xmalloc(sizeof(struct out_blk) + out_granul); oblk->size = out_granul; oblk->blk_sz = eb->ds.block_size; rv = eb->status; if (rv == OK) rv = emit(&eb->ds, oblk + 1, &oblk->size); oblk->size = out_granul - oblk->size; oblk->status = rv; oblk->base = eb->base; if (rv == MORE) { oblk->end_offset = 0; eb->base.minor++; sched_lock(); enqueue(emit_q, eb); } else { oblk->end_offset = eb->end_offset; oblk->crc = eb->ds.crc; decoder_free(&eb->ds); free(eb); sched_lock(); work_units++; } enqueue(reord_q, oblk); check_invariants(); } static bool can_reorder(void) { return (!empty(reord_q) && ((!empty(order_q) && pos_le(peek(reord_q)->base, dq_get(order_q, 0).base)) || (empty(order_q) && parsing_done))); } static void do_reorder(void) { struct out_blk *oblk; struct head_blk ord; uintmax_t offs_incr; if (empty(order_q) || pos_lt(peek(reord_q)->base, dq_get(order_q, 0).base)) { Trace(("Rejected bogus block at {%u}", nbsx2(peek(reord_q)->base))); free(dequeue(reord_q)); out_slots++; check_invariants(); return; } ord = shift(order_q); oblk = dequeue(reord_q); offs_incr = (reord_offs < oblk->end_offset ? oblk->end_offset - reord_offs : 0u); reord_offs += offs_incr; if (oblk->blk_sz > ord.hdr.bs100k * 100000u) oblk->status = ERR_OVERFLOW; if (oblk->status == MORE) { ord.base.minor++; unshift(order_q, ord); } else { if (oblk->status == OK && oblk->crc != ord.hdr.crc) oblk->status = ERR_BLKCRC; if (oblk->status != OK) failf(&ispec, "compressed data error: %s", err2str(oblk->status)); } sink_write_buffer(oblk + 1, oblk->size, 4 * offs_incr); check_invariants(); } static bool can_scan(void) { return ((work_units > SCAN_THRESH || (work_units > 0u && !parse_token)) && !empty(scan_q) && can_attach(*peek(scan_q))); } static void do_scan(void) { struct detached_bitstream *bs; int scan_result; unsigned skip; struct bitstream true_bitstream; assert(!parsing_done); work_units--; bs = dequeue(scan_q); skip = 0u; assert(bs->pos.major >= parser_bs.pos.major); if (bs->pos.major == parser_bs.pos.major && bs->pos.minor < parser_bs.pos.minor) { skip = (parser_bs.pos.minor - bs->pos.minor) >> 27; } true_bitstream = attach(*bs); scan_result = scan(&true_bitstream, skip); *bs = detach(true_bitstream); if (scan_result != OK || parsing_done) { work_units++; free(bs); check_invariants(); return; } if (pos_le(bs->pos, parser_bs.pos)) { Trace(("Scanner found a known pattern at {%lu}", 32ul + 32ul * bs->offset - bs->live)); work_units++; } else { struct unord_blk *ub; struct retr_blk *rb; Trace(("Scanner found a unique match at {%lu}", 32ul + 32ul * bs->offset - bs->live)); ub = XMALLOC(struct unord_blk); ub->base = bs->pos; ub->end_pos = *bs; ub->complete = false; enqueue(unord_q, ub); rb = XMALLOC(struct retr_blk); rb->unord_link = ub; decoder_init(&rb->ds); rb->curr_pos = *bs; rb->base = bs->pos; enqueue(retr_q, rb); } if (true_bitstream.data != true_bitstream.limit && bs->offset >= head_offs) { enqueue(scan_q, bs); } else { free(bs); } check_invariants(); } static bool can_terminate(void) { return (eof && parsing_done && parse_token && work_units == num_worker && out_slots == total_out_slots); } static void on_input_avail(void *buffer, size_t size) { struct in_blk *iblk; struct detached_bitstream *scan_task; unsigned missing; iblk = XMALLOC(struct in_blk);; iblk->buffer = buffer; iblk->size = (size + 3) / 4; iblk->ref_count = 1u; iblk->offset = tail_offs; /* tail_offs is a protected variable, but current thread is the only one that can possibly alter it, so here we can read tail_offs without locking. */ missing = -size % 4; bzero((char *)buffer + size, missing); scan_task = XMALLOC(struct detached_bitstream); *scan_task = bits_init(tail_offs); sched_lock(); if (parsing_done) { sched_unlock(); free(iblk); free(scan_task); source_release_buffer(buffer); return; } eof_missing = missing; tail_offs += iblk->size; push(input_q, iblk); enqueue(scan_q, scan_task); check_invariants(); sched_unlock(); } static void on_write_complete(void *buffer) { struct out_blk *oblk = buffer; free(oblk - 1); sched_lock(); ++out_slots; check_invariants(); sched_unlock(); } static void init(void) { deque_init(input_q, in_slots); pqueue_init(scan_q, in_slots); pqueue_init(retr_q, work_units); pqueue_init(emit_q, work_units); pqueue_init(unord_q, (work_units + out_slots > UNORD_THRESH ? work_units + out_slots - UNORD_THRESH : 0)); deque_init(order_q, work_units + out_slots); pqueue_init(reord_q, out_slots); head_offs = 0; tail_offs = 0; eof_missing = 0; parsing_done = false; parse_token = true; reord_offs = 0; parser_bs = bits_init(0); parser_init(&par, bs100k); } static void uninit(void) { assert(parsing_done); assert(head_offs == tail_offs); assert(parse_token); pqueue_uninit(scan_q); pqueue_uninit(unord_q); deque_uninit(order_q); pqueue_uninit(reord_q); pqueue_uninit(emit_q); pqueue_uninit(retr_q); deque_uninit(input_q); } static const struct task task_list[] = { { "reorder", can_reorder, do_reorder }, { "parse", can_parse, do_parse }, { "emit", can_emit, do_emit }, { "retrieve", can_retrieve, do_retrieve }, { "scan", can_scan, do_scan }, { NULL, NULL, NULL }, }; const struct process expansion = { task_list, init, uninit, can_terminate, on_input_avail, on_write_complete, }; lbzip2-2.3/src/main.c000066400000000000000000000701471221755423000144510ustar00rootroot00000000000000/*- main.c -- main module Copyright (C) 2011, 2012 Mikolaj Izdebski Copyright (C) 2008, 2009, 2010 Laszlo Ersek This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include /* unlink() */ #include /* SIGPIPE */ #include /* va_list */ #include /* vfprintf() */ #include /* strcpy() */ #include /* lstat() */ #include /* open() */ #include "stat-time.h" /* get_stat_atime() */ #include "utimens.h" /* fdutimens() */ #include "signals.h" /* setup_signals() */ #include "main.h" /* pname */ unsigned num_worker; /* -n */ size_t max_mem; /* -m */ bool decompress; /* -d */ unsigned bs100k = 9; /* -1..-9 */ bool force; /* -f */ bool keep; /* -k */ bool verbose; /* -v */ bool print_cctrs; /* -S */ bool small; /* -s */ struct filespec ispec; struct filespec ospec; #define EX_OK 0 #define EX_WARN 4 static char *opathn; static const char *pname; static bool warned; /* Called just before abnormal program termination. */ void cleanup(void) { if (opathn != NULL) { (void)unlink(opathn); /* Don't release "opathn" -- the muxer might encounter a write error and access *opathn via "ofmt" for error reporting before the main thread re-raises the signal here. This is a deliberate leak, but we're on our (short) way out anyway. 16-Feb-2010 lacos */ opathn = NULL; } } /* Called when one of xalloc functions fails. */ void xalloc_die(void) { failx(errno, "xalloc"); } /* Logging utilities. */ static void log_generic(const struct filespec *fs, int code, const char *fmt, va_list args, int nl) __attribute__((format(printf, 3, 0))); static void log_generic(const struct filespec *fs, int code, const char *fmt, va_list args, int nl) { if (0 > fprintf(stderr, "%s: ", pname) || (fs && 0 > fprintf(stderr, "%s%s%s: ", fs->sep, fs->fmt, fs->sep)) || 0 > vfprintf(stderr, fmt, args) || (0 != code && 0 > fprintf(stderr, ": %s", strerror(code))) || (nl && 0 > fprintf(stderr, "\n")) || 0 != fflush(stderr)) bailout(); } #define DEF(proto, f, x, warn, bail, nl) \ void proto \ { \ va_list args; \ \ flockfile(stderr); \ va_start(args, fmt); \ if (!bail || (EPIPE != x && EFBIG != x)) { \ log_generic(f, x, fmt, args, nl); \ } \ \ if (!bail) { \ va_end(args); \ if (warn) \ warned = 1; \ funlockfile(stderr); \ } \ else \ bailout(); \ } DEF(info ( const char *fmt, ...), 0,0,0,0,1) DEF(infof (const struct filespec *f, const char *fmt, ...), f,0,0,0,1) DEF(infox ( int x, const char *fmt, ...), 0,x,0,0,1) DEF(infofx (const struct filespec *f, int x, const char *fmt, ...), f,x,0,0,1) DEF(warn ( const char *fmt, ...), 0,0,1,0,1) DEF(warnf (const struct filespec *f, const char *fmt, ...), f,0,1,0,1) DEF(warnx ( int x, const char *fmt, ...), 0,x,1,0,1) DEF(warnfx (const struct filespec *f, int x, const char *fmt, ...), f,x,1,0,1) DEF(fail ( const char *fmt, ...), 0,0,0,1,1) DEF(failf (const struct filespec *f, const char *fmt, ...), f,0,0,1,1) DEF(failx ( int x, const char *fmt, ...), 0,x,0,1,1) DEF(failfx (const struct filespec *f, int x, const char *fmt, ...), f,x,0,1,1) DEF(display( /* WITH NO ADVANCING :) */ const char *fmt, ...), 0,0,0,0,0) #undef DEF enum outmode { OM_STDOUT, /* Write all output to stdout, -c. */ OM_DISCARD, /* Discard output, -t. */ OM_REGF /* Write output to files; neither -t nor -c */ }; static enum outmode outmode = OM_REGF; /* How to store output, -c/-t. */ /* Names of other recognized environment variables. */ static const char *const ev_name[] = { "LBZIP2", "BZIP2", "BZIP" }; /* Separator characters in environment variable values. No escaping. */ static const char envsep[] = " \t"; static uintmax_t xstrtol(const char *str, int source, uintmax_t lower, uintmax_t upper) { long tmp; char *endptr; unsigned shift; uintmax_t val; const char *suffix = "EePpTtGgMmKk"; if ('\0' == *str) goto fail; errno = 0; tmp = strtol(str, &endptr, 10); if (0 != errno || tmp < 0 || (endptr[0] != '\0' && endptr[1] != '\0')) goto fail; val = tmp; endptr = index(suffix, *endptr); if (endptr == NULL) goto fail; shift = (index(suffix, '\0') - endptr + 1) / 2 * 10; if (val > (UINTMAX_MAX >> shift)) goto fail; val <<= shift; if (val < lower || val > upper) { fail: fail("failed to parse \"%s\" from \"-%c\" as an integer in [%ju..%ju]," " specify \"-h\" for help", str, source, lower, upper); } return val; } /* The usage message, displayed when user gives us `--help' option. The following macro definition was generated by pretty-usage.pl script. To alter the message, simply edit and run pretty-usage.pl. It will patch the macro definition automatically. */ #define USAGE_STRING "%s%s%s%s%s%s", "Usage:\n1. PROG [-n WTHRS] [-k|-c|-t] [-\ d|-z] [-1 .. -9] [-f] [-v] [-S] {FILE}\n2. PROG -h|-V\n\nRecognized PROG names\ :\n\n bunzip2, lbunzip2 : Decompress. Forceable with `-d'.\n bzcat, lbzcat \ : Decompress to stdout. Forceable with `-cd'.\n : Com\ press. Forceable with `-z'.\n\nEnvironment variables:\n\n LBZIP2, BZIP2,\n B\ ZIP : Insert arguments between PROG and the rest of the\n \ command line. Tokens are separated by spaces and tabs;\n \ ", " no escaping.\n\nOptions:\n\n -n WTHRS : Set th\ e number of (de)compressor threads to WTHRS, where\n WTH\ RS is a positive integer.\n -k, --keep : Don't remove FILE operands. \ Open regular input files\n with more than one link.\n -\ c, --stdout : Write output to stdout even with FILE operands. Implies\n \ `-k'. Incompatible with `-t'.\n -t, --test : Te\ st decompression; discard output instead of writing it\n ", " \ to files or stdout. Implies `-k'. Incompatible with\n \ `-c'.\n -d, --decompress : Force decompression over the selection by PRO\ G.\n -z, --compress : Force compression over the selection by PROG.\n -1\ .. -9 : Set the compression block size to 100K .. 900K.\n --fast \ : Alias for `-1'.\n --best : Alias for `-9'. This is t\ he default.\n -f, --force : Open non-regular input files. Open input f\ iles with more\n than one", " link. Try to remove each o\ utput file before\n opening it. With `-cd' copy files no\ t in bzip2 format.\n -v, --verbose : Log each (de)compression start to s\ tderr. Display\n compression ratio and space savings. Di\ splay progress\n information if stderr is connected to a\ terminal.\n -S : Print condition variable statistics to stde\ rr.\n -s, --small, -q,\n --quiet,\n --repetitive-fast,\n --repetitive-best\ ,\n --exponential ", ": Accepted for compatibility, otherwise ignored.\n\ -h, --help : Print this help to stdout and exit.\n -L, --license, -\ V,\n --version : Print version information to stdout and exit.\n\nOp\ erands:\n\n FILE : Specify files to compress or decompress. If \ no FILE is\n given, work as a filter. FILEs with `.bz2',\ `.tbz',\n `.tbz2' and `.tz2' name suffixes will be skip\ ped when\n compressing. When decompressing, `.bz2' suffi\ x", "es will be\n removed in output filenames; `.tbz', `\ .tbz2' and `.tz2'\n suffixes will be replaced by `.tar';\ other filenames\n will be suffixed with `.out'.\n" #define HELP_STRING "%s version %s\n%s\n\n%s%s", \ PACKAGE_NAME, PACKAGE_VERSION, "http://lbzip2.org/", \ "Copyright (C) 2011, 2012 Mikolaj Izdebski\n" \ "Copyright (C) 2008, 2009, 2010 Laszlo Ersek\n" \ "\n" \ "This program is free software: you can redistribute it and/or modify\n" \ "it under the terms of the GNU General Public License as published by\n" \ "the Free Software Foundation, either version 3 of the License, or\n" \ "(at your option) any later version.\n" \ "\n", \ "This program is distributed in the hope that it will be useful,\n" \ "but WITHOUT ANY WARRANTY; without even the implied warranty of\n" \ "MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n" \ "GNU General Public License for more details.\n" \ "\n" \ "You should have received a copy of the GNU General Public License\nal" \ "ong with this program. If not, see .\n" static void _Noreturn usage(void) { if (0 > printf(USAGE_STRING)) failx(errno, "printf()"); if (0 != fclose(stdout)) failx(errno, "fclose(stdout)"); _exit(EX_OK); } static void _Noreturn version(void) { if (0 > printf(HELP_STRING)) failx(errno, "printf()"); if (0 != fclose(stdout)) failx(errno, "fclose(stdout)"); _exit(EX_OK); } struct arg { struct arg *next; const char *val; }; static void opts_outmode(char ch) { assert('c' == ch || 't' == ch); if (('c' == ch ? OM_DISCARD : OM_STDOUT) == outmode) { fail("\"-c\" and \"-t\" are incompatible, specify \"-h\" for help"); } if ('c' == ch) { outmode = OM_STDOUT; } else { outmode = OM_DISCARD; decompress = 1; } } static void opts_decompress(char ch) { assert('d' == ch || 'z' == ch); decompress = ('d' == ch); if (OM_DISCARD == outmode) { outmode = OM_REGF; } } static void opts_setup(struct arg **operands, size_t argc, char **argv) { struct arg **link_at; uintmax_t mx_worker; /* Create a homogeneous argument list from the recognized environment variables and from the command line. */ *operands = 0; link_at = operands; { size_t ofs; for (ofs = 0u; ofs < sizeof ev_name / sizeof ev_name[0]; ++ofs) { char *ev_val; ev_val = getenv(ev_name[ofs]); if (0 != ev_val) { char *tok; for (tok = strtok(ev_val, envsep); 0 != tok; tok = strtok(0, envsep)) { struct arg *arg; arg = XMALLOC(struct arg); arg->next = NULL; arg->val = tok; *link_at = arg; link_at = &arg->next; } } } for (ofs = 1u; ofs < argc; ++ofs) { struct arg *arg; arg = XMALLOC(struct arg); arg->next = NULL; arg->val = argv[ofs]; *link_at = arg; link_at = &arg->next; } } /* Effectuate option defaults. */ #ifdef _SC_THREAD_THREADS_MAX mx_worker = sysconf(_SC_THREAD_THREADS_MAX); #else mx_worker = -1; #endif mx_worker = min(mx_worker, min(UINT_MAX, SIZE_MAX / sizeof(pthread_t))); if (strcmp(pname, "bunzip2") == 0 || strcmp(pname, "lbunzip2") == 0) { decompress = 1; } else if (strcmp(pname, "bzcat") == 0 || strcmp(pname, "lbzcat") == 0) { outmode = OM_STDOUT; decompress = 1; } /* Process and remove all arguments that are options or option arguments. The remaining arguments are the operands. */ link_at = operands; { enum { AS_CONTINUE, /* Continue argument processing. */ AS_STOP, /* Processing finished because of "--". */ AS_USAGE, /* User asked for help. */ AS_VERSION /* User asked for version. */ } args_state; struct arg *arg, *next; args_state = AS_CONTINUE; for (arg = *operands; 0 != arg && AS_CONTINUE == args_state; arg = next) { const char *argscan; argscan = arg->val; if ('-' != *argscan) { /* This is an operand, keep it. */ link_at = &arg->next; next = arg->next; } else { /* This argument holds options and possibly an option argument. */ ++argscan; if ('-' == *argscan) { ++argscan; if ('\0' == *argscan) { args_state = AS_STOP; } else if (0 == strcmp("stdout", argscan)) { opts_outmode('c'); } else if (0 == strcmp("test", argscan)) { opts_outmode('t'); } else if (0 == strcmp("decompress", argscan)) { opts_decompress('d'); } else if (0 == strcmp("compress", argscan)) { opts_decompress('z'); } else if (0 == strcmp("fast", argscan)) { bs100k = 1; } else if (0 == strcmp("best", argscan)) { bs100k = 9; } else if (0 == strcmp("force", argscan)) { force = 1; } else if (0 == strcmp("keep", argscan)) { keep = 1; } else if (0 == strcmp("small", argscan)) { small = 1; } else if (0 == strcmp("verbose", argscan)) { verbose = 1; } else if (0 == strcmp("help", argscan)) { args_state = AS_USAGE; } else if (0 == strcmp("license", argscan) || 0 == strcmp("version", argscan)) { args_state = AS_VERSION; } else if (0 != strcmp("quiet", argscan) && 0 != strcmp("repetitive-fast", argscan) && 0 != strcmp("repetitive-best", argscan) && 0 != strcmp("exponential", argscan)) { fail("unknown option \"%s\", specify \"-h\" for help", arg->val); } } /* long option */ else { int cont; cont = 1; do { int opt; opt = *argscan; switch (opt) { case '\0': cont = 0; break; case 'c': case 't': opts_outmode(opt); break; case 'd': case 'z': opts_decompress(opt); break; case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': bs100k = opt - '0'; break; case 'f': force = 1; break; case 'k': keep = 1; break; case 's': small = 1; break; case 'v': verbose = 1; break; case 'S': print_cctrs = 1; break; case 'q': break; case 'h': args_state = AS_USAGE; cont = 0; break; case 'L': case 'V': args_state = AS_VERSION; cont = 0; break; case 'n': case 'm': ++argscan; if ('\0' == *argscan) { /* Drop this argument, as it wasn't an operand. */ next = arg->next; free(arg); *link_at = next; /* Move to next argument, which is an option argument. */ arg = next; if (NULL == arg) { fail("option \"-%.1s\" requires an argument," " specify \"-h\" for help", argscan - 1); } argscan = arg->val; } if (opt == 'n') num_worker = xstrtol(argscan, opt, 1, mx_worker); else max_mem = xstrtol(argscan, opt, 1, SIZE_MAX); cont = 0; break; default: fail("unknown option \"-%c\", specify \"-h\" for" " help", opt); } /* switch (*argscan) */ ++argscan; } while (cont); } /* cluster of short options */ /* This wasn't an operand, drop it. */ next = arg->next; free(arg); *link_at = next; } /* argument holds options */ } /* arguments loop */ if (AS_USAGE == args_state || AS_VERSION == args_state) { for (arg = *operands; 0 != arg; arg = next) { next = arg->next; free(arg); } if (AS_USAGE == args_state) usage(); else version(); } } /* process arguments */ /* Finalize options. */ if (OM_REGF == outmode && 0 == *operands) { outmode = OM_STDOUT; } if (decompress) { if (0 == *operands && isatty(STDIN_FILENO)) { fail("won't read compressed data from a terminal, specify" " \"-h\" for help"); } } else { if (OM_STDOUT == outmode && isatty(STDOUT_FILENO)) { fail("won't write compressed data to a terminal, specify" " \"-h\" for help"); } } if (0u == num_worker) { #ifdef _SC_NPROCESSORS_ONLN long num_online; num_online = sysconf(_SC_NPROCESSORS_ONLN); if (-1 == num_online) { fail("number of online processors unavailable, specify \"-h\" for help"); } assert(1L <= num_online); num_worker = min(mx_worker, (unsigned long)num_online); #else fail("WORKER-THREADS not set, specify \"-h\" for help"); #endif } } /* Dual purpose: a) Is the current operand already compressed? b) What decompressed suffix corresponds to the current compressed suffix? */ struct suffix { const char *compr; /* Suffix of compressed file. */ size_t compr_len; /* Its length (not size). */ const char *decompr; /* Suffix of decompressed file. */ size_t decompr_len; /* Its length (not size). */ int chk_compr; /* If "compr" is suited for purpose "a". */ }; #define SUF(c, dc, c1) { c, sizeof c - 1u, dc, sizeof dc - 1u, c1 } static const struct suffix suffix[] = { SUF(".bz2", "", 1), SUF(".tbz2", ".tar", 1), SUF(".tbz", ".tar", 1), SUF(".tz2", ".tar", 1), SUF("", ".out", 0) }; #undef SUF /* If "decompr_pathname" is NULL: check if "compr_pathname" has a compressed suffix. If "decompr_pathname" is not NULL: allocate and format a pathname for storing the decompressed output -- this always returns 1. */ static int suffix_xform(const char *compr_pathname, char **decompr_pathname) { size_t len, ofs; len = strlen(compr_pathname); for (ofs = 0u; ofs < sizeof suffix / sizeof suffix[0]; ++ofs) { if ((suffix[ofs].chk_compr || 0 != decompr_pathname) && len >= suffix[ofs].compr_len) { size_t prefix_len; prefix_len = len - suffix[ofs].compr_len; if (0 == strcmp(compr_pathname + prefix_len, suffix[ofs].compr)) { if (0 != decompr_pathname) { if (SIZE_MAX - prefix_len < suffix[ofs].decompr_len + 1u) { fail("\"%s\": size_t overflow in dpn_alloc\n", compr_pathname); } *decompr_pathname = xmalloc(prefix_len + suffix[ofs].decompr_len + 1u); (void)memcpy(*decompr_pathname, compr_pathname, prefix_len); (void)strcpy(*decompr_pathname + prefix_len, suffix[ofs].decompr); } return 1; } } } assert(0 == decompr_pathname); return 0; } /* If input is unavailable (skipping), return -1. Otherwise: - return 0, - store the file descriptor to read from (might be -1 if discarding), - if input is coming from a successfully opened FILE operand, fill in "*sbuf" via fstat() -- but "*sbuf" may be modified without this, too, - set up "ispec.sep" and "ispec.fmt" for logging; the character arrays pointed to by them won't need to be released (or at least not through these aliases). */ static int input_init(const struct arg *operand, struct stat *sbuf) { ispec.total = 0u; if (0 == operand) { ispec.fd = STDIN_FILENO; ispec.sep = ""; ispec.fmt = "stdin"; ispec.size = 0u; return 0; } if (!force) { if (-1 == lstat(operand->val, sbuf)) { warnx(errno, "skipping \"%s\": lstat()", operand->val); return -1; } if (OM_REGF == outmode && !S_ISREG(sbuf->st_mode)) { warn("skipping \"%s\": not a regular file", operand->val); return -1; } if (OM_REGF == outmode && !keep && sbuf->st_nlink > (nlink_t) 1) { warn("skipping \"%s\": more than one links", operand->val); return -1; } } if (!decompress && suffix_xform(operand->val, 0)) { warn("skipping \"%s\": compressed suffix", operand->val); } else { int infd; infd = open(operand->val, O_RDONLY | O_NOCTTY); if (-1 == infd) { warnx(errno, "skipping \"%s\": open()", operand->val); } else { if (-1 != fstat(infd, sbuf)) { ispec.fd = infd; ispec.sep = "\""; ispec.fmt = operand->val; assert(0 <= sbuf->st_size); ispec.size = sbuf->st_size; return 0; } warnx(errno, "skipping \"%s\": fstat()", operand->val); if (-1 == close(infd)) { failx(errno, "close(\"%s\")", operand->val); } } } return -1; } static void input_oprnd_rm(const struct arg *operand) { assert(0 != operand); if (-1 == unlink(operand->val) && ENOENT != errno) { warnx(errno, "unlink(\"%s\")", operand->val); } } static void input_uninit(void) { if (-1 == close(ispec.fd)) { failx(errno, "close(%s%s%s)", ispec.sep, ispec.fmt, ispec.sep); } } /* If skipping (output unavailable), return -1. Otherwise: - return 0, - store the file descriptor to write to (might be -1 if discarding), - if we write to a regular file, store the dynamically allocated output pathname, - set up "ospec.sep" and "ospec.fmt" for logging; the character arrays pointed to by them won't need to be released (or at least not through these aliases). */ static int output_init(const struct arg *operand, const struct stat *sbuf) { ospec.total = 0u; switch (outmode) { case OM_STDOUT: ospec.fd = STDOUT_FILENO; ospec.sep = ""; ospec.fmt = "stdout"; return 0; case OM_DISCARD: ospec.fd = -1; ospec.sep = ""; ospec.fmt = "the bit bucket"; return 0; case OM_REGF: assert(0 != operand); { char *tmp; if (decompress) { (void) suffix_xform(operand->val, &tmp); } else { size_t len; len = strlen(operand->val); if (SIZE_MAX - sizeof ".bz2" < len) { fail("\"%s\": size_t overflow in cpn_alloc\n", operand->val); } tmp = xmalloc(len + sizeof ".bz2"); (void)memcpy(tmp, operand->val, len); (void)strcpy(tmp + len, ".bz2"); } if (force && -1 == unlink(tmp) && ENOENT != errno) { /* This doesn't warrant a warning in itself, just an explanation if the following open() fails. */ infox(errno, "unlink(\"%s\")", tmp); } ospec.fd = open(tmp, O_WRONLY | O_CREAT | O_EXCL, sbuf->st_mode & (S_IRUSR | S_IWUSR)); if (-1 == ospec.fd) { warnx(errno, "skipping \"%s\": open(\"%s\")", operand->val, tmp); free(tmp); } else { opathn = tmp; ospec.sep = "\""; ospec.fmt = tmp; return 0; } } break; default: assert(0); } return -1; } static void output_regf_uninit(int outfd, const struct stat *sbuf) { assert(0 != opathn); if (-1 == fchown(outfd, sbuf->st_uid, sbuf->st_gid)) { /* File stays with euid:egid, and at most 0600. */ warnx(errno, "fchown(\"%s\")", opathn); } else { if (sbuf->st_mode & (S_ISUID | S_ISGID | S_ISVTX)) { warn("\"%s\": won't restore any of setuid, setgid, sticky", opathn); } if (-1 == fchmod(outfd, sbuf->st_mode & (S_IRWXU | S_IRWXG | S_IRWXO))) { /* File stays with orig-uid:orig-gid, and at most 0600. */ warnx(errno, "fchmod(\"%s\")", opathn); } } { struct timespec ts[2]; ts[0] = get_stat_atime(sbuf); ts[1] = get_stat_mtime(sbuf); if (-1 == fdutimens(outfd, opathn, ts)) { warnx(errno, "fdutimens(\"%s\")", opathn); } } if (-1 == close(outfd)) { failx(errno, "close(\"%s\")", opathn); } free(opathn); opathn = 0; } int main(int argc, char **argv) { struct arg *operands; static char stderr_buf[BUFSIZ]; pname = strrchr(argv[0], '/'); pname = pname ? pname + 1 : argv[0]; setbuf(stderr, stderr_buf); setup_signals(); opts_setup(&operands, argc, argv); /* TODO: For now --small is ignored as it wasn't tested enough... Test it, enable, and document (in manpage and usage string) */ small = 0; do { /* Process operand. */ { int ret; struct stat instat; ret = input_init(operands, &instat); if (-1 != ret) { cli(); if (-1 != output_init(operands, &instat)) { work(); if (OM_REGF == outmode) { output_regf_uninit(ospec.fd, &instat); if (!keep) { input_oprnd_rm(operands); } } /* Display data compression ratio and space savings, but only if the user desires so. */ if (verbose && 0u < ispec.total && 0u < ospec.total) { uintmax_t plain_size, compr_size; double ratio, savings, ratio_magnitude; /* Do the math. Note that converting from uintmax_t to double *may* result in precision loss, but that shouldn't matter. */ plain_size = !decompress ? ispec.total : ospec.total; compr_size = ispec.total ^ ospec.total ^ plain_size; ratio = (double)compr_size / plain_size; savings = 1 - ratio; ratio_magnitude = ratio < 1 ? 1 / ratio : ratio; infof(&ispec, "compression ratio is %s%.3f%s, space savings is " "%.2f%%", ratio < 1 ? "1:" : "", ratio_magnitude, ratio < 1 ? "" : ":1", 100 * savings); } } /* output available or discarding */ sti(); input_uninit(); } /* input available */ } /* Move to next operand. */ if (0 != operands) { struct arg *next; next = operands->next; free(operands); operands = next; } } while (0 != operands); assert(0 == opathn); if (OM_STDOUT == outmode && -1 == close(STDOUT_FILENO)) { failx(errno, "close(stdout)"); } _exit(warned ? EX_WARN : EX_OK); } lbzip2-2.3/src/main.h000066400000000000000000000062231221755423000144500ustar00rootroot00000000000000/*- main.h -- main module header Copyright (C) 2011, 2012 Mikolaj Izdebski Copyright (C) 2008, 2009, 2010 Laszlo Ersek This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include /* CHAR_BIT */ #if 8 != CHAR_BIT #error "Environments where 8 != CHAR_BIT are not supported." #endif /* The file specifier. The pointers "sep" and "fmt" point to character arrays that either don't need to be released, or need to be released through different aliases. These are prepared solely for logging. This is why the pointed to chars are qualified as const. */ struct filespec { int fd; /* the file descriptor; -1 if none */ const char *sep; /* name separator; either "" or "\"" */ const char *fmt; /* either file name or a special name */ uintmax_t total; /* total number of bytes transferred */ uintmax_t size; /* file size or 0 if unknown */ }; extern unsigned num_worker; /* -n */ extern size_t max_mem; /* -m */ extern bool decompress; /* -d */ extern unsigned bs100k; /* -1..-9 */ extern bool force; /* -f */ extern bool keep; /* -k */ extern bool verbose; /* -v */ extern bool print_cctrs; /* -S */ extern bool small; /* -s */ extern struct filespec ispec; extern struct filespec ospec; void info(const char *fmt, ...) __attribute__((format(printf, 1, 2))); void infof(const struct filespec *f, const char *fmt, ...) __attribute__((format(printf, 2, 3))); void infox(int x, const char *fmt, ...) __attribute__((format(printf, 2, 3))); void infofx(const struct filespec *f, int x, const char *fmt, ...) __attribute__((format(printf, 3, 4))); void warn(const char *fmt, ...) __attribute__((format(printf, 1, 2))); void warnf(const struct filespec *f, const char *fmt, ...) __attribute__((format(printf, 2, 3))); void warnx(int x, const char *fmt, ...) __attribute__((format(printf, 2, 3))); void warnfx(const struct filespec *f, int x, const char *fmt, ...) __attribute__((format(printf, 3, 4))); void _Noreturn fail(const char *fmt, ...) __attribute__((format(printf, 1, 2))); void _Noreturn failf(const struct filespec *f, const char *fmt, ...) __attribute__((format(printf, 2, 3))); void _Noreturn failx(int x, const char *fmt, ...) __attribute__((format(printf, 2, 3))); void _Noreturn failfx(const struct filespec *f, int x, const char *fmt, ...) __attribute__((format(printf, 3, 4))); void display(const char *fmt, ...) __attribute__((format(printf, 1, 2))); void work(void); lbzip2-2.3/src/parse.c000066400000000000000000000144531221755423000146350ustar00rootroot00000000000000/*- parse.c -- find block boundraries Copyright (C) 2011, 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" /* OK */ #include /* htonl() */ #include "main.h" /* Trace() */ #include "decode.h" /* bits_need() */ #include "scantab.h" #define bits_need(bs,n) \ ((n) <= (bs)->live \ ? \ OK \ : \ unlikely ((bs)->data == (bs)->limit) \ ? \ (bs)->eof \ ? \ FINISH \ : \ MORE \ : \ ((bs)->buff |= (uint64_t) ntohl \ (*(bs)->data) << (32u - (bs)->live), \ (bs)->data++, \ (bs)->live += 32u, \ OK)) #define bits_peek(bs,n) ((bs)->buff >> (64u - (n))) #define bits_dump(bs,n) \ ((bs)->buff <<= (n), \ (bs)->live -= (n), \ (void) 0) #define bits_align(bs) \ (bits_dump (bs, (bs)->live % 8u), \ (void) 0) #define bits_consume(bs) \ (bits_dump (bs, (bs)->live), \ (bs)->data = (bs)->limit, \ (void) 0) enum { STREAM_MAGIC_1, STREAM_MAGIC_2, BLOCK_MAGIC_1, BLOCK_MAGIC_2, BLOCK_MAGIC_3, BLOCK_CRC_1, BLOCK_CRC_2, EOS_2, EOS_3, EOS_CRC_1, EOS_CRC_2, }; void parser_init(struct parser_state *ps, int my_bs100k) { ps->state = BLOCK_MAGIC_1; ps->prev_bs100k = my_bs100k; ps->bs100k = -1; ps->computed_crc = 0u; } int parse(struct parser_state *restrict ps, struct header *restrict hd, struct bitstream *bs, unsigned *garbage) { assert(ps->state != ACCEPT); if (ps->state == ACCEPT) { return FINISH; } while (OK == bits_need(bs, 16)) { unsigned word = bits_peek(bs, 16); bits_dump(bs, 16); switch (ps->state) { case STREAM_MAGIC_1: if (0x425Au != word) { hd->bs100k = -1; hd->crc = 0; ps->state = ACCEPT; *garbage = 16; return FINISH; } ps->state = STREAM_MAGIC_2; continue; case STREAM_MAGIC_2: if (0x6839u < word || 0x6831 > word) { hd->bs100k = -1; hd->crc = 0; ps->state = ACCEPT; *garbage = 32; return FINISH; } ps->prev_bs100k = word & 15u; ps->bs100k = -1; ps->state = BLOCK_MAGIC_1; continue; case BLOCK_MAGIC_1: if (0x1772u == word) { ps->state = EOS_2; continue; } if (0x3141u != word) return ERR_HEADER; ps->state = BLOCK_MAGIC_2; continue; case BLOCK_MAGIC_2: if (0x5926u != word) return ERR_HEADER; ps->state = BLOCK_MAGIC_3; continue; case BLOCK_MAGIC_3: if (0x5359u != word) return ERR_HEADER; ps->state = BLOCK_CRC_1; continue; case BLOCK_CRC_1: ps->stored_crc = word; ps->state = BLOCK_CRC_2; continue; case BLOCK_CRC_2: hd->crc = (ps->stored_crc << 16) | word; hd->bs100k = ps->prev_bs100k; ps->computed_crc = (ps->computed_crc << 1) ^ (ps->computed_crc >> 31) ^ hd->crc; ps->prev_bs100k = ps->bs100k; ps->state = BLOCK_MAGIC_1; return OK; case EOS_2: if (0x4538u != word) return ERR_HEADER; ps->state = EOS_3; continue; case EOS_3: if (0x5090u != word) return ERR_HEADER; ps->state = EOS_CRC_1; continue; case EOS_CRC_1: ps->stored_crc = word; ps->state = EOS_CRC_2; continue; case EOS_CRC_2: ps->stored_crc = (ps->stored_crc << 16) | word; if (ps->stored_crc != ps->computed_crc) return ERR_STRMCRC; ps->computed_crc = 0u; bits_align(bs); ps->state = STREAM_MAGIC_1; continue; default: break; } assert(0); } if (FINISH != bits_need(bs, 16)) return MORE; if (ps->state == STREAM_MAGIC_1) { ps->state = ACCEPT; *garbage = 0; return FINISH; } if (ps->state == STREAM_MAGIC_2) { ps->state = ACCEPT; *garbage = 16; return FINISH; } return ERR_EOF; } int scan(struct bitstream *bs, unsigned skip) { unsigned state = 0; const uint32_t *data, *limit; if (skip > bs->live) { skip -= bs->live; bits_dump(bs, bs->live); skip = (skip + 31u) / 32u; if (bs->limit - bs->data < skip) bs->data = bs->limit; else bs->data += skip; } again: assert(state < ACCEPT); while (bs->live > 0) { unsigned bit = bits_peek(bs, 1); bits_dump(bs, 1); state = mini_dfa[state][bit]; if (state == ACCEPT) { if (bits_need(bs, 32) == OK) { bits_dump(bs, 32); return OK; } else { bits_consume(bs); return MORE; } } } data = bs->data; limit = bs->limit; while (data < limit) { unsigned bt_state = state; uint32_t word = *data; word = ntohl(word); state = big_dfa[state][word >> 24]; state = big_dfa[state][(uint8_t)(word >> 16)]; state = big_dfa[state][(uint8_t)(word >> 8)]; state = big_dfa[state][(uint8_t)word]; if (unlikely(state == ACCEPT)) { state = bt_state; bs->data = data; (void)bits_need(bs, 1u); goto again; } data++; } bs->data = data; return MORE; } lbzip2-2.3/src/process.c000066400000000000000000000341601221755423000151760ustar00rootroot00000000000000/*- process.c -- priority scheduling Copyright (C) 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include /* ntohl() */ #include /* pthread_t */ #include /* SIGUSR2 */ #include /* write() */ #include "timespec.h" /* struct timespec */ #include "main.h" /* work() */ #include "process.h" /* struct process */ #include "signals.h" /* halt() */ /* JOB SCHEDULING Jobs of different importance for the whole process are divided into several categories. Each category has a distinct priority assigned to it. Those priorities are static -- they were defined at lbzip2 design time and they can't change during run time. Any job can be scheduled only if there are no available jobs of higher priority. Therefore the primary scheduling algorithm of lbzip2 is static priority scheduling. The secondary algorithm is EDF (Earliest Deadline First). If two jobs of the same priority are to be scheduled (which implies they fall to the same job category) their scheduling order is determined basing on their underlying blocks order within the bzip2 file. Blocks that need to be outputed sooner have respectively higher priority than blocks that can be written later. The scheduler maintains several priority queues. Each queue contains jobs of the same type, ordered by their deadline. When there is some free computing power available (that is a worker thread is idle), the scheduler picks the first non-empty priority queue of the highest priority and removes from it the job of the earliest deadline. This job is then passed to the worker thread, which is executing it. The EDF algorithm is proven to be optimal in single-worker configuration. In this case it uses the least possible amount of resources (time, memory). In case of two or more worker threads much more resources may be needed, but the overall time should be no longer than in the case with one worker thread. */ /* Error-checking POSIX thread macros. For most pthread functions, non-zero return value means a programming bug. In this case aborting seems wiser than printing error message and exiting because abort() can produce code dumps that can be useful in debugging. */ #define xjoin(t) ((void)(pthread_join((t), NULL) && (abort(), 0))) #define xlock(m) ((void)(pthread_mutex_lock(m) && (abort(), 0))) #define xunlock(m) ((void)(pthread_mutex_unlock(m) && (abort(), 0))) #define xwait(c,m) ((void)(pthread_cond_wait((c),(m)) && (abort(), 0))) #define xsignal(c) ((void)(pthread_cond_signal(c) && (abort(), 0))) #define xbroadcast(c) ((void)(pthread_cond_broadcast(c) && (abort(), 0))) static void * thread_entry(void *real_entry) { ((void (*)(void))real_entry)(); return NULL; } /* Create a POSIX thread with error checking. */ static pthread_t xcreate(void (*entry)(void)) { int err; pthread_t thread; err = pthread_create(&thread, NULL, thread_entry, entry); if (err != 0) failx(err, "unable to create a POSIX thread"); return thread; } void xread(void *vbuf, size_t *vacant) { char *buffer = vbuf; assert(*vacant > 0); do { ssize_t rd; rd = read(ispec.fd, buffer, *vacant > (size_t)SSIZE_MAX ? (size_t)SSIZE_MAX : *vacant); /* End of file. */ if (0 == rd) break; /* Read error. */ if (-1 == rd) { failfx(&ispec, errno, "read()"); } *vacant -= (size_t)rd; buffer += (size_t)rd; ispec.total += (size_t)rd; } while (*vacant > 0); } void xwrite(const void *vbuf, size_t size) { const char *buffer = vbuf; ospec.total += size; if (size > 0 && ospec.fd != -1) { do { ssize_t wr; wr = write(ospec.fd, buffer, size > (size_t)SSIZE_MAX ? (size_t)SSIZE_MAX : size); /* Write error. */ if (-1 == wr) { failfx(&ospec, errno, "write()"); } size -= (size_t)wr; buffer += (size_t)wr; } while (size > 0); } } /* Parent and left child indices. */ #define parent(i) (((i)-1)/2) #define left(i) ((i)*2+1) void up_heap(void *vroot, unsigned size) { struct position **root = vroot; struct position *el; unsigned j; j = size; el = root[j]; while (j > 0 && pos_lt(*el, *root[parent(j)])) { root[j] = root[parent(j)]; j = parent(j); } root[j] = el; } void down_heap(void *vroot, unsigned size) { struct position **root = vroot; struct position *el; unsigned j; el = root[size]; root[size] = root[0]; j = 0; while (left(j) < size) { unsigned child = left(j); if (child + 1 < size && pos_lt(*root[child + 1], *root[child])) child++; if (pos_le(*el, *root[child])) break; root[j] = root[child]; j = child; } root[j] = el; } static pthread_mutex_t source_mutex = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t source_cond = PTHREAD_COND_INITIALIZER; static pthread_mutex_t sink_mutex = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t sink_cond = PTHREAD_COND_INITIALIZER; static pthread_mutex_t sched_mutex = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t sched_cond = PTHREAD_COND_INITIALIZER; static const struct process *process; bool eof; unsigned work_units; unsigned in_slots; unsigned out_slots; unsigned total_in_slots; unsigned total_out_slots; size_t in_granul; size_t out_granul; static bool request_close; static pthread_t source_thread; static pthread_t sink_thread; static pthread_t *worker_thread; struct block { void *buffer; size_t size; size_t weight; }; static struct deque(struct block) output_q; static bool finish; unsigned thread_id; /* Highest priority runnable task or NULL if there are no runnable tasks. */ static const struct task *next_task; static void source_thread_proc(void) { Trace((" source: spawned")); for (;;) { void *buffer; size_t vacant, avail; xlock(&source_mutex); while (in_slots == 0 && !request_close) { Trace((" source: stalled")); xwait(&source_cond, &source_mutex); } if (request_close) { Trace((" source: received premature close requtest")); xunlock(&source_mutex); break; } Trace((" source: reading data (%u free slots)", in_slots)); in_slots--; xunlock(&source_mutex); vacant = in_granul; avail = vacant; buffer = XNMALLOC(vacant, uint8_t); xread(buffer, &vacant); avail -= vacant; Trace((" source: block of %u bytes read", (unsigned)avail)); if (avail == 0u) source_release_buffer(buffer); else process->on_block(buffer, avail); if (vacant > 0u) break; } sched_lock(); eof = 1; sched_unlock(); Trace((" source: terminating")); } void source_release_buffer(void *buffer) { free(buffer); xlock(&source_mutex); if (in_slots++ == 0) xsignal(&source_cond); xunlock(&source_mutex); } void source_close(void) { xlock(&source_mutex); request_close = true; if (in_slots == 0) xsignal(&source_cond); xunlock(&source_mutex); } void sink_write_buffer(void *buffer, size_t size, size_t weight) { struct block block; block.buffer = buffer; block.size = size; block.weight = weight; xlock(&sink_mutex); push(output_q, block); xsignal(&sink_cond); xunlock(&sink_mutex); } static void sink_thread_proc(void) { bool progress_enabled; uintmax_t processed; struct timespec start_time; struct timespec next_time; struct timespec update_interval; struct block block; static const double UPDATE_INTERVAL = 0.1; Trace((" sink: spawned")); /* Progress info is displayed only if all the following conditions are met: 1) the user has specified -v or --verbose option 2) stderr is connected to a terminal device 3) the input file is a regular file 4) the input file is nonempty */ progress_enabled = (verbose && ispec.size > 0 && isatty(STDERR_FILENO)); processed = 0u; gettime(&start_time); next_time = start_time; update_interval = dtotimespec(UPDATE_INTERVAL); for (;;) { xlock(&sink_mutex); while (empty(output_q) && !finish) { Trace((" sink: stalled")); xwait(&sink_cond, &sink_mutex); } if (empty(output_q)) break; block = shift(output_q); xunlock(&sink_mutex); Trace((" sink: writing data (%u bytes)", (unsigned)block.size)); xwrite(block.buffer, block.size); Trace((" sink: releasing output slot")); process->on_written(block.buffer); if (progress_enabled) { struct timespec time_now; double completed, elapsed; processed = min(processed + block.weight, ispec.size); gettime(&time_now); if (timespec_cmp(time_now, next_time) > 0) { next_time = timespec_add(time_now, update_interval); elapsed = timespectod(timespec_sub(time_now, start_time)); completed = (double)processed / ispec.size; if (elapsed < 5) display("progress: %.2f%%\r", 100 * completed); else display("progress: %.2f%%, ETA: %.0f s \r", 100 * completed, elapsed * (1 / completed - 1)); } } } xunlock(&sink_mutex); Trace((" sink: terminating")); } static void select_task(void) { const struct task *task; for (task = process->tasks; task->ready != NULL; ++task) { if (task->ready()) { next_task = task; return; } } next_task = NULL; } static void worker_thread_proc(void) { unsigned id; (void)id; xlock(&sched_mutex); Trace(("worker[%2u]: spawned", (id = thread_id++))); for (;;) { while (next_task != NULL) { Trace(("worker[%2u]: scheduling task '%s'...", id, next_task->name)); next_task->run(); select_task(); } if (process->finished()) break; Trace(("worker[%2u]: stalled", id)); xwait(&sched_cond, &sched_mutex); } xbroadcast(&sched_cond); xunlock(&sched_mutex); Trace(("worker[%2u]: terminating", id)); } /* Enter scheduler monitor. */ void sched_lock(void) { xlock(&sched_mutex); } /* Leave scheduler monitor. */ void sched_unlock(void) { select_task(); if (next_task != NULL || process->finished()) xsignal(&sched_cond); xunlock(&sched_mutex); } static void init_io(void) { request_close = false; finish = false; deque_init(output_q, out_slots); sink_thread = xcreate(sink_thread_proc); source_thread = xcreate(source_thread_proc); } static void uninit_io(void) { xjoin(source_thread); xlock(&sink_mutex); finish = true; xsignal(&sink_cond); xunlock(&sink_mutex); xjoin(sink_thread); deque_uninit(output_q); } static void primary_thread(void) { unsigned i; thread_id = 0; eof = false; in_slots = total_in_slots; out_slots = total_out_slots; work_units = num_worker; process->init(); select_task(); init_io(); for (i = 1u; i < num_worker; ++i) worker_thread[i] = xcreate(worker_thread_proc); worker_thread_proc(); for (i = 1u; i < num_worker; ++i) xjoin(worker_thread[i]); uninit_io(); process->uninit(); assert(eof); assert(in_slots == total_in_slots); assert(out_slots == total_out_slots); assert(work_units == num_worker); xraise(SIGUSR2); } static void copy_on_input_avail(void *buffer, size_t size) { sched_lock(); out_slots--; sched_unlock(); sink_write_buffer(buffer, size, size); } static void copy_on_write_complete(void *buffer) { source_release_buffer(buffer); sched_lock(); out_slots++; sched_unlock(); } static bool copy_terminate(void) { if (eof && out_slots == total_out_slots) xraise(SIGUSR2); return false; } static void copy(void) { static const struct task null_task = { NULL, NULL, NULL }; static const struct process pseudo_process = { &null_task, NULL, NULL, copy_terminate, copy_on_input_avail, copy_on_write_complete, }; eof = false; in_slots = 2; out_slots = 2; total_out_slots = 2; in_granul = 65536; process = &pseudo_process; init_io(); halt(); uninit_io(); } static void schedule(const struct process *proc) { process = proc; worker_thread = XNMALLOC(num_worker, pthread_t); *worker_thread = xcreate(primary_thread); halt(); xjoin(*worker_thread); free(worker_thread); } /* TODO: Support -m switch */ static void set_memory_constraints(void) { if (!decompress) { total_in_slots = 2u * num_worker; total_out_slots = 2u * num_worker + /*TRANSM_THRESH*/2; in_granul = bs100k * 100000u; out_granul = -1; /* ignored during compression */ } else if (!small) { total_in_slots = 4u * num_worker; total_out_slots = 16u * num_worker; in_granul = 256u * 1024u; out_granul = MAX_BLOCK_SIZE; } else { total_in_slots = 2u; total_out_slots = 2u * num_worker; in_granul = 32768u; out_granul = 900000u; } } void work(void) { if (verbose) { info(decompress ? "decompressing %s%s%s to %s%s%s" : "compressing %s%s%s to %s%s%s", ispec.sep, ispec.fmt, ispec.sep, ospec.sep, ospec.fmt, ospec.sep); } set_memory_constraints(); if (!decompress) { schedule(&compression); } else { uint32_t header; size_t vacant = sizeof(header); xread(&header, &vacant); #define MAGIC(k) (0x425A6830u + (k)) if (vacant == 0 && (ntohl(header) >= MAGIC(1) && ntohl(header) <= MAGIC(9))) { bs100k = ntohl(header) - MAGIC(0); schedule(&expansion); } else if (force && ospec.fd == STDOUT_FILENO) { xwrite(&header, sizeof(header) - vacant); copy(); } else { failf(&ispec, "not a valid bzip2 file"); } } } lbzip2-2.3/src/process.h000066400000000000000000000117271221755423000152070ustar00rootroot00000000000000/*- process.h -- priority scheduling header Copyright (C) 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ struct task { const char *name; bool (*ready)(void); void (*run)(void); }; struct process { const struct task *tasks; void (*init)(void); void (*uninit)(void); bool (*finished)(void); void (*on_block)(void *buffer, size_t size); void (*on_written)(void *buffer); }; struct position { uint64_t major; uint64_t minor; }; #define pqueue(T) \ { \ T *restrict root; \ unsigned size; \ } #define deque(T) \ { \ T *restrict root; \ unsigned size; \ unsigned modulus; \ unsigned head; \ } extern bool eof; extern unsigned work_units; extern unsigned in_slots; extern unsigned out_slots; extern unsigned total_work_units; extern unsigned total_in_slots; extern unsigned total_out_slots; extern size_t in_granul; extern size_t out_granul; extern const struct process compression; extern const struct process expansion; #define pos_eq(a,b) ((a).major == (b).major && \ (a).minor == (b).minor) #define pos_lt(a,b) ((a).major < (b).major || \ ((a).major == (b).major && (a).minor < (b).minor)) #define pos_le(a,b) (!pos_lt(b,a)) #define pqueue_init(q,n) ((q).root = xmalloc((n) * sizeof(*(q).root)), \ (void)((q).size = 0)) #define pqueue_uninit(q) (assert(empty(q)), free((q).root)) #define peek(q) (*(q).root) #define enqueue(q,e) ((q).root[(q).size] = (e), \ up_heap((q).root, (q).size++)) #define dequeue(q) (down_heap((q).root, --(q).size), \ (q).root[(q).size]) #define deque_init(q,n) ((q).root = xmalloc((n) * sizeof(*(q).root)), \ (q).modulus = (n), \ (void)((q).head = (q).size = 0)) #define deque_uninit(q) (assert(empty(q)), free((q).root)) #define size(q) (+(q).size) #define empty(q) (!(q).size) #define dq_get(q,i) (assert((i) < (q).size), \ (q).root[min((q).head + (i) + 1, \ (q).head + (i) + 1 - (q).modulus)]) #define dq_set(q,i,e) (assert((i) < (q).size), \ (q).root[min((q).head + (i) + 1, \ (q).head + (i) + 1 - (q).modulus)] = (e)) #define shift(q) (assert(!empty(q)), \ (q).size--, \ (q).head = min((q).head + 1u, \ (q).head + 1u - (q).modulus), \ (q).root[(q).head]) #define unshift(q,e) (assert((q).size < (q).modulus), \ (q).size++, \ (q).root[(q).head] = (e), \ (q).head = min((q).head - 1, \ (q).head - 1 + (q).modulus), \ (void)0) #define push(q,e) (assert((q).size < (q).modulus), \ (q).size++, \ (q).root[min((q).head + (q).size, \ (q).head + (q).size - (q).modulus)] = (e)) #define pop(q) (assert(!empty(q)), \ (q).size--, \ (q).root[min((q).head + (q).size + 1, \ (q).head + (q).size + 1 - (q).modulus)]) void sched_lock(void); void sched_unlock(void); void source_close(void); void source_release_buffer(void *buffer); void sink_write_buffer(void *buffer, size_t size, size_t weight); void xread(void *vbuf, size_t *vacant); void xwrite(const void *vbuf, size_t size); void up_heap(void *root, unsigned size); void down_heap(void *root, unsigned size); lbzip2-2.3/src/signals.c000066400000000000000000000156141221755423000151630ustar00rootroot00000000000000/*- signals.c -- signal handling Copyright (C) 2011, 2012, 2013 Mikolaj Izdebski Copyright (C) 2008, 2009, 2010 Laszlo Ersek This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ #include "common.h" #include /* kill() */ #include /* pthread_sigmask() */ #include /* getpid() */ #include "signals.h" #define EX_FAIL 1 static void xempty(sigset_t *set) { if (sigemptyset(set) != 0) abort(); } static void xadd(sigset_t *set, int sig) { if (sigaddset(set, sig) != 0) abort(); } static void xmask(int how, const sigset_t *set, sigset_t *oset) { if (pthread_sigmask(how, set, oset) != 0) abort(); } static void xpending(sigset_t *set) { if (sigpending(set) != 0) abort(); } static bool xmember(const sigset_t *set, int sig) { unsigned rv; /* Cast return value to unsigned to save one comparison. */ rv = sigismember(set, sig); if (rv > 1) abort(); return rv; } static void xaction(int sig, void (*handler)(int)) { struct sigaction act; act.sa_handler = handler; xempty(&act.sa_mask); act.sa_flags = 0; if (sigaction(sig, &act, NULL) != 0) abort(); } /* SIGPIPE and SIGXFSZ will be blocked in all sub-threads during the entire lifetime of the process. Any EPIPE or EFBIG write() condition will be handled just as before lbzip2-0.23, when these signals were ignored. However, starting with lbzip2-0.23, SIGPIPE and/or SIGXFSZ will be generated for the offending thread(s) in addition. bailout(), when called by such sub-threads in response to EPIPE or EFBIG, or when called by other sub-threads failing concurrently (but a bit later) for any other reason, will forward (regenerate) the pending signal(s) for the whole process. The main thread will unblock these signals right before exiting with EX_FAIL in bailout(). 2010-03-03 lacos */ static const int blocked_signals[] = { SIGPIPE, SIGXFSZ, }; static const int handled_signals[] = { SIGUSR1, SIGUSR2, SIGINT, SIGTERM, }; /* Iterate over signal table. */ #define foreach(p,tab) for ((p) = (tab); \ (p) < (tab) + sizeof(tab) / sizeof(*(tab)); \ (p)++) /* sig_atomic_t is nowhere required to be able to hold signal values. */ static volatile sig_atomic_t caught_index; static void signal_handler(int caught) { const int *sig; foreach (sig, handled_signals) { if (*sig == caught) { caught_index = sig - handled_signals; return; } } abort(); } static pid_t pid; static pthread_t main_thread; static sigset_t blocked; static sigset_t handled; static sigset_t saved; void setup_signals(void) { const int *sig; pid = getpid(); main_thread = pthread_self(); xempty(&blocked); foreach (sig, blocked_signals) xadd(&blocked, *sig); xempty(&handled); foreach (sig, handled_signals) xadd(&handled, *sig); xmask(SIG_BLOCK, &blocked, 0); } /* Block signals. */ void cli(void) { const int *sig; xmask(SIG_BLOCK, &handled, &saved); foreach (sig, handled_signals) xaction(*sig, signal_handler); } /* Unblock signals. */ void sti(void) { const int *sig; foreach (sig, handled_signals) xaction(*sig, SIG_DFL); xmask(SIG_UNBLOCK, &handled, NULL); } /* Terminate the process with a signal. */ static void _Noreturn terminate(int sig) { sigset_t set; /* We might have inherited a SIG_IGN from the parent, but that would make no sense here. 24-OCT-2009 lacos */ xaction(sig, SIG_DFL); xraise(sig); xempty(&set); xadd(&set, sig); xmask(SIG_UNBLOCK, &set, NULL); _exit(EX_FAIL); } /* Unblock signals, wait for them, then block them again. */ void halt(void) { int sig; int ret; /* We could wait for signals with either sigwait() or sigsuspend(). SUSv2 states about sigwait() that its effect on signal actions is unspecified. SUSv3 still claims the same. The SUSv2 description of sigsuspend() talks about both the thread and the whole process being suspended until a signal arrives, although thread suspension seems much more likely from the wording. They note that they filed a clarification request for this. SUSv3 cleans this up and chooses thread suspension which was more logical anyway. I favor sigsuspend() because I need to re-raise SIGTERM and SIGINT, and unspecified action behavior with sigwait() seems messy. 13-OCT-2009 lacos */ ret = sigsuspend(&saved); assert(-1 == ret && EINTR == errno); sig = handled_signals[caught_index]; assert(xmember(&handled, sig)); switch (sig) { default: cleanup(); terminate(sig); case SIGUSR1: /* Error from a non-main thread via bailout(). */ bailout(); case SIGUSR2: /* Muxer thread joined other sub-threads and finished successfully. */ break; } } void xraise(int sig) { if (kill(pid, sig) != 0) abort(); } /* Promote signals pending on current thread to the process level. */ static void promote(void) { const int *sig; sigset_t pending; xempty(&pending); xpending(&pending); foreach (sig, blocked_signals) if (xmember(&pending, *sig)) xraise(*sig); } /* The treatment for fatal errors. If called from the main thread, remove any current output file and bail out. Primarily by unblocking any pending SIGPIPE/SIGXFSZ signals; both those forwarded by any sub-thread to the process level, and those generated for the main thread specifically, in response to an EPIPE/EFBIG write() condition. If no such signal is pending, or SIG_IGN was inherited through exec() as their actions, then bail out with the failure exit status. If called (indirectly) from any other thread, resignal the process with any pending SIGPIPE/SIGXFSZ. This will promote any such signal to the process level, if it was originally generated for the calling thread, accompanying an EPIPE/EFBIG write() condition. If the pending signal was already pending on the whole process, this will result in an idempotent kill(). Thereafter, send SIGUSR1 to the process, in order to signal the fatal error in the sub-thread. Finally, terminate the thread. */ void bailout(void) { if (pthread_equal(pthread_self(), main_thread)) { cleanup(); xmask(SIG_UNBLOCK, &blocked, NULL); _exit(EX_FAIL); } promote(); xraise(SIGUSR1); pthread_exit(NULL); } lbzip2-2.3/src/signals.h000066400000000000000000000015601221755423000151630ustar00rootroot00000000000000/*- signals.h -- signal handling header Copyright (C) 2012 Mikolaj Izdebski This file is part of lbzip2. lbzip2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. lbzip2 is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ void setup_signals(void); void cli(void); void sti(void); void halt(void); void xraise(int sig); void _Noreturn bailout(void); void cleanup(void); lbzip2-2.3/tests/000077500000000000000000000000001221755423000137235ustar00rootroot00000000000000lbzip2-2.3/tests/32767.bz2000066400000000000000000000100541221755423000150320ustar00rootroot00000000000000BZh91AY&SY>LF /i4f^.p } lbzip2-2.3/tests/32767.diff000066400000000000000000000053771221755423000152610ustar00rootroot00000000000000--- bzip2-0.1pl2.c 1997-08-30 01:32:15.000000000 +0200 +++ 32767.c 2011-08-20 12:02:19.000000000 +0200 @@ -1,18 +1,13 @@ - -/*-----------------------------------------------------------*/ -/*--- A block-sorting, lossless compressor bzip2.c ---*/ -/*-----------------------------------------------------------*/ - -/*-- - This program is bzip2, a lossless, block-sorting data compressor, - version 0.1pl2, dated 29-Aug-1997. - +/* This is an ancient version of bzip2 (0.1pl2) modified to produce + blocks with constant number of 32767 selectors. + Note that this program is full of bugs! Use with care! + */ +/*- + Copyright (C) 2011 Mikolaj Izdebski Copyright (C) 1996, 1997 by Julian Seward. - Guildford, Surrey, UK - email: jseward@acm.org - This program is free software; you can redistribute it and/or modify + This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or + the Free Software Foundation, either version 3 of the License, or (at your option) any later version. @@ -23,8 +18,18 @@ You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + along with this program. If not, see . +*/ - The GNU General Public License is contained in the file LICENSE. +/*-----------------------------------------------------------*/ +/*--- A block-sorting, lossless compressor bzip2.c ---*/ +/*-----------------------------------------------------------*/ + +/*-- + This program is bzip2, a lossless, block-sorting data compressor, + version 0.1pl2, dated 29-Aug-1997. + + Copyright (C) 1996, 1997 by Julian Seward. + Guildford, Surrey, UK + email: jseward@acm.org This program is based on (at least) the work of: @@ -160,5 +165,5 @@ run rather slowly. gcc version 2.x is recommended. --*/ - #ifdef __GNUC__ + #if 0 #define INLINE inline #define NORETURN __attribute__ ((noreturn)) @@ -476,5 +481,5 @@ #define N_ITERS 4 -#define MAX_SELECTORS (2 + (900000 / G_SIZE)) +#define MAX_SELECTORS 32767 Bool inUse[256]; @@ -1337,7 +1342,4 @@ if (!(nGroups < 8)) panic ( "sendMTFValues(1)" ); - if (!(nSelectors < 32768 && - nSelectors <= (2 + (900000 / G_SIZE)))) - panic ( "sendMTFValues(2)" ); @@ -1398,4 +1400,5 @@ /*--- Now the selectors. ---*/ + nSelectors = 32767; nBytes = bytesOut; bsW ( 3, nGroups ); @@ -1443,5 +1446,4 @@ selCtr++; } - if (!(selCtr == nSelectors)) panic ( "sendMTFValues(5)" ); if (verbosity >= 3) lbzip2-2.3/tests/Makefile.am000066400000000000000000000024311221755423000157570ustar00rootroot00000000000000#- # Copyright (C) 2011, 2012 Mikolaj Izdebski # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # check_PROGRAMS = minbzcat timeout minbzcat_SOURCES = minbzcat.c minbzcat_LDADD = $(top_builddir)/lib/libgnu.a timeout_SOURCES = timeout.c timeout_LDADD = $(top_builddir)/lib/libgnu.a TESTS_ENVIRONMENT = $(SHELL) $(srcdir)/Tester TESTS = \ 32767.bz2 \ ch255.bz2 \ concat.bz2 \ crc1.bz2 \ crc2.bz2 \ cve.bz2 \ cve2.bz2 \ empty.bz2 \ fib.bz2 \ gap.bz2 \ incomp-1.bz2 \ incomp-2.bz2 \ load0.bz2 \ overrun.bz2 \ rand.bz2 \ repet.bz2 \ trash.bz2 \ void.bz2 EXTRA_DIST = $(TESTS) 32767.diff bzip2-0.1pl2.c ch255.c crc2.diff cve.c fib.c \ Tester lbzip2-2.3/tests/README000066400000000000000000000042111221755423000146010ustar00rootroot00000000000000Test cases for lbzip2. -*- outline -*- * Compressor tests ** fib Check how compressor copes with a Fibonacci word on input. I think that Fibonacci word is the worst case for some BWT implementations. ** repet Check how compressor handles very repetitive data (a sequence of alternating two different characters). This seems to be the worst case for some bzip2 implementations (like ant). * Decompressor tests ** 32767 Test if decompressor supports blocks with more than 18001 selectors (in particular 32767 selectors). Some implementations (p7zip, ant) choke on this. ** ch255 Test how decompressor copes with a `zip bomb'. ** concat Check whether decompressor supports concatenated bzip2 streams. Some implementations don't support that (including older versions of the original bzip2). ** crc1 Determine whether decompressor checks block CRCs. Some old versions of yambi didn't check them. ** crc2 Determine whether decompressor checks stream CRCs. Pre-2.0 versions of lbzip2 don't check stream CRC. ** cve Test if decompressor is vulnerable to CVE-2010-0405 (integer overflow in BZ2_decompress). Most decompressors based on bzip2-1.0.5 or earlier are vulnerable. ** empty Check if decompressor supports empty compressed files. ** gap Check if decompressor allows a gap consisting of garbage bytes between bzip2 streams. Pre-2.0 lbzip2 allows that. ** incomp-[12] Check if cyclic, non-randomized files are supported. See comments in `incomp' file for more infomation. ** overrun Check if decompressor correctly rejects blocks that overrun maximal block size declared in bzip2 header. Pre-2.0 lbzip2 bounded block sizes on a flat value of 900000, ignoring the information in stream header. ** rand Check if decompressor supports randomized blocks. Old yambi versions lacked that. Some implementations (eg. busybox) don't seem to care about randomized blocks at all. ** trash Check if decompressor ignores trailing garbage after bzip2 stream. ** void Check if decompressor treats empty bz2 files as empty streams. lbzip2-2.3/tests/Tester000066400000000000000000000025211221755423000151140ustar00rootroot00000000000000#! /bin/sh # Copyright (C) 2011 Mikolaj Izdebski set -e -C IFS=' '' '' ' test -r "$1" # Try limiting virtual memory usage to 1 GiB. This can save users of systems # with memory overcommit enabled and no ulimit set. ulimit -v $(expr 1024 \* 1024) || : # Unset variables that may affect behavior of lbzip2. # (Some shells don't like unsetting variables that are not set.) LBZIP2= unset LBZIP2 BZIP2= unset BZIP2 BZIP= unset BZIP # Create temp files in current working directory. bn=$(echo "$1" | sed 's@^\(.*/\)*\([^/.][^/.]*\)\.[^/]*$@\2@') r1=tmp.$bn.1.$$ r2=tmp.$bn.2.$$ # Run decompressors, temporarly disabling error checking. set +e ./timeout ./minbzcat <"$1" >$r1 2>/dev/null; rc1=$? ./timeout ../src/lbzip2 -dcqn4 <"$1" >$r2 2>/dev/null; rc2=$? set -e # Make sure none of decompressors crashed (exit code >= 128) test $rc1 -lt 128 -a $rc2 -lt 128 # bad - bad bz2 file according to lbzip2 # xbad - bad bz2 file according to minbzcat (expected bad) test $rc1 = 0 && xbad= || xbad=1 test $rc2 = 0 && bad= || bad=1 # pdksh needs whitespace between the two left parentheses. # dash builtin test needs this redundand quoting before $bad and $xbad, # see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=598238 ( (test $xbad && test $bad) || (test ! ''$xbad && test ! ''$bad && cmp $r1 $r2 2>/dev/null)) && rc= || rc=1 rm -f $r1 $r2 exit $rc lbzip2-2.3/tests/bzip2-0.1pl2.c000066400000000000000000003451031221755423000160350ustar00rootroot00000000000000 /*-----------------------------------------------------------*/ /*--- A block-sorting, lossless compressor bzip2.c ---*/ /*-----------------------------------------------------------*/ /*-- This program is bzip2, a lossless, block-sorting data compressor, version 0.1pl2, dated 29-Aug-1997. Copyright (C) 1996, 1997 by Julian Seward. Guildford, Surrey, UK email: jseward@acm.org This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. The GNU General Public License is contained in the file LICENSE. This program is based on (at least) the work of: Mike Burrows David Wheeler Peter Fenwick Alistair Moffat Radford Neal Ian H. Witten Robert Sedgewick Jon L. Bentley For more information on these sources, see the file ALGORITHMS. --*/ /*----------------------------------------------------*/ /*--- IMPORTANT ---*/ /*----------------------------------------------------*/ /*-- WARNING: This program (attempts to) compress data by performing several non-trivial transformations on it. Unless you are 100% familiar with *all* the algorithms contained herein, and with the consequences of modifying them, you should NOT meddle with the compression or decompression machinery. Incorrect changes can and very likely *will* lead to disasterous loss of data. DISCLAIMER: I TAKE NO RESPONSIBILITY FOR ANY LOSS OF DATA ARISING FROM THE USE OF THIS PROGRAM, HOWSOEVER CAUSED. Every compression of a file implies an assumption that the compressed file can be decompressed to reproduce the original. Great efforts in design, coding and testing have been made to ensure that this program works correctly. However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program. DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. That is not to say this program is inherently unreliable. Indeed, I very much hope the opposite is true. bzip2 has been carefully constructed and extensively tested. PATENTS: To the best of my knowledge, bzip2 does not use any patented algorithms. However, I do not have the resources available to carry out a full patent search. Therefore I cannot give any guarantee of the above statement. --*/ /*----------------------------------------------------*/ /*--- and now for something much more pleasant :-) ---*/ /*----------------------------------------------------*/ /*---------------------------------------------*/ /*-- Place a 1 beside your platform, and 0 elsewhere. --*/ /*-- Generic 32-bit Unix. Also works on 64-bit Unix boxes. --*/ #define BZ_UNIX 1 /*-- Win32, as seen by Jacob Navia's excellent port of (Chris Fraser & David Hanson)'s excellent lcc compiler. --*/ #define BZ_LCCWIN32 0 /*---------------------------------------------*/ /*-- Some stuff for all platforms. --*/ #include #include #if DEBUG #include #endif #include #include #include #define ERROR_IF_EOF(i) { if ((i) == EOF) ioError(); } #define ERROR_IF_NOT_ZERO(i) { if ((i) != 0) ioError(); } #define ERROR_IF_MINUS_ONE(i) { if ((i) == (-1)) ioError(); } /*---------------------------------------------*/ /*-- Platform-specific stuff. --*/ #if BZ_UNIX #include #include #include #include #include #include #define Int32 int #define UInt32 unsigned int #define Char char #define UChar unsigned char #define Int16 short #define UInt16 unsigned short #define PATH_SEP '/' #define MY_LSTAT lstat #define MY_S_IFREG S_ISREG #define MY_STAT stat #define APPEND_FILESPEC(root, name) \ root=snocString((root), (name)) #define SET_BINARY_MODE(fd) /**/ /*-- You should try very hard to persuade your C compiler to inline the bits marked INLINE. Otherwise bzip2 will run rather slowly. gcc version 2.x is recommended. --*/ #ifdef __GNUC__ #define INLINE inline #define NORETURN __attribute__ ((noreturn)) #else #define INLINE /**/ #define NORETURN /**/ #endif #endif #if BZ_LCCWIN32 #include #include #include #define Int32 int #define UInt32 unsigned int #define Int16 short #define UInt16 unsigned short #define Char char #define UChar unsigned char #define INLINE /**/ #define NORETURN /**/ #define PATH_SEP '\\' #define MY_LSTAT _stat #define MY_STAT _stat #define MY_S_IFREG(x) ((x) & _S_IFREG) #if 0 /*-- lcc-win32 seems to expand wildcards itself --*/ #define APPEND_FILESPEC(root, spec) \ do { \ if ((spec)[0] == '-') { \ root = snocString((root), (spec)); \ } else { \ struct _finddata_t c_file; \ long hFile; \ hFile = _findfirst((spec), &c_file); \ if ( hFile == -1L ) { \ root = snocString ((root), (spec)); \ } else { \ int anInt = 0; \ while ( anInt == 0 ) { \ root = snocString((root), \ &c_file.name[0]); \ anInt = _findnext(hFile, &c_file); \ } \ } \ } \ } while ( 0 ) #else #define APPEND_FILESPEC(root, name) \ root = snocString ((root), (name)) #endif #define SET_BINARY_MODE(fd) \ do { \ int retVal = setmode ( fileno ( fd ), \ O_BINARY ); \ ERROR_IF_MINUS_ONE ( retVal ); \ } while ( 0 ) #endif /*---------------------------------------------*/ /*-- Some more stuff for all platforms :-) --*/ #define Bool unsigned char #define True 1 #define False 0 /*-- IntNative is your platform's `native' int size. Only here to avoid probs with 64-bit platforms. --*/ #define IntNative int /*-- change to 1, or compile with -DDEBUG=1 to debug --*/ #ifndef DEBUG #define DEBUG 0 #endif /*---------------------------------------------------*/ /*--- ---*/ /*---------------------------------------------------*/ /*-- Implementation notes, July 1997 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Memory allocation ~~~~~~~~~~~~~~~~~ All large data structures are allocated on the C heap, for better or for worse. That includes the various arrays of pointers, striped words, bytes, frequency tables and buffers for compression and decompression. bzip2 can operate at various block-sizes, ranging from 100k to 900k in 100k steps, and it allocates only as much as it needs to. When compressing, we know from the command-line options what the block-size is going to be, so all allocation can be done at start-up; if that succeeds, there can be no further allocation problems. Decompression is more complicated. Each compressed file contains, in its header, a byte indicating the block size used for compression. This means bzip2 potentially needs to reallocate memory for each file it deals with, which in turn opens the possibility for a memory allocation failure part way through a run of files, by encountering a file requiring a much larger block size than all the ones preceding it. The policy is to simply give up if a memory allocation failure occurs. During decompression, it would be possible to move on to subsequent files in the hope that some might ask for a smaller block size, but the complications for doing this seem more trouble than they are worth. Compressed file formats ~~~~~~~~~~~~~~~~~~~~~~~ [This is now entirely different from both 0.21, and from any previous Huffman-coded variant of bzip. See the associated file bzip2.txt for details.] Error conditions ~~~~~~~~~~~~~~~~ Dealing with error conditions is the least satisfactory aspect of bzip2. The policy is to try and leave the filesystem in a consistent state, then quit, even if it means not processing some of the files mentioned in the command line. `A consistent state' means that a file exists either in its compressed or uncompressed form, but not both. This boils down to the rule `delete the output file if an error condition occurs, leaving the input intact'. Input files are only deleted when we can be pretty sure the output file has been written and closed successfully. Errors are a dog because there's so many things to deal with. The following can happen mid-file, and require cleaning up. internal `panics' -- indicating a bug corrupted or inconsistent compressed file can't allocate enough memory to decompress this file I/O error reading/writing/opening/closing signal catches -- Control-C, SIGTERM, SIGHUP. Other conditions, primarily pertaining to file names, can be checked in-between files, which makes dealing with them easier. --*/ /*---------------------------------------------------*/ /*--- Misc (file handling) data decls ---*/ /*---------------------------------------------------*/ UInt32 bytesIn, bytesOut; Int32 verbosity; Bool keepInputFiles, smallMode, testFailsExist; UInt32 globalCrc; Int32 numFileNames, numFilesProcessed; /*-- source modes; F==file, I==stdin, O==stdout --*/ #define SM_I2O 1 #define SM_F2O 2 #define SM_F2F 3 /*-- operation modes --*/ #define OM_Z 1 #define OM_UNZ 2 #define OM_TEST 3 Int32 opMode; Int32 srcMode; Int32 longestFileName; Char inName[1024]; Char outName[1024]; Char *progName; Char progNameReally[1024]; FILE *outputHandleJustInCase; void panic ( Char* ) NORETURN; void ioError ( void ) NORETURN; void compressOutOfMemory ( Int32, Int32 ) NORETURN; void uncompressOutOfMemory ( Int32, Int32 ) NORETURN; void blockOverrun ( void ) NORETURN; void badBlockHeader ( void ) NORETURN; void badBGLengths ( void ) NORETURN; void crcError ( UInt32, UInt32 ) NORETURN; void bitStreamEOF ( void ) NORETURN; void cleanUpAndFail ( Int32 ) NORETURN; void compressedStreamEOF ( void ) NORETURN; void* myMalloc ( Int32 ); /*---------------------------------------------------*/ /*--- Data decls for the front end ---*/ /*---------------------------------------------------*/ /*-- The overshoot bytes allow us to avoid most of the cost of pointer renormalisation during comparison of rotations in sorting. The figure of 20 is derived as follows: qSort3 allows an overshoot of up to 10. It then calls simpleSort, which calls fullGtU, also with max overshoot 10. fullGtU does up to 10 comparisons without renormalising, giving 10+10 == 20. --*/ #define NUM_OVERSHOOT_BYTES 20 /*-- These are the main data structures for the Burrows-Wheeler transform. --*/ /*-- Pointers to compression and decompression structures. Set by allocateCompressStructures and setDecompressStructureSizes The structures are always set to be suitable for a block of size 100000 * blockSize100k. --*/ UChar *block; /*-- compress --*/ UInt16 *quadrant; /*-- compress --*/ Int32 *zptr; /*-- compress --*/ UInt16 *szptr; /*-- overlays zptr ---*/ Int32 *ftab; /*-- compress --*/ UInt16 *ll16; /*-- small decompress --*/ UChar *ll4; /*-- small decompress --*/ Int32 *tt; /*-- fast decompress --*/ UChar *ll8; /*-- fast decompress --*/ /*-- freq table collected to save a pass over the data during decompression. --*/ Int32 unzftab[256]; /*-- index of the last char in the block, so the block size == last + 1. --*/ Int32 last; /*-- index in zptr[] of original string after sorting. --*/ Int32 origPtr; /*-- always: in the range 0 .. 9. The current block size is 100000 * this number. --*/ Int32 blockSize100k; /*-- Used when sorting. If too many long comparisons happen, we stop sorting, randomise the block slightly, and try again. --*/ Int32 workFactor; Int32 workDone; Int32 workLimit; Bool blockRandomised; Bool firstAttempt; Int32 nBlocksRandomised; /*---------------------------------------------------*/ /*--- Data decls for the back end ---*/ /*---------------------------------------------------*/ #define MAX_ALPHA_SIZE 258 #define MAX_CODE_LEN 23 #define RUNA 0 #define RUNB 1 #define N_GROUPS 6 #define G_SIZE 50 #define N_ITERS 4 #define MAX_SELECTORS (2 + (900000 / G_SIZE)) Bool inUse[256]; Int32 nInUse; UChar seqToUnseq[256]; UChar unseqToSeq[256]; UChar selector [MAX_SELECTORS]; UChar selectorMtf[MAX_SELECTORS]; Int32 nMTF; Int32 mtfFreq[MAX_ALPHA_SIZE]; UChar len [N_GROUPS][MAX_ALPHA_SIZE]; /*-- decompress only --*/ Int32 limit [N_GROUPS][MAX_ALPHA_SIZE]; Int32 base [N_GROUPS][MAX_ALPHA_SIZE]; Int32 perm [N_GROUPS][MAX_ALPHA_SIZE]; Int32 minLens[N_GROUPS]; /*-- compress only --*/ Int32 code [N_GROUPS][MAX_ALPHA_SIZE]; Int32 rfreq[N_GROUPS][MAX_ALPHA_SIZE]; /*---------------------------------------------------*/ /*--- 32-bit CRC grunge ---*/ /*---------------------------------------------------*/ /*-- I think this is an implementation of the AUTODIN-II, Ethernet & FDDI 32-bit CRC standard. Vaguely derived from code by Rob Warnock, in Section 51 of the comp.compression FAQ. --*/ UInt32 crc32Table[256] = { /*-- Ugly, innit? --*/ 0x00000000UL, 0x04c11db7UL, 0x09823b6eUL, 0x0d4326d9UL, 0x130476dcUL, 0x17c56b6bUL, 0x1a864db2UL, 0x1e475005UL, 0x2608edb8UL, 0x22c9f00fUL, 0x2f8ad6d6UL, 0x2b4bcb61UL, 0x350c9b64UL, 0x31cd86d3UL, 0x3c8ea00aUL, 0x384fbdbdUL, 0x4c11db70UL, 0x48d0c6c7UL, 0x4593e01eUL, 0x4152fda9UL, 0x5f15adacUL, 0x5bd4b01bUL, 0x569796c2UL, 0x52568b75UL, 0x6a1936c8UL, 0x6ed82b7fUL, 0x639b0da6UL, 0x675a1011UL, 0x791d4014UL, 0x7ddc5da3UL, 0x709f7b7aUL, 0x745e66cdUL, 0x9823b6e0UL, 0x9ce2ab57UL, 0x91a18d8eUL, 0x95609039UL, 0x8b27c03cUL, 0x8fe6dd8bUL, 0x82a5fb52UL, 0x8664e6e5UL, 0xbe2b5b58UL, 0xbaea46efUL, 0xb7a96036UL, 0xb3687d81UL, 0xad2f2d84UL, 0xa9ee3033UL, 0xa4ad16eaUL, 0xa06c0b5dUL, 0xd4326d90UL, 0xd0f37027UL, 0xddb056feUL, 0xd9714b49UL, 0xc7361b4cUL, 0xc3f706fbUL, 0xceb42022UL, 0xca753d95UL, 0xf23a8028UL, 0xf6fb9d9fUL, 0xfbb8bb46UL, 0xff79a6f1UL, 0xe13ef6f4UL, 0xe5ffeb43UL, 0xe8bccd9aUL, 0xec7dd02dUL, 0x34867077UL, 0x30476dc0UL, 0x3d044b19UL, 0x39c556aeUL, 0x278206abUL, 0x23431b1cUL, 0x2e003dc5UL, 0x2ac12072UL, 0x128e9dcfUL, 0x164f8078UL, 0x1b0ca6a1UL, 0x1fcdbb16UL, 0x018aeb13UL, 0x054bf6a4UL, 0x0808d07dUL, 0x0cc9cdcaUL, 0x7897ab07UL, 0x7c56b6b0UL, 0x71159069UL, 0x75d48ddeUL, 0x6b93dddbUL, 0x6f52c06cUL, 0x6211e6b5UL, 0x66d0fb02UL, 0x5e9f46bfUL, 0x5a5e5b08UL, 0x571d7dd1UL, 0x53dc6066UL, 0x4d9b3063UL, 0x495a2dd4UL, 0x44190b0dUL, 0x40d816baUL, 0xaca5c697UL, 0xa864db20UL, 0xa527fdf9UL, 0xa1e6e04eUL, 0xbfa1b04bUL, 0xbb60adfcUL, 0xb6238b25UL, 0xb2e29692UL, 0x8aad2b2fUL, 0x8e6c3698UL, 0x832f1041UL, 0x87ee0df6UL, 0x99a95df3UL, 0x9d684044UL, 0x902b669dUL, 0x94ea7b2aUL, 0xe0b41de7UL, 0xe4750050UL, 0xe9362689UL, 0xedf73b3eUL, 0xf3b06b3bUL, 0xf771768cUL, 0xfa325055UL, 0xfef34de2UL, 0xc6bcf05fUL, 0xc27dede8UL, 0xcf3ecb31UL, 0xcbffd686UL, 0xd5b88683UL, 0xd1799b34UL, 0xdc3abdedUL, 0xd8fba05aUL, 0x690ce0eeUL, 0x6dcdfd59UL, 0x608edb80UL, 0x644fc637UL, 0x7a089632UL, 0x7ec98b85UL, 0x738aad5cUL, 0x774bb0ebUL, 0x4f040d56UL, 0x4bc510e1UL, 0x46863638UL, 0x42472b8fUL, 0x5c007b8aUL, 0x58c1663dUL, 0x558240e4UL, 0x51435d53UL, 0x251d3b9eUL, 0x21dc2629UL, 0x2c9f00f0UL, 0x285e1d47UL, 0x36194d42UL, 0x32d850f5UL, 0x3f9b762cUL, 0x3b5a6b9bUL, 0x0315d626UL, 0x07d4cb91UL, 0x0a97ed48UL, 0x0e56f0ffUL, 0x1011a0faUL, 0x14d0bd4dUL, 0x19939b94UL, 0x1d528623UL, 0xf12f560eUL, 0xf5ee4bb9UL, 0xf8ad6d60UL, 0xfc6c70d7UL, 0xe22b20d2UL, 0xe6ea3d65UL, 0xeba91bbcUL, 0xef68060bUL, 0xd727bbb6UL, 0xd3e6a601UL, 0xdea580d8UL, 0xda649d6fUL, 0xc423cd6aUL, 0xc0e2d0ddUL, 0xcda1f604UL, 0xc960ebb3UL, 0xbd3e8d7eUL, 0xb9ff90c9UL, 0xb4bcb610UL, 0xb07daba7UL, 0xae3afba2UL, 0xaafbe615UL, 0xa7b8c0ccUL, 0xa379dd7bUL, 0x9b3660c6UL, 0x9ff77d71UL, 0x92b45ba8UL, 0x9675461fUL, 0x8832161aUL, 0x8cf30badUL, 0x81b02d74UL, 0x857130c3UL, 0x5d8a9099UL, 0x594b8d2eUL, 0x5408abf7UL, 0x50c9b640UL, 0x4e8ee645UL, 0x4a4ffbf2UL, 0x470cdd2bUL, 0x43cdc09cUL, 0x7b827d21UL, 0x7f436096UL, 0x7200464fUL, 0x76c15bf8UL, 0x68860bfdUL, 0x6c47164aUL, 0x61043093UL, 0x65c52d24UL, 0x119b4be9UL, 0x155a565eUL, 0x18197087UL, 0x1cd86d30UL, 0x029f3d35UL, 0x065e2082UL, 0x0b1d065bUL, 0x0fdc1becUL, 0x3793a651UL, 0x3352bbe6UL, 0x3e119d3fUL, 0x3ad08088UL, 0x2497d08dUL, 0x2056cd3aUL, 0x2d15ebe3UL, 0x29d4f654UL, 0xc5a92679UL, 0xc1683bceUL, 0xcc2b1d17UL, 0xc8ea00a0UL, 0xd6ad50a5UL, 0xd26c4d12UL, 0xdf2f6bcbUL, 0xdbee767cUL, 0xe3a1cbc1UL, 0xe760d676UL, 0xea23f0afUL, 0xeee2ed18UL, 0xf0a5bd1dUL, 0xf464a0aaUL, 0xf9278673UL, 0xfde69bc4UL, 0x89b8fd09UL, 0x8d79e0beUL, 0x803ac667UL, 0x84fbdbd0UL, 0x9abc8bd5UL, 0x9e7d9662UL, 0x933eb0bbUL, 0x97ffad0cUL, 0xafb010b1UL, 0xab710d06UL, 0xa6322bdfUL, 0xa2f33668UL, 0xbcb4666dUL, 0xb8757bdaUL, 0xb5365d03UL, 0xb1f740b4UL }; /*---------------------------------------------*/ void initialiseCRC ( void ) { globalCrc = 0xffffffffUL; } /*---------------------------------------------*/ UInt32 getFinalCRC ( void ) { return ~globalCrc; } /*---------------------------------------------*/ UInt32 getGlobalCRC ( void ) { return globalCrc; } /*---------------------------------------------*/ void setGlobalCRC ( UInt32 newCrc ) { globalCrc = newCrc; } /*---------------------------------------------*/ #define UPDATE_CRC(crcVar,cha) \ { \ crcVar = (crcVar << 8) ^ \ crc32Table[(crcVar >> 24) ^ \ ((UChar)cha)]; \ } /*---------------------------------------------------*/ /*--- Bit stream I/O ---*/ /*---------------------------------------------------*/ UInt32 bsBuff; Int32 bsLive; FILE* bsStream; Bool bsWriting; /*---------------------------------------------*/ void bsSetStream ( FILE* f, Bool wr ) { if (bsStream != NULL) panic ( "bsSetStream" ); bsStream = f; bsLive = 0; bsBuff = 0; bytesOut = 0; bytesIn = 0; bsWriting = wr; } /*---------------------------------------------*/ void bsFinishedWithStream ( void ) { if (bsWriting) while (bsLive > 0) { fputc ( (UChar)(bsBuff >> 24), bsStream ); bsBuff <<= 8; bsLive -= 8; bytesOut++; } bsStream = NULL; } /*---------------------------------------------*/ #define bsNEEDR(nz) \ { \ while (bsLive < nz) { \ Int32 zzi = fgetc ( bsStream ); \ if (zzi == EOF) compressedStreamEOF(); \ bsBuff = (bsBuff << 8) | (zzi & 0xffL); \ bsLive += 8; \ } \ } /*---------------------------------------------*/ #define bsNEEDW(nz) \ { \ while (bsLive >= 8) { \ fputc ( (UChar)(bsBuff >> 24), \ bsStream ); \ bsBuff <<= 8; \ bsLive -= 8; \ bytesOut++; \ } \ } /*---------------------------------------------*/ #define bsR1(vz) \ { \ bsNEEDR(1); \ vz = (bsBuff >> (bsLive-1)) & 1; \ bsLive--; \ } /*---------------------------------------------*/ INLINE UInt32 bsR ( Int32 n ) { UInt32 v; bsNEEDR ( n ); v = (bsBuff >> (bsLive-n)) & ((1 << n)-1); bsLive -= n; return v; } /*---------------------------------------------*/ INLINE void bsW ( Int32 n, UInt32 v ) { bsNEEDW ( n ); bsBuff |= (v << (32 - bsLive - n)); bsLive += n; } /*---------------------------------------------*/ UChar bsGetUChar ( void ) { return (UChar)bsR(8); } /*---------------------------------------------*/ void bsPutUChar ( UChar c ) { bsW(8, (UInt32)c ); } /*---------------------------------------------*/ Int32 bsGetUInt32 ( void ) { UInt32 u; u = 0; u = (u << 8) | bsR(8); u = (u << 8) | bsR(8); u = (u << 8) | bsR(8); u = (u << 8) | bsR(8); return u; } /*---------------------------------------------*/ UInt32 bsGetIntVS ( UInt32 numBits ) { return (UInt32)bsR(numBits); } /*---------------------------------------------*/ UInt32 bsGetInt32 ( void ) { return (Int32)bsGetUInt32(); } /*---------------------------------------------*/ void bsPutUInt32 ( UInt32 u ) { bsW ( 8, (u >> 24) & 0xffL ); bsW ( 8, (u >> 16) & 0xffL ); bsW ( 8, (u >> 8) & 0xffL ); bsW ( 8, u & 0xffL ); } /*---------------------------------------------*/ void bsPutInt32 ( Int32 c ) { bsPutUInt32 ( (UInt32)c ); } /*---------------------------------------------*/ void bsPutIntVS ( Int32 numBits, UInt32 c ) { bsW ( numBits, c ); } /*---------------------------------------------------*/ /*--- Huffman coding low-level stuff ---*/ /*---------------------------------------------------*/ #define WEIGHTOF(zz0) ((zz0) & 0xffffff00) #define DEPTHOF(zz1) ((zz1) & 0x000000ff) #define MYMAX(zz2,zz3) ((zz2) > (zz3) ? (zz2) : (zz3)) #define ADDWEIGHTS(zw1,zw2) \ (WEIGHTOF(zw1)+WEIGHTOF(zw2)) | \ (1 + MYMAX(DEPTHOF(zw1),DEPTHOF(zw2))) #define UPHEAP(z) \ { \ Int32 zz, tmp; \ zz = z; tmp = heap[zz]; \ while (weight[tmp] < weight[heap[zz >> 1]]) { \ heap[zz] = heap[zz >> 1]; \ zz >>= 1; \ } \ heap[zz] = tmp; \ } #define DOWNHEAP(z) \ { \ Int32 zz, yy, tmp; \ zz = z; tmp = heap[zz]; \ while (True) { \ yy = zz << 1; \ if (yy > nHeap) break; \ if (yy < nHeap && \ weight[heap[yy+1]] < weight[heap[yy]]) \ yy++; \ if (weight[tmp] < weight[heap[yy]]) break; \ heap[zz] = heap[yy]; \ zz = yy; \ } \ heap[zz] = tmp; \ } /*---------------------------------------------*/ void hbMakeCodeLengths ( UChar *len, Int32 *freq, Int32 alphaSize, Int32 maxLen ) { /*-- Nodes and heap entries run from 1. Entry 0 for both the heap and nodes is a sentinel. --*/ Int32 nNodes, nHeap, n1, n2, i, j, k; Bool tooLong; Int32 heap [ MAX_ALPHA_SIZE + 2 ]; Int32 weight [ MAX_ALPHA_SIZE * 2 ]; Int32 parent [ MAX_ALPHA_SIZE * 2 ]; for (i = 0; i < alphaSize; i++) weight[i+1] = (freq[i] == 0 ? 1 : freq[i]) << 8; while (True) { nNodes = alphaSize; nHeap = 0; heap[0] = 0; weight[0] = 0; parent[0] = -2; for (i = 1; i <= alphaSize; i++) { parent[i] = -1; nHeap++; heap[nHeap] = i; UPHEAP(nHeap); } if (!(nHeap < (MAX_ALPHA_SIZE+2))) panic ( "hbMakeCodeLengths(1)" ); while (nHeap > 1) { n1 = heap[1]; heap[1] = heap[nHeap]; nHeap--; DOWNHEAP(1); n2 = heap[1]; heap[1] = heap[nHeap]; nHeap--; DOWNHEAP(1); nNodes++; parent[n1] = parent[n2] = nNodes; weight[nNodes] = ADDWEIGHTS(weight[n1], weight[n2]); parent[nNodes] = -1; nHeap++; heap[nHeap] = nNodes; UPHEAP(nHeap); } if (!(nNodes < (MAX_ALPHA_SIZE * 2))) panic ( "hbMakeCodeLengths(2)" ); tooLong = False; for (i = 1; i <= alphaSize; i++) { j = 0; k = i; while (parent[k] >= 0) { k = parent[k]; j++; } len[i-1] = j; if (j > maxLen) tooLong = True; } if (! tooLong) break; for (i = 1; i < alphaSize; i++) { j = weight[i] >> 8; j = 1 + (j / 2); weight[i] = j << 8; } } } /*---------------------------------------------*/ void hbAssignCodes ( Int32 *code, UChar *length, Int32 minLen, Int32 maxLen, Int32 alphaSize ) { Int32 n, vec, i; vec = 0; for (n = minLen; n <= maxLen; n++) { for (i = 0; i < alphaSize; i++) if (length[i] == n) { code[i] = vec; vec++; }; vec <<= 1; } } /*---------------------------------------------*/ void hbCreateDecodeTables ( Int32 *limit, Int32 *base, Int32 *perm, UChar *length, Int32 minLen, Int32 maxLen, Int32 alphaSize ) { Int32 pp, i, j, vec; pp = 0; for (i = minLen; i <= maxLen; i++) for (j = 0; j < alphaSize; j++) if (length[j] == i) { perm[pp] = j; pp++; }; for (i = 0; i < MAX_CODE_LEN; i++) base[i] = 0; for (i = 0; i < alphaSize; i++) base[length[i]+1]++; for (i = 1; i < MAX_CODE_LEN; i++) base[i] += base[i-1]; for (i = 0; i < MAX_CODE_LEN; i++) limit[i] = 0; vec = 0; for (i = minLen; i <= maxLen; i++) { vec += (base[i+1] - base[i]); limit[i] = vec-1; vec <<= 1; } for (i = minLen + 1; i <= maxLen; i++) base[i] = ((limit[i-1] + 1) << 1) - base[i]; } /*---------------------------------------------------*/ /*--- Undoing the reversible transformation ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ #define SET_LL4(i,n) \ { if (((i) & 0x1) == 0) \ ll4[(i) >> 1] = (ll4[(i) >> 1] & 0xf0) | (n); else \ ll4[(i) >> 1] = (ll4[(i) >> 1] & 0x0f) | ((n) << 4); \ } #define GET_LL4(i) \ (((UInt32)(ll4[(i) >> 1])) >> (((i) << 2) & 0x4) & 0xF) #define SET_LL(i,n) \ { ll16[i] = (UInt16)(n & 0x0000ffff); \ SET_LL4(i, n >> 16); \ } #define GET_LL(i) \ (((UInt32)ll16[i]) | (GET_LL4(i) << 16)) /*---------------------------------------------*/ /*-- Manage memory for compression/decompression. When compressing, a single block size applies to all files processed, and that's set when the program starts. But when decompressing, each file processed could have been compressed with a different block size, so we may have to free and reallocate on a per-file basis. A call with argument of zero means `free up everything.' And a value of zero for blockSize100k means no memory is currently allocated. --*/ /*---------------------------------------------*/ void allocateCompressStructures ( void ) { Int32 n = 100000 * blockSize100k; block = malloc ( (n + 1 + NUM_OVERSHOOT_BYTES) * sizeof(UChar) ); quadrant = malloc ( (n + NUM_OVERSHOOT_BYTES) * sizeof(Int16) ); zptr = malloc ( n * sizeof(Int32) ); ftab = malloc ( 65537 * sizeof(Int32) ); if (block == NULL || quadrant == NULL || zptr == NULL || ftab == NULL) { Int32 totalDraw = (n + 1 + NUM_OVERSHOOT_BYTES) * sizeof(UChar) + (n + NUM_OVERSHOOT_BYTES) * sizeof(Int16) + n * sizeof(Int32) + 65537 * sizeof(Int32); compressOutOfMemory ( totalDraw, n ); } /*-- Since we want valid indexes for block of -1 to n + NUM_OVERSHOOT_BYTES - 1 inclusive. --*/ block++; /*-- The back end needs a place to store the MTF values whilst it calculates the coding tables. We could put them in the zptr array. However, these values will fit in a short, so we overlay szptr at the start of zptr, in the hope of reducing the number of cache misses induced by the multiple traversals of the MTF values when calculating coding tables. Seems to improve compression speed by about 1%. --*/ szptr = (UInt16*)zptr; } /*---------------------------------------------*/ void setDecompressStructureSizes ( Int32 newSize100k ) { if (! (0 <= newSize100k && newSize100k <= 9 && 0 <= blockSize100k && blockSize100k <= 9)) panic ( "setDecompressStructureSizes" ); if (newSize100k == blockSize100k) return; blockSize100k = newSize100k; if (ll16 != NULL) free ( ll16 ); if (ll4 != NULL) free ( ll4 ); if (ll8 != NULL) free ( ll8 ); if (tt != NULL) free ( tt ); if (newSize100k == 0) return; if (smallMode) { Int32 n = 100000 * newSize100k; ll16 = malloc ( n * sizeof(UInt16) ); ll4 = malloc ( ((n+1) >> 1) * sizeof(UChar) ); if (ll4 == NULL || ll16 == NULL) { Int32 totalDraw = n * sizeof(Int16) + ((n+1) >> 1) * sizeof(UChar); uncompressOutOfMemory ( totalDraw, n ); } } else { Int32 n = 100000 * newSize100k; ll8 = malloc ( n * sizeof(UChar) ); tt = malloc ( n * sizeof(Int32) ); if (ll8 == NULL || tt == NULL) { Int32 totalDraw = n * sizeof(UChar) + n * sizeof(UInt32); uncompressOutOfMemory ( totalDraw, n ); } } } /*---------------------------------------------------*/ /*--- The new back end ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ void makeMaps ( void ) { Int32 i; nInUse = 0; for (i = 0; i < 256; i++) if (inUse[i]) { seqToUnseq[nInUse] = i; unseqToSeq[i] = nInUse; nInUse++; } } /*---------------------------------------------*/ void generateMTFValues ( void ) { UChar yy[256]; Int32 i, j; UChar tmp; UChar tmp2; Int32 zPend; Int32 wr; Int32 EOB; makeMaps(); EOB = nInUse+1; for (i = 0; i <= EOB; i++) mtfFreq[i] = 0; wr = 0; zPend = 0; for (i = 0; i < nInUse; i++) yy[i] = (UChar) i; for (i = 0; i <= last; i++) { UChar ll_i; #if DEBUG assert (wr <= i); #endif ll_i = unseqToSeq[block[zptr[i] - 1]]; #if DEBUG assert (ll_i < nInUse); #endif j = 0; tmp = yy[j]; while ( ll_i != tmp ) { j++; tmp2 = tmp; tmp = yy[j]; yy[j] = tmp2; }; yy[0] = tmp; if (j == 0) { zPend++; } else { if (zPend > 0) { zPend--; while (True) { switch (zPend % 2) { case 0: szptr[wr] = RUNA; wr++; mtfFreq[RUNA]++; break; case 1: szptr[wr] = RUNB; wr++; mtfFreq[RUNB]++; break; }; if (zPend < 2) break; zPend = (zPend - 2) / 2; }; zPend = 0; } szptr[wr] = j+1; wr++; mtfFreq[j+1]++; } } if (zPend > 0) { zPend--; while (True) { switch (zPend % 2) { case 0: szptr[wr] = RUNA; wr++; mtfFreq[RUNA]++; break; case 1: szptr[wr] = RUNB; wr++; mtfFreq[RUNB]++; break; }; if (zPend < 2) break; zPend = (zPend - 2) / 2; }; } szptr[wr] = EOB; wr++; mtfFreq[EOB]++; nMTF = wr; } /*---------------------------------------------*/ #define LESSER_ICOST 0 #define GREATER_ICOST 15 void sendMTFValues ( void ) { Int32 v, t, i, j, gs, ge, totc, bt, bc, iter; Int32 nSelectors, alphaSize, minLen, maxLen, selCtr; Int32 nGroups, nBytes; /*-- UChar len [N_GROUPS][MAX_ALPHA_SIZE]; is a global since the decoder also needs it. Int32 code[N_GROUPS][MAX_ALPHA_SIZE]; Int32 rfreq[N_GROUPS][MAX_ALPHA_SIZE]; are also globals only used in this proc. Made global to keep stack frame size small. --*/ UInt16 cost[N_GROUPS]; Int32 fave[N_GROUPS]; if (verbosity >= 3) fprintf ( stderr, " %d in block, %d after MTF & 1-2 coding, %d+2 syms in use\n", last+1, nMTF, nInUse ); alphaSize = nInUse+2; for (t = 0; t < N_GROUPS; t++) for (v = 0; v < alphaSize; v++) len[t][v] = GREATER_ICOST; /*--- Decide how many coding tables to use ---*/ if (nMTF <= 0) panic ( "sendMTFValues(0)" ); if (nMTF < 200) nGroups = 2; else if (nMTF < 800) nGroups = 4; else nGroups = 6; /*--- Generate an initial set of coding tables ---*/ { Int32 nPart, remF, tFreq, aFreq; nPart = nGroups; remF = nMTF; gs = 0; while (nPart > 0) { tFreq = remF / nPart; ge = gs-1; aFreq = 0; while (aFreq < tFreq && ge < alphaSize-1) { ge++; aFreq += mtfFreq[ge]; } if (ge > gs && nPart != nGroups && nPart != 1 && ((nGroups-nPart) % 2 == 1)) { aFreq -= mtfFreq[ge]; ge--; } if (verbosity >= 3) fprintf ( stderr, " initial group %d, [%d .. %d], has %d syms (%4.1f%%)\n", nPart, gs, ge, aFreq, (100.0 * (float)aFreq) / (float)nMTF ); for (v = 0; v < alphaSize; v++) if (v >= gs && v <= ge) len[nPart-1][v] = LESSER_ICOST; else len[nPart-1][v] = GREATER_ICOST; nPart--; gs = ge+1; remF -= aFreq; } } /*--- Iterate up to N_ITERS times to improve the tables. ---*/ for (iter = 0; iter < N_ITERS; iter++) { for (t = 0; t < nGroups; t++) fave[t] = 0; for (t = 0; t < nGroups; t++) for (v = 0; v < alphaSize; v++) rfreq[t][v] = 0; nSelectors = 0; totc = 0; gs = 0; while (True) { /*--- Set group start & end marks. --*/ if (gs >= nMTF) break; ge = gs + G_SIZE - 1; if (ge >= nMTF) ge = nMTF-1; /*-- Calculate the cost of this group as coded by each of the coding tables. --*/ for (t = 0; t < nGroups; t++) cost[t] = 0; if (nGroups == 6) { register UInt16 cost0, cost1, cost2, cost3, cost4, cost5; cost0 = cost1 = cost2 = cost3 = cost4 = cost5 = 0; for (i = gs; i <= ge; i++) { UInt16 icv = szptr[i]; cost0 += len[0][icv]; cost1 += len[1][icv]; cost2 += len[2][icv]; cost3 += len[3][icv]; cost4 += len[4][icv]; cost5 += len[5][icv]; } cost[0] = cost0; cost[1] = cost1; cost[2] = cost2; cost[3] = cost3; cost[4] = cost4; cost[5] = cost5; } else { for (i = gs; i <= ge; i++) { UInt16 icv = szptr[i]; for (t = 0; t < nGroups; t++) cost[t] += len[t][icv]; } } /*-- Find the coding table which is best for this group, and record its identity in the selector table. --*/ bc = 999999999; bt = -1; for (t = 0; t < nGroups; t++) if (cost[t] < bc) { bc = cost[t]; bt = t; }; totc += bc; fave[bt]++; selector[nSelectors] = bt; nSelectors++; /*-- Increment the symbol frequencies for the selected table. --*/ for (i = gs; i <= ge; i++) rfreq[bt][ szptr[i] ]++; gs = ge+1; } if (verbosity >= 3) { fprintf ( stderr, " pass %d: size is %d, grp uses are ", iter+1, totc/8 ); for (t = 0; t < nGroups; t++) fprintf ( stderr, "%d ", fave[t] ); fprintf ( stderr, "\n" ); } /*-- Recompute the tables based on the accumulated frequencies. --*/ for (t = 0; t < nGroups; t++) hbMakeCodeLengths ( &len[t][0], &rfreq[t][0], alphaSize, 20 ); } if (!(nGroups < 8)) panic ( "sendMTFValues(1)" ); if (!(nSelectors < 32768 && nSelectors <= (2 + (900000 / G_SIZE)))) panic ( "sendMTFValues(2)" ); /*--- Compute MTF values for the selectors. ---*/ { UChar pos[N_GROUPS], ll_i, tmp2, tmp; for (i = 0; i < nGroups; i++) pos[i] = i; for (i = 0; i < nSelectors; i++) { ll_i = selector[i]; j = 0; tmp = pos[j]; while ( ll_i != tmp ) { j++; tmp2 = tmp; tmp = pos[j]; pos[j] = tmp2; }; pos[0] = tmp; selectorMtf[i] = j; } }; /*--- Assign actual codes for the tables. --*/ for (t = 0; t < nGroups; t++) { minLen = 32; maxLen = 0; for (i = 0; i < alphaSize; i++) { if (len[t][i] > maxLen) maxLen = len[t][i]; if (len[t][i] < minLen) minLen = len[t][i]; } if (maxLen > 20) panic ( "sendMTFValues(3)" ); if (minLen < 1) panic ( "sendMTFValues(4)" ); hbAssignCodes ( &code[t][0], &len[t][0], minLen, maxLen, alphaSize ); } /*--- Transmit the mapping table. ---*/ { Bool inUse16[16]; for (i = 0; i < 16; i++) { inUse16[i] = False; for (j = 0; j < 16; j++) if (inUse[i * 16 + j]) inUse16[i] = True; } nBytes = bytesOut; for (i = 0; i < 16; i++) if (inUse16[i]) bsW(1,1); else bsW(1,0); for (i = 0; i < 16; i++) if (inUse16[i]) for (j = 0; j < 16; j++) if (inUse[i * 16 + j]) bsW(1,1); else bsW(1,0); if (verbosity >= 3) fprintf ( stderr, " bytes: mapping %d, ", bytesOut-nBytes ); } /*--- Now the selectors. ---*/ nBytes = bytesOut; bsW ( 3, nGroups ); bsW ( 15, nSelectors ); for (i = 0; i < nSelectors; i++) { for (j = 0; j < selectorMtf[i]; j++) bsW(1,1); bsW(1,0); } if (verbosity >= 3) fprintf ( stderr, "selectors %d, ", bytesOut-nBytes ); /*--- Now the coding tables. ---*/ nBytes = bytesOut; for (t = 0; t < nGroups; t++) { Int32 curr = len[t][0]; bsW ( 5, curr ); for (i = 0; i < alphaSize; i++) { while (curr < len[t][i]) { bsW(2,2); curr++; /* 10 */ }; while (curr > len[t][i]) { bsW(2,3); curr--; /* 11 */ }; bsW ( 1, 0 ); } } if (verbosity >= 3) fprintf ( stderr, "code lengths %d, ", bytesOut-nBytes ); /*--- And finally, the block data proper ---*/ nBytes = bytesOut; selCtr = 0; gs = 0; while (True) { if (gs >= nMTF) break; ge = gs + G_SIZE - 1; if (ge >= nMTF) ge = nMTF-1; for (i = gs; i <= ge; i++) { #if DEBUG assert (selector[selCtr] < nGroups); #endif bsW ( len [selector[selCtr]] [szptr[i]], code [selector[selCtr]] [szptr[i]] ); } gs = ge+1; selCtr++; } if (!(selCtr == nSelectors)) panic ( "sendMTFValues(5)" ); if (verbosity >= 3) fprintf ( stderr, "codes %d\n", bytesOut-nBytes ); } /*---------------------------------------------*/ void moveToFrontCodeAndSend ( void ) { bsPutIntVS ( 24, origPtr ); generateMTFValues(); sendMTFValues(); } /*---------------------------------------------*/ void recvDecodingTables ( void ) { Int32 i, j, t, nGroups, nSelectors, alphaSize; Int32 minLen, maxLen; Bool inUse16[16]; /*--- Receive the mapping table ---*/ for (i = 0; i < 16; i++) if (bsR(1) == 1) inUse16[i] = True; else inUse16[i] = False; for (i = 0; i < 256; i++) inUse[i] = False; for (i = 0; i < 16; i++) if (inUse16[i]) for (j = 0; j < 16; j++) if (bsR(1) == 1) inUse[i * 16 + j] = True; makeMaps(); alphaSize = nInUse+2; /*--- Now the selectors ---*/ nGroups = bsR ( 3 ); nSelectors = bsR ( 15 ); for (i = 0; i < nSelectors; i++) { j = 0; while (bsR(1) == 1) j++; selectorMtf[i] = j; } /*--- Undo the MTF values for the selectors. ---*/ { UChar pos[N_GROUPS], tmp, v; for (v = 0; v < nGroups; v++) pos[v] = v; for (i = 0; i < nSelectors; i++) { v = selectorMtf[i]; tmp = pos[v]; while (v > 0) { pos[v] = pos[v-1]; v--; } pos[0] = tmp; selector[i] = tmp; } } /*--- Now the coding tables ---*/ for (t = 0; t < nGroups; t++) { Int32 curr = bsR ( 5 ); for (i = 0; i < alphaSize; i++) { while (bsR(1) == 1) { if (bsR(1) == 0) curr++; else curr--; } len[t][i] = curr; } } /*--- Create the Huffman decoding tables ---*/ for (t = 0; t < nGroups; t++) { minLen = 32; maxLen = 0; for (i = 0; i < alphaSize; i++) { if (len[t][i] > maxLen) maxLen = len[t][i]; if (len[t][i] < minLen) minLen = len[t][i]; } hbCreateDecodeTables ( &limit[t][0], &base[t][0], &perm[t][0], &len[t][0], minLen, maxLen, alphaSize ); minLens[t] = minLen; } } /*---------------------------------------------*/ #define GET_MTF_VAL(lval) \ { \ Int32 zt, zn, zvec, zj; \ if (groupPos == 0) { \ groupNo++; \ groupPos = G_SIZE; \ } \ groupPos--; \ zt = selector[groupNo]; \ zn = minLens[zt]; \ zvec = bsR ( zn ); \ while (zvec > limit[zt][zn]) { \ zn++; bsR1(zj); \ zvec = (zvec << 1) | zj; \ }; \ lval = perm[zt][zvec - base[zt][zn]]; \ } /*---------------------------------------------*/ void getAndMoveToFrontDecode ( void ) { UChar yy[256]; Int32 i, j, nextSym, limitLast; Int32 EOB, groupNo, groupPos; limitLast = 100000 * blockSize100k; origPtr = bsGetIntVS ( 24 ); recvDecodingTables(); EOB = nInUse+1; groupNo = -1; groupPos = 0; /*-- Setting up the unzftab entries here is not strictly necessary, but it does save having to do it later in a separate pass, and so saves a block's worth of cache misses. --*/ for (i = 0; i <= 255; i++) unzftab[i] = 0; for (i = 0; i <= 255; i++) yy[i] = (UChar) i; last = -1; GET_MTF_VAL(nextSym); while (True) { if (nextSym == EOB) break; if (nextSym == RUNA || nextSym == RUNB) { UChar ch; Int32 s = -1; Int32 N = 1; do { if (nextSym == RUNA) s = s + (0+1) * N; else if (nextSym == RUNB) s = s + (1+1) * N; N = N * 2; GET_MTF_VAL(nextSym); } while (nextSym == RUNA || nextSym == RUNB); s++; ch = seqToUnseq[yy[0]]; unzftab[ch] += s; if (smallMode) while (s > 0) { last++; ll16[last] = ch; s--; } else while (s > 0) { last++; ll8[last] = ch; s--; }; if (last >= limitLast) blockOverrun(); continue; } else { UChar tmp; last++; if (last >= limitLast) blockOverrun(); tmp = yy[nextSym-1]; unzftab[seqToUnseq[tmp]]++; if (smallMode) ll16[last] = seqToUnseq[tmp]; else ll8[last] = seqToUnseq[tmp]; /*-- This loop is hammered during decompression, hence the unrolling. for (j = nextSym-1; j > 0; j--) yy[j] = yy[j-1]; --*/ j = nextSym-1; for (; j > 3; j -= 4) { yy[j] = yy[j-1]; yy[j-1] = yy[j-2]; yy[j-2] = yy[j-3]; yy[j-3] = yy[j-4]; } for (; j > 0; j--) yy[j] = yy[j-1]; yy[0] = tmp; GET_MTF_VAL(nextSym); continue; } } } /*---------------------------------------------------*/ /*--- Block-sorting machinery ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ /*-- Compare two strings in block. We assume (see discussion above) that i1 and i2 have a max offset of 10 on entry, and that the first bytes of both block and quadrant have been copied into the "overshoot area", ie into the subscript range [last+1 .. last+NUM_OVERSHOOT_BYTES]. --*/ INLINE Bool fullGtU ( Int32 i1, Int32 i2 ) { Int32 k; UChar c1, c2; UInt16 s1, s2; #if DEBUG /*-- shellsort shouldn't ask to compare something with itself. --*/ assert (i1 != i2); #endif c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); i1++; i2++; k = last + 1; do { c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); s1 = quadrant[i1]; s2 = quadrant[i2]; if (s1 != s2) return (s1 > s2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); s1 = quadrant[i1]; s2 = quadrant[i2]; if (s1 != s2) return (s1 > s2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); s1 = quadrant[i1]; s2 = quadrant[i2]; if (s1 != s2) return (s1 > s2); i1++; i2++; c1 = block[i1]; c2 = block[i2]; if (c1 != c2) return (c1 > c2); s1 = quadrant[i1]; s2 = quadrant[i2]; if (s1 != s2) return (s1 > s2); i1++; i2++; if (i1 > last) { i1 -= last; i1--; }; if (i2 > last) { i2 -= last; i2--; }; k -= 4; workDone++; } while (k >= 0); return False; } /*---------------------------------------------*/ /*-- Knuth's increments seem to work better than Incerpi-Sedgewick here. Possibly because the number of elems to sort is usually small, typically <= 20. --*/ Int32 incs[14] = { 1, 4, 13, 40, 121, 364, 1093, 3280, 9841, 29524, 88573, 265720, 797161, 2391484 }; void simpleSort ( Int32 lo, Int32 hi, Int32 d ) { Int32 i, j, h, bigN, hp; Int32 v; bigN = hi - lo + 1; if (bigN < 2) return; hp = 0; while (incs[hp] < bigN) hp++; hp--; for (; hp >= 0; hp--) { h = incs[hp]; if (verbosity >= 5) fprintf ( stderr, " shell increment %d\n", h ); i = lo + h; while (True) { /*-- copy 1 --*/ if (i > hi) break; v = zptr[i]; j = i; while ( fullGtU ( zptr[j-h]+d, v+d ) ) { zptr[j] = zptr[j-h]; j = j - h; if (j <= (lo + h - 1)) break; } zptr[j] = v; i++; /*-- copy 2 --*/ if (i > hi) break; v = zptr[i]; j = i; while ( fullGtU ( zptr[j-h]+d, v+d ) ) { zptr[j] = zptr[j-h]; j = j - h; if (j <= (lo + h - 1)) break; } zptr[j] = v; i++; /*-- copy 3 --*/ if (i > hi) break; v = zptr[i]; j = i; while ( fullGtU ( zptr[j-h]+d, v+d ) ) { zptr[j] = zptr[j-h]; j = j - h; if (j <= (lo + h - 1)) break; } zptr[j] = v; i++; if (workDone > workLimit && firstAttempt) return; } } } /*---------------------------------------------*/ /*-- The following is an implementation of an elegant 3-way quicksort for strings, described in a paper "Fast Algorithms for Sorting and Searching Strings", by Robert Sedgewick and Jon L. Bentley. --*/ #define swap(lv1, lv2) \ { Int32 tmp = lv1; lv1 = lv2; lv2 = tmp; } INLINE void vswap ( Int32 p1, Int32 p2, Int32 n ) { while (n > 0) { swap(zptr[p1], zptr[p2]); p1++; p2++; n--; } } INLINE UChar med3 ( UChar a, UChar b, UChar c ) { UChar t; if (a > b) { t = a; a = b; b = t; }; if (b > c) { t = b; b = c; c = t; }; if (a > b) b = a; return b; } #define min(a,b) ((a) < (b)) ? (a) : (b) typedef struct { Int32 ll; Int32 hh; Int32 dd; } StackElem; #define push(lz,hz,dz) { stack[sp].ll = lz; \ stack[sp].hh = hz; \ stack[sp].dd = dz; \ sp++; } #define pop(lz,hz,dz) { sp--; \ lz = stack[sp].ll; \ hz = stack[sp].hh; \ dz = stack[sp].dd; } #define SMALL_THRESH 20 #define DEPTH_THRESH 10 /*-- If you are ever unlucky/improbable enough to get a stack overflow whilst sorting, increase the following constant and try again. In practice I have never seen the stack go above 27 elems, so the following limit seems very generous. --*/ #define QSORT_STACK_SIZE 1000 void qSort3 ( Int32 loSt, Int32 hiSt, Int32 dSt ) { Int32 unLo, unHi, ltLo, gtHi, med, n, m; Int32 sp, lo, hi, d; StackElem stack[QSORT_STACK_SIZE]; sp = 0; push ( loSt, hiSt, dSt ); while (sp > 0) { if (sp >= QSORT_STACK_SIZE) panic ( "stack overflow in qSort3" ); pop ( lo, hi, d ); if (hi - lo < SMALL_THRESH || d > DEPTH_THRESH) { simpleSort ( lo, hi, d ); if (workDone > workLimit && firstAttempt) return; continue; } med = med3 ( block[zptr[ lo ]+d], block[zptr[ hi ]+d], block[zptr[ (lo+hi)>>1 ]+d] ); unLo = ltLo = lo; unHi = gtHi = hi; while (True) { while (True) { if (unLo > unHi) break; n = ((Int32)block[zptr[unLo]+d]) - med; if (n == 0) { swap(zptr[unLo], zptr[ltLo]); ltLo++; unLo++; continue; }; if (n > 0) break; unLo++; } while (True) { if (unLo > unHi) break; n = ((Int32)block[zptr[unHi]+d]) - med; if (n == 0) { swap(zptr[unHi], zptr[gtHi]); gtHi--; unHi--; continue; }; if (n < 0) break; unHi--; } if (unLo > unHi) break; swap(zptr[unLo], zptr[unHi]); unLo++; unHi--; } #if DEBUG assert (unHi == unLo-1); #endif if (gtHi < ltLo) { push(lo, hi, d+1 ); continue; } n = min(ltLo-lo, unLo-ltLo); vswap(lo, unLo-n, n); m = min(hi-gtHi, gtHi-unHi); vswap(unLo, hi-m+1, m); n = lo + unLo - ltLo - 1; m = hi - (gtHi - unHi) + 1; push ( lo, n, d ); push ( n+1, m-1, d+1 ); push ( m, hi, d ); } } /*---------------------------------------------*/ #define BIGFREQ(b) (ftab[((b)+1) << 8] - ftab[(b) << 8]) #define SETMASK (1 << 21) #define CLEARMASK (~(SETMASK)) void sortIt ( void ) { Int32 i, j, ss, sb; Int32 runningOrder[256]; Int32 copy[256]; Bool bigDone[256]; UChar c1, c2; Int32 numQSorted; /*-- In the various block-sized structures, live data runs from 0 to last+NUM_OVERSHOOT_BYTES inclusive. First, set up the overshoot area for block. --*/ if (verbosity >= 4) fprintf ( stderr, " sort initialise ...\n" ); for (i = 0; i < NUM_OVERSHOOT_BYTES; i++) block[last+i+1] = block[i % (last+1)]; for (i = 0; i <= last+NUM_OVERSHOOT_BYTES; i++) quadrant[i] = 0; block[-1] = block[last]; if (last < 4000) { /*-- Use simpleSort(), since the full sorting mechanism has quite a large constant overhead. --*/ if (verbosity >= 4) fprintf ( stderr, " simpleSort ...\n" ); for (i = 0; i <= last; i++) zptr[i] = i; firstAttempt = False; workDone = workLimit = 0; simpleSort ( 0, last, 0 ); if (verbosity >= 4) fprintf ( stderr, " simpleSort done.\n" ); } else { numQSorted = 0; for (i = 0; i <= 255; i++) bigDone[i] = False; if (verbosity >= 4) fprintf ( stderr, " bucket sorting ...\n" ); for (i = 0; i <= 65536; i++) ftab[i] = 0; c1 = block[-1]; for (i = 0; i <= last; i++) { c2 = block[i]; ftab[(c1 << 8) + c2]++; c1 = c2; } for (i = 1; i <= 65536; i++) ftab[i] += ftab[i-1]; c1 = block[0]; for (i = 0; i < last; i++) { c2 = block[i+1]; j = (c1 << 8) + c2; c1 = c2; ftab[j]--; zptr[ftab[j]] = i; } j = (block[last] << 8) + block[0]; ftab[j]--; zptr[ftab[j]] = last; /*-- Now ftab contains the first loc of every small bucket. Calculate the running order, from smallest to largest big bucket. --*/ for (i = 0; i <= 255; i++) runningOrder[i] = i; { Int32 vv; Int32 h = 1; do h = 3 * h + 1; while (h <= 256); do { h = h / 3; for (i = h; i <= 255; i++) { vv = runningOrder[i]; j = i; while ( BIGFREQ(runningOrder[j-h]) > BIGFREQ(vv) ) { runningOrder[j] = runningOrder[j-h]; j = j - h; if (j <= (h - 1)) goto zero; } zero: runningOrder[j] = vv; } } while (h != 1); } /*-- The main sorting loop. --*/ for (i = 0; i <= 255; i++) { /*-- Process big buckets, starting with the least full. --*/ ss = runningOrder[i]; /*-- Complete the big bucket [ss] by quicksorting any unsorted small buckets [ss, j]. Hopefully previous pointer-scanning phases have already completed many of the small buckets [ss, j], so we don't have to sort them at all. --*/ for (j = 0; j <= 255; j++) { sb = (ss << 8) + j; if ( ! (ftab[sb] & SETMASK) ) { Int32 lo = ftab[sb] & CLEARMASK; Int32 hi = (ftab[sb+1] & CLEARMASK) - 1; if (hi > lo) { if (verbosity >= 4) fprintf ( stderr, " qsort [0x%x, 0x%x] done %d this %d\n", ss, j, numQSorted, hi - lo + 1 ); qSort3 ( lo, hi, 2 ); numQSorted += ( hi - lo + 1 ); if (workDone > workLimit && firstAttempt) return; } ftab[sb] |= SETMASK; } } /*-- The ss big bucket is now done. Record this fact, and update the quadrant descriptors. Remember to update quadrants in the overshoot area too, if necessary. The "if (i < 255)" test merely skips this updating for the last bucket processed, since updating for the last bucket is pointless. --*/ bigDone[ss] = True; if (i < 255) { Int32 bbStart = ftab[ss << 8] & CLEARMASK; Int32 bbSize = (ftab[(ss+1) << 8] & CLEARMASK) - bbStart; Int32 shifts = 0; while ((bbSize >> shifts) > 65534) shifts++; for (j = 0; j < bbSize; j++) { Int32 a2update = zptr[bbStart + j]; UInt16 qVal = (UInt16)(j >> shifts); quadrant[a2update] = qVal; if (a2update < NUM_OVERSHOOT_BYTES) quadrant[a2update + last + 1] = qVal; } if (! ( ((bbSize-1) >> shifts) <= 65535 )) panic ( "sortIt" ); } /*-- Now scan this big bucket so as to synthesise the sorted order for small buckets [t, ss] for all t != ss. --*/ for (j = 0; j <= 255; j++) copy[j] = ftab[(j << 8) + ss] & CLEARMASK; for (j = ftab[ss << 8] & CLEARMASK; j < (ftab[(ss+1) << 8] & CLEARMASK); j++) { c1 = block[zptr[j]-1]; if ( ! bigDone[c1] ) { zptr[copy[c1]] = zptr[j] == 0 ? last : zptr[j] - 1; copy[c1] ++; } } for (j = 0; j <= 255; j++) ftab[(j << 8) + ss] |= SETMASK; } if (verbosity >= 4) fprintf ( stderr, " %d pointers, %d sorted, %d scanned\n", last+1, numQSorted, (last+1) - numQSorted ); } } /*---------------------------------------------------*/ /*--- Stuff for randomising repetitive blocks ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ Int32 rNums[512] = { 619, 720, 127, 481, 931, 816, 813, 233, 566, 247, 985, 724, 205, 454, 863, 491, 741, 242, 949, 214, 733, 859, 335, 708, 621, 574, 73, 654, 730, 472, 419, 436, 278, 496, 867, 210, 399, 680, 480, 51, 878, 465, 811, 169, 869, 675, 611, 697, 867, 561, 862, 687, 507, 283, 482, 129, 807, 591, 733, 623, 150, 238, 59, 379, 684, 877, 625, 169, 643, 105, 170, 607, 520, 932, 727, 476, 693, 425, 174, 647, 73, 122, 335, 530, 442, 853, 695, 249, 445, 515, 909, 545, 703, 919, 874, 474, 882, 500, 594, 612, 641, 801, 220, 162, 819, 984, 589, 513, 495, 799, 161, 604, 958, 533, 221, 400, 386, 867, 600, 782, 382, 596, 414, 171, 516, 375, 682, 485, 911, 276, 98, 553, 163, 354, 666, 933, 424, 341, 533, 870, 227, 730, 475, 186, 263, 647, 537, 686, 600, 224, 469, 68, 770, 919, 190, 373, 294, 822, 808, 206, 184, 943, 795, 384, 383, 461, 404, 758, 839, 887, 715, 67, 618, 276, 204, 918, 873, 777, 604, 560, 951, 160, 578, 722, 79, 804, 96, 409, 713, 940, 652, 934, 970, 447, 318, 353, 859, 672, 112, 785, 645, 863, 803, 350, 139, 93, 354, 99, 820, 908, 609, 772, 154, 274, 580, 184, 79, 626, 630, 742, 653, 282, 762, 623, 680, 81, 927, 626, 789, 125, 411, 521, 938, 300, 821, 78, 343, 175, 128, 250, 170, 774, 972, 275, 999, 639, 495, 78, 352, 126, 857, 956, 358, 619, 580, 124, 737, 594, 701, 612, 669, 112, 134, 694, 363, 992, 809, 743, 168, 974, 944, 375, 748, 52, 600, 747, 642, 182, 862, 81, 344, 805, 988, 739, 511, 655, 814, 334, 249, 515, 897, 955, 664, 981, 649, 113, 974, 459, 893, 228, 433, 837, 553, 268, 926, 240, 102, 654, 459, 51, 686, 754, 806, 760, 493, 403, 415, 394, 687, 700, 946, 670, 656, 610, 738, 392, 760, 799, 887, 653, 978, 321, 576, 617, 626, 502, 894, 679, 243, 440, 680, 879, 194, 572, 640, 724, 926, 56, 204, 700, 707, 151, 457, 449, 797, 195, 791, 558, 945, 679, 297, 59, 87, 824, 713, 663, 412, 693, 342, 606, 134, 108, 571, 364, 631, 212, 174, 643, 304, 329, 343, 97, 430, 751, 497, 314, 983, 374, 822, 928, 140, 206, 73, 263, 980, 736, 876, 478, 430, 305, 170, 514, 364, 692, 829, 82, 855, 953, 676, 246, 369, 970, 294, 750, 807, 827, 150, 790, 288, 923, 804, 378, 215, 828, 592, 281, 565, 555, 710, 82, 896, 831, 547, 261, 524, 462, 293, 465, 502, 56, 661, 821, 976, 991, 658, 869, 905, 758, 745, 193, 768, 550, 608, 933, 378, 286, 215, 979, 792, 961, 61, 688, 793, 644, 986, 403, 106, 366, 905, 644, 372, 567, 466, 434, 645, 210, 389, 550, 919, 135, 780, 773, 635, 389, 707, 100, 626, 958, 165, 504, 920, 176, 193, 713, 857, 265, 203, 50, 668, 108, 645, 990, 626, 197, 510, 357, 358, 850, 858, 364, 936, 638 }; #define RAND_DECLS \ Int32 rNToGo = 0; \ Int32 rTPos = 0; \ #define RAND_MASK ((rNToGo == 1) ? 1 : 0) #define RAND_UPD_MASK \ if (rNToGo == 0) { \ rNToGo = rNums[rTPos]; \ rTPos++; if (rTPos == 512) rTPos = 0; \ } \ rNToGo--; /*---------------------------------------------------*/ /*--- The Reversible Transformation (tm) ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ void randomiseBlock ( void ) { Int32 i; RAND_DECLS; for (i = 0; i < 256; i++) inUse[i] = False; for (i = 0; i <= last; i++) { RAND_UPD_MASK; block[i] ^= RAND_MASK; inUse[block[i]] = True; } } /*---------------------------------------------*/ void doReversibleTransformation ( void ) { Int32 i; if (verbosity >= 2) fprintf ( stderr, "\n" ); workLimit = workFactor * last; workDone = 0; blockRandomised = False; firstAttempt = True; sortIt (); if (verbosity >= 3) fprintf ( stderr, " %d work, %d block, ratio %5.2f\n", workDone, last, (float)workDone / (float)(last) ); if (workDone > workLimit && firstAttempt) { if (verbosity >= 2) fprintf ( stderr, " sorting aborted; randomising block\n" ); randomiseBlock (); workLimit = workDone = 0; blockRandomised = True; firstAttempt = False; sortIt(); if (verbosity >= 3) fprintf ( stderr, " %d work, %d block, ratio %f\n", workDone, last, (float)workDone / (float)(last) ); } origPtr = -1; for (i = 0; i <= last; i++) if (zptr[i] == 0) { origPtr = i; break; }; if (origPtr == -1) panic ( "doReversibleTransformation" ); } /*---------------------------------------------*/ INLINE Int32 indexIntoF ( Int32 indx, Int32 *cftab ) { Int32 nb, na, mid; nb = 0; na = 256; do { mid = (nb + na) >> 1; if (indx >= cftab[mid]) nb = mid; else na = mid; } while (na - nb != 1); return nb; } #define GET_SMALL(cccc) \ \ cccc = indexIntoF ( tPos, cftab ); \ tPos = GET_LL(tPos); void undoReversibleTransformation_small ( FILE* dst ) { Int32 cftab[257], cftabAlso[257]; Int32 i, j, tmp, tPos; UChar ch; /*-- We assume here that the global array unzftab will already be holding the frequency counts for ll8[0 .. last]. --*/ /*-- Set up cftab to facilitate generation of indexIntoF --*/ cftab[0] = 0; for (i = 1; i <= 256; i++) cftab[i] = unzftab[i-1]; for (i = 1; i <= 256; i++) cftab[i] += cftab[i-1]; /*-- Make a copy of it, used in generation of T --*/ for (i = 0; i <= 256; i++) cftabAlso[i] = cftab[i]; /*-- compute the T vector --*/ for (i = 0; i <= last; i++) { ch = (UChar)ll16[i]; SET_LL(i, cftabAlso[ch]); cftabAlso[ch]++; } /*-- Compute T^(-1) by pointer reversal on T. This is rather subtle, in that, if the original block was two or more (in general, N) concatenated copies of the same thing, the T vector will consist of N cycles, each of length blocksize / N, and decoding will involve traversing one of these cycles N times. Which particular cycle doesn't matter -- they are all equivalent. The tricky part is to make sure that the pointer reversal creates a correct reversed cycle for us to traverse. So, the code below simply reverses whatever cycle origPtr happens to fall into, without regard to the cycle length. That gives one reversed cycle, which for normal blocks, is the entire block-size long. For repeated blocks, it will be interspersed with the other N-1 non-reversed cycles. Providing that the F-subscripting phase which follows starts at origPtr, all then works ok. --*/ i = origPtr; j = GET_LL(i); do { tmp = GET_LL(j); SET_LL(j, i); i = j; j = tmp; } while (i != origPtr); /*-- We recreate the original by subscripting F through T^(-1). The run-length-decoder below requires characters incrementally, so tPos is set to a starting value, and is updated by the GET_SMALL macro. --*/ tPos = origPtr; /*-------------------------------------------------*/ /*-- This is pretty much a verbatim copy of the run-length decoder present in the distribution bzip-0.21; it has to be here to avoid creating block[] as an intermediary structure. As in 0.21, this code derives from some sent to me by Christian von Roques. It allows dst==NULL, so as to support the test (-t) option without slowing down the fast decompression code. --*/ { IntNative retVal; Int32 i2, count, chPrev, ch2; UInt32 localCrc; count = 0; i2 = 0; ch2 = 256; /*-- not a char and not EOF --*/ localCrc = getGlobalCRC(); { RAND_DECLS; while ( i2 <= last ) { chPrev = ch2; GET_SMALL(ch2); if (blockRandomised) { RAND_UPD_MASK; ch2 ^= (UInt32)RAND_MASK; } i2++; if (dst) retVal = putc ( ch2, dst ); UPDATE_CRC ( localCrc, (UChar)ch2 ); if (ch2 != chPrev) { count = 1; } else { count++; if (count >= 4) { Int32 j2; UChar z; GET_SMALL(z); if (blockRandomised) { RAND_UPD_MASK; z ^= RAND_MASK; } for (j2 = 0; j2 < (Int32)z; j2++) { if (dst) retVal = putc (ch2, dst); UPDATE_CRC ( localCrc, (UChar)ch2 ); } i2++; count = 0; } } } } setGlobalCRC ( localCrc ); } /*-- end of the in-line run-length-decoder. --*/ } #undef GET_SMALL /*---------------------------------------------*/ #define GET_FAST(cccc) \ \ cccc = ll8[tPos]; \ tPos = tt[tPos]; void undoReversibleTransformation_fast ( FILE* dst ) { Int32 cftab[257]; Int32 i, tPos; UChar ch; /*-- We assume here that the global array unzftab will already be holding the frequency counts for ll8[0 .. last]. --*/ /*-- Set up cftab to facilitate generation of T^(-1) --*/ cftab[0] = 0; for (i = 1; i <= 256; i++) cftab[i] = unzftab[i-1]; for (i = 1; i <= 256; i++) cftab[i] += cftab[i-1]; /*-- compute the T^(-1) vector --*/ for (i = 0; i <= last; i++) { ch = (UChar)ll8[i]; tt[cftab[ch]] = i; cftab[ch]++; } /*-- We recreate the original by subscripting L through T^(-1). The run-length-decoder below requires characters incrementally, so tPos is set to a starting value, and is updated by the GET_FAST macro. --*/ tPos = tt[origPtr]; /*-------------------------------------------------*/ /*-- This is pretty much a verbatim copy of the run-length decoder present in the distribution bzip-0.21; it has to be here to avoid creating block[] as an intermediary structure. As in 0.21, this code derives from some sent to me by Christian von Roques. --*/ { IntNative retVal; Int32 i2, count, chPrev, ch2; UInt32 localCrc; count = 0; i2 = 0; ch2 = 256; /*-- not a char and not EOF --*/ localCrc = getGlobalCRC(); if (blockRandomised) { RAND_DECLS; while ( i2 <= last ) { chPrev = ch2; GET_FAST(ch2); RAND_UPD_MASK; ch2 ^= (UInt32)RAND_MASK; i2++; retVal = putc ( ch2, dst ); UPDATE_CRC ( localCrc, (UChar)ch2 ); if (ch2 != chPrev) { count = 1; } else { count++; if (count >= 4) { Int32 j2; UChar z; GET_FAST(z); RAND_UPD_MASK; z ^= RAND_MASK; for (j2 = 0; j2 < (Int32)z; j2++) { retVal = putc (ch2, dst); UPDATE_CRC ( localCrc, (UChar)ch2 ); } i2++; count = 0; } } } } else { while ( i2 <= last ) { chPrev = ch2; GET_FAST(ch2); i2++; retVal = putc ( ch2, dst ); UPDATE_CRC ( localCrc, (UChar)ch2 ); if (ch2 != chPrev) { count = 1; } else { count++; if (count >= 4) { Int32 j2; UChar z; GET_FAST(z); for (j2 = 0; j2 < (Int32)z; j2++) { retVal = putc (ch2, dst); UPDATE_CRC ( localCrc, (UChar)ch2 ); } i2++; count = 0; } } } } /*-- if (blockRandomised) --*/ setGlobalCRC ( localCrc ); } /*-- end of the in-line run-length-decoder. --*/ } #undef GET_FAST /*---------------------------------------------------*/ /*--- The block loader and RLEr ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ /* Top 16: run length, 1 to 255. * Lower 16: the char, or MY_EOF for EOF. */ #define MY_EOF 257 INLINE Int32 getRLEpair ( FILE* src ) { Int32 runLength; IntNative ch, chLatest; ch = getc ( src ); /*--- Because I have no idea what kind of a value EOF is. ---*/ if (ch == EOF) { ERROR_IF_NOT_ZERO ( ferror(src)); return (1 << 16) | MY_EOF; } runLength = 0; do { chLatest = getc ( src ); runLength++; bytesIn++; } while (ch == chLatest && runLength < 255); if ( chLatest != EOF ) { if ( ungetc ( chLatest, src ) == EOF ) panic ( "getRLEpair: ungetc failed" ); } else { ERROR_IF_NOT_ZERO ( ferror(src) ); } /*--- Conditional is just a speedup hack. ---*/ if (runLength == 1) { UPDATE_CRC ( globalCrc, (UChar)ch ); return (1 << 16) | ch; } else { Int32 i; for (i = 1; i <= runLength; i++) UPDATE_CRC ( globalCrc, (UChar)ch ); return (runLength << 16) | ch; } } /*---------------------------------------------*/ void loadAndRLEsource ( FILE* src ) { Int32 ch, allowableBlockSize, i; last = -1; ch = 0; for (i = 0; i < 256; i++) inUse[i] = False; /*--- 20 is just a paranoia constant ---*/ allowableBlockSize = 100000 * blockSize100k - 20; while (last < allowableBlockSize && ch != MY_EOF) { Int32 rlePair, runLen; rlePair = getRLEpair ( src ); ch = rlePair & 0xFFFF; runLen = (UInt32)rlePair >> 16; #if DEBUG assert (runLen >= 1 && runLen <= 255); #endif if (ch != MY_EOF) { inUse[ch] = True; switch (runLen) { case 1: last++; block[last] = (UChar)ch; break; case 2: last++; block[last] = (UChar)ch; last++; block[last] = (UChar)ch; break; case 3: last++; block[last] = (UChar)ch; last++; block[last] = (UChar)ch; last++; block[last] = (UChar)ch; break; default: inUse[runLen-4] = True; last++; block[last] = (UChar)ch; last++; block[last] = (UChar)ch; last++; block[last] = (UChar)ch; last++; block[last] = (UChar)ch; last++; block[last] = (UChar)(runLen-4); break; } } } } /*---------------------------------------------------*/ /*--- Processing of complete files and streams ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ void compressStream ( FILE *stream, FILE *zStream ) { IntNative retVal; UInt32 blockCRC, combinedCRC; Int32 blockNo; blockNo = 0; bytesIn = 0; bytesOut = 0; nBlocksRandomised = 0; SET_BINARY_MODE(stream); SET_BINARY_MODE(zStream); ERROR_IF_NOT_ZERO ( ferror(stream) ); ERROR_IF_NOT_ZERO ( ferror(zStream) ); bsSetStream ( zStream, True ); /*--- Write `magic' bytes B and Z, then h indicating file-format == huffmanised, followed by a digit indicating blockSize100k. ---*/ bsPutUChar ( 'B' ); bsPutUChar ( 'Z' ); bsPutUChar ( 'h' ); bsPutUChar ( '0' + blockSize100k ); combinedCRC = 0; if (verbosity >= 2) fprintf ( stderr, "\n" ); while (True) { blockNo++; initialiseCRC (); loadAndRLEsource ( stream ); ERROR_IF_NOT_ZERO ( ferror(stream) ); if (last == -1) break; blockCRC = getFinalCRC (); combinedCRC = (combinedCRC << 1) | (combinedCRC >> 31); combinedCRC ^= blockCRC; if (verbosity >= 2) fprintf ( stderr, " block %d: crc = 0x%8x, combined CRC = 0x%8x, size = %d", blockNo, blockCRC, combinedCRC, last+1 ); /*-- sort the block and establish posn of original string --*/ doReversibleTransformation (); /*-- A 6-byte block header, the value chosen arbitrarily as 0x314159265359 :-). A 32 bit value does not really give a strong enough guarantee that the value will not appear by chance in the compressed datastream. Worst-case probability of this event, for a 900k block, is about 2.0e-3 for 32 bits, 1.0e-5 for 40 bits and 4.0e-8 for 48 bits. For a compressed file of size 100Gb -- about 100000 blocks -- only a 48-bit marker will do. NB: normal compression/ decompression do *not* rely on these statistical properties. They are only important when trying to recover blocks from damaged files. --*/ bsPutUChar ( 0x31 ); bsPutUChar ( 0x41 ); bsPutUChar ( 0x59 ); bsPutUChar ( 0x26 ); bsPutUChar ( 0x53 ); bsPutUChar ( 0x59 ); /*-- Now the block's CRC, so it is in a known place. --*/ bsPutUInt32 ( blockCRC ); /*-- Now a single bit indicating randomisation. --*/ if (blockRandomised) { bsW(1,1); nBlocksRandomised++; } else bsW(1,0); /*-- Finally, block's contents proper. --*/ moveToFrontCodeAndSend (); ERROR_IF_NOT_ZERO ( ferror(zStream) ); } if (verbosity >= 2 && nBlocksRandomised > 0) fprintf ( stderr, " %d block%s needed randomisation\n", nBlocksRandomised, nBlocksRandomised == 1 ? "" : "s" ); /*-- Now another magic 48-bit number, 0x177245385090, to indicate the end of the last block. (sqrt(pi), if you want to know. I did want to use e, but it contains too much repetition -- 27 18 28 18 28 46 -- for me to feel statistically comfortable. Call me paranoid.) --*/ bsPutUChar ( 0x17 ); bsPutUChar ( 0x72 ); bsPutUChar ( 0x45 ); bsPutUChar ( 0x38 ); bsPutUChar ( 0x50 ); bsPutUChar ( 0x90 ); bsPutUInt32 ( combinedCRC ); if (verbosity >= 2) fprintf ( stderr, " final combined CRC = 0x%x\n ", combinedCRC ); /*-- Close the files in an utterly paranoid way. --*/ bsFinishedWithStream (); ERROR_IF_NOT_ZERO ( ferror(zStream) ); retVal = fflush ( zStream ); ERROR_IF_EOF ( retVal ); retVal = fclose ( zStream ); ERROR_IF_EOF ( retVal ); ERROR_IF_NOT_ZERO ( ferror(stream) ); retVal = fclose ( stream ); ERROR_IF_EOF ( retVal ); if (bytesIn == 0) bytesIn = 1; if (bytesOut == 0) bytesOut = 1; if (verbosity >= 1) fprintf ( stderr, "%6.3f:1, %6.3f bits/byte, " "%5.2f%% saved, %d in, %d out.\n", (float)bytesIn / (float)bytesOut, (8.0 * (float)bytesOut) / (float)bytesIn, 100.0 * (1.0 - (float)bytesOut / (float)bytesIn), bytesIn, bytesOut ); } /*---------------------------------------------*/ Bool uncompressStream ( FILE *zStream, FILE *stream ) { UChar magic1, magic2, magic3, magic4; UChar magic5, magic6; UInt32 storedBlockCRC, storedCombinedCRC; UInt32 computedBlockCRC, computedCombinedCRC; Int32 currBlockNo; IntNative retVal; SET_BINARY_MODE(stream); SET_BINARY_MODE(zStream); ERROR_IF_NOT_ZERO ( ferror(stream) ); ERROR_IF_NOT_ZERO ( ferror(zStream) ); bsSetStream ( zStream, False ); /*-- A bad magic number is `recoverable from'; return with False so the caller skips the file. --*/ magic1 = bsGetUChar (); magic2 = bsGetUChar (); magic3 = bsGetUChar (); magic4 = bsGetUChar (); if (magic1 != 'B' || magic2 != 'Z' || magic3 != 'h' || magic4 < '1' || magic4 > '9') { bsFinishedWithStream(); retVal = fclose ( stream ); ERROR_IF_EOF ( retVal ); return False; } setDecompressStructureSizes ( magic4 - '0' ); computedCombinedCRC = 0; if (verbosity >= 2) fprintf ( stderr, "\n " ); currBlockNo = 0; while (True) { magic1 = bsGetUChar (); magic2 = bsGetUChar (); magic3 = bsGetUChar (); magic4 = bsGetUChar (); magic5 = bsGetUChar (); magic6 = bsGetUChar (); if (magic1 == 0x17 && magic2 == 0x72 && magic3 == 0x45 && magic4 == 0x38 && magic5 == 0x50 && magic6 == 0x90) break; if (magic1 != 0x31 || magic2 != 0x41 || magic3 != 0x59 || magic4 != 0x26 || magic5 != 0x53 || magic6 != 0x59) badBlockHeader(); storedBlockCRC = bsGetUInt32 (); if (bsR(1) == 1) blockRandomised = True; else blockRandomised = False; currBlockNo++; if (verbosity >= 2) fprintf ( stderr, "[%d: huff+mtf ", currBlockNo ); getAndMoveToFrontDecode (); ERROR_IF_NOT_ZERO ( ferror(zStream) ); initialiseCRC(); if (verbosity >= 2) fprintf ( stderr, "rt+rld" ); if (smallMode) undoReversibleTransformation_small ( stream ); else undoReversibleTransformation_fast ( stream ); ERROR_IF_NOT_ZERO ( ferror(stream) ); computedBlockCRC = getFinalCRC(); if (verbosity >= 3) fprintf ( stderr, " {0x%x, 0x%x}", storedBlockCRC, computedBlockCRC ); if (verbosity >= 2) fprintf ( stderr, "] " ); /*-- A bad CRC is considered a fatal error. --*/ if (storedBlockCRC != computedBlockCRC) crcError ( storedBlockCRC, computedBlockCRC ); computedCombinedCRC = (computedCombinedCRC << 1) | (computedCombinedCRC >> 31); computedCombinedCRC ^= computedBlockCRC; }; if (verbosity >= 2) fprintf ( stderr, "\n " ); storedCombinedCRC = bsGetUInt32 (); if (verbosity >= 2) fprintf ( stderr, "combined CRCs: stored = 0x%x, computed = 0x%x\n ", storedCombinedCRC, computedCombinedCRC ); if (storedCombinedCRC != computedCombinedCRC) crcError ( storedCombinedCRC, computedCombinedCRC ); bsFinishedWithStream (); ERROR_IF_NOT_ZERO ( ferror(zStream) ); retVal = fclose ( zStream ); ERROR_IF_EOF ( retVal ); ERROR_IF_NOT_ZERO ( ferror(stream) ); retVal = fflush ( stream ); ERROR_IF_NOT_ZERO ( retVal ); if (stream != stdout) { retVal = fclose ( stream ); ERROR_IF_EOF ( retVal ); } return True; } /*---------------------------------------------*/ Bool testStream ( FILE *zStream ) { UChar magic1, magic2, magic3, magic4; UChar magic5, magic6; UInt32 storedBlockCRC, storedCombinedCRC; UInt32 computedBlockCRC, computedCombinedCRC; Int32 currBlockNo; IntNative retVal; SET_BINARY_MODE(zStream); ERROR_IF_NOT_ZERO ( ferror(zStream) ); bsSetStream ( zStream, False ); magic1 = bsGetUChar (); magic2 = bsGetUChar (); magic3 = bsGetUChar (); magic4 = bsGetUChar (); if (magic1 != 'B' || magic2 != 'Z' || magic3 != 'h' || magic4 < '1' || magic4 > '9') { bsFinishedWithStream(); fclose ( zStream ); fprintf ( stderr, "\n%s: bad magic number (ie, not created by bzip2)\n", inName ); return False; } smallMode = True; setDecompressStructureSizes ( magic4 - '0' ); computedCombinedCRC = 0; if (verbosity >= 2) fprintf ( stderr, "\n" ); currBlockNo = 0; while (True) { magic1 = bsGetUChar (); magic2 = bsGetUChar (); magic3 = bsGetUChar (); magic4 = bsGetUChar (); magic5 = bsGetUChar (); magic6 = bsGetUChar (); if (magic1 == 0x17 && magic2 == 0x72 && magic3 == 0x45 && magic4 == 0x38 && magic5 == 0x50 && magic6 == 0x90) break; currBlockNo++; if (magic1 != 0x31 || magic2 != 0x41 || magic3 != 0x59 || magic4 != 0x26 || magic5 != 0x53 || magic6 != 0x59) { bsFinishedWithStream(); fclose ( zStream ); fprintf ( stderr, "\n%s, block %d: bad header (not == 0x314159265359)\n", inName, currBlockNo ); return False; } storedBlockCRC = bsGetUInt32 (); if (bsR(1) == 1) blockRandomised = True; else blockRandomised = False; if (verbosity >= 2) fprintf ( stderr, " block [%d: huff+mtf ", currBlockNo ); getAndMoveToFrontDecode (); ERROR_IF_NOT_ZERO ( ferror(zStream) ); initialiseCRC(); if (verbosity >= 2) fprintf ( stderr, "rt+rld" ); undoReversibleTransformation_small ( NULL ); computedBlockCRC = getFinalCRC(); if (verbosity >= 3) fprintf ( stderr, " {0x%x, 0x%x}", storedBlockCRC, computedBlockCRC ); if (verbosity >= 2) fprintf ( stderr, "] " ); if (storedBlockCRC != computedBlockCRC) { bsFinishedWithStream(); fclose ( zStream ); fprintf ( stderr, "\n%s, block %d: computed CRC does not match stored one\n", inName, currBlockNo ); return False; } if (verbosity >= 2) fprintf ( stderr, "ok\n" ); computedCombinedCRC = (computedCombinedCRC << 1) | (computedCombinedCRC >> 31); computedCombinedCRC ^= computedBlockCRC; }; storedCombinedCRC = bsGetUInt32 (); if (verbosity >= 2) fprintf ( stderr, " combined CRCs: stored = 0x%x, computed = 0x%x\n ", storedCombinedCRC, computedCombinedCRC ); if (storedCombinedCRC != computedCombinedCRC) { bsFinishedWithStream(); fclose ( zStream ); fprintf ( stderr, "\n%s: computed CRC does not match stored one\n", inName ); return False; } bsFinishedWithStream (); ERROR_IF_NOT_ZERO ( ferror(zStream) ); retVal = fclose ( zStream ); ERROR_IF_EOF ( retVal ); return True; } /*---------------------------------------------------*/ /*--- Error [non-] handling grunge ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ void cadvise ( void ) { fprintf ( stderr, "\nIt is possible that the compressed file(s) have become corrupted.\n" "You can use the -tvv option to test integrity of such files.\n\n" "You can use the `bzip2recover' program to *attempt* to recover\n" "data from undamaged sections of corrupted files.\n\n" ); } /*---------------------------------------------*/ void showFileNames ( void ) { fprintf ( stderr, "\tInput file = %s, output file = %s\n", inName==NULL ? "(null)" : inName, outName==NULL ? "(null)" : outName ); } /*---------------------------------------------*/ void cleanUpAndFail ( Int32 ec ) { IntNative retVal; if ( srcMode == SM_F2F && opMode != OM_TEST ) { fprintf ( stderr, "%s: Deleting output file %s, if it exists.\n", progName, outName==NULL ? "(null)" : outName ); if (outputHandleJustInCase != NULL) fclose ( outputHandleJustInCase ); retVal = remove ( outName ); if (retVal != 0) fprintf ( stderr, "%s: WARNING: deletion of output file (apparently) failed.\n", progName ); } if (numFileNames > 0 && numFilesProcessed < numFileNames) { fprintf ( stderr, "%s: WARNING: some files have not been processed:\n" "\t%d specified on command line, %d not processed yet.\n\n", progName, numFileNames, numFileNames - numFilesProcessed ); } exit ( ec ); } /*---------------------------------------------*/ void panic ( Char* s ) { fprintf ( stderr, "\n%s: PANIC -- internal consistency error:\n" "\t%s\n" "\tThis is a BUG. Please report it to me at:\n" "\tjseward@acm.org\n", progName, s ); showFileNames(); cleanUpAndFail( 3 ); } /*---------------------------------------------*/ void badBGLengths ( void ) { fprintf ( stderr, "\n%s: error when reading background model code lengths,\n" "\twhich probably means the compressed file is corrupted.\n", progName ); showFileNames(); cadvise(); cleanUpAndFail( 2 ); } /*---------------------------------------------*/ void crcError ( UInt32 crcStored, UInt32 crcComputed ) { fprintf ( stderr, "\n%s: Data integrity error when decompressing.\n" "\tStored CRC = 0x%x, computed CRC = 0x%x\n", progName, crcStored, crcComputed ); showFileNames(); cadvise(); cleanUpAndFail( 2 ); } /*---------------------------------------------*/ void compressedStreamEOF ( void ) { fprintf ( stderr, "\n%s: Compressed file ends unexpectedly;\n\t" "perhaps it is corrupted? *Possible* reason follows.\n", progName ); perror ( progName ); showFileNames(); cadvise(); cleanUpAndFail( 2 ); } /*---------------------------------------------*/ void ioError ( ) { fprintf ( stderr, "\n%s: I/O or other error, bailing out. Possible reason follows.\n", progName ); perror ( progName ); showFileNames(); cleanUpAndFail( 1 ); } /*---------------------------------------------*/ void blockOverrun () { fprintf ( stderr, "\n%s: block overrun during decompression,\n" "\twhich probably means the compressed file\n" "\tis corrupted.\n", progName ); showFileNames(); cadvise(); cleanUpAndFail( 2 ); } /*---------------------------------------------*/ void badBlockHeader () { fprintf ( stderr, "\n%s: bad block header in the compressed file,\n" "\twhich probably means it is corrupted.\n", progName ); showFileNames(); cadvise(); cleanUpAndFail( 2 ); } /*---------------------------------------------*/ void bitStreamEOF () { fprintf ( stderr, "\n%s: read past the end of compressed data,\n" "\twhich probably means it is corrupted.\n", progName ); showFileNames(); cadvise(); cleanUpAndFail( 2 ); } /*---------------------------------------------*/ void mySignalCatcher ( IntNative n ) { fprintf ( stderr, "\n%s: Control-C (or similar) caught, quitting.\n", progName ); cleanUpAndFail(1); } /*---------------------------------------------*/ void mySIGSEGVorSIGBUScatcher ( IntNative n ) { if (opMode == OM_Z) fprintf ( stderr, "\n%s: Caught a SIGSEGV or SIGBUS whilst compressing,\n" "\twhich probably indicates a bug in bzip2. Please\n" "\treport it to me at: jseward@acm.org\n", progName ); else fprintf ( stderr, "\n%s: Caught a SIGSEGV or SIGBUS whilst decompressing,\n" "\twhich probably indicates that the compressed data\n" "\tis corrupted.\n", progName ); showFileNames(); if (opMode == OM_Z) cleanUpAndFail( 3 ); else { cadvise(); cleanUpAndFail( 2 ); } } /*---------------------------------------------*/ void uncompressOutOfMemory ( Int32 draw, Int32 blockSize ) { fprintf ( stderr, "\n%s: Can't allocate enough memory for decompression.\n" "\tRequested %d bytes for a block size of %d.\n" "\tTry selecting space-economic decompress (with flag -s)\n" "\tand failing that, find a machine with more memory.\n", progName, draw, blockSize ); showFileNames(); cleanUpAndFail(1); } /*---------------------------------------------*/ void compressOutOfMemory ( Int32 draw, Int32 blockSize ) { fprintf ( stderr, "\n%s: Can't allocate enough memory for compression.\n" "\tRequested %d bytes for a block size of %d.\n" "\tTry selecting a small block size (with flag -s).\n", progName, draw, blockSize ); showFileNames(); cleanUpAndFail(1); } /*---------------------------------------------------*/ /*--- The main driver machinery ---*/ /*---------------------------------------------------*/ /*---------------------------------------------*/ void pad ( Char *s ) { Int32 i; if ( (Int32)strlen(s) >= longestFileName ) return; for (i = 1; i <= longestFileName - (Int32)strlen(s); i++) fprintf ( stderr, " " ); } /*---------------------------------------------*/ Bool fileExists ( Char* name ) { FILE *tmp = fopen ( name, "rb" ); Bool exists = (tmp != NULL); if (tmp != NULL) fclose ( tmp ); return exists; } /*---------------------------------------------*/ /*-- if in doubt, return True --*/ Bool notABogStandardFile ( Char* name ) { IntNative i; struct MY_STAT statBuf; i = MY_LSTAT ( name, &statBuf ); if (i != 0) return True; if (MY_S_IFREG(statBuf.st_mode)) return False; return True; } /*---------------------------------------------*/ void copyDateAndPermissions ( Char *srcName, Char *dstName ) { #if BZ_UNIX IntNative retVal; struct MY_STAT statBuf; struct utimbuf uTimBuf; retVal = MY_LSTAT ( srcName, &statBuf ); ERROR_IF_NOT_ZERO ( retVal ); uTimBuf.actime = statBuf.st_atime; uTimBuf.modtime = statBuf.st_mtime; retVal = chmod ( dstName, statBuf.st_mode ); ERROR_IF_NOT_ZERO ( retVal ); retVal = utime ( dstName, &uTimBuf ); ERROR_IF_NOT_ZERO ( retVal ); #endif } /*---------------------------------------------*/ Bool endsInBz2 ( Char* name ) { Int32 n = strlen ( name ); if (n <= 4) return False; return (name[n-4] == '.' && name[n-3] == 'b' && name[n-2] == 'z' && name[n-1] == '2'); } /*---------------------------------------------*/ Bool containsDubiousChars ( Char* name ) { Bool cdc = False; for (; *name != '\0'; name++) if (*name == '?' || *name == '*') cdc = True; return cdc; } /*---------------------------------------------*/ void compress ( Char *name ) { FILE *inStr; FILE *outStr; if (name == NULL && srcMode != SM_I2O) panic ( "compress: bad modes\n" ); switch (srcMode) { case SM_I2O: strcpy ( inName, "(stdin)" ); strcpy ( outName, "(stdout)" ); break; case SM_F2F: strcpy ( inName, name ); strcpy ( outName, name ); strcat ( outName, ".bz2" ); break; case SM_F2O: strcpy ( inName, name ); strcpy ( outName, "(stdout)" ); break; } if ( srcMode != SM_I2O && containsDubiousChars ( inName ) ) { fprintf ( stderr, "%s: There are no files matching `%s'.\n", progName, inName ); return; } if ( srcMode != SM_I2O && !fileExists ( inName ) ) { fprintf ( stderr, "%s: Input file %s doesn't exist, skipping.\n", progName, inName ); return; } if ( srcMode != SM_I2O && endsInBz2 ( inName )) { fprintf ( stderr, "%s: Input file name %s ends in `.bz2', skipping.\n", progName, inName ); return; } if ( srcMode != SM_I2O && notABogStandardFile ( inName )) { fprintf ( stderr, "%s: Input file %s is not a normal file, skipping.\n", progName, inName ); return; } if ( srcMode == SM_F2F && fileExists ( outName ) ) { fprintf ( stderr, "%s: Output file %s already exists, skipping.\n", progName, outName ); return; } switch ( srcMode ) { case SM_I2O: inStr = stdin; outStr = stdout; if ( isatty ( fileno ( stdout ) ) ) { fprintf ( stderr, "%s: I won't write compressed data to a terminal.\n", progName ); fprintf ( stderr, "%s: For help, type: `%s --help'.\n", progName, progName ); return; }; break; case SM_F2O: inStr = fopen ( inName, "rb" ); outStr = stdout; if ( isatty ( fileno ( stdout ) ) ) { fprintf ( stderr, "%s: I won't write compressed data to a terminal.\n", progName ); fprintf ( stderr, "%s: For help, type: `%s --help'.\n", progName, progName ); return; }; if ( inStr == NULL ) { fprintf ( stderr, "%s: Can't open input file %s, skipping.\n", progName, inName ); return; }; break; case SM_F2F: inStr = fopen ( inName, "rb" ); outStr = fopen ( outName, "wb" ); if ( outStr == NULL) { fprintf ( stderr, "%s: Can't create output file %s, skipping.\n", progName, outName ); return; } if ( inStr == NULL ) { fprintf ( stderr, "%s: Can't open input file %s, skipping.\n", progName, inName ); return; }; break; default: panic ( "compress: bad srcMode" ); break; } if (verbosity >= 1) { fprintf ( stderr, " %s: ", inName ); pad ( inName ); fflush ( stderr ); } /*--- Now the input and output handles are sane. Do the Biz. ---*/ outputHandleJustInCase = outStr; compressStream ( inStr, outStr ); outputHandleJustInCase = NULL; /*--- If there was an I/O error, we won't get here. ---*/ if ( srcMode == SM_F2F ) { copyDateAndPermissions ( inName, outName ); if ( !keepInputFiles ) { IntNative retVal = remove ( inName ); ERROR_IF_NOT_ZERO ( retVal ); } } } /*---------------------------------------------*/ void uncompress ( Char *name ) { FILE *inStr; FILE *outStr; Bool magicNumberOK; if (name == NULL && srcMode != SM_I2O) panic ( "uncompress: bad modes\n" ); switch (srcMode) { case SM_I2O: strcpy ( inName, "(stdin)" ); strcpy ( outName, "(stdout)" ); break; case SM_F2F: strcpy ( inName, name ); strcpy ( outName, name ); if (endsInBz2 ( outName )) outName [ strlen ( outName ) - 4 ] = '\0'; break; case SM_F2O: strcpy ( inName, name ); strcpy ( outName, "(stdout)" ); break; } if ( srcMode != SM_I2O && containsDubiousChars ( inName ) ) { fprintf ( stderr, "%s: There are no files matching `%s'.\n", progName, inName ); return; } if ( srcMode != SM_I2O && !fileExists ( inName ) ) { fprintf ( stderr, "%s: Input file %s doesn't exist, skipping.\n", progName, inName ); return; } if ( srcMode != SM_I2O && !endsInBz2 ( inName )) { fprintf ( stderr, "%s: Input file name %s doesn't end in `.bz2', skipping.\n", progName, inName ); return; } if ( srcMode != SM_I2O && notABogStandardFile ( inName )) { fprintf ( stderr, "%s: Input file %s is not a normal file, skipping.\n", progName, inName ); return; } if ( srcMode == SM_F2F && fileExists ( outName ) ) { fprintf ( stderr, "%s: Output file %s already exists, skipping.\n", progName, outName ); return; } switch ( srcMode ) { case SM_I2O: inStr = stdin; outStr = stdout; if ( isatty ( fileno ( stdin ) ) ) { fprintf ( stderr, "%s: I won't read compressed data from a terminal.\n", progName ); fprintf ( stderr, "%s: For help, type: `%s --help'.\n", progName, progName ); return; }; break; case SM_F2O: inStr = fopen ( inName, "rb" ); outStr = stdout; if ( inStr == NULL ) { fprintf ( stderr, "%s: Can't open input file %s, skipping.\n", progName, inName ); return; }; break; case SM_F2F: inStr = fopen ( inName, "rb" ); outStr = fopen ( outName, "wb" ); if ( outStr == NULL) { fprintf ( stderr, "%s: Can't create output file %s, skipping.\n", progName, outName ); return; } if ( inStr == NULL ) { fprintf ( stderr, "%s: Can't open input file %s, skipping.\n", progName, inName ); return; }; break; default: panic ( "uncompress: bad srcMode" ); break; } if (verbosity >= 1) { fprintf ( stderr, " %s: ", inName ); pad ( inName ); fflush ( stderr ); } /*--- Now the input and output handles are sane. Do the Biz. ---*/ outputHandleJustInCase = outStr; magicNumberOK = uncompressStream ( inStr, outStr ); outputHandleJustInCase = NULL; /*--- If there was an I/O error, we won't get here. ---*/ if ( magicNumberOK ) { if ( srcMode == SM_F2F ) { copyDateAndPermissions ( inName, outName ); if ( !keepInputFiles ) { IntNative retVal = remove ( inName ); ERROR_IF_NOT_ZERO ( retVal ); } } } else { if ( srcMode == SM_F2F ) { IntNative retVal = remove ( outName ); ERROR_IF_NOT_ZERO ( retVal ); } } if ( magicNumberOK ) { if (verbosity >= 1) fprintf ( stderr, "done\n" ); } else { if (verbosity >= 1) fprintf ( stderr, "not a bzip2 file, skipping.\n" ); else fprintf ( stderr, "%s: %s is not a bzip2 file, skipping.\n", progName, inName ); } } /*---------------------------------------------*/ void testf ( Char *name ) { FILE *inStr; Bool allOK; if (name == NULL && srcMode != SM_I2O) panic ( "testf: bad modes\n" ); strcpy ( outName, "(none)" ); switch (srcMode) { case SM_I2O: strcpy ( inName, "(stdin)" ); break; case SM_F2F: strcpy ( inName, name ); break; case SM_F2O: strcpy ( inName, name ); break; } if ( srcMode != SM_I2O && containsDubiousChars ( inName ) ) { fprintf ( stderr, "%s: There are no files matching `%s'.\n", progName, inName ); return; } if ( srcMode != SM_I2O && !fileExists ( inName ) ) { fprintf ( stderr, "%s: Input file %s doesn't exist, skipping.\n", progName, inName ); return; } if ( srcMode != SM_I2O && !endsInBz2 ( inName )) { fprintf ( stderr, "%s: Input file name %s doesn't end in `.bz2', skipping.\n", progName, inName ); return; } if ( srcMode != SM_I2O && notABogStandardFile ( inName )) { fprintf ( stderr, "%s: Input file %s is not a normal file, skipping.\n", progName, inName ); return; } switch ( srcMode ) { case SM_I2O: if ( isatty ( fileno ( stdin ) ) ) { fprintf ( stderr, "%s: I won't read compressed data from a terminal.\n", progName ); fprintf ( stderr, "%s: For help, type: `%s --help'.\n", progName, progName ); return; }; inStr = stdin; break; case SM_F2O: case SM_F2F: inStr = fopen ( inName, "rb" ); if ( inStr == NULL ) { fprintf ( stderr, "%s: Can't open input file %s, skipping.\n", progName, inName ); return; }; break; default: panic ( "testf: bad srcMode" ); break; } if (verbosity >= 1) { fprintf ( stderr, " %s: ", inName ); pad ( inName ); fflush ( stderr ); } /*--- Now the input handle is sane. Do the Biz. ---*/ allOK = testStream ( inStr ); if (allOK && verbosity >= 1) fprintf ( stderr, "ok\n" ); if (!allOK) testFailsExist = True; } /*---------------------------------------------*/ void license ( void ) { fprintf ( stderr, "bzip2, a block-sorting file compressor. " "Version 0.1pl2, 29-Aug-97.\n" " \n" " Copyright (C) 1996, 1997 by Julian Seward.\n" " \n" " This program is free software; you can redistribute it and/or modify\n" " it under the terms of the GNU General Public License as published by\n" " the Free Software Foundation; either version 2 of the License, or\n" " (at your option) any later version.\n" " \n" " This program is distributed in the hope that it will be useful,\n" " but WITHOUT ANY WARRANTY; without even the implied warranty of\n" " MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n" " GNU General Public License for more details.\n" " \n" " You should have received a copy of the GNU General Public License\n" " along with this program; if not, write to the Free Software\n" " Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.\n" " \n" " The GNU General Public License is contained in the file LICENSE.\n" " \n" ); } /*---------------------------------------------*/ void usage ( Char *fullProgName ) { fprintf ( stderr, "bzip2, a block-sorting file compressor. " "Version 0.1pl2, 29-Aug-97.\n" "\n usage: %s [flags and input files in any order]\n" "\n" " -h --help print this message\n" " -d --decompress force decompression\n" " -f --compress force compression\n" " -t --test test compressed file integrity\n" " -c --stdout output to standard out\n" " -v --verbose be verbose (a 2nd -v gives more)\n" " -k --keep keep (don't delete) input files\n" " -L --license display software version & license\n" " -V --version display software version & license\n" " -s --small use less memory (at most 2500k)\n" " -1 .. -9 set block size to 100k .. 900k\n" " --repetitive-fast compress repetitive blocks faster\n" " --repetitive-best compress repetitive blocks better\n" "\n" " If invoked as `bzip2', the default action is to compress.\n" " as `bunzip2', the default action is to decompress.\n" "\n" " If no file names are given, bzip2 compresses or decompresses\n" " from standard input to standard output. You can combine\n" " flags, so `-v -4' means the same as -v4 or -4v, &c.\n" #if BZ_UNIX "\n" #endif , fullProgName ); } /*---------------------------------------------*/ /*-- All the garbage from here to main() is purely to implement a linked list of command-line arguments, into which main() copies argv[1 .. argc-1]. The purpose of this ridiculous exercise is to facilitate the expansion of wildcard characters * and ? in filenames for halfwitted OSs like MSDOS, Windows 95 and NT. The actual Dirty Work is done by the platform-specific macro APPEND_FILESPEC. --*/ typedef struct zzzz { Char *name; struct zzzz *link; } Cell; /*---------------------------------------------*/ void *myMalloc ( Int32 n ) { void* p; p = malloc ( (size_t)n ); if (p == NULL) { fprintf ( stderr, "%s: `malloc' failed on request for %d bytes.\n", progName, n ); exit ( 1 ); } return p; } /*---------------------------------------------*/ Cell *mkCell ( void ) { Cell *c; c = (Cell*) myMalloc ( sizeof ( Cell ) ); c->name = NULL; c->link = NULL; return c; } /*---------------------------------------------*/ Cell *snocString ( Cell *root, Char *name ) { if (root == NULL) { Cell *tmp = mkCell(); tmp->name = (Char*) myMalloc ( 5 + strlen(name) ); strcpy ( tmp->name, name ); return tmp; } else { Cell *tmp = root; while (tmp->link != NULL) tmp = tmp->link; tmp->link = snocString ( tmp->link, name ); return root; } } /*---------------------------------------------*/ #define ISFLAG(s) (strcmp(aa->name, (s))==0) IntNative main ( IntNative argc, Char *argv[] ) { Int32 i, j; Char *tmp; Cell *argList; Cell *aa; #if DEBUG fprintf ( stderr, "bzip2: *** compiled with debugging ON ***\n" ); #endif /*-- Be really really really paranoid :-) --*/ if (sizeof(Int32) != 4 || sizeof(UInt32) != 4 || sizeof(Int16) != 2 || sizeof(UInt16) != 2 || sizeof(Char) != 1 || sizeof(UChar) != 1) { fprintf ( stderr, "bzip2: I'm not configured correctly for this platform!\n" "\tI require Int32, Int16 and Char to have sizes\n" "\tof 4, 2 and 1 bytes to run properly, and they don't.\n" "\tProbably you can fix this by defining them correctly,\n" "\tand recompiling. Bye!\n" ); exit(1); } /*-- Set up signal handlers --*/ signal (SIGINT, mySignalCatcher); signal (SIGTERM, mySignalCatcher); signal (SIGSEGV, mySIGSEGVorSIGBUScatcher); #if BZ_UNIX signal (SIGHUP, mySignalCatcher); signal (SIGBUS, mySIGSEGVorSIGBUScatcher); #endif /*-- Initialise --*/ outputHandleJustInCase = NULL; ftab = NULL; ll4 = NULL; ll16 = NULL; ll8 = NULL; tt = NULL; block = NULL; zptr = NULL; smallMode = False; keepInputFiles = False; verbosity = 0; blockSize100k = 9; testFailsExist = False; bsStream = NULL; numFileNames = 0; numFilesProcessed = 0; workFactor = 30; strcpy ( inName, "(none)" ); strcpy ( outName, "(none)" ); strcpy ( progNameReally, argv[0] ); progName = &progNameReally[0]; for (tmp = &progNameReally[0]; *tmp != '\0'; tmp++) if (*tmp == PATH_SEP) progName = tmp + 1; /*-- Expand filename wildcards in arg list --*/ argList = NULL; for (i = 1; i <= argc-1; i++) APPEND_FILESPEC(argList, argv[i]); /*-- Find the length of the longest filename --*/ longestFileName = 7; numFileNames = 0; for (aa = argList; aa != NULL; aa = aa->link) if (aa->name[0] != '-') { numFileNames++; if (longestFileName < (Int32)strlen(aa->name) ) longestFileName = (Int32)strlen(aa->name); } /*-- Determine what to do (compress/uncompress/test). --*/ /*-- Note that subsequent flag handling may change this. --*/ opMode = OM_Z; if ( (strcmp ( "bunzip2", progName ) == 0) || (strcmp ( "BUNZIP2", progName ) == 0) || (strcmp ( "bunzip2.exe", progName ) == 0) || (strcmp ( "BUNZIP2.EXE", progName ) == 0) ) opMode = OM_UNZ; /*-- Determine source modes; flag handling may change this too. --*/ if (numFileNames == 0) srcMode = SM_I2O; else srcMode = SM_F2F; /*-- Look at the flags. --*/ for (aa = argList; aa != NULL; aa = aa->link) if (aa->name[0] == '-' && aa->name[1] != '-') for (j = 1; aa->name[j] != '\0'; j++) switch (aa->name[j]) { case 'c': srcMode = SM_F2O; break; case 'd': opMode = OM_UNZ; break; case 'f': opMode = OM_Z; break; case 't': opMode = OM_TEST; break; case 'k': keepInputFiles = True; break; case 's': smallMode = True; break; case '1': blockSize100k = 1; break; case '2': blockSize100k = 2; break; case '3': blockSize100k = 3; break; case '4': blockSize100k = 4; break; case '5': blockSize100k = 5; break; case '6': blockSize100k = 6; break; case '7': blockSize100k = 7; break; case '8': blockSize100k = 8; break; case '9': blockSize100k = 9; break; case 'V': case 'L': license(); break; case 'v': verbosity++; break; case 'h': usage ( progName ); exit ( 1 ); break; default: fprintf ( stderr, "%s: Bad flag `%s'\n", progName, aa->name ); usage ( progName ); exit ( 1 ); break; } /*-- And again ... --*/ for (aa = argList; aa != NULL; aa = aa->link) { if (ISFLAG("--stdout")) srcMode = SM_F2O; else if (ISFLAG("--decompress")) opMode = OM_UNZ; else if (ISFLAG("--compress")) opMode = OM_Z; else if (ISFLAG("--test")) opMode = OM_TEST; else if (ISFLAG("--keep")) keepInputFiles = True; else if (ISFLAG("--small")) smallMode = True; else if (ISFLAG("--version")) license(); else if (ISFLAG("--license")) license(); else if (ISFLAG("--repetitive-fast")) workFactor = 5; else if (ISFLAG("--repetitive-best")) workFactor = 150; else if (ISFLAG("--verbose")) verbosity++; else if (ISFLAG("--help")) { usage ( progName ); exit ( 1 ); } else if (strncmp ( aa->name, "--", 2) == 0) { fprintf ( stderr, "%s: Bad flag `%s'\n", progName, aa->name ); usage ( progName ); exit ( 1 ); } } if (opMode == OM_Z && smallMode) blockSize100k = 2; if (opMode == OM_Z && srcMode == SM_F2O && numFileNames > 1) { fprintf ( stderr, "%s: I won't compress multiple files to stdout.\n", progName ); exit ( 1 ); } if (srcMode == SM_F2O && numFileNames == 0) { fprintf ( stderr, "%s: -c expects at least one filename.\n", progName ); exit ( 1 ); } if (opMode == OM_TEST && srcMode == SM_F2O) { fprintf ( stderr, "%s: -c and -t cannot be used together.\n", progName ); exit ( 1 ); } if (opMode != OM_Z) blockSize100k = 0; if (opMode == OM_Z) { allocateCompressStructures(); if (srcMode == SM_I2O) compress ( NULL ); else for (aa = argList; aa != NULL; aa = aa->link) if (aa->name[0] != '-') { numFilesProcessed++; compress ( aa->name ); } } else if (opMode == OM_UNZ) { if (srcMode == SM_I2O) uncompress ( NULL ); else for (aa = argList; aa != NULL; aa = aa->link) if (aa->name[0] != '-') { numFilesProcessed++; uncompress ( aa->name ); } } else { testFailsExist = False; if (srcMode == SM_I2O) testf ( NULL ); else for (aa = argList; aa != NULL; aa = aa->link) if (aa->name[0] != '-') { numFilesProcessed++; testf ( aa->name ); } if (testFailsExist) { fprintf ( stderr, "\n" "You can use the `bzip2recover' program to *attempt* to recover\n" "data from undamaged sections of corrupted files.\n\n" ); exit(2); } } return 0; } /*-----------------------------------------------------------*/ /*--- end bzip2.c ---*/ /*-----------------------------------------------------------*/ lbzip2-2.3/tests/ch255.bz2000066400000000000000000000000501221755423000151630ustar00rootroot00000000000000BZh91AY&SY7& !rE8P7lbzip2-2.3/tests/ch255.c000066400000000000000000000011521221755423000147140ustar00rootroot00000000000000/* Copyright (C) 2011 Mikolaj Izdebski */ #include #include #include #include int main(int argc, char **argv) { size_t req_sz; char buf[4096]; errno = 0; req_sz = 900000; if (argc > 1) req_sz = strtol(argv[1], 0, 10); if (errno != 0) { perror("strtol"); return EXIT_FAILURE; } memset(buf, 255, 4096); while (req_sz > 0) { size_t step = req_sz; if (step > 4096) step = 4096; req_sz -= step; if (fwrite(buf, step, 1, stdout) != 1) { perror("fwrite"); return EXIT_FAILURE; } } return EXIT_SUCCESS; } lbzip2-2.3/tests/concat.bz2000066400000000000000000000001171221755423000156100ustar00rootroot00000000000000BZh91AY&SY8Vh !w$S fBZh91AY&SY !aw$S `lbzip2-2.3/tests/crc1.bz2000066400000000000000000000000551221755423000151720ustar00rootroot00000000000000BZh91AY&SY>LF !h3M"(HBAD lbzip2-2.3/tests/crc2.bz2000066400000000000000000000000551221755423000151730ustar00rootroot00000000000000BZh91AY&SYT.F !h3M"(H|&lbzip2-2.3/tests/crc2.diff000066400000000000000000000046321221755423000154130ustar00rootroot00000000000000--- bzip2-0.1pl2.c 1997-08-30 01:32:15.000000000 +0200 +++ crc2.c 2011-08-20 12:01:46.000000000 +0200 @@ -1,18 +1,13 @@ - -/*-----------------------------------------------------------*/ -/*--- A block-sorting, lossless compressor bzip2.c ---*/ -/*-----------------------------------------------------------*/ - -/*-- - This program is bzip2, a lossless, block-sorting data compressor, - version 0.1pl2, dated 29-Aug-1997. - +/* This is an ancient version of bzip2 (0.1pl2) modified to produce + blocks with incorrect block CRCs, but valid stream CRCs. + Note that this program is full of bugs! Use with care! + */ +/*- + Copyright (C) 2011 Mikolaj Izdebski Copyright (C) 1996, 1997 by Julian Seward. - Guildford, Surrey, UK - email: jseward@acm.org - This program is free software; you can redistribute it and/or modify + This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or + the Free Software Foundation, either version 3 of the License, or (at your option) any later version. @@ -23,8 +18,18 @@ You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + along with this program. If not, see . +*/ - The GNU General Public License is contained in the file LICENSE. +/*-----------------------------------------------------------*/ +/*--- A block-sorting, lossless compressor bzip2.c ---*/ +/*-----------------------------------------------------------*/ + +/*-- + This program is bzip2, a lossless, block-sorting data compressor, + version 0.1pl2, dated 29-Aug-1997. + + Copyright (C) 1996, 1997 by Julian Seward. + Guildford, Surrey, UK + email: jseward@acm.org This program is based on (at least) the work of: @@ -160,5 +165,5 @@ run rather slowly. gcc version 2.x is recommended. --*/ - #ifdef __GNUC__ + #if 0 #define INLINE inline #define NORETURN __attribute__ ((noreturn)) @@ -2740,5 +2745,5 @@ /*-- Now the block's CRC, so it is in a known place. --*/ - bsPutUInt32 ( blockCRC ); + bsPutUInt32 ( blockCRC ^ 0xDEADBEEFu ); /* write invalid CRC */ /*-- Now a single bit indicating randomisation. --*/ lbzip2-2.3/tests/cve.bz2000066400000000000000000000004301221755423000151140ustar00rootroot00000000000000BZh91AY&SY ,`lbzip2-2.3/tests/cve.c000066400000000000000000000030471221755423000146500ustar00rootroot00000000000000/* Copyright (C) 2010 Mikolaj Izdebski */ #include unsigned b,k; p(x,n) { b = (b << n) + x, k += n; while (k >= 8) putchar((b >> (k -= 8)) & 0xff); } pb(b) { p(b,8); } run(unsigned long long n) { n--; while (1) { pb((int)(n&1)); if (n<2) break; n=(n-2)/2; } } main() { int i,j; pb('B'), pb('Z'), pb('h'), pb('9'); /* magic */ p(0x314159,24), p(0x265359, 24); /* block header */ p(0,16), p(0,16); /* block crc */ p(0,1); /* block randomised? */ p(0,24); /* bwt index */ for (i=16+256-2; i--;) p(1,1); /* bitmap */ p(0,2); p(2,3), p(355,15); /* 2 groups, 355 selectors */ for (i=355; i--;) p(0,1); /* all selectors equal to 0 */ for (j=2; j--;) { /* code lens for both groups */ p(8,5); /* all codes are 8-bit */ for (i=256; i--;) p(0,1); /* delta */ } #define AMT 500000 /* now: lit={0,0} ftab={0,0} */ run((AMT/2+0x80000000) & 0xffffffff); /* now: lit={0,0} ftab={-BIG,0} */ pb(2); /* 1 */ /* now: lit={0,1} ftab={-BIG,1} */ pb(2); /* 0 */ /* now: lit={1,1} ftab={-BIG+1,1} */ run((AMT/2+0x80000000) & 0xffffffff); /* now: lit={1,1} ftab={AMT+1,1} */ pb(2); /* 1 */ /* now: lit={1,2} ftab={AMT+1,2} */ run(AMT); /* now: lit={1,AMT+2} ftab={AMT+1,AMT+2} */ pb(2); /* 0 */ /* now: lit={2,AMT+2} ftab={AMT+2,AMT+2} */ pb(2); /* 1 */ /* now: lit={2,AMT+3} ftab={AMT+2,AMT+3} */ run(-AMT & 0xffffffff); /* now: lit={2,AMT+3} ftab={AMT+2,3} */ pb(0xff); return 0; } lbzip2-2.3/tests/cve2.bz2000066400000000000000000000004301221755423000151760ustar00rootroot00000000000000BZh91AY&SY ,`lbzip2-2.3/tests/empty.bz2000066400000000000000000000000161221755423000154750ustar00rootroot00000000000000BZh9rE8Plbzip2-2.3/tests/fib.bz2000066400000000000000000000000731221755423000151020ustar00rootroot00000000000000BZh91AY&SYDpp0 MFjkl[F`%rE8PDplbzip2-2.3/tests/fib.c000066400000000000000000000012401221755423000146240ustar00rootroot00000000000000/* Copyright (C) 2011 Mikolaj Izdebski */ #include #include #include int main(int argc, char **argv) { char *p, *q, *r, *s; size_t req_sz; errno = 0; req_sz = 900000; if (argc > 1) req_sz = strtol(argv[1], 0, 10); if (errno != 0) { perror("strtol"); return EXIT_FAILURE; } p = q = r = s = malloc(req_sz+1); if (p == 0) { perror("malloc"); exit(EXIT_FAILURE); } r = q + req_sz; *q++ = 'a'; while (q < r) { if (*p++ == 'a') *q++ = 'b'; *q++ = 'a'; } if (fwrite(s, req_sz, 1, stdout) != 1) { perror("fwrite"); return EXIT_FAILURE; } free(s); return EXIT_SUCCESS; } lbzip2-2.3/tests/gap.bz2000066400000000000000000000001411221755423000151050ustar00rootroot00000000000000BZh91AY&SY>LF !h3M"(H|&TRASH BZh91AY&SY5` !A,]B@ׯlbzip2-2.3/tests/incomp000077500000000000000000000044141221755423000151410ustar00rootroot00000000000000#!/bin/sh # A script that generates incomp-[12].bz2 files. # It's commonly believed that bzip2 always creates bit-identical compressed # files for given input file, provided that the block size used is the same. # # In fact that's not true. Output file can be different because the Reversible # Transformation (aka BWT) is sometimes ambiguous. In some cases for given # input string it's possible to construct two or more different transforms. # # The simplest case is a string consisting of two identical characters, for # instance AA. There exist exactly 2 transformates for this string, namely # AA with bwt_idx=0 and AA with bwt_idx=1. # # bzip2 uses two different algorithms for block-sorting, both of them can # produce different transformates for the same input, resulting in slightly # different resulting .bz2 files. The direct reason for that is usage of # non-stable sorting algorithms, namely shellsort and quicksort. # The following code first creates input string that has exactly 10 different # transforms. Then it's compressed with different work factors. The two # resulting .bz2 files are different (at least for bzip2 1.0.6 and 1.0.5). (for x in $(seq 10); do echo -n \ A123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ B123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ C123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ D123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ E123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ F123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ G123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ H123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ I123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789\ J123456789b123456789c123456789d123456789e123456789\ f123456789g123456789h123456789i123456789j123456789; done) | bzip2 >incomp-1.bz2 bzcat incomp-1.bz2 | bzip2 --exponential >incomp-2.bz2 lbzip2-2.3/tests/incomp-1.bz2000066400000000000000000000001511221755423000157620ustar00rootroot00000000000000BZh91AY&SY^ ?? (Jd2,sN;'tSE,H"EUzؕ_%V"Ud%W*)„lbzip2-2.3/tests/incomp-2.bz2000066400000000000000000000001511221755423000157630ustar00rootroot00000000000000BZh91AY&SY^?? (Jd2,sN;'tSE,H"EUzؕ_%V"Ud%W*)„lbzip2-2.3/tests/load0.bz2000066400000000000000000000000121221755423000153320ustar00rootroot00000000000000BZh11AY&SYlbzip2-2.3/tests/minbzcat.c000066400000000000000000000315161221755423000157040ustar00rootroot00000000000000/*- minbzcat - minimalistic bzcat Copyright (C) 2011, 2012 Mikolaj Izdebski This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ /* `minbzcat' reads compressed data in bzip2 format, decompresses it and writes the result to stdout. Any command-line options are ignored. This program is written for readability, not performance -- simpler but slower algorithms were chosen deliberately. This code targets ANSI C systems, but it was tested only very roughly. */ /* The bzip2 compression stack basically consists of 3 passes: 1. RLE - the initial run-length encoding 2. RT - the Reversible Transformation, it consists of 3 sub-transformations: a) BWT - Burrows-Wheeler transformation b) MTF - Move-To-Front transformation c) ZRLE - Zero Run-Length Encoding 3. entropy coding - prefix coding (aka Huffman coding) The initial RLE used to protect sorting algorithms from the worst case behavior, but with introduction of better sorting algorithms it became obsolete. It is kept only for compatibility with older bzip2 versions. The Reversible Transformation was invented and described by Burrows and Wheeler in their src124 paper. It takes advantage of the structure of typical data formats (like text files) to reduce data entropy so it can be compressed more easily by the last pass -- the entropy coding. The entropy coding is a traditional method of reducing data redundancy. Symbols are divided into groups of max. 50 symbols each. Each group is coded by possibly different coding tree. `Retrieving a block' means parsing the internal bit-stream and decoding prefix codes. `Decoding a block' means undoing the RT. Finally, `emitting a block' means undoing the initial RLE. Bits --(retrieve)--> MTF values --(decode)--> Characters --(emit)--> Bytes */ #ifdef HAVE_CONFIG_H # include #endif /* This code can be easily embedded, just #define EMBEDDED, provide your own read(), write() and bad() functions. Call expand() to perform decompression. Note that this code is NOT meant to be reentrant -- it uses globals to store decompression state. */ #ifdef EMBEDDED /* Input a single byte. Return -1 on EOF. It shall not return unless a byte is successfully read or EOF is reached, i.e. this function does it's own I/O error handling. */ int read(void); /* Output a single byte. Like read(), it shall not return unless the byte is written -- it does its own I/O error handling. */ void write(int c); /* Called when compressed data format inconsistency is detected. It shall not return (if it does, behavior is undefined). */ void bad(void); #else /* !EMBEDDED */ #include /* getchar() */ #include /* exit() */ static void expand(void); static void err(const char *msg) { fprintf(stderr, "minbzcat: "); if (msg) perror(msg); exit(EXIT_FAILURE); } /* Read a single byte from stdin. */ static int read(void) { int c; if ((c = getchar()) != EOF) return c; if (!feof(stdin)) err("getchar()"); return -1; } /* Write a single byte to stdout. */ static void write(int c) { if (putchar(c) == EOF) err("putchar()"); } /* Print an error message and terminate. */ static void bad(void) { fprintf(stderr, "minbzcat: data error in bz2 file\n"); exit(EXIT_FAILURE); } int main(void) { expand(); if (fclose(stdin) == EOF) err("fclose(stdin)"); if (fclose(stdout) == EOF) err("fclose(stdout)"); return EXIT_SUCCESS; } #endif /* EMBEDDED */ static int bb; /* the bit-buffer (static, right-aligned) */ static int bk; /* number of bits remaining in the `bb' bit-buffer */ static unsigned long crctab[256]; /* table for fast CRC32 computation */ static unsigned long tt[900000]; /* IBWT linked cyclic list */ static unsigned long crc; /* CRC32 computed so far */ static unsigned long mbs; /* maximal block size (100k-900k in 100k steps) */ static int rnd; /* is current block randomized? (0 or 1) */ static unsigned long bs; /* current block size (1-900000) */ static unsigned long idx; /* BWT primary index (0-899999) */ static unsigned short as; /* alphabet size (number of distinct prefix codes, 3-258) */ static unsigned short nt; /* number of prefix trees used for current block (2-6) */ static unsigned short ns; /* number of selectors (1-32767) */ static unsigned long nm; /* number of MTF values */ static unsigned char blk[900000]; /* reconstructed block */ static unsigned char len[6][259]; /* code lengths for different trees (element 258 is a sentinel) */ static unsigned char sel[32767]; /* selector MTF values */ static unsigned char mtf[256]; /* IMTF register */ static unsigned short count[21]; /* number of codes of given length (element 0 is a sentinel) */ static unsigned short sorted[258]; /* symbols sorted by ascend. code length */ static unsigned short mv[900050]; /* MTF values (elements 900000-900049 are sentinels) */ /* A table used for derandomizing randomized blocks. It's a sequence of pseudo-random numbers, hardcoded in bzip2 file format. */ static const unsigned short tab[512] = { 619,720,127,481,931,816,813,233,566,247,985,724,205,454,863,491, 741,242,949,214,733,859,335,708,621,574, 73,654,730,472,419,436, 278,496,867,210,399,680,480, 51,878,465,811,169,869,675,611,697, 867,561,862,687,507,283,482,129,807,591,733,623,150,238, 59,379, 684,877,625,169,643,105,170,607,520,932,727,476,693,425,174,647, 73,122,335,530,442,853,695,249,445,515,909,545,703,919,874,474, 882,500,594,612,641,801,220,162,819,984,589,513,495,799,161,604, 958,533,221,400,386,867,600,782,382,596,414,171,516,375,682,485, 911,276, 98,553,163,354,666,933,424,341,533,870,227,730,475,186, 263,647,537,686,600,224,469, 68,770,919,190,373,294,822,808,206, 184,943,795,384,383,461,404,758,839,887,715, 67,618,276,204,918, 873,777,604,560,951,160,578,722, 79,804, 96,409,713,940,652,934, 970,447,318,353,859,672,112,785,645,863,803,350,139, 93,354, 99, 820,908,609,772,154,274,580,184, 79,626,630,742,653,282,762,623, 680, 81,927,626,789,125,411,521,938,300,821, 78,343,175,128,250, 170,774,972,275,999,639,495, 78,352,126,857,956,358,619,580,124, 737,594,701,612,669,112,134,694,363,992,809,743,168,974,944,375, 748, 52,600,747,642,182,862, 81,344,805,988,739,511,655,814,334, 249,515,897,955,664,981,649,113,974,459,893,228,433,837,553,268, 926,240,102,654,459, 51,686,754,806,760,493,403,415,394,687,700, 946,670,656,610,738,392,760,799,887,653,978,321,576,617,626,502, 894,679,243,440,680,879,194,572,640,724,926, 56,204,700,707,151, 457,449,797,195,791,558,945,679,297, 59, 87,824,713,663,412,693, 342,606,134,108,571,364,631,212,174,643,304,329,343, 97,430,751, 497,314,983,374,822,928,140,206, 73,263,980,736,876,478,430,305, 170,514,364,692,829, 82,855,953,676,246,369,970,294,750,807,827, 150,790,288,923,804,378,215,828,592,281,565,555,710, 82,896,831, 547,261,524,462,293,465,502, 56,661,821,976,991,658,869,905,758, 745,193,768,550,608,933,378,286,215,979,792,961, 61,688,793,644, 986,403,106,366,905,644,372,567,466,434,645,210,389,550,919,135, 780,773,635,389,707,100,626,958,165,504,920,176,193,713,857,265, 203, 50,668,108,645,990,626,197,510,357,358,850,858,364,936,638, }; /* Read and return `n' bits from the input stream. `n' must be <= 32. */ static unsigned long get(int n) { unsigned long x = 0; while (n--) { if (!bk-- && (bb = read()) < 0) bad(); x = 2*x + ((bb >> (bk &= 7)) & 1); } return x; } /* Initialize crctab[]. */ static void init_crc(void) { int i, k; for (i = 0; i < 256; i++) { crctab[i] = (unsigned long)i << 24; for (k = 0; k < 8; k++) crctab[i] = ((crctab[i] << 1) & 0xFFFFFFFF) ^ (0x04C11DB7L & -(crctab[i] >> 31)); } } /* Create decode tables using code lengths from `lens[t]'. `t' is the tree selector, must be in range [0,nt). */ static void make_tree(int t) { unsigned short u[21], i, s, a; for (i = 0; i <= 20; i++) count[i] = 0; for (i = 0; i < as; i++) count[len[t][i]]++; for (a = 1, s = 0, i = 0; i <= 20; i++) { u[i] = s; a *= 2; if (count[i] > a) bad(); a -= count[i]; s += count[i]; } for (i = 0; i < as; i++) sorted[u[len[t][i]]++] = i; } /* Decode a single prefix code. The algorithm used is naive and slow. */ static unsigned get_sym(void) { int s = 0, x = 0, k = 0; do { if (k == 20) bad(); x = 2*x + get(1); s += count[++k]; } while ((x -= count[k]) >= 0); return sorted[s + x]; } /* Retrieve bitmap. */ static void bmp(void) { unsigned i, j; unsigned short b = get(16); as = 0; for (i = 0; i < 16; i++) { if (b & 0x8000) { unsigned short s = get(16); for (j = 0; j < 16; j++) { if (s & 0x8000) mtf[as++] = 16*i + j; s *= 2; } } b *= 2; } as += 2; } /* Retrieve selector MTF values. */ static void smtf(void) { unsigned g; for (g = 0; g < ns; g++) { sel[g] = 0; while (sel[g] < nt && get(1)) sel[g]++; if (sel[g] == nt) bad(); } if (ns > 18001) ns = 18001; } /* Retrieve code lengths. */ static void trees(void) { unsigned t,s; for (t = 0; t < nt; t++) { len[t][0] = get(5); for (s = 0; s < as; s++) { goto in; do { len[t][s] += 1 - 2*get(1); in: if (len[t][s] < 1 || len[t][s] > 20) bad(); } while (get(1)); len[t][s+1] = len[t][s]; } } } /* Retrieve block MTF values. */ static void data(void) { unsigned g,i,t; unsigned m[6]; for (i = 0; i < 6; i++) m[i] = i; nm = 0; for (g = 0; g < ns; g++) { i = sel[g]; t = m[i]; while (i-- > 0) m[i+1] = m[i]; m[0] = t; make_tree(t); for (i = 0; i < 50; i++) if ((mv[nm++] = get_sym()) == as-1) return; } bad(); } /* Retrieve block. */ static void retr(void) { rnd = get(1); idx = get(24); bmp(); nt = get(3); if (nt < 2 || nt > 6) bad(); ns = get(15); smtf(); trees(); data(); } /* Apply IMTF transformation. */ static void imtf(void) { unsigned i,s,t,r,h; bs = r = h = 0; for (i = 0; i < nm; i++) { s = mv[i]; if (s <= 1) { r += 1 << (h + s); h++; } else { if (bs+r > mbs) bad(); while (r--) tt[bs++] = mtf[0]; if (s+1 == as) break; t = mtf[--s]; while (s-- > 0) mtf[s+1] = mtf[s]; mtf[0] = t; h = 0; r = 1; } } } /* Apply IBWT transformation. */ static void ibwt(void) { unsigned long i, c, f[256]; if (idx >= bs) bad(); for (i = 0; i < 256; i++) f[i] = 0; for (i = 0; i < bs; i++) f[tt[i]]++; for (i = c = 0; i < 256; i++) f[i] = (c += f[i]) - f[i]; for (i = 0; i < bs; i++) tt[f[tt[i] & 0xFF]++] |= i << 8; idx = tt[idx]; for (i = 0; i < bs; i++) { idx = tt[idx >> 8]; blk[i] = idx; } } /* Derandomize block if it's randomized. */ static void derand(void) { unsigned long i = 0, j = 617; while (rnd && j < bs) { blk[j] ^= 1; i = (i + 1) & 0x1FF; j += tab[i]; } } /* Emit block. RLE is undone here. */ static void emit(void) { unsigned long i, r, c, d; r = 0; c = -1; for (i = 0; i < bs; i++) { d = c; c = blk[i]; crc = ((crc << 8) & 0xFFFFFFFF) ^ crctab[(crc >> 24) ^ c]; write(c); if (c != d) r = 1; else { r++; if (r >= 4) { unsigned j; if (++i == bs) bad(); for (j = 0; j < blk[i]; j++) { crc = ((crc << 8) & 0xFFFFFFFF) ^ crctab[(crc >> 24) ^ c]; write(c); } r = 0; } } } } /* Parse stream and bock headers, decompress any blocks found. */ void expand(void) { unsigned long t = 0, c; init_crc(); if (get(24) != 0x425A68) bad(); t = get(8) - 0x31; if (t >= 9) bad(); do { c = 0; mbs = 100000 * (t+1); while ((t = get(16)) == 0x3141) { if (get(32) != 0x59265359) bad(); t = get(32); retr(); imtf(); ibwt(); derand(); crc = 0xFFFFFFFF; emit(); if ((crc ^ 0xFFFFFFFF) != t) bad(); c = ((c << 1) & 0xFFFFFFFF) ^ (c >> 31) ^ t; } if (t != 0x1772) bad(); if (get(32) != 0x45385090) bad(); if (get(32) != c) bad(); bk = 0; } while (read() == 0x42 && read() == 0x5A && read() == 0x68 && (t = get(8) - 0x31) < 9); } lbzip2-2.3/tests/overrun.bz2000066400000000000000000000000521221755423000160370ustar00rootroot00000000000000BZh11AY&SY,c` 0)E\]B@Vlbzip2-2.3/tests/rand.bz2000066400000000000000000000000551221755423000152660ustar00rootroot00000000000000BZh91AY&SY>LF !h3M"(H|&lbzip2-2.3/tests/repet.bz2000066400000000000000000000000551221755423000154610ustar00rootroot00000000000000BZh91AY&SY[u$n0 j(DHB'rE8P[u$lbzip2-2.3/tests/timeout.c000066400000000000000000000022301221755423000155520ustar00rootroot00000000000000/*- timeout.c - execute a command with one minute timeout Copyright (C) 2011 Mikolaj Izdebski This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with lbzip2. If not, see . */ /* This program basically does the same thing as the following one-liner: perl -e'alarm 60; exec @ARGV or die $!' It's written in C because we don't want to depend on perl. */ #ifdef HAVE_CONFIG_H # include #endif #include /* execv() */ #include /* perror() */ int main(int argc, char **argv) { alarm(60); argv++; execv(*argv, argv); perror("execv"); return 1; } lbzip2-2.3/tests/trash.bz2000066400000000000000000000001211221755423000154550ustar00rootroot00000000000000BZh91AY&SY>LF !h3M"(H|&TRASH BZh91AY&SYJ* "lbzip2-2.3/tests/void.bz2000066400000000000000000000000001221755423000152710ustar00rootroot00000000000000