pax_global_header00006660000000000000000000000064143252145500014513gustar00rootroot0000000000000052 comment=58fffab4056abb7062b970ff74facba79e46e3bf btrbk-0.32.5/000077500000000000000000000000001432521455000127065ustar00rootroot00000000000000btrbk-0.32.5/.github/000077500000000000000000000000001432521455000142465ustar00rootroot00000000000000btrbk-0.32.5/.github/FUNDING.yml000066400000000000000000000003041432521455000160600ustar00rootroot00000000000000# These are supported funding model platforms custom: ['https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S', 'https://btc.com/19DYtoEepxBmn9ZPspJGZrhCtySKCxPcP1'] btrbk-0.32.5/COPYING000066400000000000000000001045131432521455000137450ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . btrbk-0.32.5/ChangeLog000066400000000000000000000550341432521455000144670ustar00rootroot00000000000000btrbk-0.32.5 * Correct handling of zero-size raw info file (close #491). btrbk-0.32.4 * Fix regression: wrong deprecation warnings in some cases. btrbk-0.32.3 * Fix deletion of many subvolumes at once (close #476). * Allow disabling ssh_identity and ssh_user options. * Minor bugfixes and documentation improvements. btrbk-0.32.2 * Fix regression: archive on missing target directories. * Fix action "config print". * Print version and help message to stdout (close #452). * Check results of filesystem usage (close #469). * Add "compat ignore_receive_errors" option (close #383). * Support multiple gpg recipients (close #471). * Fix changelog: remove "compat missing_otime" item in v0.32.1. * Fix changelog: correct "warn_unknown_targets" item in v0.31.3. * Minor bugfixes and framework improvements. btrbk-0.32.1 * Fix regression: correctly read mountinfo (close #445). * Fix regression: filter paths for "lsbtr" (action "ls"). * Add backend btrfs-progs-doas (close #444). * Allow directory traversal for local command line arguments. * Display all source subvolumes in "usage" action. * Handle errors from btrfs filesystem usage. btrbk-0.32.0 * MIGRATION - If timestamp_format is not configured, explicitely set "timestamp_format short" to revert old behavior. - Update ssh_filter_btrbk.sh on remote hosts. * Change default for timestamp_format to "long". * Optional "volume" config section. * Use "volume" section only for grouping and relative paths. * Allow absolute path for "subvolume" and "snapshot_dir" (close #407, #121). * Support subvolume names with UTF-8 characters (close #392, #213, #120). Add "safe_commands" option for paranoid people. * Add "incremental_prefs" configuration option (close #387). * Change incremental prefs policy, giving snapshots created by btrbk higher preference than the global ones resolved by parent-uuid. * Change "incremental_clones" option to boolean. * Drop support of deprecated raw file format (btrbk < 0.26.0). * Drop support of deprecated "-r, --resume-only" cmdline option. * Fix ambiguous naming in --format=raw output of "list" actions. * Accept quoted values in config. * Use single quotes for raw table output. btrbk-0.31.3 * Add "warn_unknown_targets" configuration option (close #393). * Add -1,--single-column command-line option for listing actions. * Allow relative path command line argument for all actions. * Add support for zstd adaptive compression (close #397). * Add contrib/tools/btrbk_restore_raw.py (close #401). * Minor bugfixes and framework improvements. btrbk-0.31.2 * MIGRATION - Update ssh_filter_btrbk.sh on remote hosts. * ssh_filter_btrbk.sh: Fix security vulnerability. Specialy crafted commands may be executed without being propely checked. Applies to remote hosts filtering ssh commands using ssh_filter_btrbk.sh in authorized_keys. * Warn if no subvolume defined in config (close #378). btrbk-0.31.1 * Bugfix: duplicate snapshot location check (close #360). * Bugfix: action "list all" (close #367). * btrbk-mail: optionally prefix command output lines. btrbk-0.31.0 * Add action "extents [diff]": print accurate disk space usage based on block regions (extent data, FIEMAP ioctl). * Improve action "list" and "stats" table output (close #334). * Add subcommand "list all", make default for "list" action. * Change semantics on output of action "list": show only btrbk snapshots/backups (close #333). * Print human readable units in action "diff" (close #345). * Allow custom table format definitions from command line. * Add Bash completion (close #343). * Add support for zstd compression (close #358). * Bugfix: fs_path fallback for action "origin" and logging. * Bugfix: makefile bin-links race (close #341). btrbk-0.30.0 * MIGRATION - In your scripts (e.g. cron jobs, systemd units), replace "/usr/sbin/btrbk" with "/usr/bin/btrbk". * Install to "/usr/bin" instead of "/usr/sbin". * Support IPv6 addresses (close #332). * Add "compat busybox" configuration option (close #327). * Create "lsbtr" symlink (alias for "btrbk ls"). * Improve action "ls" (allow url, bugfixes). * Add "backend_local_user" configuration option. * btrbk-mail: add more elaborated email and rsync options. * Minor bugfixes and framework improvements. btrbk-0.29.1 * Allow multiple path arguments for action "ls". * Change default output to format=short for action "ls". * ssh_filter_btrbk.sh: exclude "btrfs subvolume show|list" from restrict-path (close #309). * Bugfix: fix filter statement match on wildcards (close #311). * Fix regression: print ssh errors only if needed. btrbk-0.29.0 * MIGRATION - If stream_buffer or rate_limit is configured, please install "mbuffer" (version >= 20180505). Note that mbuffer is only required on remote hosts if stream_buffer_remote or rate_limit_remote is set. - In order to mimic old behavior, replace rate_limit with rate_limit_remote, and read btrbk.conf(5). - If you are using table output other than --format=raw in your scripts, note that the column headings changed from pretty two-line to (uppercase) one-line format. Either add --pretty option to mimic old behavior, or adapt your scripts. * Dropped run-time dependency on "pv" (in favor of "mbuffer"). * Combined stream_buffer and rate_limit: for rate_limit, use "mbuffer" (on local host) instead of "pv" (on source host). * Add stream_buffer_remote and rate_limit_remote options. * Use "mbuffer" instead of "pv" for --progress display. * Add stream_buffer functionality for raw targets. * Add action "ls": list all btrfs subvolumes below given path. * Change table output: print single-line, uppercase headings. * ssh_filter_btrbk.sh: whitelist mkdir if -t,--target option is set, used by "btrbk archive" action (close #297). * Bugfix: handle subvolumes with special characters (close #288). * Bugfix: don't display "" in backup summary if volume is skipped by --exclude or noauto (close #291). * Bugfix: systemd: Use WantedBy=timers.target instead of WantedBy=multi-user.target in btrbk.timer (close #293). btrbk-0.28.3 * Bugfix: return exitcode=10 if action skipped by stray subvolumes (close: #285). * Bugfix: correctly handle root subvolumes having uuid. * Bugfix: fix mount point resolving. btrbk-0.28.2 * Fix regression: crash if raw target dir is empty (close #281). * Bugfix: honor incremental_resolve on targets: resolve from mountpoint instead of directory. * Bugfix: handle directories named "0" correctly. btrbk-0.28.1 * Fix regression: crash if deep parent/child relations exceed depth=100 (close #279). * Remove pre-generated man pages, requires "asciidoctor" for builds. btrbk-0.28.0 * MIGRATION - Replace "ssh_port" options with "ssh://hostname[:port]" notation on "volume" or "target" declarations. - Check filter statements in your scripts ("btrbk run "), especially when using groups having the same name as subvolumes. * Fix table format "resolved" (btrbk list snapshots|backups). * Add btrbk-verify: tool for automated backup integrity check based on rsync (close #256). * Use "ssh://hostname[:port]" notation instead of ambiguous "ssh_port" option (now deprecated). * Bugfix: correctly handle multiple volume sections with same host, but distinct port numbers (virtual machines setups, close #233). * Remove selected rows ("*host", "*port") from table output if all values are empty. * Consider all parent/child relations (close #255). * Disallow unrelated parents for btrfs-send if incremental=strict. * Add clone sources to btrfs-send if necessary. * Add "incremental_resolve" configuration option. * Add "incremental_clones" configuration option. * Add "noauto" configuration option. * Add "--exclude" command line option. * Make target_type optional, defaults to "send-receive". * Use more sophisticated filter arguments. * Allow whitespace separated "group" option. btrbk-0.27.2 * Bump dependency >= btrfs-progs-4.12 (as of btrbk-0.27.0). * Trigger autofs mount while resolving mountpoints (close #259). * Bugfix: fix scheduler when overriding "target_preserve_min" in combination with global "target" section. btrbk-0.27.1 * MIGRATION - Update ssh_filter_btrbk.sh on remote hosts if using backend=btrfs-progs-sudo. * Bugfix (regression): call "sudo -n readlink" for backend=btrfs-progs-sudo (close #257). * Bugfix (regression): fix failing "config print" command. * Don't print title and blank lines for --format=raw, syslog and transaction log. btrbk-0.27.0 * MIGRATION - Run "btrbk prune --dry-run --print-schedule" and check if any snapshots/backups would get deleted [1] [2]. If you want to forcibly preserve a snapshot/backup forever, rename it (btrbk ignores subvolumes with unknown naming scheme), e.g.: "mv mysubvol.YYYYMMDD mysubvol.YYYYMMDD.keep_forever" - Update ssh_filter_btrbk.sh on remote hosts (btrbk always calls "readlink" and "cat /proc/self/mountinfo"). * Add "preserve_hour_of_day" configuration option (close #202). * Allow backup of filesystem root using "subvolume ." (close #240). * Add "-S" command line option (shortcut for --print-schedule). * Bugfix: correct scheduling of "first weekly backup in month/year" (close #217) [1] [2]. * Bugfix: add "btrfs filesystem usage" backend mapping (close #222). * Bugfix: do not fail (exitcode=10) if targets are not accessible on "btrfs snapshot --preserve". * Bugfix: if "incremental no" is configured on target, do not keep latest common snapshot. * Enhance internal data structures: - Allow snapshot_dir to be a mountpoint. - Search complete target tree for correlated subvolumes. - Include snapshots from all mountpoints as candidates (disabled due to uptream bug: github.com/kdave/btrfs-progs/issues/96). - Read /proc/self/mountinfo instead of /proc/self/mounts. - Always read /proc/self/mountinfo. - Resolve realpath using readlink(1). * Fallback to "asciidoctor" for manpage generation (close #219). [1] https://github.com/digint/btrbk/issues/217 [2] https://github.com/digint/btrbk/commit/719fb5f btrbk-0.26.1 * Add "archive_exclude" configuration option. * Add warning on redefined configuration option. * Bugfix: fix parsing of "openssl_iv_size" configuration option. * Bugfix: fix filter statement matching for volume=/ (close #209). btrbk-0.26.0 * MIGRATION - If you are using raw targets, make sure to run the "raw_suffix2sidecar" utility in each target directory. * Support for btrfs-progs v4.13.2: adapt parsing of "btrfs sub list" output (close #192). * Add "resume" command, replacement for "-r, --resume-only" command line option (which is now deprecated). * Add "snapshot" command (close #150). * Add "prune" command. * Add "--preserve-snapshots" and "--preserve-backups" options. * Add "--wipe" command line option (close #173). * Change raw backup format (sidecar file instead of uuid in file). * Honor target_preserve for raw targets (delete raw targets). * Add symmetric encryption for raw targets (close #157). * Add "{snapshot,target,archive}_qgroup_destroy" configuration options (close #49, #189). * Do not run in "perl taint mode" by default: remove "perl -T" in hashbang; hardcode $PATH only if taint mode is enabled. * Remove "duration" column from transaction_log/transaction_syslog. * Resolve ancestors (recursive on parent_uuid chain) when searching for latest common subvolume. * Generate man pages from asciidoc (remove raw groff sources). * Bugfix: ssh_filter_btrbk: accept mbuffer command (stream_buffer). * Bugfix: print correct (end-)time in transaction_log. * Bugfix: check path when expanding wildcards (close #181). * Bugfix: never show failed deletes in summary (close #183). btrbk-0.25.1 * Support for btrfs-progs v4.12: fix parsing of "btrfs sub show" output, which now prints relative paths (close #171). * Add "stream_buffer" configuration option (close #154). * Bugfix: accept "no" for "transaction_log", "transaction_syslog" and "lockfile" configuration options. * Show "up-to-date" status for backups in "stats" command. * Show "correlated" status instead of "orphaned" in "stats" command. * Check source subvolumes for readonly and received_uuid flags, and abort if one of them is set. btrbk-0.25.0 * MIGRATION - If you call ssh_filter_btrbk.sh with "--sudo" option, make sure to set "backend btrfs-progs-sudo" in btrbk.conf for this host. - If "rate_limit" is enabled, update ssh_filter_btrbk.sh on remote source hosts, and make sure the "pv" command is available there. * Allow converting backup disks to source disks (close #114). * Add "backend btrfs-progs-sudo" configuration option (close #115). * Show aggregate "size" and "used" for "usage" action (close #119). * Add "raw_target_split" configuration option (close #125). * Allow trailing comments in btrbk.conf (close #129). * Bugfix: rate limiting must be done after compression (close #134). * raw_target_encrypt: Always set "gpg --no-random-seed-file": prevents creation of "~/.gnupg/random_seed" with slight perfomance penalty. btrbk-0.24.0 * MIGRATION - update ssh_filter_btrbk.sh on remote hosts if "stream_compress" is enabled. Also add "--compress" option to ssh_filter_btrbk.sh invocation if "stream_compress" is enabled. * Add "stream_compress" configuration option. * Perform extra metadata check on target subvolume after "btrfs receive" (adds an additional call to "btrfs subvolume show"). * Bugfix: Replace "realpath" with "readlink" in ssh_filter_btrbk.sh * Add "raw_target_block_size" configuration option (close #105). * Add "backend" configuration option (experimental). * Bugfix: fix "list latest" with no snapshots (close #111). * Support for btrfs-progs v4.8.3: fix parsing of "btrfs sub show" output, which has changed for toplevel subvolume. btrbk-0.23.3 * Replace "realpath -e" with "readlink -e" for mountpoint discovery (close #92). * Dynamically set BINDIR in btrbk.service by Makefile. * Add ChangeLog to DOCDIR in Makefile. btrbk-0.23.2 * Added "lockfile" configuration option and --lockfile command line option (close: #81). * Bugfix: raw targets: correctly handle multiple backups in same target directory (close: #87). * Use relative instead of absolute binary calls in btrbk-mail. btrbk-0.23.1 * Bugfix: set correct parent section when propagating targets (close: #85). * Add syslog output of transaction log (close #82). * Do not print headers to transaction log anymore. * Explain "orphaned" status in "stats" command, and suppress it on "list backups" command (close: #76). btrbk-0.23.0 * INCOMPATIBLE CONFIGURATION: * Please read "doc/upgrade_to_v0.23.0.md" for details on updating the configuration file (/etc/btrbk/btrbk.conf). * Dropped "btrfs_progs_compat" option. Need btrfs-progs >= v3.18. * Removed "resume_missing" configuration option. * Create backups only if needed to satisfy retention policy. * Preserve FIRST backup of hour/day/week/month instead of LAST. * Replaced "{snapshot,target}_preserve_{daily,weekly,monthly}" configuration options with "{snapshot,target}_preserve_min" and "{snapshot,target}_preserve NNh NNd NNw NNm NNy" options. * Added hourly/yearly retention policies (close: #36, #69). * Allow regular directories for send-receive targets (close: #77). * Allow wildcards in subvolume section (close: #71). * Propagate targets defined in "volume" or "root" context to all "subvolume" sections (close: #78). * Added "archive" command (close: #79). * Changed output format of "origin" command, add table formats. * Added configuration option "rate_limit" (close: #72). * Added new timestamp_format "long-iso", having a timezone postfix. * Added "--print-schedule" command line option. * Detect interrupted transfers of raw targets (close: #75). * Always read "readonly" flag (additional call to btrfs-progs). * Warn on receive targets at unexpected location (instead of abort). * On incremental send/receive, use all snapshots sharing same parent_uuid as candidates for "-p " instead of only older snapshots. As last resort, use subvolumes in snapshot_dir matching btrbk file name scheme as candidates (which allows incremental backups after the parent vanished, e.g. after backup restore). * Use perl built-in Time::Local instead of Date::Calc. * Improvements of internal data structures. btrbk-0.22.2 * Bugfix: fix checks on "btrfs sub show" output, which resulted in breakage on btrfs-progs < 4.1 (close: #68). btrbk-0.22.1 * Support for btrfs-progs v4.4 (close: #66). btrbk-0.22.0 * Bugfix: fix monthly schedule if older than 10 weeks (close: #59). * Bugfix: fix sprintf used by config option "timestamp_format long" when using perl-5.22.0 (close: #57). * Bugfix: fix "--progress" option (close: #64). * Added "clean" command (close: #61). * Added "-n, --dry-run" option. * Added configuration options "raw_target_compress_level", "raw_target_compress_threads" (close: #60). * Added "stats" command (close: #54). * Print "$hostname:$path" instead of "{$hostname}$path" in summary and logs. btrbk-0.21.0 * Added transaction log (configuration option "transaction_log"). * Added configuration option "group". * Allow filtering of all commands by group as well as targets. * Added "list backups|snapshots|latest|config|source|volume|target" commands (while removing "tree" command). * Added "config print" command. * Added "--format=table|long|raw" and "-t,--table" command line options, producing tabular and raw (machine-readable) output for "(dry)run", "tree" and "list" commands. * Print scheduler details if -v option is set on action run/dryrun. * Added configuration option "ssh_cipher_spec" (close: #47). * Added "target raw", with GnuPG and compression support (experimental). * Added configuration option "timestamp_format short|long". * Replaced "info" command with "usage", with tabular output. * Bugfix: correctly handle "incremental no" option. * Bugfix: return exit status 10 instead of 0 if one or more backup tasks aborted. * Better error handling for send/receive commands (close: #33). * Hardened ssh_filter_btrbk.sh script: fine-grained access control, restrict-path option, sudo option (close: #45). * Added example cron script. btrbk-0.20.0 * Added configuration option "ssh_port" (close: #39). * Added configuration option "ssh_compression" (close: #44). * Added command line option "--progress" (close: #42). * Bugfix: correct handling of empty "snapshot_dir" (close: #43). * Accept long options on command line. * Documentation: added FAQ. btrbk-0.19.3 * Bugfix: fix sorting of dates in schedule(). * Bugfix: correct parsing of btrfs subvolume list (close: #28). * Support for btrfs-progs v4.1-rc1. btrbk-0.19.2 * Bugfix: fix crash when using btrfs-progs < 3.17.3 (close: #24). btrbk-0.19.1 * Use "cgen" for snapshot comparison. * Bugfix: fix crash in summary display (close: #22). btrbk-0.19.0 * Added "snapshot_create onchange", which skips snapshot creation if the latest snapshot is up-to-date (i.e. has same generation as the source subvolume). * Improved handling of command line subvolume filter for "run", "dryrun" and "tree" actions (close: #21). * Bugfix: fixed crash in action "diff". btrbk-0.18.0 * MIGRATION - update ssh_filter_btrbk.sh on all remote hosts - update configuration file (/etc/btrbk/btrbk.conf): - "snapshot_create_always yes" -> "snapshot_create always" - "snapshot_create_always no" -> "snapshot_create ondemand" * Set PATH variable instead of using absolute "/sbin/btrfs" for compatibility with all linux distros out there, which all install 'btrfs' in different locations (close: #20). * Added configuration option "snapshot_create", replacing option "snapshot_create_always". This allows setups with multiple btrbk instances on several hosts (close: #18). * Added command line option -r (resume only). * Catch and display errors from "btrfs subvolume show". * Include systemd service and timer unit for daily backups. btrbk-0.17.1 * Bugfix: send/receive: delete possibly left-behind garbled subvolume on failure. Fail with unrecoverable error if stray target subvolume is in the way (close: #17). * Bugfix: assume unreachable target as clean on snapshot creation if snapshot_create_always is set (close: #19). btrbk-0.17.0 * New versioning scheme using more common three-level versions. * Code refactoring: cleanup of data structures and handling of btrfs subvolume tree, as well as security related code parts. * Correct handling of symlinks to btrfs subvolumes (close: #12). * Added configuration option "snapshot_name" (close: #5). * Log messages now go to stderr, only the summary is printed on stdout. * Bugfix: allow "0" as subvolume name (close: #10). * Bugfix: allow "/" as volume name (close: #15). * Bugfix: check source AND targets for determining snapshot postfix (close: #11). * Bugfix: fixed "diff" action (colses: #14). * Allow '+' character for subvolume names. * Filesystems on remote hosts are now printed as "{my.remote-host.com}" in summary and logs. btrbk-0.16 * Bugfix: correctly check retention policy for missing backups. btrbk-0.15 * Added configuration option "btrfs_progs_compat", to be enabled if using btrfs-progs < 3.17 (close: #6). * Added configuration option "resume_missing", for automatic resume of missing backups (close: #8). * Removed configuration option "receive_log" in favor of printing errors from "btrfs receive". * Bugfix: show correct exit code on external command failure. * Bugfix: no crash if "commit_delete" option is set to "no" btrbk-0.14 * Bugfix: correctly handle empty target subvolumes (blocker for all new users; close: #4). btrbk-0.13 * Bugfix: allow '@' character for subvolume names (blocker for ubuntu users, since ubuntu prefixes all subvolumes with '@' in its subvolume layout; close: #3). btrbk-0.12 * Cleaner and more generic parsing of btrfs subvolume list. * Bugfix: subvolumes are also allowed for "snapshot_dir" (close: #1, #2). btrbk-0.11 * Added option -p (preserve backups). * Added optional subvolume argument for run/dryrun actions, for explicit selection of subvolumes to be processed. * Bugfixes btrbk-0.10 * Initial revision. btrbk-0.32.5/Makefile000066400000000000000000000066051432521455000143550ustar00rootroot00000000000000# # Btrbk is a single perl script, and does not require any special # installation procedures or libraries. There is no need to run the # "all" build target if you don't want to build the man pages (see # doc/Makefile). # # Note: systemd units (file names) are hardcoded in "install-systemd" # build target for simplicity. # BIN = btrbk BIN_LINKS = lsbtr CONFIGS = btrbk.conf.example DOCS = ChangeLog \ README.md SCRIPTS = ssh_filter_btrbk.sh \ contrib/cron/btrbk-mail \ contrib/cron/btrbk-verify \ contrib/migration/raw_suffix2sidecar \ contrib/crypt/kdf_pbkdf2.py \ contrib/tools/btrbk_restore_raw.py PN = btrbk PREFIX ?= /usr CONFDIR = /etc CRONDIR = /etc/cron.daily BINDIR = $(PREFIX)/bin DOCDIR = $(PREFIX)/share/doc/$(PN) SCRIPTDIR = $(PREFIX)/share/$(PN)/scripts SYSTEMDDIR = $(PREFIX)/lib/systemd/system BASHCOMPDIR = $(PREFIX)/share/bash-completion/completions MAN1DIR = $(PREFIX)/share/man/man1 MAN5DIR = $(PREFIX)/share/man/man5 ifeq ($(COMPRESS), yes) DOCS := $(addsuffix .gz,$(DOCS)) endif replace_vars = sed \ -e "s|@PN@|$(PN)|g" \ -e "s|@CONFDIR@|$(CONFDIR)|g" \ -e "s|@CRONDIR@|$(CRONDIR)|g" \ -e "s|@BINDIR@|$(BINDIR)|g" \ -e "s|@DOCDIR@|$(DOCDIR)|g" \ -e "s|@SCRIPTDIR@|$(SCRIPTDIR)|g" \ -e "s|@SYSTEMDDIR@|$(SYSTEMDDIR)|g" \ -e "s|@BASHCOMPDIR@|$(BASHCOMPDIR)|g" \ -e "s|@MAN1DIR@|$(MAN1DIR)|g" \ -e "s|@MAN5DIR@|$(MAN5DIR)|g" all: man install: install-bin install-bin-links install-etc install-completion install-systemd install-share install-man install-doc install-bin: @echo 'installing binary...' install -d -m 755 "$(DESTDIR)$(BINDIR)" install -p -m 755 $(BIN) "$(DESTDIR)$(BINDIR)" install-bin-links: install-bin @echo 'installing symlinks...' for name in $(BIN_LINKS); do \ ln -s -n -f $(BIN) "$(DESTDIR)$(BINDIR)/$$name"; \ done install-etc: @echo 'installing example configs...' install -d -m 755 "$(DESTDIR)$(CONFDIR)/btrbk" install -p -m 644 $(CONFIGS) "$(DESTDIR)$(CONFDIR)/btrbk" install-completion: @echo 'installing bash completion...' install -d -m 755 "$(DESTDIR)$(BASHCOMPDIR)" install -p -m 644 contrib/bash/completion.bash "$(DESTDIR)$(BASHCOMPDIR)/$(BIN)" for name in $(BIN_LINKS); do \ ln -s -n -f $(BIN) "$(DESTDIR)$(BASHCOMPDIR)/$$name"; \ done install-systemd: @echo 'installing systemd service units...' install -d -m 755 "$(DESTDIR)$(SYSTEMDDIR)" $(replace_vars) contrib/systemd/btrbk.service.in > contrib/systemd/btrbk.service.tmp $(replace_vars) contrib/systemd/btrbk.timer.in > contrib/systemd/btrbk.timer.tmp install -p -m 644 contrib/systemd/btrbk.service.tmp "$(DESTDIR)$(SYSTEMDDIR)/btrbk.service" install -p -m 644 contrib/systemd/btrbk.timer.tmp "$(DESTDIR)$(SYSTEMDDIR)/btrbk.timer" rm contrib/systemd/btrbk.service.tmp rm contrib/systemd/btrbk.timer.tmp install-share: @echo 'installing auxiliary scripts...' install -d -m 755 "$(DESTDIR)$(SCRIPTDIR)" install -p -m 755 $(SCRIPTS) "$(DESTDIR)$(SCRIPTDIR)" install-man: man @echo 'installing man pages...' @$(MAKE) -C doc install-man install-doc: $(DOCS) @echo 'installing documentation...' install -d -m 755 "$(DESTDIR)$(DOCDIR)" install -p -m 644 $(DOCS) "$(DESTDIR)$(DOCDIR)" @$(MAKE) -C doc install-doc man: @echo 'generating manpages...' @$(MAKE) -C doc man clean: rm -f *.gz @$(MAKE) -C doc clean %.gz : % gzip -9 -n -c $< > $@ btrbk-0.32.5/README.md000066400000000000000000000551341432521455000141750ustar00rootroot00000000000000Introduction ============ Btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs specific capabilities to create atomic snapshots and transfer them incrementally to your backup locations. The source and target locations are specified in a config file, which allows to easily configure simple scenarios like "laptop with locally attached backup disks", as well as more complex ones, e.g. "server receiving backups from several hosts via ssh, with different retention policies". Key Features: * Atomic snapshots * Incremental backups * Flexible retention policy * Backups to multiple destinations * Transfer via ssh * Robust recovery from interrupted backups (for removable and mobile devices) * Archive to offline storage * Encrypted backups to non-btrfs storage * Wildcard subvolumes (useful for docker and lxc containers) * Transaction log * Comprehensive list and statistics output * Resolve and trace btrfs parent-child and received-from relationships * List file changes between backups * Calculate accurate disk space usage based on block regions Btrbk is designed to run as a cron job for triggering periodic snapshots and backups, as well as from the command line (e.g. for instantly creating additional snapshots). Installation ============ Btrbk is a single perl script, and does not require any special installation procedures or libraries. Download the latest [btrbk source tarball], or try latest master: wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk chmod +x btrbk sudo ./btrbk ls / For more information, read the [installation documentation]. [btrbk source tarball]: https://digint.ch/download/btrbk/releases/ [installation documentation]: doc/install.md ### Prerequisites * [btrfs-progs]: Btrfs filesystem utilities >= v4.12 * [Perl interpreter]: Probably already installed on your system * [OpenSSH]: If you want to transfer backups from/to remote locations * [mbuffer]: If you want rate limiting and progress bars [btrfs-progs]: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/ [Perl interpreter]: https://www.perl.org [OpenSSH]: https://www.openssh.com [mbuffer]: https://www.maier-komor.de/mbuffer.html Synopsis ======== Please consult the [btrbk(1)] man-page provided with this package for a full description of the command line options. [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html Configuration ============= Before running `btrbk`, you will need to create a configuration file. You might want to take a look at `btrbk.conf.example` provided with this package. For a detailed description, please consult the [btrbk.conf(5)] man-page. After a configuration change, it is highly recommended to check it by running btrbk with the `-n,--dryrun` option: # btrbk -c /path/to/myconfig -v -n run This will read all btrfs information on the source/target filesystems and show what actions would be performed (without writing anything to the disks). The examples below assume that the btrfs subvolume containing `home` and `rootfs` is mounted at `/mnt/btr_pool`. This is usually the btrfs root subvolume, which always has `subvolid=5`. Mounting `subvolid=5` is *recommended* (mandatory for btrbk < v0.32.0) if you want to backup your root filesystem `/`. /etc/fstab: /dev/sda1 /mnt/btr_pool btrfs subvolid=5,noatime 0 0 Note that some default btrfs installations (e.g. Ubuntu) use subvolume names `@` for rootfs (mounted at `/`) and `@home` for `/home`, as a naming convention. If this is the case on your file system, replace the `subvolume` declarations in the examples accordingly. [btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html Example: Local Regular Snapshots (time-machine) ----------------------------------------------- The simplest use case is to only create snapshots of your data. This will obviously not protect it against hardware failure, but can be useful for: * protection against inadvertent changes or deletions * keeping past states of copies from rsync or similar tools Let's assume you need regular snapshots of your home directory, which is located in the subvolume `home` of the volume `/mnt/btr_pool`. The snapshots are to be stored in `btrbk_snapshots` (on the same volume). /etc/btrbk/btrbk.conf: timestamp_format long snapshot_preserve_min 18h snapshot_preserve 48h volume /mnt/btr_pool snapshot_dir btrbk_snapshots subvolume home Notice that the `target` option is not provided, and btrbk will only manage snapshots located on the same volume in `snapshot_dir`. Btrbk does not create subdirs by default, the snapshot directory must first be created manually: # mkdir /mnt/btr_pool/btrbk_snapshots The "volume" section is merely used as a specifier for a base directory, and can be skipped if you prefer to configure everything using absolute paths. The above configuration can also be written as: snapshot_dir /mnt/btr_pool/btrbk_snapshots subvolume /mnt/btr_pool/home If you don't want to mount the btrfs root filesystem to `/mnt/btr_pool`, you might as well configure it like this: snapshot_dir /btrbk_snapshots subvolume /home Start a dry run: # btrbk run -n Create the first snapshot: # btrbk run If it works as expected, configure a cron job to run btrbk hourly: /etc/cron.hourly/btrbk: #!/bin/sh exec /usr/bin/btrbk -q run Snapshots will now be created every hour. All snapshots are preserved for at least 18 hours (`snapshot_preserve_min`), whether they are created by the cron job or manually by calling `sudo btrbk run` on the command line. Additionally, 48 hourly snapshots are preserved (`snapshot_preserve`). Example: Backups to USB Disk ---------------------------- In this example, we assume you have a laptop with: * a disk having a btrfs root subvolume (subvolid=5) mounted on `/mnt/btr_pool`, containing a subvolume `rootfs` for the root filesystem (i.e. mounted on `/`) and a subvolume `home` for the user data, * a directory or subvolume `/mnt/btr_pool/btrbk_snapshots` which will hold the btrbk snapshots, * a backup disk having a btrfs volume mounted as `/mnt/btr_backup`, containing a subvolume or directory `mylaptop` for the incremental backups. Retention policy: * keep all snapshots for 2 days, no matter how frequently you (or your cron job) run btrbk * keep daily snapshots for 14 days (very handy if you are on the road and the backup disk is not attached) * keep monthly backups forever * keep weekly backups for 10 weeks * keep daily backups for 20 days /etc/btrbk/btrbk-mylaptop.conf: snapshot_preserve_min 2d snapshot_preserve 14d # Create snapshots only if the backup disk is attached #snapshot_create ondemand target_preserve_min no target_preserve 20d 10w *m snapshot_dir btrbk_snapshots volume /mnt/btr_pool target /mnt/btr_backup/mylaptop subvolume rootfs subvolume home [...] /etc/cron.daily/btrbk: #!/bin/sh exec /usr/bin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run * This will create snapshots on a daily basis: * `/mnt/btr_pool/btrbk_snapshots/rootfs.YYYYMMDD` * `/mnt/btr_pool/btrbk_snapshots/home.YYYYMMDD` * And create incremental backups in: * `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD` * `/mnt/btr_backup/mylaptop/home.YYYYMMDD` If you prefer triggering the backups manually, change the cron command to run the `snapshot` action instead of `run`. Start the backups manually by running: # btrbk resume For a quick additional snapshot of your home, run: # btrbk snapshot home Example: Host-initiated Backup on Fileserver -------------------------------------------- Let's say you have a fileserver at "myserver.mydomain.com" where you want to create backups of your laptop disk. The config could look like this: ssh_identity /etc/btrbk/ssh/id_rsa volume /mnt/btr_pool subvolume rootfs target /mnt/btr_backup/mylaptop target ssh://myserver.mydomain.com/mnt/btr_backup/mylaptop In addition to the backups on your local usb-disk mounted at `/mnt/btr_backup/mylaptop`, incremental backups would also be pushed to `myserver.mydomain.com`. Example: Fileserver-initiated Backups from Several Hosts -------------------------------------------------------- If you're a sysadmin and want to trigger backups directly from your fileserver, the config would be something like: ssh_identity /etc/btrbk/ssh/id_rsa volume ssh://alpha.mydomain.com/mnt/btr_pool target /mnt/btr_backup/alpha subvolume rootfs subvolume home volume ssh://beta.mydomain.com/mnt/btr_pool target /mnt/btr_backup/beta subvolume rootfs subvolume dbdata This will pull backups from alpha/beta.mydomain.com and locally create: * `/mnt/btr_backup/alpha/rootfs.YYYYMMDD` * `/mnt/btr_backup/alpha/home.YYYYMMDD` * `/mnt/btr_backup/beta/rootfs.YYYYMMDD` * `/mnt/btr_backup/beta/dbdata.YYYYMMDD` Example: Multiple Btrbk Instances --------------------------------- Let's say we have a host (at 192.168.0.42) running btrbk with the setup of the time-machine example above, and we need a backup server to only fetch the snapshots. /etc/btrbk/btrbk.conf (on backup server): target_preserve_min no target_preserve 0d 10w *m volume ssh://192.168.0.42/mnt/btr_pool target /mnt/btr_backup/my-laptop subvolume home snapshot_dir btrbk_snapshots snapshot_preserve_min all snapshot_create no If the server runs btrbk with this config, 10 weeklies and all monthlies are received from 192.168.0.42. The source filesystem is never altered because of `snapshot_preserve_min all`. Example: Virtual Machine Setup ------------------------------ Common virtual machine setups have multiple volume sections with same host, but distinct port numbers for each machine. /etc/btrbk/btrbk.conf: # This propagates to all subvolume sections: target /mnt/btr_backup/ volume ssh://localhost:2201/mnt/btr_pool group vm vm01 subvolume home snapshot_name vm01-home subvolume data snapshot_name vm01-data volume ssh://localhost:2202/mnt/btr_pool group vm vm02 subvolume home snapshot_name vm02-home volume ssh://localhost:2203/mnt/btr_pool [...] This will create `/mnt/btr_backup/vm[NN]-home`, `vm[NN]-data`, ... Note that btrbk holds a single reference to every btrfs filesystem tree, regarding UUID's as "globally unique". If the configured subvolumes point to the same filesystem on different machines (ports), you will see log lines like this when running `btrbk -v`: ``` Assuming same filesystem: "ssh://localhost:2201/dev/sda1", "ssh://localhost:2202/dev/sda1" ``` Example: Backup from non-btrfs Source ------------------------------------- If you want to make backups from a filesystem other than btrfs (e.g. ext4 or reiserfs), you need to create a *synchronization subvolume* on the backup disk: # btrfs subvolume create /mnt/btr_backup/myhost_sync Configure btrbk to use `myhost_sync` as source subvolume: volume /mnt/btr_backup subvolume myhost_sync snapshot_name myhost snapshot_preserve_min latest snapshot_preserve 14d 20w *m The btrbk package provides the "btrbk-mail" script, which automates the synchronization using rsync, and can be run as cron job or systemd timer unit. For configuration details, see the config section in "/contrib/cron/btrbk-mail". Alternatively, you can run any synchronization software prior to running btrbk. Something like: #!/bin/sh rsync -az --delete \ --inplace --numeric-ids --acls --xattrs \ -e 'ssh -i /etc/btrbk/ssh/id_rsa' \ myhost.mydomain.com:/data/ \ /mnt/btr_backup/myhost_sync/ exec /usr/bin/btrbk -q run This will produce snapshots `/mnt/btr_backup/myhost.20150101`, with retention as defined with the snapshot_preserve option. Example: Encrypted Backup to non-btrfs Target --------------------------------------------- If your backup server does not support btrfs, you can send your subvolumes to a raw file. This is an _experimental_ feature: btrbk supports "raw" targets, meaning that similar to the "send-receive" target the btrfs subvolume is being sent using `btrfs send` (mirroring filesystem level data), but instead of instantly being received (`btrfs receive`) by the target filesystem, it is being redirected to a file, optionally compressed and piped through GnuPG. /etc/btrbk/btrbk.conf: raw_target_compress xz raw_target_encrypt gpg gpg_keyring /etc/btrbk/gpg/pubring.gpg gpg_recipient btrbk@mydomain.com volume /mnt/btr_pool subvolume home target raw ssh://cloud.example.com/backup ssh_user btrbk # incremental no This will create a GnuPG encrypted, compressed files on the target host. For each backup, two files are created: * `/backup/home.YYYYMMDD.btrfs.xz.gpg`: main data file containing the btrfs send-stream, * `/backup/home.YYYYMMDD.btrfs.xz.gpg.info`: sidecar file containing metadata used by btrbk. I you are using raw _incremental_ backups, please make sure you understand the implications (see [btrbk.conf(5)], TARGET TYPES). Setting up SSH ============== Since btrbk needs root access, it is *very advisable* to take all the security precautions you can. In most cases backups are generated periodically without user interaction, so it is not possible to protect your ssh key with a password. The steps below will give you hints on how to secure your ssh server for a backup scenario. Note that the btrbk package is not required on the remote side, but you will need the `btrfs` executable from the [btrfs-progs] package. ### Create SSH Key Pair On the client side, create a ssh key dedicated to btrbk, without password protection: # ssh-keygen -t rsa -b 4096 -f /etc/btrbk/ssh/id_rsa -C btrbk@mydomain.com -N "" The content of the public key (/etc/btrbk/ssh/id_rsa.pub) is used for authentication in "authorized_keys" on the server side (see [sshd(8)] for details). ### Allow Root Login The most straight forward setup is to allow root login on the remote host. If this is not an option for you, refer to the more complex "Dedicated Btrbk User Login" section below. /etc/ssh/sshd_config: PermitRootLogin prohibit-password Add your btrbk public key to "/root/.ssh/authorized_keys" on the server, and you are good to go. ### Restrict Access Restrict ssh access to a static IP address within your network. On the remote host, either add a "Match" block in: /etc/ssh/sshd_config: Match Address 192.168.0.42 Or restrict in authorized_keys: from="192.168.0.42" ... Consult the [sshd_config(5)] man-page for a detailed explanation and more options. Dedicated Btrbk User Login (optional) ------------------------------------- If allowing root login is not an option for you, there are several ways to restrict SSH access to a regular user. ### Option 1: Use sudo On the client side, configure btrbk use the sudo backend. This changes the ssh calls to btrfs commands to `sudo btrfs `. /etc/btrbk/btrbk.conf: backend_remote btrfs-progs-sudo On the remote host, grant root permissions for the "btrfs" command groups (subcommands) in "/etc/sudoers". If you are using [ssh_filter_btrbk(1)], also add the `ssh_filter_btrbk.sh --sudo` option in "authorized_keys" (see below). ### Option 2: Use btrfs-progs-btrbk Instead of using the all-inclusive `btrfs` command, "btrfs-progs-btrbk" allows you to restrict privileges to its subcommands using linux capabilities(7) or setuid. Note that the "btrfs-progs-btrbk" package is not available on all linux distributions, you might need to build and install it on your own (refer to [btrfs-progs-btrbk] on GitHub for more details). /etc/btrbk/btrbk.conf: backend_remote btrfs-progs-btrbk Make sure that only the required binaries with elevated privileges can be called by the btrbk user. For example, on a server acting as "btrbk source", allow only the following binaries for the "btrbk" group: # getcap /usr/bin/btrfs-* /usr/bin/btrfs-send cap_dac_read_search,cap_fowner,cap_sys_admin=ep /usr/bin/btrfs-subvolume-delete cap_dac_override,cap_sys_admin=ep /usr/bin/btrfs-subvolume-list cap_dac_read_search,cap_fowner,cap_sys_admin=ep /usr/bin/btrfs-subvolume-show cap_dac_read_search,cap_fowner,cap_sys_admin=ep /usr/bin/btrfs-subvolume-snapshot cap_dac_override,cap_dac_read_search,cap_fowner,cap_sys_admin=ep # ls -l /usr/bin/btrfs-* -rwx--x--- 1 root btrbk /usr/bin/btrfs-send -rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-delete -rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-list -rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-show -rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-snapshot Restrict Commands with "ssh_filter_btrbk.sh" (optional) ------------------------------------------------------- Btrbk comes with a shell script "ssh_filter_btrbk.sh", which restricts ssh access to sane calls to the "btrfs" command needed for snapshot creation and send/receive operations (see [ssh_filter_btrbk(1)]). Copy "ssh_filter_btrbk.sh" to "/backup/scripts/", and configure sshd to run it whenever the key is used for authentication. Example "/root/.ssh/authorized_keys": # example backup source (also allowing deletion of old snapshots) command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete" ... # example backup target (also allowing deletion of old snapshots) command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete" ... # example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no), # restricted to subvolumes within /home or /data command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data" ... [ssh_filter_btrbk(1)]: https://digint.ch/btrbk/doc/ssh_filter_btrbk.1.html [sshd(8)]: https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8 [sshd_config(5)]: https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config [btrfs-progs-btrbk]: https://github.com/digint/btrfs-progs-btrbk Restoring Backups ================= Btrbk does not provide any mechanism to restore your backups, this has to be done manually. In the instructions below, we assume that you have a btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you want to restore is at `/mnt/btr_pool/data`. **Important**: don't use `btrfs property set` to make a subvolume read-write after restoring. This is a low-level command, and leaves "Received UUID" in a false state which causes btrbk to fail on subsequent incremental backups. Instead, use `btrfs subvolume snapshot` (without `-r` flag) as described below. ### Step 0: Identify Subvolume # list snapshots managed by btrbk btrbk list snapshots # alternative: list all subvolumes btrbk ls / btrbk ls -L / From the list, identify the snapshot you want to restore. Let's say it's `/mnt/btr_pool/_btrbk_snap/data.20150101`. ### Step 1: Restore Backup (skip this step if you restore from a snapshot) # locally mounted backup disk btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/ # from / to remote host ssh root@remote btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/ btrfs send /mnt/btr_backup/data.20150101 | ssh root@remote btrfs receive /mnt/btr_pool/ **Hint**: Try to send-receive backups incrementally if possible. In case you still have common snapshot / backup pairs (i.e. both "snapshot_subvol" and "target_subvol" are listed above), use `btrfs send -p `. From this point on, `data.20150101` on both disks can be used as parents for subsequent send-receive operations, and a *received_uuid* relationship is established (see below). ### Step 2: Create read-write Subvolume # if still present, move broken subvolume away mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN # create read-write subvolume btrfs subvolume snapshot /mnt/btr_pool/data.20150101 /mnt/btr_pool/data Your `data` subvolume is restored, you can carry on with incremental backups to `/mnt/btr_backup`. ### Step 3: Cleanup # if everything went fine, delete the broken subvolume btrfs subvolume delete /mnt/btr_pool/data.BROKEN Make sure to keep `data.20150101` subvolumes on both disks at least until you created a new backup using btrbk, in order to keep the incremental chain alive. Btrfs Relationship (technical note) ----------------------------------- btrbk origin -t /mnt/btr_backup/data.20150101 btrbk ls -L /mnt/btr_pool /mnt/btr_backup * **received_uuid** relationship: *correlated*, *identical* read-only subvolumes, cross-filesystem. a.received_uuid = b.received_uuid a.received_uuid = b.uuid * Required for subvolumes used as parent (or clone-src) of send-receive operations. * Present on subvolumes created by `btrfs send | btrfs receive`. * `/mnt/btr_pool/data.20150101 === /mnt/btr_backup/data.20150101` * **parent_uuid** relationship: "is-snapshot-of" a.parent_uuid = b.uuid * Present on subvolumes created by `btrfs subvolume snapshot` or `btrfs send -p | btrfs receive`. * Used by btrbk to determine best parent. * `/mnt/btr_pool/data.20150101 <-- /mnt/btr_pool/data` FAQ === Make sure to also read the [btrbk FAQ page](doc/FAQ.md). Help improve it by asking! Donate ====== So btrbk saved your day? I will definitively continue to develop btrbk for free. If you want to support my hard work with a donation, you are welcome to do so! [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S) Development =========== Source Code Repository ---------------------- The source code for btrbk is managed using Git. Official repository: git clone https://dev.tty0.ch/btrbk.git Mirror on GitHub: git clone https://github.com/digint/btrbk.git How to Contribute ----------------- Your contributions are welcome! If you would like to contribute or have found bugs: * Visit the [btrbk project page on GitHub] and use the [issues tracker] there. * Talk to us on [Libera.Chat] in `#btrbk`. * Contact the author via email (the email address can be found in the sources). Any feedback is appreciated! [btrbk project page on GitHub]: https://github.com/digint/btrbk [issues tracker]: https://github.com/digint/btrbk/issues [Libera.Chat]: https://libera.chat License ======= btrbk is free software, available under the [GNU General Public License, Version 3 or later][GPL-3.0-or-later]. [GPL-3.0-or-later]: https://www.gnu.org/licenses/gpl.html btrbk-0.32.5/btrbk000077500000000000000000010410641432521455000137460ustar00rootroot00000000000000#!/usr/bin/perl # # btrbk - Create snapshots and remote backups of btrfs subvolumes # # Copyright (C) 2014-2022 Axel Burri # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # --------------------------------------------------------------------- # The official btrbk website is located at: # https://digint.ch/btrbk/ # # Author: # Axel Burri # --------------------------------------------------------------------- use strict; use warnings FATAL => qw( all ), NONFATAL => qw( deprecated ); use Carp qw(confess); use Getopt::Long qw(GetOptions); use Time::Local qw( timelocal timegm timegm_nocheck ); use IPC::Open3 qw(open3); use Symbol qw(gensym); use Cwd qw(abs_path); our $VERSION = '0.32.5'; our $AUTHOR = 'Axel Burri '; our $PROJECT_HOME = ''; our $BTRFS_PROGS_MIN = '4.12'; # required since btrbk-v0.27.0 my $VERSION_INFO = "btrbk command line client, version $VERSION"; my @config_src = ("/etc/btrbk.conf", "/etc/btrbk/btrbk.conf"); my %compression = ( # NOTE: also adapt "compress_list" in ssh_filter_btrbk.sh if you change this gzip => { name => 'gzip', format => 'gz', compress_cmd => [ 'gzip', '-c' ], decompress_cmd => [ 'gzip', '-d', '-c' ], level_min => 1, level_max => 9 }, pigz => { name => 'pigz', format => 'gz', compress_cmd => [ 'pigz', '-c' ], decompress_cmd => [ 'pigz', '-d', '-c' ], level_min => 1, level_max => 9, threads => '-p' }, bzip2 => { name => 'bzip2', format => 'bz2', compress_cmd => [ 'bzip2', '-c' ], decompress_cmd => [ 'bzip2', '-d', '-c' ], level_min => 1, level_max => 9 }, pbzip2 => { name => 'pbzip2', format => 'bz2', compress_cmd => [ 'pbzip2', '-c' ], decompress_cmd => [ 'pbzip2', '-d', '-c' ], level_min => 1, level_max => 9, threads => '-p' }, xz => { name => 'xz', format => 'xz', compress_cmd => [ 'xz', '-c' ], decompress_cmd => [ 'xz', '-d', '-c' ], level_min => 0, level_max => 9, threads => '-T' }, lzo => { name => 'lzo', format => 'lzo', compress_cmd => [ 'lzop', '-c' ], decompress_cmd => [ 'lzop', '-d', '-c' ], level_min => 1, level_max => 9 }, lz4 => { name => 'lz4', format => 'lz4', compress_cmd => [ 'lz4', '-c' ], decompress_cmd => [ 'lz4', '-d', '-c' ], level_min => 1, level_max => 9 }, zstd => { name => 'zstd', format => 'zst', compress_cmd => [ 'zstd', '-c' ], decompress_cmd => [ 'zstd', '-d', '-c' ], level_min => 1, level_max => 19, threads => '-T', long => '--long=', adapt => '--adapt' }, ); my $compress_format_alt = join '|', map { $_->{format} } values %compression; # note: this contains duplicate alternations my $ipv4_addr_match = qr/(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])/; my $ipv6_addr_match = qr/[a-fA-F0-9]*:[a-fA-F0-9]*:[a-fA-F0-9:]+/; # simplified (contains at least two colons), matches "::1", "2001:db8::7" my $host_name_match = qr/(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])/; my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/; my $btrbk_timestamp_match = qr/(?[0-9]{4})(?[0-9]{2})(?
[0-9]{2})(T(?[0-9]{2})(?[0-9]{2})((?[0-9]{2})(?(Z|[+-][0-9]{4})))?)?(_(?[0-9]+))?/; # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]" my $raw_postfix_match = qr/\.btrfs(\.($compress_format_alt))?(\.(gpg|encrypted))?/; # matches ".btrfs[.gz|bz2|xz][.gpg|encrypted]" my $safe_file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/; # note: ubuntu uses '@' in the subvolume layout: my $group_match = qr/[a-zA-Z0-9_:-]+/; my $config_split_match = qr/\s*[,\s]\s*/; my %day_of_week_map = ( sunday => 0, monday => 1, tuesday => 2, wednesday => 3, thursday => 4, friday => 5, saturday => 6 ); my @syslog_facilities = qw( user mail daemon auth lpr news cron authpriv local0 local1 local2 local3 local4 local5 local6 local7 ); my @incremental_prefs_avail = qw(sro srn sao san aro arn); my @incremental_prefs_default = qw(sro:1 srn:1 sao:1 san:1 aro:1 arn:1); my $incremental_prefs_match = "(defaults|(" . join("|", @incremental_prefs_avail) . ")(:[0-9]+)?)"; my %config_options = ( # NOTE: the parser always maps "no" to undef # NOTE: keys "volume", "subvolume" and "target" are hardcoded # NOTE: files "." and "no" map to timestamp_format => { default => "long", accept => [qw( short long long-iso )], context => [qw( global volume subvolume )] }, snapshot_dir => { default => undef, accept_file => { relative => 1, absolute => 1 }, context => [qw( global volume subvolume )] }, snapshot_name => { c_default => 1, accept_file => { name_only => 1 }, context => [qw( subvolume )], deny_glob_context => 1 }, # NOTE: defaults to the subvolume name (hardcoded) snapshot_create => { default => "always", accept => [qw( no always ondemand onchange )], context => [qw( global volume subvolume )] }, incremental => { default => "yes", accept => [qw( yes no strict )] }, incremental_prefs => { default => \@incremental_prefs_default, accept => [ qr/$incremental_prefs_match/ ], split => 1 }, incremental_clones => { default => "yes", accept => [qw( yes no )] }, incremental_resolve => { default => "mountpoint", accept => [qw( mountpoint directory _all_accessible )] }, preserve_day_of_week => { default => "sunday", accept => [ (keys %day_of_week_map) ] }, preserve_hour_of_day => { default => 0, accept => [ (0..23) ] }, snapshot_preserve => { default => undef, accept => [qw( no )], accept_preserve_matrix => 1, context => [qw( global volume subvolume )], }, snapshot_preserve_min => { default => "all", accept => [qw( all latest ), qr/[1-9][0-9]*[hdwmy]/ ], context => [qw( global volume subvolume )], }, target_preserve => { default => undef, accept => [qw( no )], accept_preserve_matrix => 1 }, target_preserve_min => { default => "all", accept => [qw( all latest no ), qr/[0-9]+[hdwmy]/ ] }, archive_preserve => { default => undef, accept => [qw( no )], accept_preserve_matrix => 1, context => [qw( global )] }, archive_preserve_min => { default => "all", accept => [qw( all latest no ), qr/[0-9]+[hdwmy]/ ], context => [qw( global )] }, ssh_identity => { default => undef, accept => [qw( no ) ], accept_file => { absolute => 1 } }, ssh_user => { default => "root", accept => [qw( no ), qr/[a-z_][a-z0-9_-]*/ ] }, ssh_compression => { default => undef, accept => [qw( yes no )] }, ssh_cipher_spec => { default => [ "default" ], accept => [qw( default ), qr/[a-z0-9][a-z0-9@.-]+/ ], split => 1 }, transaction_log => { default => undef, accept => [qw( no )], accept_file => { absolute => 1 }, context => [qw( global )] }, transaction_syslog => { default => undef, accept => [qw( no ), @syslog_facilities ], context => [qw( global )] }, lockfile => { default => undef, accept => [qw( no )], accept_file => { absolute => 1 }, context => [qw( global )] }, rate_limit => { default => undef, accept => [qw( no ), qr/[0-9]+[kmgtKMGT]?/ ], require_bin => 'mbuffer' }, rate_limit_remote => { default => undef, accept => [qw( no ), qr/[0-9]+[kmgtKMGT]?/ ] }, # NOTE: requires 'mbuffer' command on remote hosts stream_buffer => { default => undef, accept => [qw( no ), qr/[0-9]+[kmgKMG%]?/ ], require_bin => 'mbuffer' }, stream_buffer_remote => { default => undef, accept => [qw( no ), qr/[0-9]+[kmgKMG%]?/ ] }, # NOTE: requires 'mbuffer' command on remote hosts stream_compress => { default => undef, accept => [qw( no ), (keys %compression) ] }, stream_compress_level => { default => "default", accept => [qw( default ), qr/[0-9]+/ ] }, stream_compress_long => { default => "default", accept => [qw( default ), qr/[0-9]+/ ] }, stream_compress_threads => { default => "default", accept => [qw( default ), qr/[0-9]+/ ] }, stream_compress_adapt => { default => undef, accept => [qw( yes no )] }, raw_target_compress => { default => undef, accept => [qw( no ), (keys %compression) ] }, raw_target_compress_level => { default => "default", accept => [qw( default ), qr/[0-9]+/ ] }, raw_target_compress_long => { default => "default", accept => [qw( default ), qr/[0-9]+/ ] }, raw_target_compress_threads => { default => "default", accept => [qw( default ), qr/[0-9]+/ ] }, raw_target_encrypt => { default => undef, accept => [qw( no gpg openssl_enc )] }, raw_target_block_size => { default => "128K", accept => [ qr/[0-9]+[kmgKMG]?/ ] }, raw_target_split => { default => undef, accept => [qw( no ), qr/[0-9]+([kmgtpezyKMGTPEZY][bB]?)?/ ] }, gpg_keyring => { default => undef, accept_file => { absolute => 1 } }, gpg_recipient => { default => undef, accept => [ qr/[0-9a-zA-Z_@\+\-\.]+/ ], split => 1 }, openssl_ciphername => { default => "aes-256-cbc", accept => [ qr/[0-9a-zA-Z\-]+/ ] }, openssl_iv_size => { default => undef, accept => [qw( no ), qr/[0-9]+/ ] }, openssl_keyfile => { default => undef, accept_file => { absolute => 1 } }, kdf_backend => { default => undef, accept_file => { absolute => 1 } }, kdf_keysize => { default => "32", accept => [ qr/[0-9]+/ ] }, kdf_keygen => { default => "once", accept => [qw( once each )] }, group => { default => undef, accept => [ qr/$group_match/ ], allow_multiple => 1, split => 1 }, noauto => { default => undef, accept => [qw( yes no )] }, backend => { default => "btrfs-progs", accept => [qw( btrfs-progs btrfs-progs-btrbk btrfs-progs-sudo btrfs-progs-doas )] }, backend_local => { default => undef, accept => [qw( no btrfs-progs btrfs-progs-btrbk btrfs-progs-sudo btrfs-progs-doas )] }, backend_remote => { default => undef, accept => [qw( no btrfs-progs btrfs-progs-btrbk btrfs-progs-sudo btrfs-progs-doas )] }, backend_local_user => { default => undef, accept => [qw( no btrfs-progs btrfs-progs-btrbk btrfs-progs-sudo btrfs-progs-doas )] }, compat => { default => undef, accept => [qw( no busybox ignore_receive_errors )], split => 1 }, compat_local => { default => undef, accept => [qw( no busybox ignore_receive_errors )], split => 1 }, compat_remote => { default => undef, accept => [qw( no busybox ignore_receive_errors )], split => 1 }, safe_commands => { default => undef, accept => [qw( yes no )], context => [qw( global )] }, btrfs_commit_delete => { default => undef, accept => [qw( yes no after each )], deprecated => { MATCH => { regex => qr/^(?:after|each)$/, warn => 'Please use "btrfs_commit_delete yes|no"', replace_key => "btrfs_commit_delete", replace_value => "yes" } } }, snapshot_qgroup_destroy => { default => undef, accept => [qw( yes no )], context => [qw( global volume subvolume )] }, target_qgroup_destroy => { default => undef, accept => [qw( yes no )] }, archive_qgroup_destroy => { default => undef, accept => [qw( yes no )], context => [qw( global )] }, archive_exclude => { default => undef, accept_file => { wildcards => 1 }, allow_multiple => 1, context => [qw( global )] }, archive_exclude_older => { default => undef, accept => [qw( yes no )] }, cache_dir => { default => undef, accept_file => { absolute => 1 }, allow_multiple => 1, context => [qw( global )] }, ignore_extent_data_inline => { default => "yes", accept => [qw( yes no )] }, warn_unknown_targets => { default => undef, accept => [qw( yes no )] }, # deprecated options ssh_port => { default => "default", accept => [qw( default ), qr/[0-9]+/ ], deprecated => { DEFAULT => { warn => 'Please use "ssh://hostname[:port]" notation in the "volume" and "target" configuration lines.' } } }, btrfs_progs_compat => { default => undef, accept => [qw( yes no )], deprecated => { DEFAULT => { ABORT => 1, warn => 'This feature has been dropped in btrbk-v0.23.0. Please update to newest btrfs-progs, AT LEAST >= $BTRFS_PROGS_MIN' } } }, snapshot_preserve_daily => { default => 'all', accept => [qw( all ), qr/[0-9]+/ ], context => [qw( global volume subvolume )], deprecated => { DEFAULT => { FAILSAFE_PRESERVE => 1, warn => 'Please use "snapshot_preserve" and/or "snapshot_preserve_min"' } } }, snapshot_preserve_weekly => { default => 0, accept => [qw( all ), qr/[0-9]+/ ], context => [qw( global volume subvolume )], deprecated => { DEFAULT => { FAILSAFE_PRESERVE => 1, warn => 'Please use "snapshot_preserve" and/or "snapshot_preserve_min"' } } }, snapshot_preserve_monthly => { default => 'all', accept => [qw( all ), qr/[0-9]+/ ], context => [qw( global volume subvolume )], deprecated => { DEFAULT => { FAILSAFE_PRESERVE => 1, warn => 'Please use "snapshot_preserve" and/or "snapshot_preserve_min"' } } }, target_preserve_daily => { default => 'all', accept => [qw( all ), qr/[0-9]+/ ], deprecated => { DEFAULT => { FAILSAFE_PRESERVE => 1, warn => 'Please use "target_preserve" and/or "target_preserve_min"' } } }, target_preserve_weekly => { default => 0, accept => [qw( all ), qr/[0-9]+/ ], deprecated => { DEFAULT => { FAILSAFE_PRESERVE => 1, warn => 'Please use "target_preserve" and/or "target_preserve_min"' } } }, target_preserve_monthly => { default => 'all', accept => [qw( all ), qr/[0-9]+/ ], deprecated => { DEFAULT => { FAILSAFE_PRESERVE => 1, warn => 'Please use "target_preserve" and/or "target_preserve_min"' } } }, resume_missing => { default => "yes", accept => [qw( yes no )], deprecated => { yes => { warn => 'ignoring (missing backups are always resumed since btrbk v0.23.0)' }, no => { FAILSAFE_PRESERVE => 1, warn => 'Please use "target_preserve_min latest" and "target_preserve no" if you want to keep only the latest backup', }, DEFAULT => {} } }, snapshot_create_always => { default => undef, accept => [qw( yes no )], deprecated => { yes => { warn => "Please use \"snapshot_create always\"", replace_key => "snapshot_create", replace_value => "always", }, no => { warn => "Please use \"snapshot_create no\" or \"snapshot_create ondemand\"", replace_key => "snapshot_create", replace_value => "ondemand", }, DEFAULT => {}, }, }, receive_log => { default => undef, accept => [qw( sidecar no )], accept_file => { absolute => 1 }, deprecated => { DEFAULT => { warn => "ignoring" } }, } ); my @config_target_types = qw(send-receive raw); # first in list is default my %table_formats = ( config_volume => { table => [ qw( -volume_host -volume_port volume_path ) ], long => [ qw( volume_host -volume_port volume_path -volume_rsh ) ], raw => [ qw( volume_url volume_host volume_port volume_path volume_rsh ) ], single_column => [ qw( volume_url ) ], }, config_source => { table => [ qw( -source_host -source_port source_subvolume snapshot_path snapshot_name ) ], long => [ qw( source_host -source_port source_subvolume snapshot_path snapshot_name -source_rsh ) ], raw => [ qw( source_url source_host source_port source_subvolume snapshot_path snapshot_name source_rsh ) ], single_column => [ qw( source_url ) ], }, config_target => { table => [ qw( -target_host -target_port target_path ) ], long => [ qw( target_host -target_port target_path -target_rsh ) ], raw => [ qw( target_url target_host target_port target_path target_rsh ) ], single_column => [ qw( target_url ) ], }, config => { table => [ qw( -source_host -source_port source_subvolume snapshot_path snapshot_name -target_host -target_port target_path ) ], long => [ qw( -source_host -source_port source_subvolume snapshot_path snapshot_name -target_host -target_port target_path target_type snapshot_preserve target_preserve ) ], raw => [ qw( source_url source_host source_port source_subvolume snapshot_path snapshot_name target_url target_host target_port target_path target_type snapshot_preserve target_preserve source_rsh target_rsh ) ], }, resolved => { table => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status -target_host -target_port target_subvolume ) ], long => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status -target_host -target_port target_subvolume target_type ) ], raw => [ qw( type source_url source_host source_port source_subvolume snapshot_subvolume snapshot_name status target_url target_host target_port target_subvolume target_type source_rsh target_rsh ) ], }, snapshots => { table => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status ) ], long => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status ) ], raw => [ qw( source_url source_host source_port source_subvolume snapshot_subvolume snapshot_name status source_rsh ) ], single_column => [ qw( snapshot_url ) ], }, backups => { # same as resolved, except for single_column table => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status -target_host -target_port target_subvolume ) ], long => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status -target_host -target_port target_subvolume target_type ) ], raw => [ qw( type source_url source_host source_port source_subvolume snapshot_subvolume snapshot_name status target_url target_host target_port target_subvolume target_type source_rsh target_rsh ) ], single_column => [ qw( target_url ) ], }, latest => { # same as resolved, except hiding target if not present table => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status -target_host -target_port -target_subvolume ) ], long => [ qw( -source_host -source_port source_subvolume snapshot_subvolume status -target_host -target_port -target_subvolume -target_type ) ], raw => [ qw( type source_url source_host source_port source_subvolume snapshot_subvolume snapshot_name status target_url target_host target_port target_subvolume target_type source_rsh target_rsh ) ], }, stats => { table => [ qw( -source_host -source_port source_subvolume snapshot_subvolume -target_host -target_port -target_subvolume snapshots -backups ) ], long => [ qw( -source_host -source_port source_subvolume snapshot_subvolume -target_host -target_port -target_subvolume snapshot_status backup_status snapshots -backups -correlated -orphaned -incomplete ) ], raw => [ qw( source_url source_host source_port source_subvolume snapshot_subvolume snapshot_name target_url target_host target_port target_subvolume snapshot_status backup_status snapshots backups correlated orphaned incomplete ) ], RALIGN => { snapshots=>1, backups=>1, correlated=>1, orphaned=>1, incomplete=>1 }, }, schedule => { table => [ qw( action -host -port subvolume scheme reason ) ], long => [ qw( action -host -port subvolume scheme reason ) ], raw => [ qw( topic action url host port path hod dow min h d w m y) ], }, usage => { table => [ qw( -host -port mount_source path size used free ) ], long => [ qw( type -host -port mount_source path size used device_size device_allocated device_unallocated device_missing device_used free free_min data_ratio metadata_ratio global_reserve global_reserve_used ) ], raw => [ qw( type host port mount_source path size used device_size device_allocated device_unallocated device_missing device_used free free_min data_ratio metadata_ratio global_reserve global_reserve_used ) ], RALIGN => { size=>1, used=>1, device_size=>1, device_allocated=>1, device_unallocated=>1, device_missing=>1, device_used=>1, free=>1, free_min=>1, data_ratio=>1, metadata_ratio=>1, global_reserve=>1, global_reserve_used=>1 }, }, transaction => { table => [ qw( type status -target_host -target_port target_subvolume -source_host -source_port source_subvolume parent_subvolume ) ], long => [ qw( localtime type status duration target_host -target_port target_subvolume source_host -source_port source_subvolume parent_subvolume message ) ], tlog => [ qw( localtime type status target_url source_url parent_url message ) ], syslog => [ qw( type status target_url source_url parent_url message ) ], raw => [ qw( time localtime type status duration target_url source_url parent_url message ) ], }, origin_tree => { table => [ qw( tree uuid parent_uuid received_uuid ) ], long => [ qw( tree uuid parent_uuid received_uuid recursion ) ], raw => [ qw( tree uuid parent_uuid received_uuid recursion ) ], }, diff => { table => [ qw( flags count size file ) ], long => [ qw( flags count size file ) ], raw => [ qw( flags count size file ) ], RALIGN => { count=>1, size=>1 }, }, fs_list => { table => [ qw( -host mount_source mount_subvol mount_point id flags subvolume_path path ) ], short => [ qw( -host mount_source id flags path ) ], long => [ qw( -host mount_source id top cgen gen uuid parent_uuid received_uuid flags path ) ], raw => [ qw( host mount_source mount_subvol mount_point mount_subvolid id top_level cgen gen uuid parent_uuid received_uuid readonly path subvolume_path subvolume_rel_path url ) ], single_column => [ qw( url ) ], RALIGN => { id=>1, top=>1, cgen=>1, gen=>1 }, }, extent_diff => { table => [ qw( total exclusive -diff -set subvol ) ], long => [ qw( id cgen gen total exclusive -diff -set subvol ) ], raw => [ qw( id cgen gen total exclusive -diff -set subvol ) ], RALIGN => { id=>1, cgen=>1, gen=>1, total=>1, exclusive=>1, diff=>1, set=>1 }, }, ); my @btrfs_cmd = ( "btrfs subvolume list", "btrfs subvolume show", "btrfs subvolume snapshot", "btrfs subvolume delete", "btrfs send", "btrfs receive", "btrfs filesystem usage", "btrfs qgroup destroy", ); my @system_cmd = ( "readlink", "test", ); my %backend_cmd_map = ( "btrfs-progs-btrbk" => { map +( $_ => [ s/ /-/gr ] ), @btrfs_cmd }, "btrfs-progs-sudo" => { map +( $_ => [ qw( sudo -n ), split(" ", $_) ] ), @btrfs_cmd, @system_cmd }, "btrfs-progs-doas" => { map +( $_ => [ qw( doas -n ), split(" ", $_) ] ), @btrfs_cmd, @system_cmd }, ); # keys used in raw target sidecar files (.info): my %raw_info_sort = ( TYPE => 1, FILE => 2, RECEIVED_UUID => 3, RECEIVED_PARENT_UUID => 4, compress => 10, split => 11, encrypt => 12, cipher => 13, iv => 14, # kdf_* (generated by kdf_backend) INCOMPLETE => 100, ); my %raw_url_cache; # map URL to (fake) btr_tree node my %mountinfo_cache; # map MACHINE_ID to mount points (sorted descending by file length) my %mount_source_cache; # map URL_PREFIX:mount_source (aka device) to btr_tree node my %uuid_cache; # map UUID to btr_tree node my %realpath_cache; # map URL to realpath (symlink target). empty string denotes an error. my $tree_inject_id = 0; # fake subvolume id for injected nodes (negative) my $fake_uuid_prefix = 'XXXXXXXX-XXXX-XXXX-XXXX-'; # plus 0-padded inject_id: XXXXXXXX-XXXX-XXXX-XXXX-000000000000 my $program_name; # "btrbk" or "lsbtr", default to "btrbk" my $safe_commands; my $dryrun; my $loglevel = 1; my $quiet; my @exclude_vf; my $do_dumper; my $do_trace; my $show_progress = 0; my $output_format; my $output_pretty = 0; my @output_unit; my $lockfile; my $tlog_fh; my $syslog_enabled = 0; my $current_transaction; my @transaction_log; my %config_override; my @tm_now; # current localtime ( sec, min, hour, mday, mon, year, wday, yday, isdst ) my @stderr; # stderr of last run_cmd my %warn_once; my %kdf_vars; my $kdf_session_key; $SIG{__DIE__} = sub { print STDERR "\nERROR: process died unexpectedly (btrbk v$VERSION)"; print STDERR "\nPlease contact the author: $AUTHOR\n\n"; print STDERR "Stack Trace:\n----------------------------------------\n"; Carp::confess @_; }; $SIG{INT} = sub { print STDERR "\nERROR: Caught SIGINT, dumping transaction log:\n"; action("signal", status => "SIGINT"); print_formatted("transaction", \@transaction_log, output_format => "tlog", outfile => *STDERR); exit 1; }; sub VERSION_MESSAGE { print $VERSION_INFO . "\n"; } sub ERROR_HELP_MESSAGE { return if($quiet); print STDERR "See '$program_name --help'.\n"; } sub HELP_MESSAGE { return if($quiet); #80----------------------------------------------------------------------------- if($program_name eq "lsbtr") { print <<"END_HELP_LSBTR"; usage: lsbtr [] [[--] ...] options: -h, --help display this help message --version display version information -l, --long use long listing format -u, --uuid print uuid table (parent/received relations) -1, --single-column Print path column only --raw print raw table format -v, --verbose increase output verbosity -c, --config=FILE specify btrbk configuration file --override=KEY=VALUE globally override a configuration option For additional information, see $PROJECT_HOME END_HELP_LSBTR } else { print <<"END_HELP_BTRBK"; usage: btrbk [] [[--] ...] options: -h, --help display this help message --version display version information -c, --config=FILE specify configuration file -n, --dry-run perform a trial run with no changes made --exclude=FILTER exclude configured sections -p, --preserve preserve all (do not delete anything) --preserve-snapshots preserve snapshots (do not delete snapshots) --preserve-backups preserve backups (do not delete backups) --wipe delete all but latest snapshots -v, --verbose be more verbose (increase logging level) -q, --quiet be quiet (do not print backup summary) -l, --loglevel=LEVEL set logging level (error, warn, info, debug, trace) -t, --table change output to table format -L, --long change output to long format --format=FORMAT change output format, FORMAT=table|long|raw -S, --print-schedule print scheduler details (for the "run" command) --progress show progress bar on send-receive operation --lockfile=FILE create and check lockfile --override=KEY=VALUE globally override a configuration option commands: run run snapshot and backup operations dryrun don't run btrfs commands; show what would be executed snapshot run snapshot operations only resume run backup operations, and delete snapshots prune only delete snapshots and backups archive recursively copy all subvolumes clean delete incomplete (garbled) backups stats print snapshot/backup statistics list available subcommands are (default "all"): all snapshots and backups snapshots snapshots backups backups and correlated snapshots latest most recent snapshots and backups config configured source/snapshot/target relations source configured source/snapshot relations volume configured volume sections target configured targets usage print filesystem usage ls list all btrfs subvolumes below path origin print origin information for subvolume diff list file changes between related subvolumes extents [diff] calculate accurate disk space usage For additional information, see $PROJECT_HOME END_HELP_BTRBK } #80----------------------------------------------------------------------------- } sub _log_cont { my $p = shift; print STDERR $p . join("\n${p}... ", grep defined, @_) . "\n"; } sub TRACE { print STDERR map { "___ $_\n" } @_ if($loglevel >= 4) } sub DEBUG { _log_cont("", @_) if($loglevel >= 3) } sub INFO { _log_cont("", @_) if($loglevel >= 2) } sub WARN { _log_cont("WARNING: ", @_) if($loglevel >= 1) } sub ERROR { _log_cont("ERROR: ", @_) } sub INFO_ONCE { my $t = shift; if($warn_once{INFO}{$t}) { TRACE("INFO(again): $t", @_) if($do_trace); return 0; } else { $warn_once{INFO}{$t} = 1; INFO($t, @_); return 1; } } sub WARN_ONCE { my $t = shift; if($warn_once{WARN}{$t}) { TRACE("WARNING(again): $t", @_) if($do_trace); return 0; } else { $warn_once{WARN}{$t} = 1; WARN($t, @_); return 1; } } sub VINFO { return undef unless($do_dumper); my $vinfo = shift; my $t = shift || "vinfo"; my $maxdepth = shift // 2; print STDERR Data::Dumper->new([$vinfo], [$t])->Maxdepth($maxdepth)->Dump(); } sub SUBVOL_LIST { return undef unless($do_dumper); my $vol = shift; my $t = shift // "SUBVOL_LIST"; my $svl = vinfo_subvol_list($vol); print STDERR "$t:\n " . join("\n ", map { "$vol->{PRINT}/./$_->{SUBVOL_PATH}\t$_->{node}{id}" } @$svl) . "\n"; } sub ABORTED($$;$) { my $config = shift; my $abrt_key = shift // die; my $abrt = shift; $config = $config->{CONFIG} if($config->{CONFIG}); # accept vinfo for $config unless(defined($abrt)) { # no key (only text) set: switch arguments, use default key $abrt = $abrt_key; $abrt_key = "abort_" . $config->{CONTEXT}; } unless($abrt_key =~ /^skip_/) { # keys starting with "skip_" are not actions $abrt =~ s/\n/\\\\/g; $abrt =~ s/\r//g; action($abrt_key, status => "ABORT", vinfo_prefixed_keys("target", vinfo($config->{url}, $config)), message => $abrt, ); } $config->{ABORTED} = { key => $abrt_key, text => $abrt }; } sub IS_ABORTED($;$) { my $config = shift; $config = $config->{CONFIG} if($config->{CONFIG}); # accept vinfo for $config return undef unless(defined($config->{ABORTED})); my $abrt_key = $config->{ABORTED}->{key}; return undef unless(defined($abrt_key)); my $filter_prefix = shift; return undef if($filter_prefix && ($abrt_key !~ /^$filter_prefix/)); return $abrt_key; } sub ABORTED_TEXT($) { my $config = shift; $config = $config->{CONFIG} if($config->{CONFIG}); # accept vinfo for $config return "" unless(defined($config->{ABORTED})); return $config->{ABORTED}->{text} // ""; } sub FIX_MANUALLY($$) { # treated as error, but does not abort config section my $config = shift; $config = $config->{CONFIG} if($config->{CONFIG}); # accept vinfo for $config my $msg = shift // die; $config->{FIX_MANUALLY} //= []; push(@{$config->{FIX_MANUALLY}}, $msg); } sub eval_quiet(&) { local $SIG{__DIE__}; return eval { $_[0]->() } } sub require_data_dumper { if(eval_quiet { require Data::Dumper; }) { Data::Dumper->import("Dumper"); $Data::Dumper::Sortkeys = 1; $Data::Dumper::Quotekeys = 0; $do_dumper = 1; # silence perl warning: Name "Data::Dumper::Sortkeys" used only once: possible typo at... TRACE "Successfully loaded Dumper module: sortkeys=$Data::Dumper::Sortkeys, quotekeys=$Data::Dumper::Quotekeys" if($do_trace); } else { WARN "Perl module \"Data::Dumper\" not found: data trace dumps disabled!" if($do_trace); } } sub init_transaction_log($$) { my $file = shift; my $config_syslog_facility = shift; if(defined($file) && (not $dryrun)) { if(open($tlog_fh, '>>', $file)) { # print headers (disabled) # print_formatted("transaction", [ ], output_format => "tlog", outfile => $tlog_fh); INFO "Using transaction log: $file"; } else { $tlog_fh = undef; ERROR "Failed to open transaction log '$file': $!"; } } if(defined($config_syslog_facility) && (not $dryrun)) { DEBUG "Opening syslog"; if(eval_quiet { require Sys::Syslog; }) { $syslog_enabled = 1; Sys::Syslog::openlog("btrbk", "", $config_syslog_facility); DEBUG "Syslog enabled"; } else { WARN "Syslog disabled: $@"; } } action("DEFERRED", %$_) foreach (@transaction_log); } sub close_transaction_log() { if($tlog_fh) { DEBUG "Closing transaction log"; close $tlog_fh || ERROR "Failed to close transaction log: $!"; } if($syslog_enabled) { DEBUG "Closing syslog"; eval_quiet { Sys::Syslog::closelog(); }; } } sub action($@) { my $type = shift // die; my $h = { @_ }; unless($type eq "DEFERRED") { my $time = $h->{time} // time; $h->{type} = $type; $h->{time} = $time; $h->{localtime} = timestamp($time, 'debug-iso'); push @transaction_log, $h; } print_formatted("transaction", [ $h ], output_format => "tlog", no_header => 1, outfile => $tlog_fh) if($tlog_fh); print_formatted("transaction", [ $h ], output_format => "syslog", no_header => 1) if($syslog_enabled); # dirty hack, this calls syslog() return $h; } sub start_transaction($@) { my $type = shift // die; my $time = time; die("start_transaction() while transaction is running") if($current_transaction); my @actions = (ref($_[0]) eq "HASH") ? @_ : { @_ }; # single action is not hashref $current_transaction = []; foreach (@actions) { push @$current_transaction, action($type, %$_, status => ($dryrun ? "dryrun_starting" : "starting"), time => $time); } } sub end_transaction($$) { my $type = shift // die; my $success = shift; # scalar or coderef: if scalar, status is set for all current transitions my $time = time; die("end_transaction() while no transaction is running") unless($current_transaction); foreach (@$current_transaction) { die("end_transaction() has different type") unless($_->{type} eq $type); my $status = (ref($success) ? &{$success} ($_) : $success) ? "success" : "ERROR"; $status = "dryrun_" . $status if($dryrun); action($type, %$_, status => $status, time => $time, duration => ($dryrun ? undef : ($time - $_->{time}))); } $current_transaction = undef; } sub syslog($) { return undef unless($syslog_enabled); my $line = shift; eval_quiet { Sys::Syslog::syslog("info", $line); }; } sub check_exe($) { my $cmd = shift // die; foreach my $path (split(":", $ENV{PATH})) { return 1 if( -x "$path/$cmd" ); } return 0; } sub stream_buffer_cmd_text($) { my $opts = shift; my $rl_in = $opts->{rate_limit_in} // $opts->{rate_limit}; # maximum read rate: b,k,M,G my $rl_out = $opts->{rate_limit_out}; # maximum write rate: b,k,M,G my $bufsize = $opts->{stream_buffer}; # b,k,M,G,% (default: 2%) my $blocksize = $opts->{blocksize}; # defaults to 10k my $progress = $opts->{show_progress}; # return empty array if mbuffer is not needed return () unless($rl_in || $rl_out || $bufsize || $progress); # NOTE: mbuffer takes defaults from /etc/mbuffer.rc my @cmd = ( "mbuffer" ); push @cmd, ( "-v", "1" ); # disable warnings (they arrive asynchronously and cant be caught) push @cmd, "-q" unless($progress); push @cmd, ( "-s", $blocksize ) if($blocksize); push @cmd, ( "-m", lc($bufsize) ) if($bufsize); push @cmd, ( "-r", lc($rl_in) ) if($rl_in); push @cmd, ( "-R", lc($rl_out) ) if($rl_out); return { cmd_text => join(' ', @cmd) }; } sub compress_cmd_text($;$) { my $def = shift // die; my $decompress = shift; my $cc = $compression{$def->{key}}; my @cmd = $decompress ? @{$cc->{decompress_cmd}} : @{$cc->{compress_cmd}}; if((not $decompress) && defined($def->{level}) && ($def->{level} ne "default")) { my $level = $def->{level}; if($level < $cc->{level_min}) { WARN_ONCE "Compression level capped to minimum for '$cc->{name}': $cc->{level_min}"; $level = $cc->{level_min}; } if($level > $cc->{level_max}) { WARN_ONCE "Compression level capped to maximum for '$cc->{name}': $cc->{level_max}"; $level = $cc->{level_max}; } push @cmd, '-' . $level; } if(defined($def->{threads}) && ($def->{threads} ne "default")) { my $thread_opt = $cc->{threads}; if($thread_opt) { push @cmd, $thread_opt . $def->{threads}; } else { WARN_ONCE "Threading is not supported for '$cc->{name}', ignoring"; } } if(defined($def->{long}) && ($def->{long} ne "default")) { my $long_opt = $cc->{long}; if($long_opt) { push @cmd, $long_opt . $def->{long}; } else { WARN_ONCE "Long distance matching is not supported for '$cc->{name}', ignoring"; } } if(defined($def->{adapt})) { my $adapt_opt = $cc->{adapt}; if($adapt_opt) { push @cmd, $adapt_opt; } else { WARN_ONCE "Adaptive compression is not supported for '$cc->{name}', ignoring"; } } return { cmd_text => join(' ', @cmd) }; } sub decompress_cmd_text($) { return compress_cmd_text($_[0], 1); } sub _piped_cmd_txt($) { my $cmd_pipe = shift; my $cmd = ""; my $pipe = ""; my $last; foreach (map $_->{cmd_text}, @$cmd_pipe) { die if($last); if(/^>/) { # can't be first, must be last die unless($pipe); $last = 1; $pipe = ' '; } $cmd .= $pipe . $_; $pipe = ' | '; } return $cmd; } sub quoteshell(@) { # replace ' -> '\'' join ' ', map { "'" . s/'/'\\''/gr . "'" } @_ } sub _safe_cmd($;$) { # hashes of form: "{ unsafe => 'string' }" get translated to "'string'" my $aref = shift; my $offending = shift; return join ' ', map { if(ref($_)) { my $prefix = $_->{prefix} // ""; my $postfix = $_->{postfix} // ""; $_ = $_->{unsafe}; die "cannot quote leading dash for command: $_" if(/^-/); # NOTE: all files must be absolute if($offending) { push @$offending, $_ unless(defined(check_file($_, { absolute => 1 }))); push @$offending, $_ unless(!$safe_commands || /^($safe_file_match)$/); } $_ = $prefix . quoteshell($_) . $postfix; } $_ } @$aref; } sub run_cmd(@) { # IPC::Open3 based implementation. # NOTE: multiple filters are not supported! my @cmd_pipe_in = (ref($_[0]) eq "HASH") ? @_ : { @_ }; die unless(scalar(@cmd_pipe_in)); @stderr = (); my $destructive = 0; my @cmd_pipe; my @unsafe_cmd; my $compressed = undef; my $large_output; my $stream_options = $cmd_pipe_in[0]->{stream_options} // {}; my @filter_stderr; my $fatal_stderr; my $has_rsh; $cmd_pipe_in[0]->{stream_source} = 1; $cmd_pipe_in[-1]->{stream_sink} = 1; foreach my $href (@cmd_pipe_in) { die if(defined($href->{cmd_text})); push @filter_stderr, ((ref($href->{filter_stderr}) eq "ARRAY") ? @{$href->{filter_stderr}} : $href->{filter_stderr}) if($href->{filter_stderr}); $fatal_stderr = $href->{fatal_stderr} if($href->{fatal_stderr}); $destructive = 1 unless($href->{non_destructive}); $has_rsh = $href->{rsh} if($href->{rsh}); $large_output = 1 if($href->{large_output}); if($href->{redirect_to_file}) { die unless($href->{stream_sink}); $href->{cmd_text} = _safe_cmd([ '>', $href->{redirect_to_file} ], \@unsafe_cmd); } elsif($href->{append_to_file}) { die unless($href->{stream_sink}); $href->{cmd_text} = _safe_cmd([ '>>', $href->{append_to_file} ], \@unsafe_cmd); } elsif($href->{compress_stdin}) { # does nothing if already compressed correctly by stream_compress if($compressed && ($compression{$compressed->{key}}->{format} ne $compression{$href->{compress_stdin}->{key}}->{format})) { # re-compress with different algorithm push @cmd_pipe, decompress_cmd_text($compressed); $compressed = undef; } unless($compressed) { push @cmd_pipe, compress_cmd_text($href->{compress_stdin}); $compressed = $href->{compress_stdin}; } next; } elsif($href->{cmd}) { $href->{cmd_text} = _safe_cmd($href->{cmd}, \@unsafe_cmd); } return undef unless(defined($href->{cmd_text})); my @rsh_compress_in; my @rsh_compress_out; my @decompress_in; # input stream compression: local, in front of rsh_cmd_pipe if($href->{rsh} && $stream_options->{stream_compress} && (not $href->{stream_source})) { if($compressed && ($compression{$compressed->{key}}->{format} ne $compression{$stream_options->{stream_compress}->{key}}->{format})) { # re-compress with different algorithm, should be avoided! push @rsh_compress_in, decompress_cmd_text($compressed); $compressed = undef; } if(not $compressed) { $compressed = $stream_options->{stream_compress}; push @rsh_compress_in, compress_cmd_text($compressed); } } if($compressed && (not ($href->{compressed_ok}))) { push @decompress_in, decompress_cmd_text($compressed); $compressed = undef; } # output stream compression: remote, at end of rsh_cmd_pipe if($href->{rsh} && $stream_options->{stream_compress} && (not $href->{stream_sink}) && (not $compressed)) { $compressed = $stream_options->{stream_compress}; push @rsh_compress_out, compress_cmd_text($compressed); } if($href->{rsh}) { # honor stream_buffer_remote, rate_limit_remote for stream source / sink my @rsh_stream_buffer_in = $href->{stream_sink} ? stream_buffer_cmd_text($stream_options->{rsh_sink}) : (); my @rsh_stream_buffer_out = $href->{stream_source} ? stream_buffer_cmd_text($stream_options->{rsh_source}) : (); my @rsh_cmd_pipe = ( @decompress_in, @rsh_stream_buffer_in, $href, @rsh_stream_buffer_out, @rsh_compress_out, ); @decompress_in = (); # fixup redirect_to_file if((scalar(@rsh_cmd_pipe) == 1) && ($rsh_cmd_pipe[0]->{redirect_to_file} || $rsh_cmd_pipe[0]->{append_to_file})) { # NOTE: direct redirection in ssh command does not work: "ssh '> outfile'" # we need to assemble: "ssh 'cat > outfile'" unshift @rsh_cmd_pipe, { cmd_text => 'cat' }; } my $rsh_text = _safe_cmd($href->{rsh}, \@unsafe_cmd); return undef unless(defined($rsh_text)); $href->{cmd_text} = $rsh_text . ' ' . quoteshell(_piped_cmd_txt(\@rsh_cmd_pipe)); } # local stream_buffer, rate_limit and show_progress in front of stream sink my @stream_buffer_in = $href->{stream_sink} ? stream_buffer_cmd_text($stream_options->{local_sink}) : (); push @cmd_pipe, ( @decompress_in, # empty if rsh @stream_buffer_in, @rsh_compress_in, # empty if not rsh $href, # command or rsh_cmd_pipe ); } my $cmd = _piped_cmd_txt(\@cmd_pipe); if(scalar(@unsafe_cmd)) { ERROR "Unsafe command `$cmd`", map "Offending string: \"$_\"", @unsafe_cmd; return undef; } if($dryrun && $destructive) { DEBUG "### (dryrun) $cmd"; return []; } DEBUG "### $cmd"; # execute command my ($pid, $out_fh, $err_fh, @stdout); $err_fh = gensym; if(eval_quiet { $pid = open3(undef, $out_fh, $err_fh, $cmd); }) { chomp(@stdout = readline($out_fh)); chomp(@stderr = readline($err_fh)); waitpid($pid, 0); if($do_trace) { if($large_output) { TRACE "Command output lines=" . scalar(@stdout) . " (large_output, not dumped)"; } else { TRACE map("[stdout] $_", @stdout); } TRACE map("[stderr] $_", @stderr); } } else { ERROR "Command execution failed ($!): `$cmd`"; return undef; } # fatal errors if($? == -1) { ERROR "Command execution failed ($!): `$cmd`"; return undef; } elsif ($? & 127) { my $signal = $? & 127; ERROR "Command execution failed (child died with signal $signal): `$cmd`"; return undef; } my $exitcode = $? >> 8; # call hooks: fatal_stderr, filter_stderr if(($exitcode == 0) && $fatal_stderr) { $exitcode = -1 if(grep &{$fatal_stderr}(), @stderr); } foreach my $filter_fn (@filter_stderr) { @stderr = map { &{$filter_fn} ($exitcode); $_ // () } @stderr; } if($exitcode) { unshift @stderr, "sh: $cmd"; if($has_rsh && ($exitcode == 255)) { # SSH returns exit status 255 if an error occurred (including # network errors, dns failures). unshift @stderr, "(note: option \"ssh_identity\" is not set, using ssh defaults)" unless(grep /^-i$/, @$has_rsh); unshift @stderr, "SSH command failed (exitcode=$exitcode)"; } else { unshift @stderr, "Command execution failed (exitcode=$exitcode)"; } DEBUG @stderr; return undef; } else { DEBUG "Command execution successful"; } return \@stdout; } sub _btrfs_filter_stderr { if(/^usage: / || /(unrecognized|invalid) option/) { WARN_ONCE "Using unsupported btrfs-progs < v$BTRFS_PROGS_MIN"; } # strip error prefix (we print our own) # note that this also affects ssh_filter_btrbk.sh error strings s/^ERROR: //; } sub btrfs_filesystem_show($) { my $vol = shift || die; my $path = $vol->{PATH} // die; return run_cmd( cmd => vinfo_cmd($vol, "btrfs filesystem show", { unsafe => $path } ), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, ); } sub btrfs_filesystem_df($) { my $vol = shift || die; my $path = $vol->{PATH} // die; return run_cmd( cmd => vinfo_cmd($vol, "btrfs filesystem df", { unsafe => $path }), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, ); } sub btrfs_filesystem_usage($) { my $vol = shift || die; my $path = $vol->{PATH} // die; my $ret = run_cmd( cmd => vinfo_cmd($vol, "btrfs filesystem usage", { unsafe => $path } ), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, ); unless(defined($ret)) { ERROR "Failed to fetch btrfs filesystem usage for: $vol->{PRINT}", @stderr; return undef; } return undef unless(defined($ret)); my %detail; foreach(@$ret) { $detail{device_size} = $1, next if(/^\s+Device size:\s+(\S+)/); $detail{device_allocated} = $1, next if(/^\s+Device allocated:\s+(\S+)/); $detail{device_unallocated} = $1, next if(/^\s+Device unallocated:\s+(\S+)/); $detail{device_missing} = $1, next if(/^\s+Device missing:\s+(\S+)/); $detail{device_used} = $1, next if(/^\s+Used:\s+(\S+)/); @detail{qw(free free_min)} = ($1,$2), next if(/^\s+Free \(estimated\):\s+(\S+)\s+\(min: (\S+)\)/); $detail{data_ratio} = $1, next if(/^\s+Data ratio:\s+([0-9]+\.[0-9]+)/); $detail{metadata_ratio} = $1, next if(/^\s+Metadata ratio:\s+([0-9]+\.[0-9]+)/); $detail{used} = $1, next if(/^\s+Used:\s+(\S+)/); @detail{qw(global_reserve global_reserve_used)} = ($1,$2), next if(/^\s+Global reserve:\s+(\S+)\s+\(used: (\S+)\)/); TRACE "Failed to parse filesystem usage line \"$_\" for: $vol->{PRINT}" if($do_trace); } DEBUG "Parsed " . scalar(keys %detail) . " filesystem usage detail items: $vol->{PRINT}"; foreach (qw(device_size device_used data_ratio)) { unless(defined($detail{$_})) { ERROR "Failed to parse filesystem usage detail (unsupported btrfs-progs) for: $vol->{PRINT}"; return undef; } } # calculate aggregate size / usage if($detail{device_size} =~ /^([0-9]+\.[0-9]+)(.*)/) { $detail{size} = sprintf('%.2f%s', $1 / $detail{data_ratio}, $2); } if($detail{device_used} =~ /^([0-9]+\.[0-9]+)(.*)/) { $detail{used} = sprintf('%.2f%s', $1 / $detail{data_ratio}, $2); } TRACE(Data::Dumper->Dump([\%detail], ["btrfs_filesystem_usage($vol->{URL})"])) if($do_trace && $do_dumper); return \%detail; } # returns hashref with keys: (uuid parent_uuid id gen cgen top_level) # for btrfs root, returns at least: (id is_root) # for btrfs-progs >= 4.1, also returns key: "received_uuid" # if present, also returns (unvalidated) keys: (name creation_time flags) sub btrfs_subvolume_show($;@) { my $vol = shift || die; my %opts = @_; my @cmd_options; push(@cmd_options, '--rootid=' . $opts{rootid}) if($opts{rootid}); # btrfs-progs >= 4.12 my $path = $vol->{PATH} // die; my $ret = run_cmd(cmd => vinfo_cmd($vol, "btrfs subvolume show", @cmd_options, { unsafe => $path }), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, ); return undef unless(defined($ret)); unless(scalar(@$ret)) { ERROR "Failed to parse subvolume detail (unsupported btrfs-progs) for: $vol->{PRINT}"; return undef; } # NOTE: the first line starts with a path: # - btrfs-progs < 4.12 prints the full (absolute, resolved) path # - btrfs-progs >= 4.12 prints the relative path to btrfs root (or "/" if it is the root) my %detail; if($ret->[0] =~ / is (btrfs root|toplevel subvolume)$/) { # btrfs-progs < 4.4 prints: " is btrfs root" # btrfs-progs >= 4.4 prints: " is toplevel subvolume" # btrfs-progs >= 4.8.3 does not enter here, as output shares format with regular subvolumes $detail{id} = 5; } else { my %trans = ( "Name" => "name", "uuid" => "uuid", "UUID" => "uuid", # btrfs-progs >= 4.1 "Parent uuid" => "parent_uuid", "Parent UUID" => "parent_uuid", # btrfs-progs >= 4.1 "Received UUID" => "received_uuid", # btrfs-progs >= 4.1 "Creation time" => "creation_time", "Object ID" => "id", "Subvolume ID" => "id", # btrfs-progs >= 4.1 "Generation (Gen)" => "gen", "Generation" => "gen", # btrfs-progs >= 4.1 "Gen at creation" => "cgen", "Parent" => "parent_id", "Parent ID" => "parent_id", # btrfs-progs >= 4.1 "Top Level" => "top_level", "Top level ID" => "top_level", # btrfs-progs >= 4.1 "Flags" => "flags", "Send transid" => "send_transid", # btrfs-progs >= 5.14.2 "Send time" => "send_time", # btrfs-progs >= 5.14.2 "Receive transid" => "receive_transid", # btrfs-progs >= 5.14.2 "Receive time" => "receive_time", # btrfs-progs >= 5.14.2 ); foreach(@$ret) { next unless /^\s+(.+):\s+(.*)$/; my ($key, $value) = ($1, $2); if($trans{$key}) { $detail{$trans{$key}} = $value; } else { DEBUG "Ignoring subvolume detail \"$key: $value\" for: $vol->{PRINT}"; } } DEBUG "Parsed " . scalar(keys %detail) . " subvolume detail items: $vol->{PRINT}"; # NOTE: as of btrfs-progs v4.6.1, flags are either "-" or "readonly" $detail{readonly} = ($detail{flags} =~ /readonly/) ? 1 : 0 if($detail{flags}); # validate required keys unless((defined($detail{parent_uuid}) && (($detail{parent_uuid} eq '-') || ($detail{parent_uuid} =~ /^$uuid_match$/))) && (defined($detail{id}) && ($detail{id} =~ /^\d+$/) && ($detail{id} >= 5)) && (defined($detail{gen}) && ($detail{gen} =~ /^\d+$/)) && (defined($detail{cgen}) && ($detail{cgen} =~ /^\d+$/)) && (defined($detail{top_level}) && ($detail{top_level} =~ /^\d+$/)) && (defined($detail{readonly}))) { ERROR "Failed to parse subvolume detail (unsupported btrfs-progs) for: $vol->{PRINT}"; return undef; } # NOTE: filesystems created with btrfs-progs < 4.16 have no UUID for subvolid=5, # assert {uuid} is either valid or undef if(defined($detail{uuid}) && ($detail{uuid} !~ /^$uuid_match$/)) { if($detail{id} == 5) { DEBUG "No UUID on btrfs root (id=5): $vol->{PRINT}"; } else { ERROR "Failed to parse subvolume detail (unsupported btrfs-progs) for: $vol->{PRINT}"; return undef; } delete $detail{uuid}; } # NOTE: received_uuid is not required here, as btrfs-progs < 4.1 does not give us that information. # no worries, we get this from btrfs_subvolume_list() for all subvols. if(defined($detail{received_uuid}) && ($detail{received_uuid} ne '-') && ($detail{received_uuid} !~ /^$uuid_match$/)) { ERROR "Failed to parse subvolume detail (unsupported btrfs-progs) for: $vol->{PRINT}"; return undef; } VINFO(\%detail, "detail") if($loglevel >=4); } if($opts{rootid} && ($detail{id} != $opts{rootid})) { ERROR "Failed to parse subvolume detail (rootid mismatch) for: $vol->{PRINT}"; return undef; } if($detail{id} == 5) { DEBUG "Found btrfs root: $vol->{PRINT}"; $detail{is_root} = 1; } return \%detail; } sub btrfs_subvolume_list_readonly_flag($) { my $vol = shift || die; my $path = $vol->{PATH} // die; my $ret = run_cmd(cmd => vinfo_cmd($vol, "btrfs subvolume list", '-a', '-r', { unsafe => $path } ), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, ); return undef unless(defined($ret)); my %ro; foreach(@$ret) { unless(/^ID\s+([0-9]+)\s+gen\s+[0-9]+\s+top level\s+[0-9]+\s+path\s/) { ERROR "Failed to parse subvolume list (unsupported btrfs-progs) for: $vol->{PRINT}"; DEBUG "Offending line: $_"; return undef; } $ro{$1} = 1; } DEBUG "Parsed " . scalar(keys %ro) . " readonly subvolumes for filesystem at: $vol->{PRINT}"; return \%ro; } sub btrfs_subvolume_list($;@) { my $vol = shift || die; my %opts = @_; my $path = $vol->{PATH} // die; my @filter_options = ('-a'); push(@filter_options, '-o') if($opts{subvol_only}); push(@filter_options, '-d') if($opts{deleted_only}); # NOTE: btrfs-progs <= 3.17 do NOT support the '-R' flag. # NOTE: Support for btrfs-progs <= 3.17 has been dropped in # btrbk-0.23, the received_uuid flag very essential! my @display_options = ('-c', '-u', '-q', '-R'); my $ret = run_cmd(cmd => vinfo_cmd($vol, "btrfs subvolume list", @filter_options, @display_options, { unsafe => $path } ), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, ); return undef unless(defined($ret)); my @nodes; foreach(@$ret) { my %node; # NOTE: btrfs-progs >= 4.13.2 pads uuid's with 36 whitespaces unless(/^ID \s+ ([0-9]+) \s+ gen \s+ ([0-9]+) \s+ cgen \s+ ([0-9]+) \s+ top\ level \s+ ([0-9]+) \s+ parent_uuid \s+ ([0-9a-f-]+) \s+ received_uuid \s+ ([0-9a-f-]+) \s+ uuid \s+ ([0-9a-f-]+) \s+ path \s+ (.+) $/x) { ERROR "Failed to parse subvolume list (unsupported btrfs-progs) for: $vol->{PRINT}"; DEBUG "Offending line: $_"; return undef; } %node = ( id => $1, gen => $2, cgen => $3, top_level => $4, parent_uuid => $5, # note: parent_uuid="-" if no parent received_uuid => $6, uuid => $7, path => $8 # btrfs path, NOT filesystem path ); # NOTE: "btrfs subvolume list " prints prefix only if # the subvolume is reachable within . (as of btrfs-progs-3.18.2) # # NOTE: Be prepared for this to change in btrfs-progs! $node{path} =~ s/^\///; # remove "/" portion from "path". push @nodes, \%node; } DEBUG "Parsed " . scalar(@nodes) . " total subvolumes for filesystem at: $vol->{PRINT}"; return \@nodes; } sub btrfs_subvolume_list_complete($) { my $vol = shift || die; # fetch subvolume list my $nodes = btrfs_subvolume_list($vol); return undef unless($nodes); # fetch readonly flag # NOTE: the only way to get "readonly" flag is via a second call to "btrfs subvol list" with the "-r" option (as of btrfs-progs v4.3.1) my $ro = btrfs_subvolume_list_readonly_flag($vol); return undef unless(defined($ro)); foreach (@$nodes) { $_->{readonly} = $ro->{$_->{id}} // 0; } # btrfs root (id=5) is not provided by btrfs_subvolume_list above, read it separately (best-efford) my $tree_root = btrfs_subvolume_show($vol, rootid => 5); unless($tree_root) { # this is not an error: # - btrfs-progs < 4.12 does not support rootid lookup # - UUID can be missing if filesystem was created with btrfs-progs < 4.16 DEBUG "Failed to fetch subvolume detail (old btrfs-progs?) for btrfs root (id=5) on: $vol->{PRINT}"; $tree_root = { id => 5, is_root => 1 }; } unshift(@$nodes, $tree_root); return $nodes; } sub btrfs_subvolume_find_new($$;$) { my $vol = shift || die; my $path = $vol->{PATH} // die; my $lastgen = shift // die; my $ret = run_cmd(cmd => vinfo_cmd($vol, "btrfs subvolume find-new", { unsafe => $path }, $lastgen ), rsh => vinfo_rsh($vol), non_destructive => 1, filter_stderr => \&_btrfs_filter_stderr, large_output => 1, ); unless(defined($ret)) { ERROR "Failed to fetch modified files for: $vol->{PRINT}", @stderr; return undef; } my %files; my $parse_errors = 0; my $transid_marker; foreach(@$ret) { if(/^inode \S+ file offset (\S+) len (\S+) disk start \S+ offset \S+ gen (\S+) flags (\S+) (.+)$/) { my $file_offset = $1; my $len = $2; my $gen = $3; my $flags = $4; my $name = $5; $files{$name}->{len} += $len; $files{$name}->{new} = 1 if($file_offset == 0); $files{$name}->{gen}->{$gen} = 1; # count the generations if($flags ne "NONE") { $files{$name}->{flags}{$_} = 1 foreach split(/\|/, $flags); } } elsif(/^transid marker was (\S+)$/) { $transid_marker = $1; } else { ERROR "Failed to parse output from `btrfs subvolume find-new`:", $_; $parse_errors++; } } ERROR "Failed to parse $parse_errors lines from `btrfs subvolume find-new`" if($parse_errors); return { files => \%files, transid_marker => $transid_marker, parse_errors => $parse_errors, }; } # returns $target, or undef on error sub btrfs_subvolume_snapshot($$) { my $svol = shift || die; my $target_vol = shift // die; my $target_path = $target_vol->{PATH} // die; my $src_path = $svol->{PATH} // die; INFO "[snapshot] source: $svol->{PRINT}"; INFO "[snapshot] target: $target_vol->{PRINT}"; start_transaction("snapshot", vinfo_prefixed_keys("target", $target_vol), vinfo_prefixed_keys("source", $svol), ); my $ret = run_cmd(cmd => vinfo_cmd($svol, "btrfs subvolume snapshot", '-r', { unsafe => $src_path }, { unsafe => $target_path } ), rsh => vinfo_rsh($svol), filter_stderr => \&_btrfs_filter_stderr, ); end_transaction("snapshot", defined($ret)); unless(defined($ret)) { ERROR "Failed to create snapshot: $svol->{PRINT} -> $target_path", @stderr; return undef; } return $target_vol; } sub btrfs_subvolume_delete($@) { my $vol = shift // die; my %opts = @_; my $target_type = $vol->{node}{TARGET_TYPE} || ""; my $ret; INFO "[delete] target: $vol->{PRINT}"; start_transaction($opts{type} // "delete", vinfo_prefixed_keys("target", $vol)); if($target_type eq "raw") { $ret = run_cmd(cmd => [ 'rm', '-f', { unsafe => $vol->{PATH}, postfix => ($vol->{node}{BTRBK_RAW}{split} && ".split_??") }, { unsafe => $vol->{PATH}, postfix => ".info" }, ], rsh => vinfo_rsh($vol), ); } else { my @options; push @options, "--commit-each" if($opts{commit}); $ret = run_cmd(cmd => vinfo_cmd($vol, "btrfs subvolume delete", @options, { unsafe => $vol->{PATH} } ), rsh => vinfo_rsh($vol), fatal_stderr => sub { m/^ERROR: /; }, # probably not needed, "btrfs sub delete" returns correct exit status filter_stderr => \&_btrfs_filter_stderr, ); } end_transaction($opts{type} // "delete", defined($ret)); unless(defined($ret)) { ERROR "Failed to delete subvolume: $vol->{PRINT}", @stderr; return undef; } return $vol; } sub btrfs_qgroup_destroy($@) { my $vol = shift // die; my %opts = @_; my $vol_id = $vol->{node}{id}; unless($vol_id) { ERROR "Unknown subvolume_id for: $vol->{PRINT}"; return undef; } my $path = $vol->{PATH} // die; my $qgroup_id = "0/$vol_id"; INFO "[qgroup-destroy] qgroup_id: $qgroup_id"; INFO "[qgroup-destroy] subvolume: $vol->{PRINT}"; start_transaction($opts{type} // "qgroup_destroy", vinfo_prefixed_keys("target", $vol)); my $ret = run_cmd(cmd => vinfo_cmd($vol, "btrfs qgroup destroy", $qgroup_id, { unsafe => $path }), rsh => vinfo_rsh($vol), filter_stderr => \&_btrfs_filter_stderr, ); end_transaction($opts{type} // "qgroup_destroy", defined($ret)); unless(defined($ret)) { ERROR "Failed to destroy qgroup \"$qgroup_id\" for subvolume: $vol->{PRINT}", @stderr; return undef; } return $vol; } sub btrfs_send_receive($$;$$$) { my $snapshot = shift || die; my $target = shift || die; my $parent = shift; my $clone_src = shift // []; my $ret_vol_received = shift; my $snapshot_path = $snapshot->{PATH} // die; my $target_path = $target->{PATH} // die; my $parent_path = $parent ? $parent->{PATH} : undef; my $vol_received = vinfo_child($target, $snapshot->{NAME}); $$ret_vol_received = $vol_received if(ref $ret_vol_received); print STDOUT "Creating backup: $vol_received->{PRINT}\n" if($show_progress && (not $dryrun)); INFO "[send/receive] target: $vol_received->{PRINT}"; INFO "[send/receive] source: $snapshot->{PRINT}"; INFO "[send/receive] parent: $parent->{PRINT}" if($parent); INFO "[send/receive] clone-src: $_->{PRINT}" foreach(@$clone_src); my $stream_options = config_stream_hash($snapshot, $target); my $compat_ignore_err = config_key_lru($target, "compat", "ignore_receive_errors"); my @send_options; my @receive_options; push(@send_options, '-p', { unsafe => $parent_path} ) if($parent_path); push(@send_options, '-c', { unsafe => $_ } ) foreach(map { $_->{PATH} } @$clone_src); # push(@send_options, '-v') if($loglevel >= 3); # push(@receive_options, '-v') if($loglevel >= 3); push(@receive_options, '--max-errors=0') if($compat_ignore_err); my @cmd_pipe; push @cmd_pipe, { cmd => vinfo_cmd($snapshot, "btrfs send", @send_options, { unsafe => $snapshot_path } ), rsh => vinfo_rsh($snapshot, disable_compression => $stream_options->{stream_compress}), stream_options => $stream_options, filter_stderr => [ \&_btrfs_filter_stderr, sub { $_ = undef if(/^At subvol/) } ], }; push @cmd_pipe, { cmd => vinfo_cmd($target, "btrfs receive", @receive_options, { unsafe => $target_path . '/' } ), rsh => vinfo_rsh($target, disable_compression => $stream_options->{stream_compress}), fatal_stderr => sub { # NOTE: btrfs-progs < 4.11: if "btrfs send" fails, "btrfs receive" returns 0! if($compat_ignore_err && s/^ERROR: (.*)//) { WARN "Ignoring btrfs receive error (compat=ignore_receive_errors): $1"; } m/^ERROR: /; }, }; my $send_receive_error = 0; start_transaction("send-receive", vinfo_prefixed_keys("target", $vol_received), vinfo_prefixed_keys("source", $snapshot), vinfo_prefixed_keys("parent", $parent), ); my $ret = run_cmd(@cmd_pipe); my @cmd_err; unless(defined($ret)) { @cmd_err = @stderr; # save for later $send_receive_error = 1; } # Read in target subvolume metadata (btrfs subvolume show): # Double checking the output increases robustness against exotic # revisions of external commands (btrfs-progs, pv, xz, lz4, ...). # # NOTE: we cannot rely on the underlying shell to have # "pipefail" functionality. # # NOTE: btrfs-progs < 4.11: # "cat /dev/null | btrfs receive" returns with exitcode=0 and no # error message, having the effect that silently no subvolume is # created if any command in @cmd_pipe fail. my $is_garbled; if($dryrun) { INFO "[send/receive] (dryrun, skip) checking target metadata: $vol_received->{PRINT}"; } else { INFO "[send/receive] checking target metadata: $vol_received->{PRINT}"; my $detail = btrfs_subvolume_show($vol_received); if(defined($detail)) { unless($send_receive_error) { # plausibility checks on target detail unless($detail->{readonly}) { push @cmd_err, "target is not readonly: $vol_received->{PRINT}"; $send_receive_error = 1; } if($detail->{received_uuid} && ($detail->{received_uuid} eq '-')) { # NOTE: received_uuid is not in @required_keys (needs btrfs-progs >= 4.1 (BTRFS_PROGS_MIN)) # so we only check it if it's really present push @cmd_err, "received_uuid is not set on target: $vol_received->{PRINT}"; $send_receive_error = 1; } if($parent && ($detail->{parent_uuid} eq '-')) { push @cmd_err, "parent_uuid is not set on target: $vol_received->{PRINT}"; $send_receive_error = 1; } if((not $parent) && ($detail->{parent_uuid} ne '-')) { push @cmd_err, "parent_uuid is set on target: $vol_received->{PRINT}"; $send_receive_error = 1; } } # incomplete received (garbled) subvolumes are not readonly and have no received_uuid $is_garbled = ((not $detail->{readonly}) && defined($detail->{received_uuid}) && ($detail->{received_uuid} eq '-')); } else { push @cmd_err, "failed to check target subvolume: $vol_received->{PRINT}", @stderr; $send_receive_error = 1; } } end_transaction("send-receive", not $send_receive_error); if($send_receive_error) { ERROR "Failed to send/receive subvolume: $snapshot->{PRINT} " . ($parent_path ? "[$parent_path]" : "") . " -> $vol_received->{PRINT}", @cmd_err; } if($is_garbled) { # NOTE: btrfs-progs does not delete incomplete received (garbled) subvolumes, # we need to do this by hand. # TODO: remove this as soon as btrfs-progs handle receive errors correctly. if(btrfs_subvolume_delete($vol_received, commit => "after", type => "delete_garbled")) { WARN "Deleted partially received (garbled) subvolume: $vol_received->{PRINT}"; } else { WARN "Deletion of partially received (garbled) subvolume failed, assuming clean environment: $vol_received->{PRINT}"; } } return $send_receive_error ? undef : 1; } sub btrfs_send_to_file($$$;$$) { my $source = shift || die; my $target = shift || die; my $parent = shift; my $ret_vol_received = shift; my $ret_raw_info = shift; my $source_path = $source->{PATH} // die; my $target_path = $target->{PATH} // die; my $parent_path = $parent ? $parent->{PATH} : undef; my $parent_uuid = $parent ? $parent->{node}{uuid} : undef ; my $received_uuid = $source->{node}{uuid}; die unless($received_uuid); die if($parent && !$parent_uuid); # prepare raw_info (for vinfo_inject_child) my %raw_info = ( TYPE => 'raw', RECEIVED_UUID => $received_uuid, INCOMPLETE => 1, ); my $target_filename = $source->{NAME} || die; $target_filename .= ".btrfs"; my $compress = config_compress_hash($target, "raw_target_compress"); my $encrypt = config_encrypt_hash($target, "raw_target_encrypt"); my $split = config_key($target, "raw_target_split"); my $stream_options = config_stream_hash($source, $target); # make sure we dont re-compress, override "stream_compress" with "raw_target_compress" $stream_options->{stream_compress} = $compress if($compress); my @send_options; push(@send_options, '-p', { unsafe => $parent_path } ) if($parent_path); #push(@send_options, '-v') if($loglevel >= 3); my @cmd_pipe; push @cmd_pipe, { cmd => vinfo_cmd($source, "btrfs send", @send_options, { unsafe => $source_path } ), rsh => vinfo_rsh($source, disable_compression => $stream_options->{stream_compress}), stream_options => $stream_options, filter_stderr => [ \&_btrfs_filter_stderr, sub { $_ = undef if(/^At subvol/) } ], fatal_stderr => sub { m/^ERROR: /; }, }; if($compress) { $raw_info{compress} = $compression{$compress->{key}}->{format}; $target_filename .= '.' . $compression{$compress->{key}}->{format}; push @cmd_pipe, { compress_stdin => $compress }; # does nothing if already compressed by stream_compress } if($encrypt) { $target_filename .= ($encrypt->{type} eq "gpg") ? '.gpg' : '.encrypted'; } # NOTE: $ret_vol_received must always be set when function returns! my $vol_received = vinfo_child($target, $target_filename); $$ret_vol_received = $vol_received if(ref $ret_vol_received); if($encrypt) { $raw_info{encrypt} = $encrypt->{type}; if($encrypt->{type} eq "gpg") { # NOTE: We set "--no-random-seed-file" since one of the btrbk # design principles is to never create any files unasked. Enabling # "--no-random-seed-file" creates ~/.gnupg/random_seed, and as # such depends on $HOME to be set correctly (which e.g. is set to # "/" by some cron daemons). From gpg2(1) man page: # --no-random-seed-file GnuPG uses a file to store its # internal random pool over invocations This makes random # generation faster; however sometimes write operations are not # desired. This option can be used to achieve that with the cost # of slower random generation. my @gpg_options = ( '--batch', '--no-tty', '--no-random-seed-file', '--trust-model', 'always' ); push @gpg_options, ( '--compress-algo', 'none' ) if($compress); # NOTE: if --compress-algo is not set, gpg might still compress according to OpenPGP standard. push(@gpg_options, ( '--no-default-keyring', '--keyring', { unsafe => $encrypt->{keyring} } )) if($encrypt->{keyring}); if($encrypt->{recipient}) { push(@gpg_options, '--no-default-recipient'); push(@gpg_options, map +( '--recipient', $_ ), @{$encrypt->{recipient}}); } push @cmd_pipe, { cmd => [ 'gpg', @gpg_options, '--encrypt' ], compressed_ok => ($compress ? 1 : 0), }; } elsif($encrypt->{type} eq "openssl_enc") { # encrypt using "openssl enc" $raw_info{cipher} = $encrypt->{ciphername}; # NOTE: iv is always generated locally! my $iv_size = $encrypt->{iv_size}; my $iv; if($iv_size) { INFO "Generating iv for openssl encryption (cipher=$encrypt->{ciphername})"; $iv = system_urandom($iv_size, 'hex'); unless($iv) { ERROR "Failed generate IV for openssl_enc: $source->{PRINT}"; return undef; } $raw_info{iv} = $iv; } my $encrypt_key; if($encrypt->{keyfile}) { if($encrypt->{kdf_backend}) { WARN "Both openssl_keyfile and kdf_backend are configured, ignoring kdf_backend!"; } $encrypt_key = '$(cat ' . quoteshell($encrypt->{keyfile}) . ')'; } elsif($encrypt->{kdf_backend}) { if($encrypt->{kdf_keygen_each}) { $kdf_session_key = undef; %kdf_vars = (); } if($kdf_session_key) { INFO "Reusing session key for: $vol_received->{PRINT}"; } else { # run kdf backend, set session key and vars DEBUG "Generating session key for: $vol_received->{PRINT}"; my $key_target_text = $encrypt->{kdf_keygen_each} ? "\"$vol_received->{PRINT}\"" : "all raw backups"; print STDOUT "\nGenerate session key for $key_target_text:\n"; my $kdf_values = run_cmd(cmd => [ { unsafe => $encrypt->{kdf_backend} }, $encrypt->{kdf_keysize} ], non_destructive => 1, ); unless(defined($kdf_values)) { ERROR "Failed to generate session key for $key_target_text", @stderr; return undef; } return undef unless(defined($kdf_values)); foreach(@$kdf_values) { chomp; next if /^\s*$/; # ignore empty lines if(/^KEY=([0-9a-fA-f]+)/) { $kdf_session_key = $1; } elsif(/^([a-z_]+)=(.*)/) { my $info_key = 'kdf_' . $1; my $info_val = $2; DEBUG "Adding raw_info from kdf_backend: $info_key=$info_val"; $kdf_vars{$info_key} = $info_val; } else { ERROR "Ambiguous line from kdf_backend: $encrypt->{kdf_backend}"; return undef; } } unless($kdf_session_key && (length($kdf_session_key) == ($encrypt->{kdf_keysize} * 2))) { ERROR "Ambiguous key value from kdf_backend: $encrypt->{kdf_backend}"; return undef; } INFO "Generated session key for: $vol_received->{PRINT}"; } $encrypt_key = $kdf_session_key; %raw_info = ( %kdf_vars, %raw_info ); } my @openssl_options = ( '-' . $encrypt->{ciphername}, '-K', $encrypt_key, ); push @openssl_options, ('-iv', $iv) if($iv); push @cmd_pipe, { cmd => [ 'openssl', 'enc', '-e', @openssl_options ], compressed_ok => ($compress ? 1 : 0), }; } else { die "Usupported encryption type (raw_target_encrypt)"; } } if($split) { # NOTE: we do not append a ".split" suffix on $target_filename here, as this propagates to ".info" file $raw_info{split} = $split; push @cmd_pipe, { cmd => [ 'split', '-b', uc($split), '-', { unsafe => "${target_path}/${target_filename}.split_" } ], rsh => vinfo_rsh($target, disable_compression => $stream_options->{stream_compress}), compressed_ok => ($compress ? 1 : 0), } } else { push @cmd_pipe, { # NOTE: We use "dd" instead of shell redirections here, as it is # common to have special filesystems (like NFS, SMB, FUSE) mounted # on $target_path. By using "dd" we make sure to write in # reasonably large blocks (default=128K), which is not always the # case when using redirections (e.g. "gpg > outfile" writes in 8K # blocks). # Another approach would be to always pipe through "cat", which # uses st_blksize from fstat(2) (with a minimum of 128K) to # determine the block size. cmd => [ 'dd', 'status=none', 'bs=' . config_key($target, "raw_target_block_size"), { prefix => "of=", unsafe => "${target_path}/${target_filename}" } ], #redirect_to_file => { unsafe => "${target_path}/${target_filename}" }, # alternative (use shell redirection), less overhead on local filesystems (barely measurable): rsh => vinfo_rsh($target, disable_compression => $stream_options->{stream_compress}), compressed_ok => ($compress ? 1 : 0), }; } $raw_info{FILE} = $target_filename; $raw_info{RECEIVED_PARENT_UUID} = $parent_uuid if($parent_uuid); # disabled for now, as its not very useful and might leak information: # $raw_info{parent_url} = $parent->{URL} if($parent); # $raw_info{target_url} = $vol_received->{URL}; $$ret_raw_info = \%raw_info if($ret_raw_info); print STDOUT "Creating raw backup: $vol_received->{PRINT}\n" if($show_progress && (not $dryrun)); INFO "[send-to-raw] target: $vol_received->{PRINT}"; INFO "[send-to-raw] source: $source->{PRINT}"; INFO "[send-to-raw] parent: $parent->{PRINT}" if($parent); start_transaction("send-to-raw", vinfo_prefixed_keys("target", $vol_received), vinfo_prefixed_keys("source", $source), vinfo_prefixed_keys("parent", $parent), ); my $ret; $ret = system_write_raw_info($vol_received, \%raw_info); my @cmd_err; if(defined($ret)) { $ret = run_cmd(@cmd_pipe); @cmd_err = @stderr unless(defined($ret)); # save for later } else { push @cmd_err, "failed to write raw .info file: $vol_received->{PATH}.info", @stderr; } if(defined($ret)) { # Test target file for "exists and size > 0" after writing, as we # can not rely on the exit status of the command pipe, and a shell # redirection as well as "dd" always creates the target file. # Note that "split" does not create empty files. my $test_postfix = ($split ? ".split_aa" : ""); my $check_file = "${target_path}/${target_filename}${test_postfix}"; DEBUG "Testing target data file (non-zero size): $check_file"; $ret = run_cmd(cmd => [ 'test', '-s', { unsafe => $check_file } ], rsh => vinfo_rsh($target), ); if(defined($ret)) { delete $raw_info{INCOMPLETE}; $ret = system_write_raw_info($vol_received, { INCOMPLETE => 0 }, append => 1); } else { push @cmd_err, "failed to check target file (not present or zero length): $check_file"; } } end_transaction("send-to-raw", defined($ret)); unless(defined($ret)) { ERROR "Failed to send btrfs subvolume to raw file: $source->{PRINT} " . ($parent_path ? "[$parent_path]" : "") . " -> $vol_received->{PRINT}", @cmd_err; return undef; } return 1; } sub system_list_mountinfo($) { my $vol = shift // die; my $file = '/proc/self/mountinfo'; # NOTE: /proc/self/mounts is deprecated my $ret = run_cmd(cmd => [ 'cat', $file ], rsh => vinfo_rsh($vol), non_destructive => 1, ); return undef unless(defined($ret)); unless(@$ret) { ERROR "Failed to parse \"$vol->{URL_PREFIX}$file\": no output"; return undef; } my @mountinfo; foreach(@$ret) { # https://www.kernel.org/doc/Documentation/filesystems/proc.txt unless(/^(?[0-9]+) # mount ID: unique identifier of the mount (may be reused after umount) \s(?[0-9]+) # parent ID: ID of parent (or of self for the top of the mount tree) \s(?[0-9]+:[0-9]+) # major:minor: value of st_dev for files on filesystem \s(?\S+) # root: root of the mount within the filesystem \s(?\S+) # mount point: mount point relative to the process's root \s(?\S+) # mount options: per mount options (\s\S+)* # optional fields: zero or more fields of the form "tag[:value]" \s- # separator: marks the end of the optional fields \s(?\S+) # filesystem type: name of filesystem of the form "type[.subtype]" \s(?\S+) # mount source: filesystem specific information or "none" \s(?\S+)$ # super options: per super block options /x) { ERROR "Failed to parse \"$vol->{URL_PREFIX}$file\""; DEBUG "Offending line: $_"; return undef; } my %line = %+; unless(defined(check_file($line{mount_point}, { absolute => 1 }))) { ERROR "Ambiguous mount point in \"$vol->{URL_PREFIX}$file\": $line{mount_point}"; return undef; } # merge super_options and mount_options to MNTOPS. my %mntops; foreach (split(',', delete($line{super_options})), split(',', delete($line{mount_options}))) { if(/^(.+?)=(.+)$/) { $mntops{$1} = $2; } else { $mntops{$_} = 1; } } $mntops{rw} = 0 if($mntops{ro}); # e.g. mount_options="ro", super_options="rw" # decode values (octal, e.g. "\040" = whitespace) s/\\([0-7]{3})/chr(oct($1))/eg foreach(values %line, values %mntops); $line{MNTOPS} = \%mntops; push @mountinfo, \%line; } # TRACE(Data::Dumper->Dump([\@mountinfo], ["mountinfo"])) if($do_trace && $do_dumper); return \@mountinfo; } sub system_testdir($) { my $vol = shift // die; my $path = $vol->{PATH} // die; my $ret = run_cmd(cmd => vinfo_cmd($vol, "test", '-d', { unsafe => $path } ), rsh => vinfo_rsh($vol), non_destructive => 1, ); return undef unless(defined($ret)); DEBUG "Directory exists: $vol->{PRINT}"; return 1; } sub system_realpath($) { my $vol = shift // die; my $path = $vol->{PATH} // die; my $compat = config_key_lru($vol, "compat", "busybox"); my @options = ("-v"); # report error messages push @options, "-e" unless($compat); # all components must exist (not available in busybox!) push @options, "-f" if($compat); # all but the last component must exist. my $ret = run_cmd(cmd => vinfo_cmd($vol, "readlink", @options, { unsafe => $path } ), rsh => vinfo_rsh($vol), non_destructive => 1, ); return undef unless(defined($ret)); my $realpath = scalar(@$ret) ? (check_file($ret->[0], { absolute => 1 }) // "") : ""; unless($realpath) { ERROR "Failed to parse output of `realpath` for \"$vol->{PRINT}\": \"$ret->[0]\""; return undef; } DEBUG "Real path for \"$vol->{PRINT}\" is: $realpath"; return undef if($compat && !system_testdir($vol)); return $realpath; } sub system_mkdir($) { my $vol = shift // die; my $path = $vol->{PATH} // die;; INFO "Creating directory: $vol->{PRINT}/"; start_transaction("mkdir", vinfo_prefixed_keys("target", $vol)); my $ret = run_cmd(cmd => [ 'mkdir', '-p', { unsafe => $path } ], rsh => vinfo_rsh($vol), ); end_transaction("mkdir", defined($ret)); return undef unless(defined($ret)); delete $realpath_cache{$vol->{URL}}; return 1; } sub system_read_raw_info_dir($) { my $droot = shift // die; my $ret = run_cmd( # NOTE: we cannot simply "cat" all files here, as it will fail if no files found cmd => [ 'find', { unsafe => $droot->{PATH} }, '-maxdepth', '1', '-type', 'f', '!', '-size', '0', '-name', '\*.btrfs.\*info', # match ".btrfs[.gz|bz2|xz][.gpg].info" '-exec', 'echo INFO_FILE=\{\} \;', '-exec', 'cat \{\} \;' ], rsh => vinfo_rsh($droot), non_destructive => 1, ); unless(defined($ret)) { ERROR("Failed to read *.btrfs.*.info files in: $droot->{PATH}"); return undef; } my @raw_targets; my $cur_target; foreach(@$ret) { if(/^INFO_FILE=/) { push @raw_targets, $cur_target if($cur_target); $cur_target = {}; } next unless($cur_target); $cur_target->{$1} = $2 if /^([a-zA-Z_]+)=(.*)/; } push @raw_targets, $cur_target if($cur_target); # input validation (we need to abort here, or the backups will be resumed) foreach my $raw_info (@raw_targets) { unless($raw_info->{INFO_FILE}) { ERROR("Error while parsing command output for: $droot->{PATH}"); return undef; } unless($raw_info->{FILE}) { ERROR("Missing \"FILE=\" in raw info file: " . $raw_info->{INFO_FILE}); return undef; } unless(check_file($raw_info->{FILE}, { name_only => 1 })) { ERROR("Ambiguous \"FILE=\" in raw info file: " . $raw_info->{INFO_FILE}); return undef; } unless($raw_info->{TYPE} && ($raw_info->{TYPE} eq 'raw')) { ERROR("Unsupported \"type\" in raw info file: " . $raw_info->{INFO_FILE}); return undef; } unless($raw_info->{RECEIVED_UUID} && ($raw_info->{RECEIVED_UUID} =~ /^$uuid_match$/)) { ERROR("Missing/Illegal \"received_uuid\" in raw info file: " . $raw_info->{INFO_FILE}); return undef; } if(defined $raw_info->{RECEIVED_PARENT_UUID}) { unless(($raw_info->{RECEIVED_PARENT_UUID} eq '-') || ($raw_info->{RECEIVED_PARENT_UUID} =~ /^$uuid_match$/)) { ERROR("Illegal \"RECEIVED_PARENT_UUID\" in raw info file: " . $raw_info->{INFO_FILE}); return undef; } } else { $raw_info->{RECEIVED_PARENT_UUID} = '-'; } } DEBUG("Parsed " . @raw_targets . " raw info files in path: $droot->{PATH}"); TRACE(Data::Dumper->Dump([\@raw_targets], ["system_read_raw_info_dir($droot->{URL})"])) if($do_trace && $do_dumper); return \@raw_targets; } sub system_write_raw_info($$;@) { my $vol = shift // die; my $raw_info = shift // die; my %opts = @_; my $append = $opts{append}; my $info_file = $vol->{PATH} . '.info'; # sort by %raw_info_sort, then by key my @line = $append ? () : ("#btrbk-v$VERSION", "# Do not edit this file"); push @line, '#t=' . time; foreach(sort { (($raw_info_sort{$a} // 99) <=> ($raw_info_sort{$b} // 99)) || ($a cmp $b) } keys %$raw_info) { push @line, ($_ . '=' . $raw_info->{$_}); } DEBUG "Writing (" . ($append ? "append:" . join(",", keys %$raw_info) : "create") . ") raw info file: $info_file"; my $ret = run_cmd( { cmd => [ 'echo', '-e', '-n', '"' . (join '\n', @line) . '\n"' ] }, { ($append ? "append_to_file" : "redirect_to_file") => { unsafe => $info_file }, rsh => vinfo_rsh($vol), }); return undef unless(defined($ret)); return $info_file; } sub system_urandom($;$) { my $size = shift; my $format = shift || 'hex'; die unless(($size > 0) && ($size <= 256)); # sanity check unless(open(URANDOM, '<', '/dev/urandom')) { ERROR "Failed to open /dev/urandom: $!"; return undef; } binmode URANDOM; my $rand; my $rlen = read(URANDOM, $rand, $size); close(FILE); unless(defined($rand) && ($rlen == $size)) { ERROR "Failed to read from /dev/urandom: $!"; return undef; } if($format eq 'hex') { my $hex = unpack('H*', $rand); die unless(length($hex) == ($size * 2)); # paranoia check return $hex; } elsif($format eq 'bin') { return $rand; } die "unsupported format"; } sub read_extentmap_cache($) { my $vol = shift; my $cache_dir = config_key($vol, 'cache_dir'); return undef unless($cache_dir); my $uuid = $vol->{node}{uuid} // die; foreach (@$cache_dir) { my $file = "$_/${uuid}.extentmap.bin"; next unless (-f $file); DEBUG "Reading extentmap cache: $file"; if(open(my $fh, '<:raw', $file)) { my @range; my $buf; read($fh, $buf, 24 + 8 * 2); # read header my ($v, $gen, $time) = unpack('a24Q{node}{gen}) { WARN "Subvolume generation has changed (cache=$gen, subvol=$vol->{node}{gen}), ignoring cache: $file"; next; } while(read $fh, $buf, 8 * 2) { # read / unpack two words push @range, [ unpack('Q{EXTENTMAP}; my $cache_dir = config_key($vol, 'cache_dir'); return undef unless($extmap && $cache_dir); my $uuid = $vol->{node}{uuid} // die; foreach (@$cache_dir) { unless(-d $_) { WARN_ONCE "Ignoring cache_dir (not a directory): $_"; next; } my $file = "$_/${uuid}.extentmap.bin"; INFO "Writing extentmap cache: $file"; if(open(my $fh, '>:raw', $file)) { # pack Q: unsigned quad (64bit, Documentation/filesystems/fiemap.txt) print $fh pack('a24Q{node}{gen}, time); print $fh pack('Q<*', map(@{$_}, @$extmap)); close($fh); } else { ERROR "Failed to create '$file': $!"; } } } # returns extents range (sorted array of [start,end], inclusive) from FIEMAP ioctl sub filefrag_extentmap($) { my $vol = shift || die; my $starttime = time; INFO("Fetching extent map (filefrag): $vol->{PRINT}"); # NOTE: this returns exitstatus=0 if file is not found, or no files found my $ret = run_cmd(cmd => [ 'find', { unsafe => $vol->{PATH} }, '-xdev', '-type', 'f', '-exec', 'filefrag -b1 -v \{\} +' ], large_output => 1); unless(defined($ret)) { ERROR "Failed to fetch extent map: $vol->{PRINT}", @stderr; return undef; } WARN_ONCE "Configuration option \"ignore_extent_data_inline=no\" not available for filefrag (please install \"IO::AIO\" perl module)" unless(config_key($vol, "ignore_extent_data_inline")); my @range; # array of [start,end] foreach (@$ret) { #my $file = $1 if(/^File size of (.*?) is/); if(/^\s*[0-9]+:\s*[0-9]+\.\.\s*[0-9]+:\s*([0-9]+)\.\.\s*([0-9]+):/) { # NOTE: filefrag (v1.45.5) returns wrong (?) physical_offset for # "inline" regions unless run with `-b1` (blocksize=1) option. # # For btrfs file systems it does not make much sense to consider # the "inline" extents anyways: these are stored in metadata # section and are not really part of the used disk space. # # # filefrag -v MYFILE # File size of MYFILE is 2307 (1 block of 4096 bytes) # ext: logical_offset: physical_offset: length: expected: flags: # 0: 0.. 4095: 0.. 4095: 4096: last,not_aligned,inline,eof # # filefrag -v -b1 MYFILE # File size of MYFILE is 2307 (4096 block of 1 bytes) # ext: logical_offset: physical_offset: length: expected: flags: # 0: 0.. 4095: 0.. 4095: 4096: last,not_aligned,inline,eof next if(/inline/); push @range, [ $1, $2 ]; } } DEBUG("Parsed " . scalar(@range) . " regions in " . (time - $starttime) . "s for: $vol->{PRINT}"); return extentmap_merge(\@range); } # returns extents range (sorted array of [start,end], inclusive) from FIEMAP ioctl sub aio_extentmap($) { my $vol = shift || die; my $starttime = time; my $ignore_inline = config_key($vol, "ignore_extent_data_inline"); INFO("Fetching extent map: $vol->{PRINT}"); # NOTE: this returns exitstatus=0 if file is not found, or no files found my $ret = run_cmd(cmd => [ 'find', { unsafe => $vol->{PATH} }, '-xdev', '-type', 'f' ], large_output => 1 ); unless(defined($ret)) { ERROR "Failed to find files in: $vol->{PRINT}", @stderr; return undef; } DEBUG("Reading ioctl FIEMAP of " . scalar(@$ret) . " files"); IO::AIO::max_outstanding(128); # < 1024 (max file descriptors) IO::AIO::max_poll_reqs(32); my @range; my $count = 0; my $inline_count = 0; foreach my $file (@$ret) { IO::AIO::aio_open($file, IO::AIO::O_RDONLY(), 0, sub { # graceful abort on file open errors (check $count below) return unless($_[0]); # [ $fh ] # note: aio_fiemap returns byte range (not blocks) # see: Documentation/filesystems/fiemap.rst IO::AIO::aio_fiemap($_[0], 0, undef, 0, undef, sub { $count++; foreach(@{$_[0]}) { # [ $logical, $physical, $length, $flags ] if($_->[3] & IO::AIO::FIEMAP_EXTENT_DATA_INLINE()) { $inline_count++; next if($ignore_inline); WARN_ONCE "Ambigous inline region [$_->[1] .. $_->[1] + $_->[2] - 1] for $file" if((($_->[1] != 0) || ($_->[2] != 4096))); } push @range, [ $_->[1], $_->[1] + $_->[2] - 1 ]; } }); }); # poll, or the above eats up all our filedescriptors IO::AIO::poll_cb(); # takes "max_outstanding" and "max_poll_reqs" settings } IO::AIO::flush(); WARN "Failed to open $count / " . scalar(@$ret) . " files" if($count != scalar(@$ret)); DEBUG("Parsed " . scalar(@range) . " regions (" . ($ignore_inline ? "ignored " : "") . "$inline_count \"inline\") for $count files in " . (time - $starttime) . "s for: $vol->{PRINT}"); return extentmap_merge(\@range); } sub extentmap_total_blocks($) { my $extmap = shift; my $count = 0; foreach(@{$extmap->{rmap}}) { $count += ($_->[1] - $_->[0] + 1); } return $count; } sub extentmap_size($) { my $extmap = shift; # merged ranges return undef unless($extmap); my $size = 0; foreach(@$extmap) { $size += $_->[1] - $_->[0] + 1; } return $size; } sub extentmap_merge(@) { return undef unless(scalar(@_)); my @range = sort { $a->[0] <=> $b->[0] } map @$_, @_; my @merged; my $start = -1; my $end = -2; foreach (@range) { if($_->[0] <= $end + 1) { # range overlaps the preceeding one, or is adjacent to it $end = $_->[1] if($_->[1] > $end); } else { push @merged, [ $start, $end ] if($start >= 0); $start = $_->[0]; $end = $_->[1]; } } push @merged, [ $start, $end ] if($start >= 0); DEBUG "extentmap: merged " . scalar(@range) . " regions into " . scalar(@merged) . " regions"; return \@merged; } # ( A \ B ) : data in A that is not in B (relative complement of B in A) sub extentmap_diff($$) { my $l = shift // die; # A, sorted my $r = shift; # B, sorted return $l unless($r); # A \ 0 = A my $i = 0; my $rn = scalar(@$r); my @diff; foreach(@$l) { my $l_start = $_->[0]; my $l_end = $_->[1]; while(($i < $rn) && ($r->[$i][1] < $l_start)) { # r_end < l_start # advance r to next overlapping $i++; } while(($i < $rn) && ($r->[$i][0] <= $l_end)) { # r_start <= l_end # while overlapping, advance l_start my $r_start = $r->[$i][0]; my $r_end = $r->[$i][1]; push @diff, [ $l_start, $r_start - 1 ] if($l_start < $r_start); $l_start = $r_end + 1; last if($l_start > $l_end); $i++; } push @diff, [ $l_start, $l_end ] if($l_start <= $l_end); } DEBUG "extentmap: relative complement ( B=" . scalar(@$r) . ' \ A=' . scalar(@$l) . " ) = " . scalar(@diff) . " regions"; return \@diff; } sub btr_tree($$$$) { my $vol = shift; my $vol_root_id = shift || die; my $mount_source = shift || die; # aka device my $mountpoints = shift || die; # all known mountpoints for this filesystem: arrayref of mountinfo die unless($vol_root_id >= 5); # return parsed tree from %mount_source_cache if present my $host_mount_source = $vol->{URL_PREFIX} . $mount_source; my $cached_tree = $mount_source_cache{$host_mount_source}; TRACE "mount_source_cache " . ($cached_tree ? "HIT" : "MISS") . ": $host_mount_source" if($do_trace); if($cached_tree) { TRACE "btr_tree: returning cached tree at id=$vol_root_id" if($do_trace); my $node = $cached_tree->{ID_HASH}{$vol_root_id}; ERROR "Unknown subvolid=$vol_root_id in btrfs tree of $host_mount_source" unless($node); return $node; } my $node_list = btrfs_subvolume_list_complete($vol); return undef unless(ref($node_list) eq "ARRAY"); my $vol_root; TRACE "btr_tree: processing subvolume list of: $vol->{PRINT}" if($do_trace); # return a reference to the cached root if we already know the tree, # making sure every tree is only stored once, which is essential # e.g. when injecting nodes. die if duplicate UUID exist on # different file systems (no matter if local or remote). # # note: this relies on subvolume UUID's to be "universally unique" # (which is why cloning btrfs filesystems using "dd" is a bad idea) # # note: a better way would be to always compare the UUID of # subvolid=5. unfortunately this is not possible for filesystems # created with btrfs-progs < 4.16 (no UUID for subvolid=5). foreach(@$node_list) { my $node_uuid = $_->{uuid}; next unless($node_uuid); if($uuid_cache{$node_uuid}) { # at least one uuid of $node_list is already known TRACE "uuid_cache HIT: $node_uuid" if($do_trace); $vol_root = $uuid_cache{$node_uuid}->{TREE_ROOT}->{ID_HASH}->{$vol_root_id}; unless($vol_root) { # check for deleted subvolumes: e.g. still mounted, but deleted elsewhere my $deleted_nodes = btrfs_subvolume_list($vol, deleted_only => 1); return undef unless(ref($deleted_nodes) eq "ARRAY"); if(grep ($_->{id} eq $vol_root_id), @$deleted_nodes) { ERROR "Subvolume is deleted: id=$vol_root_id mounted on: $vol->{PRINT}"; return undef; } ERROR "Subvolume id=$vol_root_id is not present on known btrfs tree: $vol->{PRINT}", "Possible causes:", " - Mismatch in mountinfo", " - Subvolume was deleted while btrbk is running", " - Duplicate UUID present on multiple filesystems: $node_uuid"; ERROR "Refusing to run on unstable environment; exiting"; exit 1; } INFO "Assuming same filesystem: \"$vol_root->{TREE_ROOT}->{host_mount_source}\", \"$host_mount_source\""; TRACE "btr_tree: returning already parsed tree at id=$vol_root->{id}" if($do_trace); $mount_source_cache{$host_mount_source} = $vol_root->{TREE_ROOT}; return $vol_root; } last; # check only first UUID (for performance) } # fill our hashes and uuid_cache my %id; my %uuid_hash; my %received_uuid_hash; my %parent_uuid_hash; my $gen_max = 0; foreach my $node (@$node_list) { my $node_id = $node->{id}; my $node_uuid = $node->{uuid}; die unless($node_id >= 5); die "duplicate node id" if(exists($id{$node_id})); $id{$node_id} = $node; if($node_uuid) { # NOTE: uuid on btrfs root (id=5) is not always present $uuid_hash{$node_uuid} = $node; $uuid_cache{$node_uuid} = $node; # hacky: if root node has no "uuid", it also has no "received_uuid" and no "gen" push(@{$received_uuid_hash{$node->{received_uuid}}}, $node) if($node->{received_uuid} ne '-'); push(@{$parent_uuid_hash{$node->{parent_uuid}}}, $node) if($node->{parent_uuid} ne '-'); $gen_max = $node->{gen} if($node->{gen} > $gen_max); } elsif(not $node->{is_root}) { die "missing uuid on subvolume"; } $node->{SUBTREE} = []; } my $tree_root = $id{5} // die "missing btrfs root"; $tree_root->{ID_HASH} = \%id; $tree_root->{UUID_HASH} = \%uuid_hash; $tree_root->{RECEIVED_UUID_HASH} = \%received_uuid_hash; $tree_root->{PARENT_UUID_HASH} = \%parent_uuid_hash; $tree_root->{GEN_MAX} = $gen_max; $tree_root->{URL_PREFIX} = $vol->{URL_PREFIX}; # hacky, first url prefix for logging # NOTE: host_mount_source is NOT dependent on MACHINE_ID: # if we return already present tree (see above), the value of # host_mount_source will still point to the mount_source of the # first machine. $tree_root->{mount_source} = $mount_source; $tree_root->{host_mount_source} = $host_mount_source; # unique identifier, e.g. "/dev/sda1" or "ssh://hostname[:port]/dev/sda1" $vol_root = $id{$vol_root_id}; unless($vol_root) { ERROR "Failed to resolve tree root for subvolid=$vol_root_id: " . ($vol->{PRINT} // $vol->{id}); return undef; } # set REL_PATH and tree references (TREE_ROOT, SUBTREE, TOP_LEVEL) foreach my $node (@$node_list) { unless($node->{is_root}) { # note: it is possible that id < top_level, e.g. after restoring my $top_level = $id{$node->{top_level}}; die "missing top_level reference" unless(defined($top_level)); push(@{$top_level->{SUBTREE}}, $node); $node->{TOP_LEVEL} = $top_level; # "path" always starts with set REL_PATH my $rel_path = $node->{path}; unless($top_level->{is_root}) { die unless($rel_path =~ s/^\Q$top_level->{path}\E\///); } $node->{REL_PATH} = $rel_path; # relative to {TOP_LEVEL}->{path} } $node->{TREE_ROOT} = $tree_root; add_btrbk_filename_info($node); } # add known mountpoints to nodes my %mountpoints_hash; foreach(@$mountpoints) { my $node_id = $_->{MNTOPS}{subvolid}; my $node = $id{$node_id}; unless($node) { WARN "Unknown subvolid=$node_id (in btrfs tree of $host_mount_source) for mountpoint: $vol->{URL_PREFIX}$_->{mount_point}"; next; } $mountpoints_hash{$node_id} = $node; push @{$node->{MOUNTINFO}}, $_; # if present, node is mounted at MOUNTINFO } $tree_root->{MOUNTED_NODES} = [ (values %mountpoints_hash) ]; # list of mounted nodes TRACE "btr_tree: returning tree at id=$vol_root->{id}" if($do_trace); VINFO($vol_root, "node") if($loglevel >=4); $mount_source_cache{$host_mount_source} = $tree_root; return $vol_root; } sub btr_tree_inject_node($$$) { my $top_node = shift; my $detail = shift; my $rel_path = shift; my $subtree = $top_node->{SUBTREE} // die; my $tree_root = $top_node->{TREE_ROOT}; die unless($detail->{parent_uuid} && $detail->{received_uuid} && exists($detail->{readonly})); $tree_inject_id -= 1; $tree_root->{GEN_MAX} += 1; my $uuid = sprintf("${fake_uuid_prefix}%012u", -($tree_inject_id)); my $node = { %$detail, # make a copy TREE_ROOT => $tree_root, SUBTREE => [], TOP_LEVEL => $top_node, REL_PATH => $rel_path, INJECTED => 1, id => $tree_inject_id, uuid => $uuid, gen => $tree_root->{GEN_MAX}, cgen => $tree_root->{GEN_MAX}, }; push(@$subtree, $node); $uuid_cache{$uuid} = $node; $tree_root->{ID_HASH}->{$tree_inject_id} = $node; $tree_root->{UUID_HASH}->{$uuid} = $node; push( @{$tree_root->{RECEIVED_UUID_HASH}->{$node->{received_uuid}}}, $node ) if($node->{received_uuid} ne '-'); push( @{$tree_root->{PARENT_UUID_HASH}->{$node->{parent_uuid}}}, $node ) if($node->{parent_uuid} ne '-'); return $node; } # returns array of { path, mountinfo } # NOTE: includes subvolumes hidden by other mountpoint sub __fs_info { my $node = shift; my $url_prefix = shift; my @ret = $node->{MOUNTINFO} ? map +{ path => $url_prefix . $_->{mount_point}, mountinfo => $_ }, @{$node->{MOUNTINFO}} : (); return @ret if($node->{is_root}); return ((map +{ path => $_->{path} . '/' . $node->{REL_PATH}, mountinfo => $_->{mountinfo} }, __fs_info($node->{TOP_LEVEL}, $url_prefix)), @ret); } sub _fs_info { my $node = shift // die; my $url_prefix = shift // $node->{TREE_ROOT}{URL_PREFIX}; my @ret = __fs_info($node, $url_prefix); @ret = ({ path => "$url_prefix<$node->{TREE_ROOT}{mount_source}>/$node->{path}", mountinfo => undef }) unless(scalar(@ret)); return @ret; } sub _fs_path { my @ret = map $_->{path}, _fs_info(@_); return wantarray ? @ret : $ret[0]; } sub _is_correlated($$) { my $a = shift; # node a my $b = shift; # node b return 0 if($a->{is_root} || $b->{is_root}); return 0 unless($a->{readonly} && $b->{readonly}); return (($a->{uuid} eq $b->{received_uuid}) || ($b->{uuid} eq $a->{received_uuid}) || (($a->{received_uuid} ne '-') && ($a->{received_uuid} eq $b->{received_uuid}))); } sub _is_same_fs_tree($$) { return ($_[0]->{TREE_ROOT}{host_mount_source} eq $_[1]->{TREE_ROOT}{host_mount_source}); } sub _is_child_of { my $node = shift; my $uuid = shift; foreach(@{$node->{SUBTREE}}) { return 1 if($_->{uuid} eq $uuid); return 1 if(_is_child_of($_, $uuid)); } return 0; } sub _get_longest_match { my $node = shift; my $path = shift; my $check_path = shift; # MUST have a trailing slash $path .= '/' unless($path =~ /\/$/); # correctly handle root path="/" return undef unless($check_path =~ /^\Q$path\E/); foreach(@{$node->{SUBTREE}}) { my $ret = _get_longest_match($_, $path . $_->{REL_PATH}, $check_path); return $ret if($ret); } return { node => $node, path => $path }; } sub vinfo($$) { my $url = shift // die; my $config = shift; my ($url_prefix, $path) = check_url($url); die "invalid url: $url" unless(defined($path)); my $print = $path; my $name = $path; $name =~ s/^.*\///; $name = '/' if($name eq ""); my $host = undef; my $port = undef; if($url_prefix) { $host = $url_prefix; die unless($host =~ s/^ssh:\/\///); $port = $1 if($host =~ s/:([1-9][0-9]*)$//); $print = $host . (defined($port) ? "[$port]:" : ":") . $path; $host =~ s/^\[//; # remove brackets from ipv6_addr $host =~ s/\]$//; # remove brackets from ipv6_addr } # Note that PATH and URL have no trailing slash, except if "/". # Note that URL and URL_PREFIX can contain ipv6 address in brackets (e.g. "[::1]"). return { HOST => $host, # hostname|ipv4_address|ipv6_address| PORT => $port, # port| NAME => $name, PATH => $path, PRINT => $print, # "hostname:/path" or "hostname[port]:/path" URL => $url_prefix . $path, # ssh://hostname[:port]/path URL_PREFIX => $url_prefix, # ssh://hostname[:port] (or "" if local) MACHINE_ID => $url_prefix || "LOCAL:", # unique: "LOCAL:" or hostname and port CONFIG => $config, # These are added in vinfo_init_root #NODE_SUBDIR => undef, #VINFO_MOUNTPOINT => undef, } } sub vinfo_child($$;$) { my $parent = shift || die; my $rel_path = shift // die; my $config = shift; # override parent config my $name = $rel_path; my $subvol_dir = ""; $subvol_dir = $1 if($name =~ s/^(.*)\///); # Note that PATH and URL intentionally contain "//" if $parent->{PATH} = "/". my $vinfo = { HOST => $parent->{HOST}, PORT => $parent->{PORT}, NAME => $name, PATH => "$parent->{PATH}/$rel_path", PRINT => "$parent->{PRINT}" . ($parent->{PRINT} =~ /\/$/ ? "" : "/") . $rel_path, URL => "$parent->{URL}/$rel_path", URL_PREFIX => $parent->{URL_PREFIX}, MACHINE_ID => $parent->{MACHINE_ID}, CONFIG => $config // $parent->{CONFIG}, VINFO_MOUNTPOINT => $parent->{VINFO_MOUNTPOINT}, # NOTE: these are NOT present in non-child vinfo, and should be used # only for printing and comparing results of vinfo_subvol_list. SUBVOL_PATH => $rel_path, SUBVOL_DIR => $subvol_dir, # SUBVOL_PATH=SUBVOL_DIR/NAME }; # TRACE "vinfo_child: created from \"$parent->{PRINT}\": $info{PRINT}" if($do_trace); return $vinfo; } sub vinfo_rsh($;@) { my $vinfo = shift || die; my %opts = @_; my $host = $vinfo->{HOST}; return undef unless(defined($host)); my $config = $vinfo->{CONFIG}; die unless($config); # as of btrbk-0.28.0, ssh port is a property of a "vinfo", set with # "ssh://hostname[:port]" in 'volume' and 'target' sections. Note # that the port number is also used for the MACHINE_ID to # distinguish virtual machines on same host with different ports. my $ssh_port = $vinfo->{PORT}; unless($ssh_port) { # PORT defaults to ssh_port (DEPRECATED) $ssh_port = config_key($config, "ssh_port") // "default"; $ssh_port = undef if($ssh_port eq "default"); } my $ssh_user = config_key($config, "ssh_user"); my $ssh_identity = config_key($config, "ssh_identity"); my $ssh_compression = config_key($config, "ssh_compression"); my $ssh_cipher_spec = join(",", @{config_key($config, "ssh_cipher_spec")}); my @ssh_options; # as of btrbk-0.29.0, we run ssh without -q (catching @stderr) push(@ssh_options, '-p', $ssh_port) if($ssh_port); push(@ssh_options, '-c', $ssh_cipher_spec) if($ssh_cipher_spec ne "default"); push(@ssh_options, '-i', { unsafe => $ssh_identity }) if($ssh_identity); # NOTE: hackily used in run_cmd on errors if($opts{disable_compression}) { push(@ssh_options, '-o', 'compression=no'); # force ssh compression=no (in case it is defined in ssh_config) } elsif($ssh_compression) { push(@ssh_options, '-C'); } my $ssh_dest = $ssh_user ? $ssh_user . '@' . $host : $host; return ['ssh', @ssh_options, $ssh_dest ]; } sub vinfo_cmd($$@) { my $vinfo = shift || die; my $cmd = shift || die; my @cmd_args = @_; my $backend = config_key_lru($vinfo, "backend") // die; my $cmd_mapped = $backend_cmd_map{$backend}{$cmd} // [ split(" ", $cmd) ]; return [ @$cmd_mapped, @cmd_args ]; } sub _get_btrbk_date(@) { my %a = @_; # named capture buffers (%+) from $btrbk_timestamp_match my @tm = ( ($a{ss} // 0), ($a{mm} // 0), ($a{hh} // 0), $a{DD}, ($a{MM} - 1), ($a{YYYY} - 1900) ); my $NN = $a{NN} // 0; my $zz = $a{zz}; my $has_exact_time = defined($a{hh}); # false if timestamp_format=short my $time; if(defined($zz)) { eval_quiet { $time = timegm(@tm); }; } else { eval_quiet { $time = timelocal(@tm); }; } unless(defined($time)) { # WARN "$@"; # sadly Time::Local croaks, which also prints the line number from here. return undef; } # handle ISO 8601 time offset if(defined($zz)) { my $offset; if($zz eq 'Z') { $offset = 0; # Zulu time == UTC } elsif($zz =~ /^([+-])([0-9][0-9])([0-9][0-9])$/) { $offset = ( $3 * 60 ) + ( $2 * 60 * 60 ); $offset *= -1 if($1 eq '-'); } else { return undef; } $time -= $offset; } return [ $time, $NN, $has_exact_time ]; } sub add_btrbk_filename_info($;$) { my $node = shift; my $raw_info = shift; my $name = $node->{REL_PATH}; return undef unless(defined($name)); # NOTE: unless long-iso file format is encountered, the timestamp is interpreted in local timezone. $name =~ s/^(.*)\///; if($raw_info && ($name =~ /^(?.+)\.$btrbk_timestamp_match$raw_postfix_match$/)) { ; } elsif((not $raw_info) && ($name =~ /^(?.+)\.$btrbk_timestamp_match$/)) { ; } else { return undef; } $name = $+{name} // die; my $btrbk_date = _get_btrbk_date(%+); # use named capture buffers of previous match unless($btrbk_date) { WARN "Illegal timestamp on subvolume \"$node->{REL_PATH}\", ignoring"; return undef; } $node->{BTRBK_BASENAME} = $name; $node->{BTRBK_DATE} = $btrbk_date; $node->{BTRBK_RAW} = $raw_info if($raw_info); return $node; } sub _find_mountpoint($$) { my $root = shift; my $path = shift; $path .= '/' unless($path =~ /\/$/); # append trailing slash while (my $tree = $root->{SUBTREE}) { my $m = undef; foreach (@$tree) { $m = $_, last if($path =~ /^\Q$_->{mount_point}\E\//); } last unless defined $m; $root = $m; } TRACE "resolved mount point for \"$path\": $root->{mount_point} (mount_source=$root->{mount_source}, subvolid=" . ($root->{MNTOPS}->{subvolid} // '') . ")" if($do_trace); return $root; } sub mountinfo_tree($) { my $vol = shift; my $mountinfo = $mountinfo_cache{$vol->{MACHINE_ID}}; TRACE "mountinfo_cache " . ($mountinfo ? "HIT" : "MISS") . ": $vol->{MACHINE_ID}" if($do_trace); unless($mountinfo) { $mountinfo = system_list_mountinfo($vol); return undef unless($mountinfo); $mountinfo_cache{$vol->{MACHINE_ID}} = $mountinfo; } return $mountinfo->[0]->{TREE_ROOT} if($mountinfo->[0]->{TREE_ROOT}); my %id = map +( $_->{mount_id} => $_ ), @$mountinfo; my $tree_root; foreach my $node (@$mountinfo) { my $parent = $id{$node->{parent_id}}; if($parent && ($node->{mount_id} != $node->{parent_id})) { $node->{PARENT} = $parent; push @{$parent->{SUBTREE}}, $node; } else { die "multiple root mount points" if($tree_root); $tree_root = $node; } # populate cache (mount points are always real paths) $realpath_cache{$vol->{URL_PREFIX} . $node->{mount_point}} = $node->{mount_point}; } die "no root mount point" unless($tree_root); $_->{TREE_ROOT} = $tree_root foreach (@$mountinfo); $tree_root->{MOUNTINFO_LIST} = $mountinfo; return $tree_root; } sub vinfo_mountpoint { my $vol = shift // die; my %args = @_; DEBUG "Resolving mount point for: $vol->{PRINT}"; my $mountinfo_root = mountinfo_tree($vol); return undef unless($mountinfo_root); my $realpath = $realpath_cache{$vol->{URL}}; unless(defined($realpath)) { $realpath = system_realpath($vol); # set to empty string on errors (try only once) $realpath_cache{$vol->{URL}} = $realpath // ""; } return undef unless($realpath); my $mountpoint = _find_mountpoint($mountinfo_root, $realpath); # handle autofs if($mountpoint->{fs_type} eq 'autofs') { if($args{autofs_retry}) { DEBUG "Non-btrfs autofs mount point for: $vol->{PRINT}"; return undef; } DEBUG "Found autofs mount point, triggering automount on $mountpoint->{mount_point} for: $vol->{PRINT}"; btrfs_subvolume_show(vinfo($vol->{URL_PREFIX} . $mountpoint->{mount_point}, $vol->{CONFIG})); $mountinfo_cache{$vol->{MACHINE_ID}} = undef; return vinfo_mountpoint($vol, %args, autofs_retry => 1); } if($args{fs_type} && ($mountpoint->{fs_type} ne $args{fs_type})) { ERROR "Not a btrfs filesystem (mountpoint=\"$mountpoint->{mount_point}\", fs_type=\"$mountpoint->{fs_type}\"): $vol->{PRINT}"; return undef; } DEBUG "Mount point for \"$vol->{PRINT}\": $mountpoint->{mount_point} (mount_source=$mountpoint->{mount_source}, fs_type=$mountpoint->{fs_type})"; return ($realpath, $mountpoint); } sub vinfo_init_root($) { my $vol = shift || die; @stderr = (); # clear @stderr (propagated for logging) my ($real_path, $mountpoint) = vinfo_mountpoint($vol, fs_type => 'btrfs'); return undef unless($mountpoint); my @same_source_mounts = grep { $_->{mount_source} eq $mountpoint->{mount_source} } @{$mountpoint->{TREE_ROOT}{MOUNTINFO_LIST}}; foreach my $mnt (grep { !defined($_->{MNTOPS}{subvolid}) } @same_source_mounts) { # kernel <= 4.2 does not have subvolid=NN in /proc/self/mounts, read it with btrfs-progs DEBUG "No subvolid provided in mounts for: $mnt->{mount_point}"; my $detail = btrfs_subvolume_show(vinfo($vol->{URL_PREFIX} . $mnt->{mount_point}, $vol->{CONFIG})); return undef unless($detail); $mnt->{MNTOPS}{subvolid} = $detail->{id} || die; # also affects %mountinfo_cache } # read btrfs tree for the mount point @stderr = (); # clear @stderr (propagated for logging) my $mnt_path = $mountpoint->{mount_point}; my $mnt_vol = vinfo($vol->{URL_PREFIX} . $mnt_path, $vol->{CONFIG}); my $mnt_tree_root = btr_tree($mnt_vol, $mountpoint->{MNTOPS}{subvolid}, $mountpoint->{mount_source}, \@same_source_mounts); return undef unless($mnt_tree_root); # find longest match in btrfs tree $real_path .= '/' unless($real_path =~ /\/$/); # correctly handle root path="/" my $ret = _get_longest_match($mnt_tree_root, $mnt_path, $real_path) // die; my $tree_root = $ret->{node}; return undef unless($tree_root); # set NODE_SUBDIR if $vol->{PATH} points to a regular (non-subvolume) directory. # in other words, "PATH=/NODE_SUBDIR" my $node_subdir = $real_path; die unless($node_subdir =~ s/^\Q$ret->{path}\E//); # NOTE: $ret->{path} has trailing slash! $node_subdir =~ s/\/+$//; $vol->{NODE_SUBDIR} = $node_subdir if($node_subdir ne ''); $vol->{node} = $tree_root; $vol->{VINFO_MOUNTPOINT} = vinfo($vol->{URL_PREFIX} . $mnt_path, $vol->{CONFIG}); $vol->{VINFO_MOUNTPOINT}{node} = $mnt_tree_root; return $tree_root; } sub vinfo_init_raw_root($;@) { my $droot = shift || die; my $tree_root = $raw_url_cache{$droot->{URL}}; TRACE "raw_url_cache " . ($tree_root ? "HIT" : "MISS") . ": URL=$droot->{URL}" if($do_trace); unless($tree_root) { if(my $real_path = $realpath_cache{$droot->{URL}}) { my $real_url = $droot->{URL_PREFIX} . $real_path; $tree_root = $raw_url_cache{$real_url}; TRACE "raw_url_cache " . ($tree_root ? "HIT" : "MISS") . ": REAL_URL=$real_url" if($do_trace); } } unless($tree_root) { DEBUG "Creating raw subvolume list: $droot->{PRINT}"; # create fake btr_tree $tree_root = { id => 5, is_root => 1, mount_source => '@raw_tree', # for _fs_path (logging) host_mount_source => $droot->{URL} . '@raw_tree', # for completeness (this is never used) GEN_MAX => 1, SUBTREE => [], UUID_HASH => {}, RECEIVED_UUID_HASH => {}, URL_PREFIX => $droot->{URL_PREFIX}, # for _fs_path (logging) MOUNTINFO => [ { mount_point => $droot->{PATH} } ], # for _fs_path (logging) }; $tree_root->{TREE_ROOT} = $tree_root; # list and parse *.info my $raw_info_ary = system_read_raw_info_dir($droot); return undef unless($raw_info_ary); # inject nodes to fake btr_tree $droot->{node} = $tree_root; my %child_uuid_list; foreach my $raw_info (@$raw_info_ary) { # Set btrfs subvolume information (received_uuid, parent_uuid) from filename info. # # NOTE: received_parent_uuid in BTRBK_RAW is the "parent of the source subvolume", NOT the # "parent of the received subvolume". my $subvol = vinfo_child($droot, $raw_info->{FILE}); unless(vinfo_inject_child($droot, $subvol, { TARGET_TYPE => $raw_info->{TYPE}, parent_uuid => '-', # NOTE: correct value gets inserted below # Incomplete raw fakes get same semantics as real subvolumes (readonly=0, received_uuid='-') received_uuid => ($raw_info->{INCOMPLETE} ? '-' : $raw_info->{RECEIVED_UUID}), readonly => ($raw_info->{INCOMPLETE} ? 0 : 1), }, $raw_info)) { ERROR("Ambiguous \"FILE=\" in raw info file: \"$raw_info->{INFO_FILE}\""); return undef; } if($raw_info->{RECEIVED_PARENT_UUID} ne '-') { $child_uuid_list{$raw_info->{RECEIVED_PARENT_UUID}} //= []; push @{$child_uuid_list{$raw_info->{RECEIVED_PARENT_UUID}}}, $subvol; } } my @subvol_list = @{vinfo_subvol_list($droot, sort => 'path')}; DEBUG "Found " . scalar(@subvol_list) . " raw subvolume backups in: $droot->{PRINT}"; foreach my $subvol (@subvol_list) { # If restoring a backup from raw btrfs images (using "incremental yes|strict"): # "btrfs send -p parent source > svol.btrfs", the backups # on the target will get corrupted (unusable!) as soon as # an any files in the chain gets deleted. # # We need to make sure btrbk will NEVER delete those: # - svol.--.btrfs : root (full) image # - svol.--[@].btrfs : incremental image foreach my $child (@{$child_uuid_list{$subvol->{node}{received_uuid}}}) { # Insert correct (i.e. fake) parent UUID $child->{node}{parent_uuid} = $subvol->{node}{uuid}; # Make sure that incremental backup chains are never broken: DEBUG "Found parent/child partners, forcing preserve of: \"$subvol->{PRINT}\", \"$child->{PRINT}\""; $subvol->{node}{FORCE_PRESERVE} = "preserve forced: parent of another raw target"; $child->{node}{FORCE_PRESERVE} ||= "preserve forced: child of another raw target"; } } # TRACE(Data::Dumper->Dump([\@subvol_list], ["vinfo_raw_subvol_list{$droot}"])); } $droot->{node} = $tree_root; $droot->{VINFO_MOUNTPOINT} = $droot; # fake mountpoint $raw_url_cache{$droot->{URL}} = $tree_root; return $tree_root; } sub _vinfo_subtree_list { my $tree = shift; my $vinfo_parent = shift; my $filter_readonly = shift; # if set, return only read-only my $filter_btrbk_direct_leaf = shift; # if set, return only read-only direct leafs matching btrbk_basename my $list = shift // []; my $path_prefix = shift // ""; my $depth = shift // 0; # if $vinfo_parent->{NODE_SUBDIR} is set, vinfo_parent->{PATH} does # not point to a subvolume directly, but to "/NODE_SUBDIR". # skip nodes wich are not in NODE_SUBDIR, or strip NODE_SUBDIR from from rel_path. my $node_subdir_filter = ($depth == 0) ? $vinfo_parent->{NODE_SUBDIR} : undef; foreach my $node (@{$tree->{SUBTREE}}) { my $rel_path = $node->{REL_PATH}; if(defined($node_subdir_filter)) { next unless($rel_path =~ s/^\Q$node_subdir_filter\E\///); } my $path = $path_prefix . $rel_path; # always points to a subvolume # filter direct leafs (SUBVOL_DIR="") matching btrbk_basename next unless(!defined($filter_btrbk_direct_leaf) || (exists($node->{BTRBK_BASENAME}) && ($node->{BTRBK_BASENAME} eq $filter_btrbk_direct_leaf) && ($rel_path !~ /\//))); # note: depth is always 0 if $filter_btrbk_direct_leaf # filter readonly, push vinfo_child if(!$filter_readonly || $node->{readonly}) { my $vinfo = vinfo_child($vinfo_parent, $path); $vinfo->{node} = $node; # add some additional information to vinfo $vinfo->{subtree_depth} = $depth; push(@$list, $vinfo); } unless(defined($filter_btrbk_direct_leaf)) { _vinfo_subtree_list($node, $vinfo_parent, $filter_readonly, undef, $list, $path . '/', $depth + 1); } } return $list; } sub vinfo_subvol_list($;@) { my $vol = shift || die; my %opts = @_; TRACE "Creating subvolume list for: $vol->{PRINT}" if($do_trace); # recurse into tree from $vol->{node}, returns arrayref of vinfo my $subvol_list = _vinfo_subtree_list($vol->{node}, $vol, $opts{readonly}, $opts{btrbk_direct_leaf}); if($opts{sort}) { if($opts{sort} eq 'path') { my @sorted = sort { $a->{SUBVOL_PATH} cmp $b->{SUBVOL_PATH} } @$subvol_list; return \@sorted; } else { die; } } return $subvol_list; } # returns vinfo_child if $node is in tree below $vol (or equal if allow_equal), or undef sub vinfo_resolved($$;@) { my $node = shift || die; my $vol = shift || die; # root vinfo node my %opts = @_; my $top_id = $vol->{node}{id}; my @path; my $nn = $node; while(($nn->{id} != $top_id) && (!$nn->{is_root})) { unshift(@path, $nn->{REL_PATH}); $nn = $nn->{TOP_LEVEL}; } if(scalar(@path) == 0) { return $vol if($opts{allow_equal} && not defined($vol->{NODE_SUBDIR})); return undef; } return undef if($nn->{is_root} && (!$vol->{node}{is_root})); my $jpath = join('/', @path); if(defined($vol->{NODE_SUBDIR})) { return undef unless($jpath =~ s/^\Q$vol->{NODE_SUBDIR}\E\///); } if(defined($opts{btrbk_direct_leaf})) { return undef if($jpath =~ /\//); return undef unless(exists($node->{BTRBK_BASENAME}) && ($node->{BTRBK_BASENAME} eq $opts{btrbk_direct_leaf})) } my $vinfo = vinfo_child($vol, $jpath); $vinfo->{node} = $node; return $vinfo; } # returns vinfo if $node is below any mountpoint of $vol sub vinfo_resolved_all_mountpoints($$) { my $node = shift || die; my $vol = shift || die; my $tree_root = $vol->{node}{TREE_ROOT}; foreach my $mnt_node (@{$tree_root->{MOUNTED_NODES}}) { foreach my $mountinfo (@{$mnt_node->{MOUNTINFO}}) { my $mnt_vol = vinfo($vol->{URL_PREFIX} . $mountinfo->{mount_point}, $vol->{CONFIG}); $mnt_vol->{node} = $mnt_node; TRACE "vinfo_resolved_all_mountpoints: trying mountpoint: $mnt_vol->{PRINT}" if($do_trace); my $vinfo = vinfo_resolved($node, $mnt_vol, allow_equal => 1); return $vinfo if($vinfo); } } return undef; } sub vinfo_subvol($$) { my $vol = shift || die; my $subvol_path = shift // die; foreach (@{vinfo_subvol_list($vol)}) { return $_ if($_->{SUBVOL_PATH} eq $subvol_path); } return undef; } sub vinfo_inject_child($$$;$) { my $vinfo = shift; my $vinfo_child = shift; my $detail = shift; my $raw_info = shift; my $node; my $node_subdir = defined($vinfo->{NODE_SUBDIR}) ? $vinfo->{NODE_SUBDIR} . '/' : ""; my $rel_path = $node_subdir . $vinfo_child->{SUBVOL_PATH}; $node = btr_tree_inject_node($vinfo->{node}, $detail, $rel_path); return undef unless(add_btrbk_filename_info($node, $raw_info)); $vinfo_child->{node} = $node; TRACE "vinfo_inject_child: injected child id=$node->{id} to $vinfo->{PRINT}" if($do_trace); return $vinfo_child; } # returns hash: ( $prefix_{url,path,host,name,subvol_path,rsh} => value, ... ) sub vinfo_prefixed_keys($$) { my $prefix = shift // die; my $vinfo = shift; return () unless($vinfo); my %ret; if($prefix) { $ret{$prefix} = $vinfo->{PRINT}; $prefix .= '_'; } foreach (qw( URL PATH HOST PORT NAME )) { $ret{$prefix . lc($_)} = $vinfo->{$_}; } $ret{$prefix . "subvolume"} = $vinfo->{PATH}; my $rsh = vinfo_rsh($vinfo); $ret{$prefix . "rsh"} = $rsh ? _safe_cmd($rsh) : undef, return %ret; } sub vinfo_assign_config($;$) { my $vinfo = shift || die; my $vinfo_snapshot_root = shift; my $config = $vinfo->{CONFIG} || die; die if($config->{VINFO}); $config->{VINFO} = $vinfo; $config->{VINFO_SNAPROOT} = $vinfo_snapshot_root; } sub vinfo_snapshot_root($) { my $vinfo = shift; return $vinfo->{CONFIG}{VINFO_SNAPROOT}; } sub config_subsection($$;$) { my $config = shift || die; my $context = shift || die; die if grep($_->{CONTEXT} ne $context, @{$config->{SUBSECTION}}); return @{$config->{SUBSECTION}}; } sub vinfo_subsection($$;$) { # if config: must have SUBSECTION key # if vinfo: must have CONFIG key my $config_or_vinfo = shift || die; my $context = shift || die; my $include_aborted = shift; my @config_list; my $vinfo_check; if(exists($config_or_vinfo->{SUBSECTION})) { @config_list = config_subsection($config_or_vinfo, $context); } else { @config_list = config_subsection($config_or_vinfo->{CONFIG}, $context); die unless($config_or_vinfo->{CONFIG}->{VINFO} == $config_or_vinfo); # check back reference } return map { die unless($_->{VINFO} == $_->{VINFO}->{CONFIG}->{VINFO}); # check all back references ($include_aborted || !$_->{ABORTED}) ? $_->{VINFO} : () } @config_list; } # allow (absolute) path / url with wildcards # allow group (exact match) # allow host[:port] (exact match) sub vinfo_filter_statement($) { my $filter = shift; my %ret = ( unparsed => $filter ); my ($url_prefix, $path) = check_url($filter, accept_wildcards => 1); unless($path) { # allow relative path with wildcards $url_prefix = ""; $path = check_file($filter, { relative => 1, wildcards => 1 }, sanitize => 1); } if($path) { # support "*some*file*", "*/*" my $regex = join('[^\/]*', map(quotemeta($_), split(/\*+/, lc($url_prefix) . $path, -1))); if($path =~ /^\//) { $ret{url_regex} = qr/^$regex$/; # absolute path, match full string } else { $ret{url_regex} = qr/\/$regex$/; # match end of string } } $ret{group_eq} = $filter if($filter =~ /^$group_match$/); if($filter =~ /^(?$host_name_match|$ipv4_addr_match|\[$ipv6_addr_match\])(:(?[1-9][0-9]*))?$/) { my ($host, $port) = ( $+{host}, $+{port} ); $host =~ s/^\[//; # remove brackets from ipv6_addr $host =~ s/\]$//; # remove brackets from ipv6_addr $ret{host_port_eq} = { host => $host, port => $port }; } elsif($filter =~ /^$ipv6_addr_match$/) { $ret{host_port_eq} = { host => $filter } ; } TRACE 'vinfo_filter_statement: filter="' . $filter . '" url_regex="' . ($ret{url_regex} // "") . '" group_eq="' . ($ret{group_eq} // "") . '" host_port_eq="' . ($ret{host_port_eq} ? $ret{host_port_eq}{host} . ":" . ($ret{host_port_eq}{port} // "") : "") . '"' if($do_trace); return undef unless(exists($ret{url_regex}) || exists($ret{group_eq}) || exists($ret{host_port_eq})); return \%ret; } sub vinfo_match($$;@) { my $filter = shift; my $vinfo = shift; my %opts = @_; my $flag_matched = $opts{flag_matched}; # never match dummy volume section return 0 if($vinfo->{CONFIG}{DUMMY}); # match URL against sane path (can contain "//", see vinfo_child), # no wildcards my ($url_prefix, $path) = check_url($vinfo->{URL}); my $url = defined($path) ? lc($url_prefix) . $path : undef; my $count = 0; foreach my $ff (@$filter) { if(defined($ff->{group_eq}) && (grep { $ff->{group_eq} eq $_ } @{$vinfo->{CONFIG}{group}})) { TRACE "filter \"$ff->{unparsed}\" equals $vinfo->{CONFIG}{CONTEXT} group: $vinfo->{PRINT}" if($do_trace); return $ff unless($flag_matched); #push @{$ff->{$flag_matched}}, 'group=' . $ff->{group_eq}; $ff->{$flag_matched} = 1; $count++; } if(defined($ff->{url_regex}) && defined($url) && ($url =~ /$ff->{url_regex}/)) { TRACE "filter \"$ff->{unparsed}\" matches $vinfo->{CONFIG}{CONTEXT} url: $vinfo->{PRINT}" if($do_trace); return $ff unless($flag_matched); #push @{$ff->{$flag_matched}}, $vinfo->{CONFIG}{CONTEXT} . '=' . $vinfo->{PRINT}; $ff->{$flag_matched} = 1; $count++; } if(defined($ff->{host_port_eq}) && defined($vinfo->{HOST})) { my $host = $ff->{host_port_eq}{host}; my $port = $ff->{host_port_eq}{port}; if((lc($host) eq lc($vinfo->{HOST})) && (!defined($port) || (defined($vinfo->{PORT}) && ($port == $vinfo->{PORT})))) { TRACE "filter \"$ff->{unparsed}\" matches $vinfo->{CONFIG}{CONTEXT} host: $vinfo->{PRINT}" if($do_trace); return $ff unless($flag_matched); #push @{$ff->{$flag_matched}}, $vinfo->{CONFIG}{CONTEXT} . '=' . $vinfo->{PRINT}; $ff->{$flag_matched} = 1; $count++; } } } return $count; } sub get_related_snapshots($$;$) { my $snaproot = shift || die; my $svol = shift // die; my $btrbk_basename = shift; # if set, also filter by direct_leaf my @ret = map( { vinfo_resolved($_, $snaproot, btrbk_direct_leaf => $btrbk_basename) // () } _related_nodes($svol->{node}, readonly => 1, omit_self => 1) ); if($do_trace) { TRACE "get_related_snapshots: found: $_->{PRINT}" foreach(@ret); } DEBUG "Found " . scalar(@ret) . " related snapshots of \"$svol->{PRINT}\" in: $snaproot->{PRINT}" . (defined($btrbk_basename) ? "/$btrbk_basename.*" : ""); return @ret; } sub _correlated_nodes($$) { my $dnode = shift || die; # any node on target filesystem my $snode = shift || die; my @ret; if($snode->{is_root}) { TRACE "Skip search for correlated targets: source subvolume is btrfs root: " . _fs_path($snode) if($do_trace); return @ret; } unless($snode->{readonly}) { TRACE "Skip search for correlated targets: source subvolume is not read-only: " . _fs_path($snode) if($do_trace); return @ret; } # find matches by comparing uuid / received_uuid my $uuid = $snode->{uuid}; my $received_uuid = $snode->{received_uuid}; $received_uuid = undef if($received_uuid eq '-'); my $received_uuid_hash = $dnode->{TREE_ROOT}{RECEIVED_UUID_HASH}; my $uuid_hash = $dnode->{TREE_ROOT}{UUID_HASH}; # match uuid/received_uuid combinations my @match; push(@match, @{ $received_uuid_hash->{$uuid} // [] }); # match src.uuid == target.received_uuid if($received_uuid) { push(@match, $uuid_hash->{$received_uuid} ); # match src.received_uuid == target.uuid push(@match, @{ $received_uuid_hash->{$received_uuid} // [] }); # match src.received_uuid == target.received_uuid } @ret = grep($_->{readonly}, @match); TRACE "correlated_nodes: dst=\"" . _fs_path($dnode) . "\", src=\"" . _fs_path($snode) . "\": [" . join(", ", map _fs_path($_),@ret) . "]" if($do_trace); return @ret; } # returns array of vinfo of receive targets matching btrbk name sub get_receive_targets($$;@) { my $droot = shift || die; my $src_vol = shift || die; my %opts = @_; my @ret; my @correlated = _correlated_nodes($droot->{node}, $src_vol->{node}); my $unexpected; foreach (@correlated) { my $vinfo = vinfo_resolved($_, $droot); # returns undef if not below $droot if(exists($_->{BTRBK_RAW})) { TRACE "get_receive_targets: found raw receive target: " . _fs_path($_) if($do_trace); } elsif($vinfo && ($vinfo->{SUBVOL_PATH} eq $src_vol->{NAME})) { # direct leaf, (SUBVOL_DIR = "", matching NAME) TRACE "get_receive_targets: found receive target (exact-match): $vinfo->{PRINT}" if($do_trace); } elsif($vinfo && (not $opts{exact})) { TRACE "get_receive_targets: found receive target (non-exact-match): $vinfo->{PRINT}" if($do_trace); } else { TRACE "get_receive_targets: skip unexpected match: " . _fs_path($_) if($do_trace); $unexpected = 1; if($opts{warn} && config_key($droot, "warn_unknown_targets")) { WARN "Receive target of \"$src_vol->{PRINT}\" exists at unknown location: " . ($vinfo ? $vinfo->{PRINT} : _fs_path($_)); } next; } push(@ret, $vinfo); } ${$opts{ret_unexpected_only}} = 1 if($opts{ret_unexpected_only} && $unexpected && !scalar(@ret)); return @ret; } # returns best correlated receive target within droot (independent of btrbk name) sub get_best_correlated($$;@) { my $droot = shift || die; my $src_vol = shift || die; my %opts = @_; my $inaccessible_nodes = $opts{push_inaccessible_nodes}; my @correlated = _correlated_nodes($droot->{node}, $src_vol->{node}); # all matching src_vol, from droot->TREE_ROOT foreach (@correlated) { my $vinfo = vinfo_resolved($_, $droot); # $vinfo is within $droot return [ $src_vol, $vinfo ] if($vinfo); } if($opts{fallback_all_mountpoints}) { foreach (@correlated) { my $vinfo = vinfo_resolved_all_mountpoints($_, $droot); # $vinfo is within any mountpoint of filesystem at $droot return [ $src_vol, $vinfo ] if($vinfo); } } push @$inaccessible_nodes, @correlated if($inaccessible_nodes); return undef; } # returns all related readonly nodes (by parent_uuid relationship), unsorted. sub _related_nodes($;@) { my $snode = shift // die; my %opts = @_; TRACE "related_nodes: resolving related subvolumes of: " . _fs_path($snode) if($do_trace); # iterate parent chain my @related_nodes; my $uuid_hash = $snode->{TREE_ROOT}{UUID_HASH}; my $parent_uuid_hash = $snode->{TREE_ROOT}{PARENT_UUID_HASH}; my $node = $snode; my $uuid = $node->{uuid}; my $abort_distance = 4096; # climb up parent chain my $distance = 0; # parent distance while(($distance < $abort_distance) && defined($node) && ($node->{parent_uuid} ne "-")) { $uuid = $node->{parent_uuid}; $node = $uuid_hash->{$uuid}; TRACE "related_nodes: d=$distance uuid=$uuid : parent: " . ($node ? _fs_path($node) : "") if($do_trace); $distance++; } if($distance >= $abort_distance) { my $logmsg = "Parent UUID chain exceeds depth=$abort_distance, ignoring related parents of uuid=$uuid for: " . _fs_path($snode); DEBUG $logmsg; WARN_ONCE $logmsg unless($opts{nowarn}); } TRACE "related_nodes: d=$distance uuid=$uuid : top of parent chain" if($do_trace); # push related children (even if parent node is missing -> siblings) my @nn; $abort_distance = $abort_distance; $distance = $distance * (-1); # child distance (from top parent) while($uuid) { push @related_nodes, $node if($node && (!$opts{readonly} || $node->{readonly})); my $children = $parent_uuid_hash->{$uuid}; if($children) { if($distance >= $abort_distance) { my $logmsg = "Parent/child relations exceed depth=$abort_distance, ignoring related children of uuid=$uuid for: " . _fs_path($snode); DEBUG $logmsg; WARN_ONCE $logmsg unless($opts{nowarn}); } else { push @nn, { MARK_UUID => $uuid, MARK_DISTANCE => ($distance + 1) }, @$children; } } if($do_trace) { if($node) { if($node->{readonly}) { TRACE "related_nodes: d=$distance uuid=$uuid : push related readonly: " . _fs_path($node); } else { TRACE "related_nodes: d=$distance uuid=$uuid : " . ($opts{readonly} ? "" : "push ") . "related not readonly: " . _fs_path($node); } } else { TRACE "related_nodes: d=$distance uuid=$uuid : related missing: "; } if($children && ($distance < $abort_distance)) { TRACE "related_nodes: d=$distance uuid=$uuid : postpone " . scalar(@$children) . " children"; } } $node = shift @nn; if(exists($node->{MARK_DISTANCE})) { # marker reached, restore distance $distance = $node->{MARK_DISTANCE}; TRACE "related_nodes: d=$distance uuid=$node->{MARK_UUID} : processing children" if($do_trace); $node = shift @nn; } $uuid = $node->{uuid}; } if($opts{omit_self}) { my $snode_id = $snode->{id}; my @filtered = grep { $_->{id} != $snode_id } @related_nodes; TRACE "related_nodes: found total=" . scalar(@filtered) . " related readonly subvolumes" if($do_trace); return @filtered; } TRACE "related_nodes: found total=" . scalar(@related_nodes) . " related readonly subvolumes (including self)" if($do_trace); return @related_nodes; } # returns parent, along with clone sources sub get_best_parent($$$;@) { my $svol = shift // die; my $snaproot = shift // die; my $droot = shift || die; my %opts = @_; my $ret_clone_src = $opts{clone_src}; my $ret_target_parent_node = $opts{target_parent_node}; my $strict_related = $opts{strict_related}; TRACE "get_best_parent: resolving best common parent for subvolume: $svol->{PRINT} (droot=$droot->{PRINT})" if($do_trace); # honor incremental_resolve option my $source_incremental_resolve = config_key($svol, "incremental_resolve"); my $target_incremental_resolve = config_key($droot, "incremental_resolve"); my $resolve_sroot = ($source_incremental_resolve eq "mountpoint") ? $snaproot->{VINFO_MOUNTPOINT} : $snaproot; my $resolve_droot = ($source_incremental_resolve eq "mountpoint") ? $droot->{VINFO_MOUNTPOINT} : $droot; # NOTE: Using parents from different mount points does NOT work, see # . # btrfs-progs-4.20.2 fails if the parent subvolume is not on same # mountpoint as the source subvolume: # - btrfs send -p: "ERROR: not on mount point: /path/to/mountpoint" # - btrfs receive: "ERROR: parent subvol is not reachable from inside the root subvol" # # Note that specifying clones from outside the mount point would work for btrfs send, # but btrfs receive fails with same error as above (tested with v5.13). my $source_fallback_all_mountpoints = ($source_incremental_resolve eq "_all_accessible"); my $target_fallback_all_mountpoints = ($target_incremental_resolve eq "_all_accessible"); my @inaccessible_nodes; my %gbc_opts = ( push_inaccessible_nodes => \@inaccessible_nodes, fallback_all_mountpoints => $target_fallback_all_mountpoints, ); # resolve correlated subvolumes by parent_uuid relationship. # no warnings on aborted search (due to deep relations). my %c_rel_id; # map id to c_related my @c_related; # candidates for parent (correlated + related), unsorted foreach (_related_nodes($svol->{node}, readonly => 1, omit_self => 1, nowarn => 1)) { my $vinfo = vinfo_resolved($_, $resolve_sroot); if((not $vinfo) && $source_fallback_all_mountpoints) { # related node is not under $resolve_sroot $vinfo = vinfo_resolved_all_mountpoints($_, $svol); } if($vinfo) { my $correlated = get_best_correlated($resolve_droot, $vinfo, %gbc_opts); push @c_related, $correlated if($correlated); $c_rel_id{$_->{id}} = $correlated; } else { DEBUG "Related subvolume is not accessible within $source_incremental_resolve \"$resolve_sroot->{PRINT}\": " . _fs_path($_); } } # sort by cgen my $cgen_ref = $svol->{node}{readonly} ? $svol->{node}{cgen} : $svol->{node}{gen}; my %c_map; # map correlated candidates to incremental_prefs strategy # all_related: by parent_uuid relationship, ordered by cgen $c_map{aro} = [ sort { ($cgen_ref - $a->[0]{node}{cgen}) <=> ($cgen_ref - $b->[0]{node}{cgen}) } grep { $_->[0]{node}{cgen} <= $cgen_ref } @c_related ]; $c_map{arn} = [ sort { ($a->[0]{node}{cgen} - $cgen_ref) <=> ($b->[0]{node}{cgen} - $cgen_ref) } grep { $_->[0]{node}{cgen} > $cgen_ref } @c_related ]; # NOTE: While _related_nodes() returns deep parent_uuid # relations, there is always a chance that these relations get # broken. # # Consider parent_uuid chain ($svol readonly) # B->A, C->B, delete B: C has no relation to A. # This is especially true for backups and archives (btrfs receive) # # For snapshots (here: S=$svol readwrite) the scenario is different: # A->S, B->S, C->S, delete B: A still has a relation to C. # # resolve correlated subvolumes in same directory matching btrbk file name scheme if(exists($svol->{node}{BTRBK_BASENAME})) { my $snaproot_btrbk_direct_leaf = vinfo_subvol_list($snaproot, readonly => 1, btrbk_direct_leaf => $svol->{node}{BTRBK_BASENAME}); my @sbdl_older = sort { cmp_date($b->{node}{BTRBK_DATE}, $a->{node}{BTRBK_DATE}) } grep { cmp_date($_->{node}{BTRBK_DATE}, $svol->{node}{BTRBK_DATE}) < 0 } @$snaproot_btrbk_direct_leaf; my @sbdl_newer = sort { cmp_date($a->{node}{BTRBK_DATE}, $b->{node}{BTRBK_DATE}) } grep { cmp_date($_->{node}{BTRBK_DATE}, $svol->{node}{BTRBK_DATE}) > 0 } @$snaproot_btrbk_direct_leaf; # snapdir_all: btrbk_direct_leaf, ordered by btrbk timestamp $c_map{sao} = [ map { $c_rel_id{$_->{node}{id}} // get_best_correlated($resolve_droot, $_, %gbc_opts) // () } @sbdl_older ]; $c_map{san} = [ map { $c_rel_id{$_->{node}{id}} // get_best_correlated($resolve_droot, $_, %gbc_opts) // () } @sbdl_newer ]; # snapdir_related: btrbk_direct_leaf with parent_uuid relationship, ordered by btrbk timestamp $c_map{sro} = [ map { $c_rel_id{$_->{node}{id}} // () } @sbdl_older ]; $c_map{srn} = [ map { $c_rel_id{$_->{node}{id}} // () } @sbdl_newer ]; } if(scalar @inaccessible_nodes) { # populated by get_best_correlated() WARN "Best common parent for \"$svol->{PRINT}\" is not accessible within target $target_incremental_resolve \"$resolve_droot->{PRINT}\", ignoring: " . join(", ", map('"' . _fs_path($_) . '"',@inaccessible_nodes)); } # resolve parent (and required clone sources) according to incremental_prefs if($do_trace) { TRACE "get_best_parent: related reference cgen=$svol->{node}{cgen}"; foreach my $search (@incremental_prefs_avail) { TRACE map("get_best_parent: ${search}: $_->[0]{PRINT} (cgen=$_->[0]{node}{cgen}) $_->[1]{PRINT}", @{$c_map{$search}}); } } my @parent; my @isk = map { $_ eq "defaults" ? @incremental_prefs_default : $_ } @{config_key($svol, "incremental_prefs")}; foreach(@isk) { TRACE "processing incremental_prefs: $_"; my ($k, $n) = split /:/; my $c_list = $c_map{$k} // next; for(1 .. ($n // @$c_list)) { my $cc = shift @$c_list // last; next if(grep { $_->[0]{node}{id} == $cc->[0]{node}{id} } @parent); DEBUG "Resolved " . (@parent ? "clone source" : "parent") . " (" . "next closest " . ($k =~ /n/ ? "newer" : "older") . " by " . ($k =~ /s/ ? "btrbk timestamp in snapdir" : "cgen") . ", " . ($k =~ /r/ ? "with" : "regardless of") . " parent_uuid relationship" . "): $cc->[0]{PRINT}" if($loglevel >= 3); push @parent, $cc; } } # assemble results unless(scalar @parent) { DEBUG("No suitable common parents of \"$svol->{PRINT}\" found in src=\"$resolve_sroot->{PRINT}/\", target=\"$resolve_droot->{PRINT}/\""); return undef; } if($strict_related && (!grep(exists($c_rel_id{$_->[0]{node}{id}}), @parent))) { # no relations by parent_uuid found WARN "No related common parent found (by parent_uuid relationship) for: $svol->{PRINT}", "Hint: setting option \"incremental\" to \"yes\" (instead of \"strict\") will use parent: " . join(", ", map $_->[0]{PRINT}, @parent); return undef; } my $ret_parent = shift @parent; $$ret_clone_src = [ map $_->[0], @parent ] if($ret_clone_src); $$ret_target_parent_node = $ret_parent->[1]{node} if($ret_target_parent_node); return $ret_parent->[0]; } sub get_latest_related_snapshot($$;$) { my $sroot = shift || die; my $svol = shift // die; my $btrbk_basename = shift; my $latest = undef; my $gen = -1; foreach (get_related_snapshots($sroot, $svol, $btrbk_basename)) { if($_->{node}{cgen} > $gen) { $latest = $_; $gen = $_->{node}{cgen}; } } if($latest) { DEBUG "Latest snapshot child for \"$svol->{PRINT}#$svol->{node}{gen}\" is: $latest->{PRINT}#$latest->{node}{cgen}"; } else { DEBUG "No latest snapshots found for: $svol->{PRINT}"; } return $latest; } sub check_file($$;@) { my $file = shift // die; my $accept = shift || die; my %opts = @_; my $sanitize = $opts{sanitize}; my $error_statement = $opts{error_statement}; # if not defined, no error messages are printed if($accept->{absolute} && $accept->{relative}) { # accepted, matches either absolute or relative } elsif($accept->{absolute}) { unless($file =~ /^\//) { ERROR "Only absolute files allowed $error_statement" if(defined($error_statement)); return undef; } } elsif($accept->{relative}) { if($file =~ /^\//) { ERROR "Only relative files allowed $error_statement" if(defined($error_statement)); return undef; } } elsif($accept->{name_only}) { if($file =~ /\//) { ERROR "Invalid file name ${error_statement}: $file" if(defined($error_statement)); return undef; } } elsif(not $accept->{wildcards}) { die("accept_type must contain either 'relative' or 'absolute'"); } # check directory traversal if(($file =~ /^\.\.$/) || ($file =~ /^\.\.\//) || ($file =~ /\/\.\.\//) || ($file =~ /\/\.\.$/)) { ERROR "Illegal directory traversal ${error_statement}: $file" if(defined($error_statement)); return undef; } if($sanitize) { $file =~ s/^\s+//; $file =~ s/\s+$//; $file =~ s/\/(\.?\/)+/\//g; # sanitize "//", "/./" -> "/" $file =~ s/\/\.$/\//; # sanitize trailing "/." -> "/" $file =~ s/\/$// unless($file eq '/'); # remove trailing slash } elsif(($file =~ /^\s/) || ($file =~ /\s$/)) { ERROR "Illegal leading/trailing whitespace ${error_statement}: \"$file\"" if(defined($error_statement)); return undef; } return $file; } sub check_url($;@) { my $url = shift // die; my %opts = @_; my $url_prefix = ""; if($url =~ /^ssh:\/\//) { if($url =~ s/^(ssh:\/\/($host_name_match|$ipv4_addr_match|\[$ipv6_addr_match\])(:[1-9][0-9]*)?)\//\//) { $url_prefix = $1; } } elsif($url =~ s/^($host_name_match|$ipv4_addr_match|\[$ipv6_addr_match\]):\//\//) { # convert "my.host.com:/my/path", "[2001:db8::7]:/my/path" to ssh url $url_prefix = "ssh://" . $1; } # if no url prefix match, treat it as file and let check_file() print errors return ( $url_prefix, check_file($url, { absolute => 1, wildcards => $opts{accept_wildcards} }, sanitize => 1, %opts) ); } sub config_key($$;$) { my $config = shift || die; my $key = shift || die; my $match = shift; $config = $config->{CONFIG} if($config->{CONFIG}); # accept vinfo for $config my $val; if(exists($config_override{$key})) { TRACE "config_key: OVERRIDE key=$key to value=" . ($config_override{$key} // "") if($do_trace); $val = $config_override{$key}; } else { while(not exists($config->{$key})) { # note: while all config keys exist in "meta" context (at least with default values), # we also allow fake configs (CONTEXT="cmdline") which have no PARENT. return undef unless($config->{PARENT}); $config = $config->{PARENT}; } $val = $config->{$key}; } return undef unless defined($val); return $val unless defined($match); if(ref($val) eq "ARRAY") { return grep(/^$match$/, @$val) ? $match : undef; } else { return ($val eq $match) ? $match : undef; } } sub config_key_lru($$;$) { my $vinfo = shift || die; my $key = shift || die; my $match = shift; my $retval; if(defined($vinfo->{HOST})) { $retval //= config_key($vinfo, $key . "_remote", $match); } else { $retval //= config_key($vinfo, $key . "_local_user", $match) if($>); # $EUID, $EFFECTIVE_USER_ID $retval //= config_key($vinfo, $key . "_local", $match); } $retval //= config_key($vinfo, $key, $match); return $retval; } sub config_preserve_hash($$;@) { my $config = shift || die; my $prefix = shift || die; my %opts = @_; if($opts{wipe}) { return { hod => 0, dow => 'sunday', min => 'latest', min_q => 'latest' }; } my $preserve = config_key($config, $prefix . "_preserve") // {}; my %ret = ( %$preserve, # make a copy (don't pollute config) hod => config_key($config, "preserve_hour_of_day"), dow => config_key($config, "preserve_day_of_week") ); my $preserve_min = config_key($config, $prefix . "_preserve_min"); if(defined($preserve_min)) { $ret{min} = $preserve_min; # used for raw schedule output if(($preserve_min eq 'all') || ($preserve_min eq 'latest')) { $ret{min_q} = $preserve_min; } elsif($preserve_min =~ /^([0-9]+)([hdwmy])$/) { $ret{min_n} = $1; $ret{min_q} = $2; } else { die; } } return \%ret; } sub config_compress_hash($$) { my $config = shift || die; my $config_key = shift || die; my $compress_key = config_key($config, $config_key); return undef unless($compress_key); return { key => $compress_key, level => config_key($config, $config_key . "_level"), long => config_key($config, $config_key . "_long"), threads => config_key($config, $config_key . "_threads"), adapt => config_key($config, $config_key . "_adapt"), }; } sub config_stream_hash($$) { my $source = shift || die; my $target = shift || die; return { stream_compress => config_compress_hash($target, "stream_compress"), # for remote source, limits read rate of ssh stream output after decompress # for remote target, limits read rate of "btrfs send" # for both local, limits read rate of "btrfs send" # for raw targets, limits read rate of "btrfs send | xz" (raw_target_compress) local_sink => { stream_buffer => config_key($target, "stream_buffer"), rate_limit => config_key($target, "rate_limit"), show_progress => $show_progress, }, # limits read rate of "btrfs send" rsh_source => { # limit read rate after "btrfs send", before compression stream_buffer => config_key($source, "stream_buffer_remote"), rate_limit => config_key($source, "rate_limit_remote"), #rate_limit_out => config_key($source, "rate_limit_remote"), # limit write rate }, # limits read rate of ssh stream output rsh_sink => { stream_buffer => config_key($target, "stream_buffer_remote"), rate_limit => config_key($target, "rate_limit_remote"), #rate_limit_in => config_key($target, "rate_limit_remote"), }, }; } sub config_encrypt_hash($$) { my $config = shift || die; my $config_key = shift || die; my $encrypt_type = config_key($config, $config_key); return undef unless($encrypt_type); return { type => $encrypt_type, keyring => config_key($config, "gpg_keyring"), recipient => config_key($config, "gpg_recipient"), iv_size => config_key($config, "openssl_iv_size"), ciphername => config_key($config, "openssl_ciphername"), keyfile => config_key($config, "openssl_keyfile"), kdf_keygen_each => (config_key($config, "kdf_keygen") eq "each"), kdf_backend => config_key($config, "kdf_backend"), kdf_keysize => config_key($config, "kdf_keysize"), }; } sub config_dump_keys($;@) { my $config = shift || die; my %opts = @_; my @ret; my $maxlen = 0; $config = $config->{CONFIG} if($config->{CONFIG}); # accept vinfo for $config foreach my $key (sort keys %config_options) { my $val; next if($config_options{$key}->{deprecated}{DEFAULT}); next unless($opts{all} || exists($config->{$key}) || exists($config_override{$key})); next if($config_options{$key}{context} && !grep(/^$config->{CONTEXT}$/, @{$config_options{$key}{context}})); $val = config_key($config, $key); my @va = (ref($val) eq "ARRAY") ? ($config_options{$key}->{split} ? join(" ", @$val) : @$val) : $val; foreach(@va) { if(defined($_)) { if($config_options{$key}->{accept_preserve_matrix}) { $_ = format_preserve_matrix($_, format => "config"); } } $_ //= grep(/^no$/, @{$config_options{$key}{accept} // []}) ? "no" : ""; my $comment = $_ eq "" ? "# " : ""; my $len = length($key); $maxlen = $len if($len > $maxlen); push @ret, { comment => $comment, key => $key, val => $_, len => $len }; } } return map { ($opts{prefix} // "") . $_->{comment} . $_->{key} . (' ' x (1 + $maxlen - $_->{len})) . ' ' . $_->{val} } @ret; } sub append_config_option($$$$;@) { my $config = shift; my $key = shift; my $value = shift; my $context = shift; my %opts = @_; my $error_statement = $opts{error_statement} // ""; my $opt = $config_options{$key}; # accept only keys listed in %config_options unless($opt) { ERROR "Unknown option \"$key\" $error_statement"; return undef; } if($opt->{context} && !grep(/^$context$/, @{$opt->{context}}) && ($context ne "OVERRIDE")) { ERROR "Option \"$key\" is only allowed in " . join(" or ", @{$opt->{context}}) . " context $error_statement"; return undef; } if($opt->{deny_glob_context} && $config->{GLOB_CONTEXT}) { ERROR "Option \"$key\" is not allowed on section with wildcards $error_statement"; return undef; } my $ovalue = $value; if($value eq "") { $value = "yes"; TRACE "option \"$key\" has no value, setting to \"yes\"" if($do_trace); } if($opt->{split}) { $value = [ split($config_split_match, $value) ]; } my $accepted; if($opt->{accept}) { $accepted = 1; foreach my $val (ref($value) ? @$value : $value) { $accepted = 0, last unless(grep { $val =~ /^$_$/ } @{$opt->{accept}}); TRACE "option \"$key=$val\" found in accept list" if($do_trace); } } if(!$accepted && $opt->{accept_file}) { # be very strict about file options, for security sake $value = check_file($value, $opt->{accept_file}, sanitize => 1, error_statement => ($error_statement ? "for option \"$key\" $error_statement" : undef)); return undef unless(defined($value)); TRACE "option \"$key=$value\" is a valid file, accepted" if($do_trace); $value = "no" if($value eq "."); # maps to undef later $accepted = 1; } if(!$accepted && $opt->{accept_preserve_matrix}) { my %preserve; my $s = ' ' . $value; while($s =~ s/\s+(\*|[0-9]+)([hdwmyHDWMY])//) { my $n = $1; my $q = lc($2); # qw( h d w m y ) $n = 'all' if($n eq '*'); if(exists($preserve{$q})) { ERROR "Value \"$value\" failed input validation for option \"$key\": multiple definitions of '$q' $error_statement"; return undef; } $preserve{$q} = $n; } unless($s eq "") { ERROR "Value \"$value\" failed input validation for option \"$key\" $error_statement"; return undef; } TRACE "adding preserve matrix $context context:" . Data::Dumper->new([\%preserve], [ $key ])->Indent(0)->Pad(' ')->Quotekeys(0)->Pair('=>')->Dump() if($do_trace && $do_dumper); $config->{$key} = \%preserve; return $config; } if(!$accepted) { if($ovalue eq "") { ERROR "Unsupported empty value for option \"$key\" $error_statement"; } else { ERROR "Unsupported value \"$ovalue\" for option \"$key\" $error_statement"; } return undef; } if($opt->{require_bin} && (not check_exe($opt->{require_bin}))) { WARN "Found option \"$key\", but required executable \"$opt->{require_bin}\" does not exist on your system. Please install \"$opt->{require_bin}\"."; WARN "Ignoring option \"$key\" $error_statement"; $value = "no"; } if($opt->{deprecated}) { my $dh = $opt->{deprecated}{$value} // $opt->{deprecated}{DEFAULT} // {}; $dh = $opt->{deprecated}{MATCH} if($opt->{deprecated}{MATCH} && ($value =~ $opt->{deprecated}{MATCH}{regex})); if($dh->{ABORT}) { ERROR "Deprecated (incompatible) option \"$key\" found $error_statement, refusing to continue", $dh->{warn}; return undef; } my @wmsg; push @wmsg, "Found deprecated option \"$key $value\" $error_statement", $dh->{warn} if($dh->{warn}); if(defined($dh->{replace_key})) { $key = $dh->{replace_key}; $value = $dh->{replace_value}; push @wmsg, "Using \"$key $value\""; } WARN @wmsg if(@wmsg); if($dh->{FAILSAFE_PRESERVE}) { unless($config_override{FAILSAFE_PRESERVE}) { # warn only once WARN "Entering failsafe mode:"; WARN " - preserving ALL snapshots for ALL subvolumes"; WARN " - ignoring ALL targets (skipping backup creation)"; WARN " - please read \"doc/upgrade_to_v0.23.0.md\""; $config_override{FAILSAFE_PRESERVE} = "Failsafe mode active (deprecated configuration)"; } $config_override{snapshot_preserve_min} = 'all'; return $config; } } if($opt->{allow_multiple}) { my $aref = $config->{$key} // []; my @val = ref($value) ? @$value : $value; push(@$aref, @val); TRACE "pushing option \"$key=[" . join(",", @val) . "]\" to $aref=[" . join(',', @$aref) . "]" if($do_trace); $value = $aref; } elsif(exists($config->{$key})) { unless($opt->{c_default}) { # note: computed defaults are already present WARN "Option \"$key\" redefined $error_statement"; } } TRACE "adding option \"$key=$value\" to $context context" if($do_trace); $value = undef if($value eq "no"); # we don't want to check for "no" all the time $config->{$key} = $value; return $config; } sub parse_config_line($$$;@) { my ($cur, $key, $value, %opts) = @_; my $root = $cur; $root = $root->{PARENT} while($root->{CONTEXT} ne "global"); my $error_statement = $opts{error_statement} // ""; if($key eq "volume") { $value =~ s/^"(.*)"$/$1/; $value =~ s/^'(.*)'$/$1/; $cur = $root; TRACE "config: context forced to: $cur->{CONTEXT}" if($do_trace); # be very strict about file options, for security sake my ($url_prefix, $path) = check_url($value, error_statement => "for option \"$key\" $error_statement"); return undef unless(defined($path)); TRACE "config: adding volume \"$url_prefix$path\" to global context" if($do_trace); die unless($cur->{CONTEXT} eq "global"); my $volume = { CONTEXT => "volume", PARENT => $cur, SUBSECTION => [], url => $url_prefix . $path, }; push(@{$cur->{SUBSECTION}}, $volume); $cur = $volume; } elsif($key eq "subvolume") { $value =~ s/^"(.*)"$/$1/; $value =~ s/^'(.*)'$/$1/; while($cur->{CONTEXT} ne "volume") { if($cur->{CONTEXT} eq "global") { TRACE "config: adding dummy volume context" if($do_trace); my $volume = { CONTEXT => "volume", PARENT => $cur, SUBSECTION => [], DUMMY => 1, url => "/dev/null", }; push(@{$cur->{SUBSECTION}}, $volume); $cur = $volume; last; } $cur = $cur->{PARENT} || die; TRACE "config: context changed to: $cur->{CONTEXT}" if($do_trace); } # be very strict about file options, for security sake my $url; if(!$cur->{DUMMY} && (my $rel_path = check_file($value, { relative => 1, wildcards => 1 }, sanitize => 1))) { $url = ($rel_path eq '.') ? $cur->{url} : $cur->{url} . '/' . $rel_path; } else { my ($url_prefix, $path) = check_url($value, accept_wildcards => 1, error_statement => "for option \"$key\"" . ($cur->{DUMMY} ? " (if no \"volume\" section is declared)" : "") . " $error_statement"); return undef unless(defined($path)); $url = $url_prefix . $path; } # snapshot_name defaults to subvolume name (or volume name if subvolume=".") my $default_snapshot_name = $url; $default_snapshot_name =~ s/^.*\///; $default_snapshot_name = 'ROOT' if($default_snapshot_name eq ""); # if volume="/" TRACE "config: adding subvolume \"$url\" to volume context: $cur->{url}" if($do_trace); my $subvolume = { CONTEXT => "subvolume", PARENT => $cur, # SUBSECTION => [], # handled by target propagation url => $url, snapshot_name => $default_snapshot_name, # computed default (c_default) }; $subvolume->{GLOB_CONTEXT} = 1 if($value =~ /\*/); push(@{$cur->{SUBSECTION}}, $subvolume); $cur = $subvolume; } elsif($key eq "target") { if($cur->{CONTEXT} eq "target") { $cur = $cur->{PARENT} || die; TRACE "config: context changed to: $cur->{CONTEXT}" if($do_trace); } # As of btrbk-0.28.0, target_type is optional and defaults to "send-receive" my $target_type = $config_target_types[0]; $target_type = lc($1) if($value =~ s/^([a-zA-Z_-]+)\s+//); unless(grep(/^\Q$target_type\E$/, @config_target_types)) { ERROR "Unknown target type \"$target_type\" $error_statement"; return undef; } $value =~ s/^"(.*)"$/$1/; $value =~ s/^'(.*)'$/$1/; my ($url_prefix, $path) = check_url($value, error_statement => "for option \"$key\" $error_statement"); return undef unless(defined($path)); TRACE "config: adding target \"$url_prefix$path\" (type=$target_type) to $cur->{CONTEXT} context" . ($cur->{url} ? ": $cur->{url}" : "") if($do_trace); my $target = { CONTEXT => "target", PARENT => $cur, target_type => $target_type, url => $url_prefix . $path, }; # NOTE: target sections are propagated to the apropriate SUBSECTION in _config_propagate_target() $cur->{TARGET} //= []; push(@{$cur->{TARGET}}, $target); $cur = $target; } else { return append_config_option($cur, $key, $value, $cur->{CONTEXT}, error_statement => $error_statement); } return $cur; } sub _config_propagate_target { my $cur = shift; foreach my $subsection (@{$cur->{SUBSECTION}}) { my @propagate_target; foreach my $target (@{$cur->{TARGET}}) { TRACE "propagating target \"$target->{url}\" from $cur->{CONTEXT} context to: $subsection->{CONTEXT} $subsection->{url}" if($do_trace); die if($target->{SUBSECTION}); # don't propagate if a target of same target_type and url already exists in subsection if($subsection->{TARGET} && grep({ ($_->{url} eq $target->{url}) && ($_->{target_type} eq $target->{target_type}) } @{$subsection->{TARGET}})) { DEBUG "Skip propagation of \"target $target->{target_type} $target->{url}\" from $cur->{CONTEXT} context to \"$subsection->{CONTEXT} $subsection->{url}\": same target already exists"; next; } my %copy = ( %$target, PARENT => $subsection ); push @propagate_target, \%copy; } $subsection->{TARGET} //= []; unshift @{$subsection->{TARGET}}, @propagate_target; # maintain config order: propagated targets go in front of already defined targets if($subsection->{CONTEXT} eq "subvolume") { # finally create missing SUBSECTION in subvolume context die if($subsection->{SUBSECTION}); $subsection->{SUBSECTION} = $subsection->{TARGET}; } else { # recurse into SUBSECTION _config_propagate_target($subsection); } } delete $cur->{TARGET}; return $cur; } sub _config_collect_values { my $config = shift; my $key = shift; my @values; push(@values, @{$config->{$key}}) if(ref($config->{$key}) eq "ARRAY"); foreach (@{$config->{SUBSECTION}}) { push(@values, _config_collect_values($_, $key)); } return @values; } sub init_config(@) { my %defaults = ( CONTEXT => "meta", @_ ); # set defaults foreach (keys %config_options) { next if $config_options{$_}->{deprecated}; # don't pollute hash with deprecated options $defaults{$_} = $config_options{$_}->{default}; } return { CONTEXT => "global", SUBSECTION => [], PARENT => \%defaults }; } sub _config_file(@) { my @config_files = @_; foreach my $file (@config_files) { TRACE "config: checking for file: $file" if($do_trace); return $file if(-r "$file"); } return undef; } sub parse_config($) { my $file = shift; return undef unless($file); my $root = init_config(SRC_FILE => $file); my $cur = $root; TRACE "config: open configuration file: $file" if($do_trace); open(FILE, '<', $file) or die $!; while () { chomp; s/((?:[^"'#]*(?:"[^"]*"|'[^']*'))*[^"'#]*)#.*/$1/; # remove comments next if /^\s*$/; # ignore empty lines s/^\s*//; # remove leading whitespace s/\s*$//; # remove trailing whitespace TRACE "config: parsing line $. with context=$cur->{CONTEXT}: \"$_\"" if($do_trace); unless(/^([a-zA-Z_]+)(?:\s+(.*))?$/) { ERROR "Parse error in \"$file\" line $."; $root = undef; last; } unless($cur = parse_config_line($cur, lc($1), $2 // "", error_statement => "in \"$file\" line $.")) { $root = undef; last; } TRACE "line processed: new context=$cur->{CONTEXT}" if($do_trace); } close FILE || ERROR "Failed to close configuration file: $!"; _config_propagate_target($root); return $root; } # sets $target->{CONFIG}->{ABORTED} on failure # sets $target->{SUBVOL_RECEIVED} sub macro_send_receive(@) { my %info = @_; my $source = $info{source} || die; my $target = $info{target} || die; my $parent = $info{parent}; my @clone_src = @{ $info{clone_src} // [] }; # copy array my $config_target = $target->{CONFIG}; my $target_type = $config_target->{target_type} || die; my $incremental = config_key($config_target, "incremental"); # check for existing target subvolume if(my $err_vol = vinfo_subvol($target, $source->{NAME})) { my $err_msg = "Please delete stray subvolume: \"btrfs subvolume delete $err_vol->{PRINT}\""; ABORTED($config_target, "Target subvolume \"$err_vol->{PRINT}\" already exists"); FIX_MANUALLY($config_target, $err_msg); ERROR ABORTED_TEXT($config_target) . ", aborting send/receive of: $source->{PRINT}"; ERROR $err_msg; return undef; } if($incremental) { # create backup from latest common if($parent) { INFO "Creating incremental backup..."; } elsif($incremental ne "strict") { INFO "No common parent subvolume present, creating non-incremental backup..."; } else { WARN "Backup to $target->{PRINT} failed: no common parent subvolume found for \"$source->{PRINT}\", and option \"incremental\" is set to \"strict\""; ABORTED($config_target, "No common parent subvolume found, and option \"incremental\" is set to \"strict\""); return undef; } unless(config_key($target, "incremental_clones")) { INFO "Ignoring " . scalar(@clone_src) . " clone sources (incremental_clones=no)" if(@clone_src); @clone_src = (); delete $info{clone_src}; } } else { INFO "Creating non-incremental backup..."; $parent = undef; @clone_src = (); delete $info{parent}; delete $info{clone_src}; } my $ret; my $vol_received; my $raw_info; if($target_type eq "send-receive") { $ret = btrfs_send_receive($source, $target, $parent, \@clone_src, \$vol_received); ABORTED($config_target, "Failed to send/receive subvolume") unless($ret); } elsif($target_type eq "raw") { unless($dryrun) { # make sure we know the source uuid if($source->{node}{uuid} =~ /^$fake_uuid_prefix/) { DEBUG "Fetching uuid of new subvolume: $source->{PRINT}"; my $detail = btrfs_subvolume_show($source); return undef unless($detail); die unless($detail->{uuid}); $source->{node}{uuid} = $detail->{uuid}; $uuid_cache{$detail->{uuid}} = $source->{node}; } } $ret = btrfs_send_to_file($source, $target, $parent, \$vol_received, \$raw_info); ABORTED($config_target, "Failed to send subvolume to raw file") unless($ret); } else { die "Illegal target type \"$target_type\""; } # inject fake vinfo # NOTE: it's not possible to add (and compare) correct target $detail # from btrfs_send_receive(), as source detail also has fake uuid. if($ret) { vinfo_inject_child($target, $vol_received, { # NOTE: this is not necessarily the correct parent_uuid (on # receive, btrfs-progs picks the uuid of the first (lowest id) # matching possible parent), whereas the target_parent is the # first from _correlated_nodes(). # # NOTE: the parent_uuid of an injected receive target is not used # anywhere in btrbk at the time of writing parent_uuid => $parent ? $info{target_parent_node}->{uuid} : '-', received_uuid => $source->{node}{received_uuid} eq '-' ? $source->{node}{uuid} : $source->{node}{received_uuid}, readonly => 1, TARGET_TYPE => $target_type, FORCE_PRESERVE => 'preserve forced: created just now', }, $raw_info); } # add info to $config->{SUBVOL_RECEIVED} $info{received_type} = $target_type || die; $info{received_subvolume} = $vol_received || die; $target->{SUBVOL_RECEIVED} //= []; push(@{$target->{SUBVOL_RECEIVED}}, \%info); unless($ret) { $info{ERROR} = 1; return undef; } return 1; } # sets $result_vinfo->{CONFIG}->{ABORTED} on failure # sets $result_vinfo->{SUBVOL_DELETED} sub macro_delete($$$$;@) { my $root_subvol = shift || die; my $subvol_basename = shift // die; my $result_vinfo = shift || die; my $schedule_options = shift || die; my %delete_options = @_; my @schedule; foreach my $vol (@{vinfo_subvol_list($root_subvol, btrbk_direct_leaf => $subvol_basename)}) { push(@schedule, { value => $vol, # name => $vol->{PRINT}, # only for logging btrbk_date => $vol->{node}{BTRBK_DATE}, preserve => $vol->{node}{FORCE_PRESERVE}, }); } my (undef, $delete) = schedule( %$schedule_options, schedule => \@schedule, preserve_date_in_future => 1, ); my @delete_success; foreach my $vol (@$delete) { # NOTE: we do not abort on qgroup destroy errors btrfs_qgroup_destroy($vol, %{$delete_options{qgroup}}) if($delete_options{qgroup}->{destroy}); if(btrfs_subvolume_delete($vol, %delete_options)) { push @delete_success, $vol; } } INFO "Deleted " . scalar(@delete_success) . " subvolumes in: $root_subvol->{PRINT}/$subvol_basename.*"; $result_vinfo->{SUBVOL_DELETED} //= []; push @{$result_vinfo->{SUBVOL_DELETED}}, @delete_success; if(scalar(@delete_success) != scalar(@$delete)) { ABORTED($result_vinfo, "Failed to delete subvolume"); return undef; } return 1; } sub macro_archive_target($$$;$) { my $sroot = shift || die; my $droot = shift || die; my $snapshot_name = shift // die; my $schedule_options = shift // {}; my @schedule; # NOTE: this is pretty much the same as "resume missing" my $has_unexpected_location = 0; foreach my $svol (@{vinfo_subvol_list($sroot, readonly => 1, btrbk_direct_leaf => $snapshot_name, sort => 'path')}) { if(my $ff = vinfo_match(\@exclude_vf, $svol)) { INFO "Skipping archive candidate \"$svol->{PRINT}\": Match on exclude pattern \"$ff->{unparsed}\""; next; } next if(get_receive_targets($droot, $svol, exact => 1, warn => 1, ret_unexpected_only => \$has_unexpected_location)); DEBUG "Adding archive candidate: $svol->{PRINT}"; push @schedule, { value => $svol, btrbk_date => $svol->{node}{BTRBK_DATE}, preserve => $svol->{node}{FORCE_PRESERVE}, }; } if($has_unexpected_location) { ABORTED($droot, "Receive targets of archive candidates exist at unexpected location"); WARN "Skipping archiving of \"$sroot->{PRINT}/${snapshot_name}.*\": " . ABORTED_TEXT($droot); return undef; } # add all present archives as informative_only: these are needed for correct results of schedule() my $last_dvol_date; foreach my $dvol (@{vinfo_subvol_list($droot, readonly => 1, btrbk_direct_leaf => $snapshot_name)}) { my $btrbk_date = $dvol->{node}{BTRBK_DATE}; push @schedule, { informative_only => 1, value => $dvol, btrbk_date => $btrbk_date, }; # find last present archive (by btrbk_date, needed for archive_exclude_older below) $last_dvol_date = $btrbk_date if((not defined($last_dvol_date)) || (cmp_date($btrbk_date, $last_dvol_date) > 0)); } my ($preserve, undef) = schedule( schedule => \@schedule, preserve => config_preserve_hash($droot, "archive"), preserve_threshold_date => (config_key($droot, "archive_exclude_older") ? $last_dvol_date : undef), result_preserve_action_text => 'archive', result_delete_action_text => '', %$schedule_options ); my @archive = grep defined, @$preserve; # remove entries with no value from list (archive subvolumes) my $archive_total = scalar @archive; my $archive_success = 0; foreach my $svol (@archive) { my ($clone_src, $target_parent_node); my $parent = get_best_parent($svol, $sroot, $droot, strict_related => 0, clone_src => \$clone_src, target_parent_node => \$target_parent_node); if(macro_send_receive(source => $svol, target => $droot, parent => $parent, # this is if no suitable parent found clone_src => $clone_src, target_parent_node => $target_parent_node, )) { $archive_success++; } else { ERROR("Error while archiving subvolumes, aborting"); last; } } if($archive_total) { INFO "Archived $archive_success/$archive_total subvolumes"; } else { INFO "No missing archives found"; } return $archive_success; } sub cmp_date($$) { return (($_[0]->[0] <=> $_[1]->[0]) || # unix time ($_[0]->[1] <=> $_[1]->[1])); # NN } sub schedule(@) { my %args = @_; my $schedule = $args{schedule} || die; my $preserve = $args{preserve} || die; my $preserve_date_in_future = $args{preserve_date_in_future}; my $preserve_threshold_date = $args{preserve_threshold_date}; my $results_list = $args{results}; my $result_hints = $args{result_hints} // {}; my $result_preserve_action_text = $args{result_preserve_action_text}; my $result_delete_action_text = $args{result_delete_action_text} // 'delete'; my $preserve_day_of_week = $preserve->{dow} || die; my $preserve_hour_of_day = $preserve->{hod} // die; my $preserve_min_n = $preserve->{min_n}; my $preserve_min_q = $preserve->{min_q}; my $preserve_hourly = $preserve->{h}; my $preserve_daily = $preserve->{d}; my $preserve_weekly = $preserve->{w}; my $preserve_monthly = $preserve->{m}; my $preserve_yearly = $preserve->{y}; DEBUG "Schedule: " . format_preserve_matrix($preserve, format => "debug_text"); # 0 1 2 3 4 5 6 7 8 # sec, min, hour, mday, mon, year, wday, yday, isdst # sort the schedule, ascending by date # regular entries come in front of informative_only my @sorted_schedule = sort { cmp_date($a->{btrbk_date}, $b->{btrbk_date} ) || (($a->{informative_only} ? ($b->{informative_only} ? 0 : 1) : ($b->{informative_only} ? -1 : 0))) } @$schedule; DEBUG "Scheduler reference time: " . timestamp(\@tm_now, 'debug-iso'); # first, do our calendar calculations # - days start on $preserve_hour_of_day (or 00:00 if timestamp_format=short) # - weeks start on $preserve_day_of_week # - months start on first $preserve_day_of_week of month # - years start on first $preserve_day_of_week of year # NOTE: leap hours are NOT taken into account for $delta_hours my $now_h = timegm_nocheck( 0, 0, $tm_now[2], $tm_now[3], $tm_now[4], $tm_now[5] ); # use timelocal() here (and below) if you want to honor leap hours foreach my $href (@sorted_schedule) { my @tm = localtime($href->{btrbk_date}->[0]); my $has_exact_time = $href->{btrbk_date}->[2]; my $delta_hours_from_hod = $tm[2] - ($has_exact_time ? $preserve_hour_of_day : 0); my $delta_days_from_eow = $tm[6] - $day_of_week_map{$preserve_day_of_week}; if($delta_hours_from_hod < 0) { $delta_hours_from_hod += 24; $delta_days_from_eow -= 1; } if($delta_days_from_eow < 0) { $delta_days_from_eow += 7; } my $month_corr = $tm[4]; # [0..11] my $year_corr = $tm[5]; if($tm[3] <= $delta_days_from_eow) { # our month/year start on first $preserve_day_of_week, corrected value $month_corr -= 1; if($month_corr < 0) { $month_corr = 11; $year_corr -= 1; } } # check timegm: ignores leap hours my $delta_hours = int(($now_h - timegm_nocheck( 0, 0, $tm[2], $tm[3], $tm[4], $tm[5] ) ) / (60 * 60)); my $delta_days = int(($delta_hours + $delta_hours_from_hod) / 24); # days from beginning of day my $delta_weeks = int(($delta_days + $delta_days_from_eow) / 7); # weeks from beginning of week my $delta_years = ($tm_now[5] - $year_corr); my $delta_months = $delta_years * 12 + ($tm_now[4] - $month_corr); $href->{delta_hours} = $delta_hours; $href->{delta_days} = $delta_days; $href->{delta_weeks} = $delta_weeks; $href->{delta_months} = $delta_months; $href->{delta_years} = $delta_years; # these are only needed for text output (format_preserve_delta) $href->{year} = $year_corr + 1900; $href->{month} = $month_corr + 1; $href->{delta_hours_from_hod} = $delta_hours_from_hod; $href->{delta_days_from_eow} = $delta_days_from_eow; $href->{real_hod} = $preserve_hour_of_day if($has_exact_time); if($preserve_date_in_future && ($delta_hours < 0)) { $href->{preserve} = "preserve forced: " . -($delta_hours) . " hours in the future"; } } my %first_in_delta_hours; my %first_in_delta_days; my %first_in_delta_weeks; my %first_weekly_in_delta_months; my %first_monthly_in_delta_years; # filter "preserve all within N days/weeks/..." foreach my $href (@sorted_schedule) { if($preserve_min_q) { if($preserve_min_q eq 'all') { $href->{preserve} = "preserve min: all"; } elsif($preserve_min_q eq 'h') { $href->{preserve} = "preserve min: $href->{delta_hours} hours ago" if($href->{delta_hours} <= $preserve_min_n); } elsif($preserve_min_q eq 'd') { $href->{preserve} = "preserve min: $href->{delta_days} days ago" if($href->{delta_days} <= $preserve_min_n); } elsif($preserve_min_q eq 'w') { $href->{preserve} = "preserve min: $href->{delta_weeks} weeks ago" if($href->{delta_weeks} <= $preserve_min_n); } elsif($preserve_min_q eq 'm') { $href->{preserve} = "preserve min: $href->{delta_months} months ago" if($href->{delta_months} <= $preserve_min_n); } elsif($preserve_min_q eq 'y') { $href->{preserve} = "preserve min: $href->{delta_years} years ago" if($href->{delta_years} <= $preserve_min_n); } } $first_in_delta_hours{$href->{delta_hours}} //= $href; } if($preserve_min_q && ($preserve_min_q eq 'latest') && (scalar @sorted_schedule)) { my $href = $sorted_schedule[-1]; $href->{preserve} = 'preserve min: latest'; } # filter hourly, daily, weekly, monthly, yearly foreach (sort {$b <=> $a} keys %first_in_delta_hours) { my $href = $first_in_delta_hours{$_} || die; if($preserve_hourly && (($preserve_hourly eq 'all') || ($href->{delta_hours} <= $preserve_hourly))) { $href->{preserve} = "preserve hourly: first of hour, $href->{delta_hours} hours ago"; } $first_in_delta_days{$href->{delta_days}} //= $href; } foreach (sort {$b <=> $a} keys %first_in_delta_days) { my $href = $first_in_delta_days{$_} || die; if($preserve_daily && (($preserve_daily eq 'all') || ($href->{delta_days} <= $preserve_daily))) { $href->{preserve} = "preserve daily: first of day" . ($href->{real_hod} ? sprintf(" (starting at %02u:00)", $href->{real_hod}) : "") . ", $href->{delta_days} days ago" . (defined($href->{real_hod}) ? ($href->{delta_hours_from_hod} ? ", $href->{delta_hours_from_hod}h after " : ", at ") . sprintf("%02u:00", $href->{real_hod}) : ""); } $first_in_delta_weeks{$href->{delta_weeks}} //= $href; } foreach (sort {$b <=> $a} keys %first_in_delta_weeks) { my $href = $first_in_delta_weeks{$_} || die; if($preserve_weekly && (($preserve_weekly eq 'all') || ($href->{delta_weeks} <= $preserve_weekly))) { $href->{preserve} = "preserve weekly: $href->{delta_weeks} weeks ago," . _format_preserve_delta($href, $preserve_day_of_week); } $first_weekly_in_delta_months{$href->{delta_months}} //= $href; } foreach (sort {$b <=> $a} keys %first_weekly_in_delta_months) { my $href = $first_weekly_in_delta_months{$_} || die; if($preserve_monthly && (($preserve_monthly eq 'all') || ($href->{delta_months} <= $preserve_monthly))) { $href->{preserve} = "preserve monthly: first weekly of month $href->{year}-" . sprintf("%02u", $href->{month}) . " ($href->{delta_months} months ago," . _format_preserve_delta($href, $preserve_day_of_week) . ")"; } $first_monthly_in_delta_years{$href->{delta_years}} //= $href; } foreach (sort {$b <=> $a} keys %first_monthly_in_delta_years) { my $href = $first_monthly_in_delta_years{$_} || die; if($preserve_yearly && (($preserve_yearly eq 'all') || ($href->{delta_years} <= $preserve_yearly))) { $href->{preserve} = "preserve yearly: first weekly of year $href->{year} ($href->{delta_years} years ago," . _format_preserve_delta($href, $preserve_day_of_week) . ")"; } } # assemble results my @delete; my @preserve; my %result_base = ( %$preserve, scheme => format_preserve_matrix($preserve), %$result_hints, ); my $count_defined = 0; foreach my $href (@sorted_schedule) { my $result_reason_text = $href->{preserve}; my $result_action_text; unless($href->{informative_only}) { if($href->{preserve}) { if($preserve_threshold_date && (cmp_date($href->{btrbk_date}, $preserve_threshold_date) <= 0)) { # older than threshold, do not add to preserve list $result_reason_text = "$result_reason_text, ignored (archive_exclude_older) older than existing archive"; } else { push(@preserve, $href->{value}); $result_action_text = $result_preserve_action_text; } } else { push(@delete, $href->{value}); $result_action_text = $result_delete_action_text; } $count_defined++; } TRACE join(" ", "schedule: $href->{value}{PRINT}", ($href->{informative_only} ? "(informative_only)" : uc($result_action_text || "-")), ($result_reason_text // "-")) if($do_trace && $href->{value} && $href->{value}{PRINT}); push @$results_list, { %result_base, action => $result_action_text, reason => $result_reason_text, value => $href->{value}, } if($results_list); } DEBUG "Preserving " . @preserve . "/" . $count_defined . " items"; return (\@preserve, \@delete); } sub _format_preserve_delta($$$) { my $href = shift; my $preserve_day_of_week = shift; my $s = ""; $s .= " $href->{delta_days_from_eow}d" if($href->{delta_days_from_eow}); $s .= " $href->{delta_hours_from_hod}h" if($href->{delta_hours_from_hod}); return ($s ? "$s after " : " at ") . $preserve_day_of_week . (defined($href->{real_hod}) ? sprintf(" %02u:00", $href->{real_hod}) : ""); } sub format_preserve_matrix($@) { my $preserve = shift || die; my %opts = @_; my $format = $opts{format} // "short"; if($format eq "debug_text") { my @out; my %trans = ( h => 'hours', d => 'days', w => 'weeks', m => 'months', y => 'years' ); if($preserve->{min_q} && ($preserve->{min_q} eq 'all')) { push @out, "all forever"; } else { push @out, "latest" if($preserve->{min_q} && ($preserve->{min_q} eq 'latest')); push @out, "all within $preserve->{min_n} $trans{$preserve->{min_q}}" if($preserve->{min_n} && $preserve->{min_q}); push @out, "first of day (starting at " . sprintf("%02u:00", $preserve->{hod}) . ") for $preserve->{d} days" if($preserve->{d}); unless($preserve->{d} && ($preserve->{d} eq 'all')) { push @out, "first daily in week (starting on $preserve->{dow}) for $preserve->{w} weeks" if($preserve->{w}); unless($preserve->{w} && ($preserve->{w} eq 'all')) { push @out, "first weekly of month for $preserve->{m} months" if($preserve->{m}); unless($preserve->{m} && ($preserve->{m} eq 'all')) { push @out, "first weekly of year for $preserve->{y} years" if($preserve->{y}); } } } } return 'preserving ' . join('; ', @out); } my $s = ""; if($preserve->{min_q} && ($preserve->{min_q} eq 'all')) { $s = '*d+'; } else { # $s .= '.+' if($preserve->{min_q} && ($preserve->{min_q} eq 'latest')); $s .= $preserve->{min_n} . $preserve->{min_q} . '+' if($preserve->{min_n} && $preserve->{min_q}); foreach (qw(h d w m y)) { my $val = $preserve->{$_} // 0; next unless($val); $val = '*' if($val eq 'all'); $s .= ($s ? ' ' : '') . $val . $_; } if(($format ne "config") && ($preserve->{d} || $preserve->{w} || $preserve->{m} || $preserve->{y})) { $s .= " ($preserve->{dow}, " . sprintf("%02u:00", $preserve->{hod}) . ")"; } } return $s; } sub timestamp($$;$) { my $time = shift // die; # unixtime, or arrayref from localtime() my $format = shift; my $tm_is_utc = shift; my @tm = ref($time) ? @$time : localtime($time); my $ts; # NOTE: can't use POSIX::strftime(), as "%z" always prints offset of local timezone! if($format eq "short") { return sprintf('%04u%02u%02u', $tm[5] + 1900, $tm[4] + 1, $tm[3]); } elsif($format eq "long") { return sprintf('%04u%02u%02uT%02u%02u', $tm[5] + 1900, $tm[4] + 1, $tm[3], $tm[2], $tm[1]); } elsif($format eq "long-iso") { $ts = sprintf('%04u%02u%02uT%02u%02u%02u', $tm[5] + 1900, $tm[4] + 1, $tm[3], $tm[2], $tm[1], $tm[0]); } elsif($format eq "debug-iso") { $ts = sprintf('%04u-%02u-%02uT%02u:%02u:%02u', $tm[5] + 1900, $tm[4] + 1, $tm[3], $tm[2], $tm[1], $tm[0]); } else { die; } if($tm_is_utc) { $ts .= '+0000'; # or 'Z' } else { my $offset = timegm(@tm) - timelocal(@tm); if($offset < 0) { $ts .= '-'; $offset = -$offset; } else { $ts .= '+'; } my $h = int($offset / (60 * 60)); die if($h > 24); # sanity check, something went really wrong $ts .= sprintf('%02u%02u', $h, int($offset / 60) % 60); } return $ts; return undef; } sub print_header(@) { my %args = @_; my $config = $args{config}; print "--------------------------------------------------------------------------------\n"; print "$args{title} ($VERSION_INFO)\n\n"; if($args{time}) { print " Date: " . localtime($args{time}) . "\n"; } if($config) { print " Config: " . config_key($config, "SRC_FILE") . "\n"; } if($dryrun) { print " Dryrun: YES\n"; } if($config && $config->{CMDLINE_FILTER_LIST}) { my @list = @{$config->{CMDLINE_FILTER_LIST}}; print " Filter: "; print join("\n ", @list); print "\n"; } if($args{info}) { print "\n" . join("\n", grep(defined, @{$args{info}})) . "\n"; } if($args{options} && (scalar @{$args{options}})) { print "\nOptions:\n "; print join("\n ", @{$args{options}}); print "\n"; } if($args{legend}) { print "\nLegend:\n "; print join("\n ", @{$args{legend}}); print "\n"; } print "--------------------------------------------------------------------------------\n"; print "\n" if($args{paragraph}); } sub print_footer($$) { my $config = shift; my $exit_status = shift; if($exit_status) { print "\nNOTE: Some errors occurred, which may result in missing backups!\n"; print "Please check warning and error messages above.\n"; my @fix_manually_text = _config_collect_values($config, "FIX_MANUALLY"); if(scalar(@fix_manually_text)) { my @unique = do { my %seen; grep { !$seen{$_}++ } @fix_manually_text }; print join("\n", @unique) . "\n"; } } if($dryrun) { print "\nNOTE: Dryrun was active, none of the operations above were actually executed!\n"; } } sub print_table($;$) { my $data = shift; my $spacing = shift // " "; my $maxlen = 0; foreach (@$data) { $maxlen = length($_->[0]) if($maxlen < length($_->[0])); } foreach (@$data) { print $_->[0] . ((' ' x ($maxlen - length($_->[0]))) . $spacing) . $_->[1] . "\n"; } } sub print_formatted(@) { my $format_key = shift || die; my $data = shift || die; my $default_format = "table"; my %args = @_; my $title = $args{title}; my $table_format = ref($format_key) ? $format_key : $table_formats{$format_key}; my $format = $args{output_format} || $output_format || $default_format; my $pretty = $args{pretty} // $output_pretty; my $no_header = $args{no_header}; my $fh = $args{outfile} // *STDOUT; my $table_spacing = 2; my $empty_cell_char = $args{empty_cell_char} // "-"; my @keys; my %ralign; my %hide_column; if($format =~ s/^col:\s*(h:)?\s*//) { $no_header = 1 if($1); foreach (split(/\s*,\s*/, $format)) { $ralign{$_} = 1 if s/:R(ALIGN)?$//i; push @keys, lc($_); } } else { unless(exists($table_format->{$format})) { WARN "Unsupported output format \"$format\", defaulting to \"$default_format\" format."; $format = $default_format; } @keys = @{$table_format->{$format}}; %ralign = %{$table_format->{RALIGN} // {}}; } # strips leading "-" from @keys %hide_column = map { $_ => 1 } grep { s/^-// } @keys; if($format eq "single_column") { # single-column: newline separated values, no headers my $key = $keys[0]; foreach (grep defined, map $_->{$key}, @$data) { print $fh $_ . "\n" if($_ ne ""); } } elsif($format eq "raw") { # output: key0="value0" key1="value1" ... foreach my $row (@$data) { print $fh "format=\"$format_key\" "; print $fh join(' ', map { "$_=" . quoteshell(($row->{$_} // "")) } @keys) . "\n"; } } elsif(($format eq "tlog") || ($format eq "syslog")) { # output: value0 value1, ... unless($no_header) { print $fh join(' ', map uc($_), @keys) . "\n"; # unaligned upper case headings } foreach my $row (@$data) { my $line = join(' ', map { ((defined($row->{$_}) && ($_ eq "message")) ? '# ' : '') . ($row->{$_} // "-") } @keys); if($format eq "syslog") { # dirty hack, ignore outfile on syslog format syslog($line); } else { print $fh ($line . "\n"); } } } else { # Text::CharWidth does it correctly with wide chars (e.g. asian) taking up two columns my $termwidth = eval_quiet { require Text::CharWidth; } ? \&Text::CharWidth::mbswidth : eval_quiet { require Encode; } ? sub { length(Encode::decode_utf8(shift)) } : sub { length(shift) }; # sanitize and calculate maxlen for each column my %maxlen = map { $_ => $no_header ? 0 : length($_) } @keys; my @formatted_data; foreach my $row (@$data) { my %formatted_row; foreach my $key (@keys) { my $val = $row->{$key}; $val = join(',', @$val) if(ref $val eq "ARRAY"); $hide_column{$key} = 0 if(defined($val)); $val = $empty_cell_char if(!defined($val) || ($val eq "")); $formatted_row{$key} = $val; my $vl = $termwidth->($val); $maxlen{$key} = $vl if($maxlen{$key} < $vl); } push @formatted_data, \%formatted_row; } my @visible_keys = grep !$hide_column{$_}, @keys; # print title if($title) { print $fh "$title\n"; print $fh '-' x length($title) . "\n"; # separator line } # print keys (headings) unless($no_header) { my $fill = 0; foreach (@visible_keys) { print $fh ' ' x $fill; $fill = $maxlen{$_} - length($_); if($pretty) { # use aligned lower case headings (with separator line below) if($ralign{$_}) { print $fh ' ' x $fill; $fill = 0; } print $fh $_; } else { print $fh uc($_); # default unaligned upper case headings } $fill += $table_spacing; } print $fh "\n"; $fill = 0; if($pretty) { # separator line after header foreach (@visible_keys) { print $fh ' ' x $fill; print $fh '-' x $maxlen{$_}; $fill = $table_spacing; } print $fh "\n"; # alternative (all above in one line ;) #print $fh join(' ' x $table_spacing, map { '-' x ($maxlen{$_}) } @keys) . "\n"; } } # print values foreach my $row (@formatted_data) { my $fill = 0; foreach (@visible_keys) { my $val = $row->{$_}; print $fh ' ' x $fill; $fill = $maxlen{$_} - $termwidth->($val); if($ralign{$_}) { print $fh ' ' x $fill; $fill = 0; } print $fh $val; $fill += $table_spacing; } print $fh "\n"; } # print additional newline for paragraphs if($args{paragraph}) { print $fh "\n"; } } } sub print_size($) { my $size = shift; if($output_format && ($output_format eq "raw")) { return $size; } return "-" if($size == 0); my ($unit, $mul); if(@output_unit) { ($unit, $mul) = @output_unit; } else { ($unit, $mul) = ("KiB", 1024); ($unit, $mul) = ("MiB", $mul * 1024) if($size > $mul * 1024); ($unit, $mul) = ("GiB", $mul * 1024) if($size > $mul * 1024); ($unit, $mul) = ("TiB", $mul * 1024) if($size > $mul * 1024); } return $size if($mul == 1); return sprintf('%.2f', ($size / $mul)) . " $unit"; } sub _origin_tree { my $prefix = shift; my $node = shift // die; my $lines = shift; my $nodelist = shift; my $depth = shift // 0; my $seen = shift // []; my $norecurse = shift; my $uuid = $node->{uuid} || die; # cache a bit, this might be large # note: root subvolumes dont have REL_PATH $nodelist //= [ (sort { ($a->{REL_PATH} // "") cmp ($b->{REL_PATH} // "") } values %uuid_cache) ]; my $prefix_spaces = ' ' x (($depth * 4) - ($prefix ? 4 : 0)); push(@$lines, { tree => "${prefix_spaces}${prefix}" . _fs_path($node), uuid => $node->{uuid}, parent_uuid => $node->{parent_uuid}, received_uuid => $node->{received_uuid}, }); # handle deep recursion return 0 if(grep /^$uuid$/, @$seen); if($node->{parent_uuid} ne '-') { my $parent_node = $uuid_cache{$node->{parent_uuid}}; if($parent_node) { if($norecurse) { push(@$lines,{ tree => "${prefix_spaces} ^-- ...", uuid => $parent_node->{uuid}, parent_uuid => $parent_node->{parent_uuid}, received_uuid => $parent_node->{received_uuid}, recursion => 'stop_recursion', }); return 0; } if($parent_node->{readonly}) { _origin_tree("^-- ", $parent_node, $lines, $nodelist, $depth + 1, undef, 1); # end recursion } else { _origin_tree("^-- ", $parent_node, $lines, $nodelist, $depth + 1); } } else { push(@$lines,{ tree => "${prefix_spaces} ^-- {parent_uuid}>" }); } } return 0 if($norecurse); push(@$seen, $uuid); if($node->{received_uuid} ne '-') { my $received_uuid = $node->{received_uuid}; my @receive_parents; # there should be only one! my @receive_twins; foreach (@$nodelist) { next if($_->{uuid} eq $uuid); if($received_uuid eq $_->{uuid} && $_->{readonly}) { _origin_tree("", $_, \@receive_parents, $nodelist, $depth, $seen); } elsif(($_->{received_uuid} ne '-') && ($received_uuid eq $_->{received_uuid}) && $_->{readonly}) { _origin_tree("", $_, \@receive_twins, $nodelist, $depth, $seen, 1); # end recursion } } push @$lines, @receive_twins; push @$lines, @receive_parents; } return 0; } sub exit_status { my $config = shift; foreach my $subsection (@{$config->{SUBSECTION}}) { return 10 if(IS_ABORTED($subsection, "abort_")); return 10 if(defined($subsection->{FIX_MANUALLY})); # treated as errors return 10 if(exit_status($subsection)); } return 0; } MAIN: { # NOTE: Since v0.26.0, btrbk does not enable taint mode (perl -T) by # default, and does not hardcode $PATH anymore. # # btrbk still does all taint checks, and can be run in taint mode. # In order to enable taint mode, run `perl -T btrbk`. # # see: perlrun(1), perlsec(1) # my $taint_mode_enabled = eval '${^TAINT}'; if($taint_mode_enabled) { # we are running in tainted mode (perl -T), sanitize %ENV delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; # in taint mode, perl needs an untainted $PATH. $ENV{PATH} = '/sbin:/bin:/usr/sbin:/usr/bin'; } Getopt::Long::Configure qw(gnu_getopt); my $start_time = time; @tm_now = localtime($start_time); my @config_override_cmdline; my @exclude_cmdline; my ($config_cmdline, $lockfile_cmdline, $print_schedule, $preserve_snapshots, $preserve_backups, $wipe_snapshots, $skip_snapshots, $skip_backups, $archive_raw, $extents_related, ); # Calling btrbk via "lsbtr" symlink acts as an alias for "btrbk ls", # while also changing the semantics of the command line options. $program_name = $0; $program_name =~ s/^.*\///; # remove path my @getopt_options = ( # common options 'help|h' => sub { HELP_MESSAGE; exit 0; }, 'version' => sub { VERSION_MESSAGE; exit 0; }, 'quiet|q' => \$quiet, 'verbose|v' => sub { $loglevel = ($loglevel =~ /^[0-9]+$/) ? $loglevel+1 : 2; }, 'loglevel|l=s' => \$loglevel, 'format=s' => \$output_format, 'single-column|1' => sub { $output_format = "single_column" }, 'pretty' => \$output_pretty, 'config|c=s' => \$config_cmdline, 'override=s' => \@config_override_cmdline, # e.g. --override=incremental=no 'lockfile=s' => \$lockfile_cmdline, ); push @getopt_options, ($program_name eq "lsbtr") ? ( # "lsbtr" options 'long|l' => sub { $output_format = "table" }, 'uuid|u' => sub { $output_format = "long" }, 'raw' => sub { $output_format = "raw" }, ) : ( # "btrbk" options 'dry-run|n' => \$dryrun, 'exclude=s' => \@exclude_cmdline, 'preserve|p' => sub { $preserve_snapshots = "preserve", $preserve_backups = "preserve" }, 'preserve-snapshots' => sub { $preserve_snapshots = "preserve-snapshots" }, 'preserve-backups' => sub { $preserve_backups = "preserve-backups" }, 'wipe' => \$wipe_snapshots, 'progress' => \$show_progress, 'related' => \$extents_related, 'table|t' => sub { $output_format = "table" }, 'long|L' => sub { $output_format = "long" }, 'print-schedule|S' => \$print_schedule, 'raw' => \$archive_raw, 'bytes' => sub { @output_unit = ("", 1 ) }, 'kbytes' => sub { @output_unit = ("KiB", 1024 ) }, 'mbytes' => sub { @output_unit = ("MiB", 1024 * 1024 ) }, 'gbytes' => sub { @output_unit = ("GiB", 1024 * 1024 * 1024 ) }, 'tbytes' => sub { @output_unit = ("TiB", 1024 * 1024 * 1024 * 1024 ) }, ); unless(GetOptions(@getopt_options)) { ERROR_HELP_MESSAGE; exit 2; } if($program_name eq "lsbtr") { unshift @ARGV, './' unless(@ARGV); # default to current path unshift @ARGV, "ls"; # implicit "btrbk ls" } my $command = shift @ARGV; unless($command) { HELP_MESSAGE; exit 2; } # assign command line options @config_src = ( $config_cmdline ) if($config_cmdline); $loglevel = { error => 0, warn => 1, warning => 1, info => 2, debug => 3, trace => 4 }->{$loglevel} // $loglevel; unless($loglevel =~ /^[0-9]+$/) { ERROR "Unknown loglevel: $loglevel"; ERROR_HELP_MESSAGE; exit 2; } $do_trace = 1 if($loglevel >= 4); require_data_dumper() if($do_trace || ($VERSION =~ /-dev$/)); # check command line options if($show_progress && (not check_exe('mbuffer'))) { WARN 'Found option "--progress", but required executable "mbuffer" does not exist on your system. Please install "mbuffer".'; $show_progress = 0; } my ($action_run, $action_usage, $action_resolve, $action_diff, $action_extents, $action_origin, $action_config_print, $action_list, $action_clean, $action_archive, $action_ls); my @filter_args; my @subvol_args; my $args_expected_min = 0; my $args_expected_max = 9999; my $fallback_default_config; my $subvol_args_allow_relative; my $subvol_args_init; if(($command eq "run") || ($command eq "dryrun")) { $action_run = 1; $dryrun = 1 if($command eq "dryrun"); @filter_args = @ARGV; } elsif($command eq "snapshot") { $action_run = 1; $skip_backups = "snapshot"; $preserve_backups = "snapshot"; @filter_args = @ARGV; } elsif($command eq "resume") { $action_run = 1; $skip_snapshots = "resume"; @filter_args = @ARGV; } elsif($command eq "prune") { $action_run = 1; $skip_snapshots = "prune"; $skip_backups = "prune"; @filter_args = @ARGV; } elsif ($command eq "clean") { $action_clean = 1; @filter_args = @ARGV; } elsif ($command eq "archive") { $action_archive = 1; $fallback_default_config = 1; $args_expected_min = $args_expected_max = 2; $subvol_args_allow_relative = 1; @subvol_args = @ARGV; } elsif ($command eq "usage") { $action_usage = 1; @filter_args = @ARGV; } elsif ($command eq "ls") { $action_ls = 1; $fallback_default_config = 1; $args_expected_min = 1; $subvol_args_allow_relative = 1; @subvol_args = @ARGV; } elsif ($command eq "diff") { $action_diff = 1; $fallback_default_config = 1; $args_expected_min = $args_expected_max = 2; $subvol_args_init = "restrict_same_fs deny_root_subvol"; $subvol_args_allow_relative = 1; @subvol_args = @ARGV; } elsif ($command eq "extents") { my $subcommand = shift @ARGV // ""; if(($subcommand eq "list") || ($subcommand eq "diff")) { $action_extents = $subcommand; } else { # defaults to "list" unshift @ARGV, $subcommand; $action_extents = "list"; } $fallback_default_config = 1; $args_expected_min = 1; $subvol_args_init = "restrict_same_fs"; $subvol_args_allow_relative = 1; my $excl; foreach(@ARGV) { # subvol_arg... "exclusive" filter_arg... if($_ eq "exclusive") { $excl = 1; } else { push @subvol_args, $_; push @filter_args, $_ if($excl); } } } elsif ($command eq "origin") { $action_origin = 1; $args_expected_min = $args_expected_max = 1; $subvol_args_init = "deny_root_subvol"; $subvol_args_allow_relative = 1; @subvol_args = @ARGV; } elsif($command eq "list") { my $subcommand = shift @ARGV // ""; if(($subcommand eq "config") || ($subcommand eq "volume") || ($subcommand eq "source") || ($subcommand eq "target")) { $action_list = $subcommand; } elsif(($subcommand eq "all") || ($subcommand eq "snapshots") || ($subcommand eq "backups") || ($subcommand eq "latest")) { $action_resolve = $subcommand; } else { $action_resolve = "all"; unshift @ARGV, $subcommand if($subcommand ne ""); } @filter_args = @ARGV; } elsif($command eq "stats") { $action_resolve = "stats"; @filter_args = @ARGV; } elsif ($command eq "config") { my $subcommand = shift @ARGV // ""; @filter_args = @ARGV; if(($subcommand eq "print") || ($subcommand eq "print-all")) { $action_config_print = $subcommand; } elsif($subcommand eq "list") { $action_list = "config"; } else { ERROR "Unknown subcommand for \"config\" command: $subcommand"; ERROR_HELP_MESSAGE; exit 2; } } else { ERROR "Unrecognized command: $command"; ERROR_HELP_MESSAGE; exit 2; } if(($args_expected_min > scalar(@ARGV)) || ($args_expected_max < scalar(@ARGV))) { ERROR "Incorrect number of arguments"; ERROR_HELP_MESSAGE; exit 2; } # input validation foreach (@subvol_args) { my ($url_prefix, $path) = check_url($_); if(!defined($path) && $subvol_args_allow_relative && ($url_prefix eq "") && (-d $_)) { $path = check_file(abs_path($_), { absolute => 1, sanitize => 1 }); } unless(defined($path)) { ERROR "Bad argument: not a subvolume declaration: $_"; ERROR_HELP_MESSAGE; exit 2; } $_ = $url_prefix . $path; } my @filter_vf; foreach (@filter_args) { my $vf = vinfo_filter_statement($_); unless($vf) { ERROR "Bad argument: invalid filter statement: $_"; ERROR_HELP_MESSAGE; exit 2; } push @filter_vf, $vf; } foreach (@exclude_cmdline) { my $vf = vinfo_filter_statement($_); unless($vf) { ERROR "Bad argument: invalid filter statement: --exclude='$_'"; ERROR_HELP_MESSAGE; exit 2; } push @exclude_vf, $vf; } foreach(@config_override_cmdline) { if(/(.*?)=(.*)/) { my $key = $1; my $value = $2; unless(append_config_option(\%config_override, $key, $value, "OVERRIDE", error_statement => "in option \"--override\"")) { ERROR_HELP_MESSAGE; exit 2; } } else { ERROR "Option \"override\" requires \"=\" format"; ERROR_HELP_MESSAGE; exit 2; } } if(defined($lockfile_cmdline)) { unless($lockfile = check_file($lockfile_cmdline, { absolute => 1, relative => 1 }, error_statement => 'for option --lockfile')) { exit 2; } } INFO "$VERSION_INFO (" . localtime($start_time) . ")"; action("startup", status => "v$VERSION", message => $VERSION_INFO, time => $start_time); # # parse config file # my $config; if(my $config_file = _config_file(@config_src)) { INFO "Using configuration: $config_file"; $config = parse_config($config_file); exit 2 unless($config); } elsif($fallback_default_config) { INFO "Configuration file not found, falling back to defaults"; $config = init_config(); } else { ERROR "Configuration file not found: " . join(', ', @config_src); exit 2; } $safe_commands = config_key($config, 'safe_commands'); unless(ref($config->{SUBSECTION}) eq "ARRAY") { ERROR "No volumes defined in configuration file"; exit 2; } # input validation (part 2, after config is initialized) @subvol_args = map { vinfo($_, $config) } @subvol_args; if($subvol_args_init) { foreach(@subvol_args) { unless(vinfo_init_root($_)) { ERROR "Failed to fetch subvolume detail for '$_->{PRINT}'" , @stderr; exit 1; } if(defined($_->{NODE_SUBDIR})) { ERROR "Argument is not a subvolume: $_->{PATH}"; exit 1; } if(($subvol_args_init =~ /deny_root_subvol/) && $_->{node}{is_root}) { ERROR "Subvolume is btrfs root: $_->{PATH}"; exit 1; } if(($subvol_args_init =~ /restrict_same_fs/) && (not _is_same_fs_tree($subvol_args[0]->{node}, $_->{node}))) { ERROR "Subvolumes are not on the same btrfs filesystem!"; exit 1; } } } if($action_diff) { # # print snapshot diff (btrfs find-new) # my $src_vol = $subvol_args[0]; my $target_vol = $subvol_args[1]; # NOTE: in some cases "cgen" differs from "gen", even for read-only snapshots (observed: gen=cgen+1) my $lastgen = $src_vol->{node}{gen} + 1; # dump files, sorted and unique my $ret = btrfs_subvolume_find_new($target_vol, $lastgen); exit 1 unless(ref($ret)); INFO "Listing changed files for subvolume: $target_vol->{PRINT} (gen=$target_vol->{node}{gen})"; INFO "Starting at generation after subvolume: $src_vol->{PRINT} (gen=$src_vol->{node}{gen})"; INFO "Listing files modified within generation range: [$lastgen..$target_vol->{node}{gen}]"; DEBUG "Newest file generation (transid marker) was: $ret->{transid_marker}"; my $files = $ret->{files}; my $total_len = 0; my @data; foreach my $name (sort keys %$files) { my $finfo = $files->{$name}; $total_len += $finfo->{len}; push @data, { flags => ($finfo->{new} ? '+' : '.') . ($finfo->{flags}->{COMPRESS} ? 'c' : '.') . ($finfo->{flags}->{INLINE} ? 'i' : '.'), count => scalar(keys(%{$finfo->{gen}})), size => print_size($finfo->{len}), file => $name, }; } my $raw = ($output_format && $output_format eq "raw"); print_formatted("diff", \@data, paragraph => 1); print "Total size: " . print_size($total_len) . "\n" unless($raw); exit 0; } if($action_extents) { # # print extents diff (filefrag) # # check system requirements my $extentmap_fn; if($dryrun) { $extentmap_fn = sub { INFO("Fetching extent information (dryrun) for: $_[0]->{PRINT}"); return undef; }; } elsif(eval_quiet { require IO::AIO; }) { # this is slightly faster (multithreaded) than filefrag $extentmap_fn=\&aio_extentmap; } elsif(check_exe("filefrag")) { INFO "IO::AIO module not present, falling back to 'filefrag' (slower)"; $extentmap_fn=\&filefrag_extentmap; } else { ERROR 'Please install either "IO::AIO" perl module or "filefrag" (from e2fsprogs package)'; exit 1; } INFO "Extent map caching disabled (consider setting \"cache_dir\" configuration option)" unless(config_key($config, 'cache_dir')); # resolve related subvolumes my @resolved_vol; if($extents_related) { # add all related subvolumes foreach my $svol (@subvol_args) { my $svol_gen = $svol->{node}{readonly} ? $svol->{node}{cgen} : $svol->{node}{gen}; my @related = map({ vinfo_resolved_all_mountpoints($_, $svol->{VINFO_MOUNTPOINT}) // () } _related_nodes($svol->{node})); # includes $svol push @resolved_vol, @related; } } else { @resolved_vol = @subvol_args; } my @data; # print results on ctrl-c $SIG{INT} = sub { print STDERR "\nERROR: Caught SIGINT, dumping incomplete list:\n"; print_formatted("extent_diff", \@data); exit 1; }; my $do_diff = ($action_extents eq "diff"); my $prev_data; # sort by gen for r/w subvolumes, cgen on readonly subvolumes, as # "gen" is increased on readonly subvolume when snapshotted. # crawl descending, but display ascending (unshift): foreach my $vol (sort { ($b->{node}{readonly} ? $b->{node}{cgen} : $b->{node}{gen}) <=> ($a->{node}{readonly} ? $a->{node}{cgen} : $a->{node}{gen}) } @resolved_vol) { if($prev_data && ($prev_data->{_vinfo}{node}{id} == $vol->{node}{id})) { INFO "Skipping duplicate of \"$prev_data->{_vinfo}{PRINT}\": $vol->{PRINT}"; next; } # read extent map if($vol->{EXTENTMAP} = read_extentmap_cache($vol)) { INFO "Using cached extent map: $vol->{PRINT}"; } else { $vol->{EXTENTMAP} = $extentmap_fn->($vol); write_extentmap_cache($vol); } next unless($vol->{EXTENTMAP}); if($do_diff && $prev_data) { my $diff_map = extentmap_diff($prev_data->{_vinfo}{EXTENTMAP}, $vol->{EXTENTMAP}); $prev_data->{diff} = print_size(extentmap_size($diff_map)); } $prev_data = { %{$vol->{node}}, # copy node total => print_size(extentmap_size($vol->{EXTENTMAP})), subvol => $vol->{PRINT}, _vinfo => $vol, }; unshift @data, $prev_data; } my @universe_set = map $_->{_vinfo}{EXTENTMAP}, @data; unless(scalar(@universe_set)) { ERROR "No extent map data, exiting"; exit -1; } my @summary; INFO "Calculating union of " . scalar(@data) . " subvolumes"; push @summary, { a => "Union (" . scalar(@data) . " subvolumes):", b => print_size(extentmap_size(extentmap_merge(@universe_set))) }; INFO "Calculating set-exclusive size for " . scalar(@data) . " subvolumes"; foreach my $d (@data) { my $vol = $d->{_vinfo}; DEBUG "Calculating exclusive for: $vol->{PRINT}"; my @others = grep { $_ != $vol->{EXTENTMAP} } @universe_set; $d->{exclusive} = print_size(extentmap_size(extentmap_diff($vol->{EXTENTMAP}, extentmap_merge(@others)))), } if(scalar(@filter_vf)) { INFO "Calculating set difference (X \\ A)"; my @excl; my @others; foreach(@data) { if(vinfo_match(\@filter_vf, $_->{_vinfo})) { $_->{set} = "X"; push @excl, $_->{_vinfo}{EXTENTMAP}; } else { $_->{set} = "A"; push @others, $_->{_vinfo}{EXTENTMAP}; } } push @summary, { a => "Exclusive data ( X \\ A ):", b => print_size(extentmap_size(extentmap_diff(extentmap_merge(@excl), extentmap_merge(@others)))), }; } unless($do_diff) { @data = sort { $a->{subvol} cmp $b->{subvol} } @data; } INFO "Printing extents map set difference: (extents \\ extents-on-prev-line)" if $do_diff; print_formatted("extent_diff", \@data, paragraph => 1); print_formatted({ table => [ qw( a b ) ], RALIGN => { b=>1 } }, \@summary, output_format => "table", no_header => 1); exit 0; } if($action_ls) { # # print accessible subvolumes for local path # my $exit_status = 0; my %data_uniq; foreach my $root_vol (@subvol_args) { my ($root_path, $mountpoint) = vinfo_mountpoint($root_vol); unless($mountpoint) { ERROR "Failed to read filesystem details for: $root_vol->{PRINT}", @stderr; $exit_status = 1; next; } $root_vol = vinfo($root_vol->{URL_PREFIX} . $root_path, $config); INFO "Listing subvolumes for directory: $root_vol->{PRINT}"; my @search = ( $mountpoint ); while(my $mnt = shift @search) { unshift @search, @{$mnt->{SUBTREE}} if($mnt->{SUBTREE}); next if($mnt->{fs_type} ne "btrfs"); my $vol = vinfo($root_vol->{URL_PREFIX} . $mnt->{mount_point}, $config); unless(vinfo_init_root($vol)) { ERROR "Failed to fetch subvolume detail for: $vol->{PRINT}", @stderr; $exit_status = 1; next; } my $subvol_list = vinfo_subvol_list($vol); my $count_added = 0; foreach my $svol ($vol, @$subvol_list) { my $svol_path = $svol->{PATH}; $svol_path =~ s/^\/\//\//; # sanitize "//" (see vinfo_child) next unless($root_path eq "/" || $svol_path =~ /^\Q$root_path\E(\/|\z)/); if(_find_mountpoint($mnt, $svol_path) ne $mnt) { DEBUG "Subvolume is hidden by another mount point: $svol->{PRINT}"; next; } $data_uniq{$svol->{PRINT}} = { %{$svol->{node}}, # copy node top => $svol->{node}{top_level}, # alias (narrow column) mount_point => $svol->{VINFO_MOUNTPOINT}{PATH}, mount_source => $svol->{node}{TREE_ROOT}{mount_source}, mount_subvolid => $mnt->{MNTOPS}{subvolid}, mount_subvol => $mnt->{MNTOPS}{subvol}, subvolume_path => $svol->{node}{path}, subvolume_rel_path => $svol->{node}{REL_PATH}, url => $svol->{URL}, host => $svol->{HOST}, path => $svol_path, flags => ($svol->{node}{readonly} ? "readonly" : undef), }; $count_added++; } DEBUG "Listing $count_added/" . (scalar(@$subvol_list) + 1) . " subvolumes for btrfs mount: $vol->{PRINT}"; } } my @sorted = sort { (($a->{host} // "") cmp ($b->{host} // "")) || ($a->{mount_point} cmp $b->{mount_point}) || ($a->{path} cmp $b->{path}) } values %data_uniq; $output_format ||= "short"; print_formatted("fs_list", \@sorted, no_header => !scalar(@sorted)); exit $exit_status; } # # try exclusive lock if set in config or command-line option # $lockfile //= config_key($config, "lockfile"); if(defined($lockfile) && (not $dryrun)) { unless(open(LOCKFILE, '>>', $lockfile)) { # NOTE: the lockfile is never deleted by design ERROR "Failed to open lock file '$lockfile': $!"; exit 3; } unless(flock(LOCKFILE, 6)) { # exclusive, non-blocking (LOCK_EX | LOCK_NB) ERROR "Failed to take lock (another btrbk instance is running): $lockfile"; exit 3; } } if($action_archive) { # # archive (clone) tree # # NOTE: This is intended to work without a config file! The only # thing used from the configuration is the SSH and transaction log # stuff. # init_transaction_log(config_key($config, "transaction_log"), config_key($config, "transaction_syslog")); my $src_root = $subvol_args[0] || die; my $archive_root = $subvol_args[1] || die; # FIXME: add command line options for preserve logic $config->{SUBSECTION} = []; # clear configured subsections, we build them dynamically unless(vinfo_init_root($src_root)) { ERROR "Failed to fetch subvolume detail for '$src_root->{PRINT}'", @stderr; exit 1; } unless($archive_raw ? vinfo_init_raw_root($archive_root) : vinfo_init_root($archive_root)) { ERROR "Failed to fetch " . ($archive_raw ? "raw target metadata" : "subvolume detail") . " for '$archive_root->{PRINT}'", @stderr; exit 1; } my %name_uniq; my @subvol_list = @{vinfo_subvol_list($src_root)}; my @sorted = sort { ($a->{subtree_depth} <=> $b->{subtree_depth}) || ($a->{SUBVOL_DIR} cmp $b->{SUBVOL_DIR}) } @subvol_list; foreach my $vol (@sorted) { next unless($vol->{node}{readonly}); my $snapshot_name = $vol->{node}{BTRBK_BASENAME}; unless(defined($snapshot_name)) { WARN "Skipping subvolume (not a btrbk subvolume): $vol->{PRINT}"; next; } my $subvol_dir = $vol->{SUBVOL_DIR}; next if($name_uniq{"$subvol_dir/$snapshot_name"}); $name_uniq{"$subvol_dir/$snapshot_name"} = 1; my $droot_url = $archive_root->{URL} . ($subvol_dir eq "" ? "" : "/$subvol_dir"); my $sroot_url = $src_root->{URL} . ($subvol_dir eq "" ? "" : "/$subvol_dir"); my $config_sroot = { CONTEXT => "archive_source", PARENT => $config, url => $sroot_url, # ABORTED() needs this snapshot_name => $snapshot_name, }; my $config_droot = { CONTEXT => "archive_target", PARENT => $config_sroot, target_type => ($archive_raw ? "raw" : "send-receive"), # macro_send_receive checks this url => $droot_url, # ABORTED() needs this }; $config_sroot->{SUBSECTION} = [ $config_droot ]; push(@{$config->{SUBSECTION}}, $config_sroot); my $sroot = vinfo($sroot_url, $config_sroot); vinfo_assign_config($sroot); unless(vinfo_init_root($sroot)) { ABORTED($sroot, "Failed to fetch subvolume detail"); WARN "Skipping archive source \"$sroot->{PRINT}\": " . ABORTED_TEXT($sroot), @stderr; next; } my $droot = vinfo($droot_url, $config_droot); vinfo_assign_config($droot); unless($archive_raw ? vinfo_init_raw_root($droot) : vinfo_init_root($droot)) { DEBUG "Failed to fetch " . ($archive_raw ? "raw target metadata" : "subvolume detail") . " for '$droot->{PRINT}'"; unless(system_mkdir($droot)) { ABORTED($droot, "Failed to create directory: $droot->{PRINT}/"); WARN "Skipping archive target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot), @stderr; next; } $droot->{SUBDIR_CREATED} = 1; if($dryrun) { # we need to fake this directory on dryrun $droot->{node} = $archive_root->{node}; $droot->{NODE_SUBDIR} = $subvol_dir; $droot->{VINFO_MOUNTPOINT} = $archive_root->{VINFO_MOUNTPOINT}; $realpath_cache{$droot->{URL}} = $droot->{PATH}; } else { # after directory is created, try to init again unless($archive_raw ? vinfo_init_raw_root($droot) : vinfo_init_root($droot)) { ABORTED($droot, "Failed to fetch subvolume detail"); WARN "Skipping archive target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot), @stderr; next; } } } if(_is_same_fs_tree($droot->{node}, $vol->{node})) { ERROR "Source and target subvolumes are on the same btrfs filesystem!"; exit 1; } } # translate archive_exclude globs, add to exclude args my $archive_exclude = config_key($config, 'archive_exclude') // []; push @exclude_vf, map(vinfo_filter_statement($_), (@$archive_exclude)); # create archives my $schedule_results = []; my $aborted; foreach my $sroot (vinfo_subsection($config, 'archive_source')) { if($aborted) { # abort all subsequent sources on any abort (we don't want to go on hammering on "disk full" errors) ABORTED($sroot, $aborted); next; } my $snapshot_name = config_key($sroot, "snapshot_name") // die; # skip on archive_exclude and --exclude option if(vinfo_match(\@exclude_vf, $sroot) || vinfo_match(\@exclude_vf, vinfo_child($sroot, $snapshot_name))) { ABORTED($sroot, "skip_archive_exclude", "Match on exclude pattern"); INFO "Skipping archive subvolumes \"$sroot->{PRINT}/${snapshot_name}.*\": " . ABORTED_TEXT($sroot); next; } foreach my $droot (vinfo_subsection($sroot, 'archive_target')) { INFO "Archiving subvolumes: $sroot->{PRINT}/${snapshot_name}.*"; macro_archive_target($sroot, $droot, $snapshot_name, { results => $schedule_results }); if(IS_ABORTED($droot)) { # also abort $sroot $aborted = "At least one target aborted earlier"; ABORTED($sroot, $aborted); WARN "Skipping archiving of \"$sroot->{PRINT}/\": " . ABORTED_TEXT($sroot); last; } } } # delete archives my $del_schedule_results; if($preserve_backups) { INFO "Preserving all archives (option \"-p\" or \"-r\" present)"; } else { $del_schedule_results = []; foreach my $sroot (vinfo_subsection($config, 'archive_source')) { my $snapshot_name = config_key($sroot, "snapshot_name") // die; foreach my $droot (vinfo_subsection($sroot, 'archive_target')) { INFO "Cleaning archive: $droot->{PRINT}/${snapshot_name}.*"; macro_delete($droot, $snapshot_name, $droot, { preserve => config_preserve_hash($droot, "archive"), results => $del_schedule_results, result_hints => { topic => "archive", root_path => $droot->{PATH} }, }, commit => config_key($droot, "btrfs_commit_delete"), type => "delete_archive", qgroup => { destroy => config_key($droot, "archive_qgroup_destroy"), type => "qgroup_destroy_archive" }, ); } } } my $exit_status = exit_status($config); my $time_elapsed = time - $start_time; INFO "Completed within: ${time_elapsed}s (" . localtime(time) . ")"; action("finished", status => $exit_status ? "partial" : "success", duration => $time_elapsed, message => $exit_status ? "At least one backup task aborted" : undef, ); close_transaction_log(); unless($quiet) { # print scheduling results if($print_schedule) { my @data = map { { %$_, vinfo_prefixed_keys("", $_->{value}) }; } @$schedule_results; print_formatted("schedule", \@data, title => "ARCHIVE SCHEDULE", paragraph => 1); } if($print_schedule && $del_schedule_results) { my @data = map { { %$_, vinfo_prefixed_keys("", $_->{value}) }; } @$del_schedule_results; print_formatted("schedule", \@data, title => "DELETE SCHEDULE", paragraph => 1); } # print summary $output_format ||= "custom"; if($output_format eq "custom") { my @out; foreach my $sroot (vinfo_subsection($config, 'archive_source', 1)) { foreach my $droot (vinfo_subsection($sroot, 'archive_target', 1)) { my @subvol_out; if($droot->{SUBDIR_CREATED}) { push @subvol_out, "++. $droot->{PRINT}/"; } foreach(@{$droot->{SUBVOL_RECEIVED} // []}) { my $create_mode = "***"; $create_mode = ">>>" if($_->{parent}); $create_mode = "!!!" if($_->{ERROR}); push @subvol_out, "$create_mode $_->{received_subvolume}->{PRINT}"; } foreach(@{$droot->{SUBVOL_DELETED} // []}) { push @subvol_out, "--- $_->{PRINT}"; } if(IS_ABORTED($droot, "abort_") || IS_ABORTED($sroot, "abort_")) { push @subvol_out, "!!! Target \"$droot->{PRINT}\" aborted: " . (ABORTED_TEXT($droot) || ABORTED_TEXT($sroot)); } elsif(IS_ABORTED($sroot, "skip_archive_exclude")) { push @subvol_out, ""; } unless(@subvol_out) { push @subvol_out, "[-] $droot->{PRINT}/$sroot->{CONFIG}->{snapshot_name}.*"; } push @out, "$sroot->{PRINT}/$sroot->{CONFIG}->{snapshot_name}.*", @subvol_out, ""; } } my @cmdline_options = map { "exclude: $_" } @exclude_cmdline; push @cmdline_options, "preserve: Preserved all archives" if($preserve_backups); print_header(title => "Archive Summary", time => $start_time, options => \@cmdline_options, legend => [ "++. created directory", "--- deleted subvolume", "*** received subvolume (non-incremental)", ">>> received subvolume (incremental)", "[-] no action", ], ); print join("\n", @out); print_footer($config, $exit_status); } else { # print action log (without transaction start messages) my @data = grep { $_->{status} !~ /starting$/ } @transaction_log; print_formatted("transaction", \@data, title => "TRANSACTION LOG"); } } exit $exit_status; } # # expand subvolume globs (wildcards) # foreach my $config_vol (config_subsection($config, "volume")) { # read-in subvolume list (and expand globs) only if needed next unless(grep defined($_->{GLOB_CONTEXT}), @{$config_vol->{SUBSECTION}}); my @vol_subsection_expanded; foreach my $config_subvol (config_subsection($config_vol, "subvolume")) { if($config_subvol->{GLOB_CONTEXT}) { my ($url_prefix, $globs) = check_url($config_subvol->{url}, accept_wildcards => 1); $globs =~ s/([^\*]*)\///; my $sroot_glob = vinfo($url_prefix . $1, $config_subvol); INFO "Expanding wildcards: $sroot_glob->{PRINT}/$globs"; unless(vinfo_init_root($sroot_glob)) { WARN "Failed to fetch subvolume detail for: $sroot_glob->{PRINT}", @stderr; WARN "No subvolumes found matching: $sroot_glob->{PRINT}/$globs"; next; } # support "*some*file*", "*/*" my $match = join('[^\/]*', map(quotemeta($_), split(/\*+/, $globs, -1))); TRACE "translated globs \"$globs\" to regex \"$match\"" if($do_trace); my $expand_count = 0; foreach my $vol (@{vinfo_subvol_list($sroot_glob, sort => 'path')}) { if($vol->{node}{readonly}) { TRACE "skipping readonly subvolume: $vol->{PRINT}" if($do_trace); next; } unless($vol->{SUBVOL_PATH} =~ /^$match$/) { TRACE "skipping non-matching subvolume: $vol->{PRINT}" if($do_trace); next; } unless(defined(check_file($vol->{SUBVOL_PATH}, { relative => 1 }))) { WARN "Ambiguous subvolume path \"$vol->{SUBVOL_PATH}\" while expanding \"$globs\", ignoring"; next; } INFO "Found source subvolume: $vol->{PRINT}"; my %conf = ( %$config_subvol, url_glob => $config_subvol->{url}, url => $vol->{URL}, snapshot_name => $vol->{NAME}, # snapshot_name defaults to subvolume name ); # deep copy of target subsection my @subsection_copy = map { { %$_, PARENT => \%conf }; } @{$config_subvol->{SUBSECTION}}; $conf{SUBSECTION} = \@subsection_copy; push @vol_subsection_expanded, \%conf; $expand_count += 1; } unless($expand_count) { WARN "No subvolumes found matching: $sroot_glob->{PRINT}/$globs"; } } else { push @vol_subsection_expanded, $config_subvol; } } $config_vol->{SUBSECTION} = \@vol_subsection_expanded; } TRACE(Data::Dumper->Dump([$config], ["config"])) if($do_trace && $do_dumper); # # create vinfo nodes (no readin yet) # foreach my $config_vol (config_subsection($config, "volume")) { my $sroot = $config_vol->{DUMMY} ? { CONFIG => $config_vol, PRINT => "*default*" } : vinfo($config_vol->{url}, $config_vol); vinfo_assign_config($sroot); foreach my $config_subvol (config_subsection($config_vol, "subvolume")) { my $svol = vinfo($config_subvol->{url}, $config_subvol); my $snapshot_dir = config_key($svol, "snapshot_dir"); my $url; if(!defined($snapshot_dir)) { if($config_vol->{DUMMY}) { ABORTED($svol, "No snapshot_dir defined for subvolume"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); } else { $url = $sroot->{URL}; } } elsif($snapshot_dir =~ /^\//) { $url = $svol->{URL_PREFIX} . $snapshot_dir; } else { if($config_vol->{DUMMY}) { ABORTED($svol, "Relative snapshot_dir path defined, but no volume context present"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); } else { $url = $sroot->{URL} . '/' . $snapshot_dir; } } $url //= "/dev/null"; # snaproot cannot be undef, even if ABORTED my $snaproot = vinfo($url, $config_subvol); vinfo_assign_config($svol, $snaproot); foreach my $config_target (@{$config_subvol->{SUBSECTION}}) { die unless($config_target->{CONTEXT} eq "target"); my $droot = vinfo($config_target->{url}, $config_target); vinfo_assign_config($droot); } } } # # filter subvolumes matching command line arguments, handle noauto option # if(scalar @filter_vf) { foreach my $sroot (vinfo_subsection($config, 'volume', 1)) { my $found_vol = 0; if(vinfo_match(\@filter_vf, $sroot, flag_matched => '_matched')) { next; } foreach my $svol (vinfo_subsection($sroot, 'subvolume', 1)) { my $found_subvol = 0; my $snaproot = vinfo_snapshot_root($svol); my $snapshot_name = config_key($svol, "snapshot_name") // die; if(vinfo_match(\@filter_vf, $svol, flag_matched => '_matched') || vinfo_match(\@filter_vf, vinfo_child($snaproot, $snapshot_name), flag_matched => '_matched')) { $found_vol = 1; next; } foreach my $droot (vinfo_subsection($svol, 'target', 1)) { if(vinfo_match(\@filter_vf, $droot, flag_matched => '_matched') || vinfo_match(\@filter_vf, vinfo_child($droot, $snapshot_name), flag_matched => '_matched')) { $found_subvol = 1; $found_vol = 1; } else { ABORTED($droot, "skip_cmdline_filter", "No match on filter command line argument"); DEBUG "Skipping target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot); } } unless($found_subvol) { ABORTED($svol, "skip_cmdline_filter", "No match on filter command line argument"); DEBUG "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); } } unless($found_vol) { ABORTED($sroot, "skip_cmdline_filter", "No match on filter command line argument"); DEBUG "Skipping volume \"$sroot->{PRINT}\": " . ABORTED_TEXT($sroot); } } # make sure all args have a match my @nomatch = map { $_->{_matched} ? () : $_->{unparsed} } @filter_vf; if(@nomatch) { foreach(@nomatch) { ERROR "Filter argument \"$_\" does not match any volume, subvolume, target or group declaration"; } exit 2; } $config->{CMDLINE_FILTER_LIST} = [ map { $_->{unparsed} } @filter_vf ]; } elsif(not $action_config_print) { # no filter_args present, abort "noauto" contexts if(config_key($config, "noauto")) { WARN "Option \"noauto\" is set in global context, and no filter argument present, exiting"; exit 0; } foreach my $sroot (vinfo_subsection($config, 'volume')) { if(config_key($sroot, "noauto")) { ABORTED($sroot, "skip_noauto", 'option "noauto" is set'); DEBUG "Skipping volume \"$sroot->{PRINT}\": " . ABORTED_TEXT($sroot); next; } foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { if(config_key($svol, "noauto")) { ABORTED($svol, "skip_noauto", 'option "noauto" is set'); DEBUG "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); next; } foreach my $droot (vinfo_subsection($svol, 'target')) { if(config_key($droot, "noauto")) { ABORTED($droot, "skip_noauto", 'option "noauto" is set'); DEBUG "Skipping target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot); } } } } } if(scalar @exclude_vf) { # handle --exclude command line option foreach my $sroot (vinfo_subsection($config, 'volume')) { if(my $ff = vinfo_match(\@exclude_vf, $sroot)) { ABORTED($sroot, "skip_cmdline_exclude", "command line argument \"--exclude=$ff->{unparsed}\""); DEBUG "Skipping volume \"$sroot->{PRINT}\": " . ABORTED_TEXT($sroot); next; } my $all_svol_aborted = 1; foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snaproot = vinfo_snapshot_root($svol); my $snapshot_name = config_key($svol, "snapshot_name") // die; if(my $ff = (vinfo_match(\@exclude_vf, $svol) || vinfo_match(\@exclude_vf, vinfo_child($snaproot, $snapshot_name)))) { ABORTED($svol, "skip_cmdline_exclude", "command line argument \"--exclude=$ff->{unparsed}\""); DEBUG "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); next; } $all_svol_aborted = 0; foreach my $droot (vinfo_subsection($svol, 'target')) { if(my $ff = (vinfo_match(\@exclude_vf, $droot) || vinfo_match(\@exclude_vf, vinfo_child($droot, $snapshot_name)))) { ABORTED($droot, "skip_cmdline_exclude", "command line argument \"--exclude=$ff->{unparsed}\""); DEBUG "Skipping target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot); next; } } } if($all_svol_aborted) { ABORTED($sroot, "skip_cmdline_exclude", "All subvolumes excluded"); DEBUG "Skipping volume \"$sroot->{PRINT}\": " . ABORTED_TEXT($sroot); } } } if($action_usage) { # # print filesystem information # my @data; my %usage_cache; my %processed; my $push_data = sub { my ($vol, $type) = @_; return if $processed{$vol->{URL}}; my $mountpoint = vinfo_mountpoint($vol, fs_type => 'btrfs'); return unless($mountpoint); my $mount_source = $mountpoint->{mount_source}; my $mid = $vol->{MACHINE_ID} . $mount_source; $usage_cache{$mid} //= btrfs_filesystem_usage(vinfo($vol->{URL_PREFIX} . $mountpoint->{mount_point}, $vol->{CONFIG})) // {}; push @data, { %{$usage_cache{$mid}}, type => $type, mount_source => $mount_source, vinfo_prefixed_keys("", $vol), }; $processed{$vol->{URL}} = 1; }; foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { $push_data->($svol, "source"); foreach my $droot (vinfo_subsection($svol, 'target')) { $push_data->($droot, "target"); } } } @data = sort { $a->{url} cmp $b->{url} } @data; print_formatted("usage", \@data); exit exit_status($config); } if($action_config_print) { # # print configuration lines, machine readable # my %opts = (all => ($action_config_print eq "print-all")); my @out; push @out, config_dump_keys($config, %opts); my $indent = ""; foreach my $sroot (vinfo_subsection($config, 'volume', 1)) { unless($sroot->{CONFIG}{DUMMY}) { push @out, ""; push @out, "volume $sroot->{URL}"; $indent .= "\t"; push @out, config_dump_keys($sroot, prefix => $indent, %opts); } foreach my $svol (vinfo_subsection($sroot, 'subvolume', 1)) { push @out, ""; push @out, "${indent}# subvolume $svol->{CONFIG}->{url_glob}" if(defined($svol->{CONFIG}->{url_glob})); push @out, "${indent}subvolume $svol->{URL}"; $indent .= "\t"; push @out, config_dump_keys($svol, prefix => $indent, %opts); foreach my $droot (vinfo_subsection($svol, 'target', 1)) { push @out, ""; push @out, "${indent}target $droot->{CONFIG}->{target_type} $droot->{URL}"; push @out, config_dump_keys($droot, prefix => "\t$indent", %opts); } $indent =~ s/\t//; } $indent = ""; } print_header(title => "Configuration Dump", config => $config, time => $start_time, ); print join("\n", @out) . "\n"; exit exit_status($config); } if($action_list) { my @vol_data; my @subvol_data; my @target_data; my @mixed_data; my %target_uniq; # # print configuration lines, machine readable # foreach my $sroot (vinfo_subsection($config, 'volume')) { my $volh = { vinfo_prefixed_keys("volume", $sroot) }; push @vol_data, $volh; foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snaproot = vinfo_snapshot_root($svol); my $subvolh = { %$volh, vinfo_prefixed_keys("source", $svol), snapshot_path => $snaproot->{PATH}, snapshot_name => config_key($svol, "snapshot_name"), snapshot_preserve => format_preserve_matrix(config_preserve_hash($svol, "snapshot")), }; push @subvol_data, $subvolh; my $found = 0; foreach my $droot (vinfo_subsection($svol, 'target')) { my $targeth = { %$subvolh, vinfo_prefixed_keys("target", $droot), target_preserve => format_preserve_matrix(config_preserve_hash($droot, "target")), target_type => $droot->{CONFIG}{target_type}, # "send-receive" or "raw" }; if($action_list eq "target") { next if($target_uniq{$droot->{URL}}); $target_uniq{$droot->{URL}} = 1; } push @target_data, $targeth; push @mixed_data, $targeth; $found = 1; } # make sure the subvol is always printed (even if no targets around) push @mixed_data, $subvolh unless($found); } } if($action_list eq "volume") { print_formatted("config_volume", \@vol_data); } elsif($action_list eq "source") { print_formatted("config_source", \@subvol_data); } elsif($action_list eq "target") { print_formatted("config_target", \@target_data); } elsif($action_list eq "config") { print_formatted("config", \@mixed_data); } else { die "unknown action_list=$action_list"; } exit exit_status($config); } # # fill vinfo hash, basic checks on configuration # # read volume btrfs tree, and make sure subvolume exist foreach my $sroot (vinfo_subsection($config, 'volume')) { DEBUG "Initializing volume section: $sroot->{PRINT}"; unless(scalar(vinfo_subsection($sroot, 'subvolume', 1))) { WARN "No subvolume configured for \"volume $sroot->{URL}\""; } foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { DEBUG "Initializing subvolume section: $svol->{PRINT}"; unless(vinfo_init_root($svol)) { ABORTED($svol, "Failed to fetch subvolume detail"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol), @stderr; next; } if((not $svol->{node}{uuid}) || ($svol->{node}{uuid} eq '-')) { ABORTED($svol, "subvolume has no UUID"); ERROR "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); next; } if($svol->{node}{readonly}) { ABORTED($svol, "subvolume is readonly"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); next; } if($svol->{node}{received_uuid} ne '-') { ABORTED($svol, "\"Received UUID\" is set"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); next; } my $snaproot = vinfo_snapshot_root($svol); unless(vinfo_init_root($snaproot)) { ABORTED($svol, "Failed to fetch subvolume detail for snapshot_dir"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol), @stderr; next; } unless(_is_same_fs_tree($snaproot->{node}, $svol->{node})) { ABORTED($svol, "Snapshot path is not on same filesystem"); WARN "Skipping subvolume \"$svol->{PRINT}\": " . ABORTED_TEXT($svol); next; } } } # read target btrfs tree if($action_run && $skip_backups && $preserve_snapshots && $preserve_backups) { # if running "btrbk snapshot --preserve", there is no need to # initialize targets, and we don't want to fail on missing targets. DEBUG "Skipping target tree readin (preserving all snapshots and backups)"; } else { foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { foreach my $droot (vinfo_subsection($svol, 'target')) { DEBUG "Initializing target section: $droot->{PRINT}"; my $target_type = $droot->{CONFIG}->{target_type} || die; if($target_type eq "send-receive") { unless(vinfo_init_root($droot)) { ABORTED($droot, "Failed to fetch subvolume detail"); WARN "Skipping target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot), @stderr; next; } } elsif($target_type eq "raw") { unless(vinfo_init_raw_root($droot)) { ABORTED($droot, "Failed to fetch raw target metadata"); WARN "Skipping target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot), @stderr; next; } } if($config_override{FAILSAFE_PRESERVE}) { ABORTED($droot, $config_override{FAILSAFE_PRESERVE}); WARN "Skipping target \"$droot->{PRINT}\": " . ABORTED_TEXT($droot); } } } } } # check for duplicate snapshot locations foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snapshot_basename = config_key($svol, "snapshot_name") // die; # check for duplicate snapshot locations if(config_key($svol, "snapshot_create")) { my $snaproot = vinfo_snapshot_root($svol); my $snaproot_subdir_path = (defined($snaproot->{NODE_SUBDIR}) ? $snaproot->{NODE_SUBDIR} . '/' : "") . $snapshot_basename; if(my $prev = $snaproot->{node}->{_SNAPSHOT_CHECK}->{$snaproot_subdir_path}) { ERROR "Subvolume \"$prev\" and \"$svol->{PRINT}\" will create same snapshot: $snaproot->{PRINT}/${snapshot_basename}.*"; ERROR "Please fix \"snapshot_name\" configuration options!"; exit 1; } $snaproot->{node}->{_SNAPSHOT_CHECK}->{$snaproot_subdir_path} = $svol->{PRINT}; } # check for duplicate target locations foreach my $droot (vinfo_subsection($svol, 'target')) { my $droot_subdir_path = (defined($droot->{NODE_SUBDIR}) ? $droot->{NODE_SUBDIR} . '/' : "") . $snapshot_basename; if(my $prev = $droot->{node}->{_BACKUP_CHECK}->{$droot_subdir_path}) { ERROR "Subvolume \"$prev\" and \"$svol->{PRINT}\" will create same backup target: $droot->{PRINT}/${snapshot_basename}.*"; ERROR "Please fix \"snapshot_name\" or \"target\" configuration options!"; exit 1; } $droot->{node}->{_BACKUP_CHECK}->{$droot_subdir_path} = $svol->{PRINT}; } } } if($action_origin) { # # print origin information # my $vol = $subvol_args[0] || die; my $lines = []; _origin_tree("", $vol->{node}, $lines); $output_format ||= "custom"; if($output_format eq "custom") { print_header(title => "Origin Tree", config => $config, time => $start_time, legend => [ "^-- : parent subvolume", "newline : received-from relationship with subvolume (identical content)", ] ); print join("\n", map { $_->{tree} } @$lines) . "\n"; } else { print_formatted('origin_tree', $lines ); } exit 0; } if($action_resolve) { my @data; my %stats = ( snapshots => 0, backups => 0, correlated => 0, incomplete => 0, orphaned => 0 ); foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snaproot = vinfo_snapshot_root($svol); my $snapshot_name = config_key($svol, "snapshot_name") // die; my @related_snapshots = get_related_snapshots($snaproot, $svol, $snapshot_name); my %svol_data = ( vinfo_prefixed_keys("source", $svol), snapshot_name => $snapshot_name, ); my @sdata = map +{ %svol_data, type => "snapshot", status => ($_->{node}{cgen} == $svol->{node}{gen}) ? "up-to-date" : "", vinfo_prefixed_keys("snapshot", $_), _vinfo => $_, _btrbk_date => $_->{node}{BTRBK_DATE}, }, @related_snapshots; my %svol_stats_data = ( %svol_data, snapshot_subvolume => "$snaproot->{PATH}/$snapshot_name.*", snapshot_status => (grep { $_->{status} eq "up-to-date" } @sdata) ? "up-to-date" : "", snapshots => scalar(@sdata), ); $stats{snapshots} += scalar(@sdata); my (@bdata, @ldata, @stdata); foreach my $droot (vinfo_subsection($svol, 'target')) { my %dstats = ( backups => 0, correlated => 0, orphaned => 0, incomplete => 0, uptodate => 0 ); my $latest_backup; foreach my $target_vol (@{vinfo_subvol_list($droot, btrbk_direct_leaf => $snapshot_name, sort => 'path')}) { my $target_data = { %svol_data, type => "backup", target_type => $target_vol->{CONFIG}{target_type}, # "send-receive" or "raw" vinfo_prefixed_keys("target", $target_vol), _btrbk_date => $target_vol->{node}{BTRBK_DATE}, }; # incomplete received (garbled) subvolumes have no received_uuid (as of btrfs-progs v4.3.1). # a subvolume in droot matching our naming is considered incomplete if received_uuid is not set! if($target_vol->{node}{received_uuid} eq '-') { $dstats{incomplete}++; $target_data->{status} = "incomplete"; push @bdata, $target_data; next; } foreach (@sdata) { if(_is_correlated($_->{_vinfo}{node}, $target_vol->{node})) { $target_data = { %$_, %$target_data, type => "snapshot,backup", _correlated => 1, }; $_->{_correlated} = 1; last; } } push @bdata, $target_data; $latest_backup = $target_data if(!defined($latest_backup) || (cmp_date($latest_backup->{_btrbk_date}, $target_data->{_btrbk_date}) < 0)); $dstats{uptodate} ||= ($target_data->{status} // "") eq "up-to-date"; $dstats{backups}++; if($target_data->{_correlated}) { $dstats{correlated}++; } else { $dstats{orphaned}++; } } push @ldata, $latest_backup; push @stdata, { %svol_stats_data, %dstats, vinfo_prefixed_keys("target", $droot), target_subvolume => "$droot->{PATH}/$snapshot_name.*", backup_status => $dstats{uptodate} ? "up-to-date" : "", }; $stats{$_} += $dstats{$_} foreach(qw(backups correlated incomplete orphaned)); } if($action_resolve eq "snapshots") { push @data, @sdata; } elsif($action_resolve eq "backups") { push @data, @bdata; } elsif($action_resolve eq "all") { push @data, sort { cmp_date($a->{_btrbk_date}, $b->{_btrbk_date}) } (@bdata, grep { !$_->{_correlated} } @sdata); } elsif($action_resolve eq "latest") { my $latest_snapshot = (sort { cmp_date($b->{_btrbk_date}, $a->{_btrbk_date}) } (@sdata, @bdata))[0]; push @data, @ldata; push @data, $latest_snapshot if($latest_snapshot && !$latest_snapshot->{_correlated}); } elsif($action_resolve eq "stats") { @stdata = ( \%svol_stats_data ) unless(@stdata); push @data, @stdata; } } } if($action_resolve eq "stats") { my $filter = $config->{CMDLINE_FILTER_LIST} ? " (" . join(", ", @{$config->{CMDLINE_FILTER_LIST}}) . ")" : ""; my @backup_total = map { $stats{$_} ? "$stats{$_} $_" : () } qw( correlated incomplete ); my $bflags = @backup_total ? "(" . join(', ', @backup_total) . ")" : undef; print_formatted("stats", \@data, paragraph => 1); print "Total${filter}:\n"; print_formatted({ table => [ qw( a b -c ) ], RALIGN => { a=>1 } }, [ { a => $stats{snapshots}, b => "snapshots" }, { a => $stats{backups}, b => "backups", c => $bflags } ], output_format => "table", no_header => 1, empty_cell_char => ""); } elsif($action_resolve eq "snapshots") { print_formatted("snapshots", \@data); } elsif($action_resolve eq "backups") { print_formatted("backups", \@data); } elsif($action_resolve eq "latest") { print_formatted("latest", \@data); } else { print_formatted("resolved", \@data); } exit exit_status($config); } if($action_clean) { # # identify and delete incomplete backups # init_transaction_log(config_key($config, "transaction_log"), config_key($config, "transaction_syslog")); my @out; foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snapshot_name = config_key($svol, "snapshot_name") // die; foreach my $droot (vinfo_subsection($svol, 'target')) { INFO "Cleaning incomplete backups in: $droot->{PRINT}/$snapshot_name.*"; push @out, "$droot->{PRINT}/$snapshot_name.*"; # incomplete received (garbled) subvolumes are not readonly and have no received_uuid (as of btrfs-progs v4.3.1). # a subvolume in droot matching our naming is considered incomplete if received_uuid is not set! my @delete = grep $_->{node}{received_uuid} eq '-', @{vinfo_subvol_list($droot, btrbk_direct_leaf => $snapshot_name, sort => 'path')}; my @delete_success; foreach my $target_vol (@delete) { DEBUG "Found incomplete target subvolume: $target_vol->{PRINT}"; if(btrfs_subvolume_delete($target_vol, commit => config_key($droot, "btrfs_commit_delete"), type => "delete_garbled")) { push(@delete_success, $target_vol); } } INFO "Deleted " . scalar(@delete_success) . " incomplete backups in: $droot->{PRINT}/$snapshot_name.*"; $droot->{SUBVOL_DELETED} //= []; push @{$droot->{SUBVOL_DELETED}}, @delete_success; push @out, map("--- $_->{PRINT}", @delete_success); if(scalar(@delete_success) != scalar(@delete)) { ABORTED($droot, "Failed to delete incomplete target subvolume"); push @out, "!!! Target \"$droot->{PRINT}\" aborted: " . ABORTED_TEXT($droot); } push(@out, "") unless(scalar(@delete)); push(@out, ""); } } } my $exit_status = exit_status($config); my $time_elapsed = time - $start_time; INFO "Completed within: ${time_elapsed}s (" . localtime(time) . ")"; action("finished", status => $exit_status ? "partial" : "success", duration => $time_elapsed, message => $exit_status ? "At least one delete operation failed" : undef, ); close_transaction_log(); # # print summary # unless($quiet) { $output_format ||= "custom"; if($output_format eq "custom") { print_header(title => "Cleanup Summary", config => $config, time => $start_time, legend => [ "--- deleted subvolume (incomplete backup)", ], ); print join("\n", @out); print_footer($config, $exit_status); } else { # print action log (without transaction start messages) my @data = grep { $_->{status} !~ /starting$/ } @transaction_log; print_formatted("transaction", \@data, title => "TRANSACTION LOG"); } } exit $exit_status; } if($action_run) { init_transaction_log(config_key($config, "transaction_log"), config_key($config, "transaction_syslog")); if($skip_snapshots) { INFO "Skipping snapshot creation (btrbk resume)"; } else { # # create snapshots # foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snaproot = vinfo_snapshot_root($svol); my $snapshot_basename = config_key($svol, "snapshot_name") // die; DEBUG "Evaluating snapshot creation for: $svol->{PRINT}"; # check if we need to create a snapshot my $snapshot_create = config_key($svol, "snapshot_create"); if(not $snapshot_create) { DEBUG "Snapshot creation disabled (snapshot_create=no)"; next; } elsif($snapshot_create eq "always") { DEBUG "Snapshot creation enabled (snapshot_create=always)"; } elsif($snapshot_create eq "onchange") { # check if latest (btrbk only!) snapshot is up-to-date with source subvolume (by generation) my $latest = get_latest_related_snapshot($snaproot, $svol, $snapshot_basename); if($latest) { if($latest->{node}{cgen} == $svol->{node}{gen}) { INFO "Snapshot creation skipped: snapshot_create=onchange, snapshot is up-to-date: $latest->{PRINT}"; $svol->{SNAPSHOT_UP_TO_DATE} = $latest; next; } DEBUG "Snapshot creation enabled: snapshot_create=onchange, gen=$svol->{node}{gen} > snapshot_cgen=$latest->{node}{cgen}"; } else { DEBUG "Snapshot creation enabled: snapshot_create=onchange, no snapshots found"; } } elsif($snapshot_create eq "ondemand") { # check if at least one target is present if(scalar vinfo_subsection($svol, 'target')) { DEBUG "Snapshot creation enabled (snapshot_create=ondemand): at least one target is present"; } else { INFO "Snapshot creation skipped: snapshot_create=ondemand, and no target is present for: $svol->{PRINT}"; next; } } else { die "illegal value for snapshot_create configuration option: $snapshot_create"; } # find unique snapshot name my $timestamp = timestamp(\@tm_now, config_key($svol, "timestamp_format")); my @unconfirmed_target_name; my @lookup = map { $_->{SUBVOL_PATH} } @{vinfo_subvol_list($snaproot)}; foreach my $droot (vinfo_subsection($svol, 'target', 1)) { if(IS_ABORTED($droot)) { push(@unconfirmed_target_name, $droot); next; } push(@lookup, map { $_->{SUBVOL_PATH} } @{vinfo_subvol_list($droot)}); } @lookup = grep /^\Q$snapshot_basename.$timestamp\E(_[0-9]+)?$/ ,@lookup; TRACE "Present snapshot names for \"$svol->{PRINT}\": " . join(', ', @lookup) if($do_trace); @lookup = map { /_([0-9]+)$/ ? $1 : 0 } @lookup; @lookup = sort { $b <=> $a } @lookup; my $postfix_counter = $lookup[0] // -1; $postfix_counter++; my $snapshot_name = $snapshot_basename . '.' . $timestamp . ($postfix_counter ? "_$postfix_counter" : ""); if(@unconfirmed_target_name) { INFO "Assuming non-present subvolume \"$snapshot_name\" in skipped targets: " . join(", ", map { "\"$_->{PRINT}\"" } @unconfirmed_target_name); } # finally create the snapshot INFO "Creating subvolume snapshot for: $svol->{PRINT}"; my $snapshot = vinfo_child($snaproot, "$snapshot_name"); if(btrfs_subvolume_snapshot($svol, $snapshot)) { vinfo_inject_child($snaproot, $snapshot, { parent_uuid => $svol->{node}{uuid}, received_uuid => '-', readonly => 1, FORCE_PRESERVE => 'preserve forced: created just now', }); $svol->{SNAPSHOT_CREATED} = $snapshot; } else { ABORTED($svol, "Failed to create snapshot: $svol->{PRINT} -> $snapshot->{PRINT}"); WARN "Skipping subvolume section: " . ABORTED_TEXT($svol); } } } } # # create backups # if($skip_backups) { INFO "Skipping backup creation (btrbk snapshot)"; } else { foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snaproot = vinfo_snapshot_root($svol); my $snapshot_basename = config_key($svol, "snapshot_name") // die; my @related_snapshots = sort({ cmp_date($a->{node}{BTRBK_DATE}, $b->{node}{BTRBK_DATE}) } get_related_snapshots($snaproot, $svol, $snapshot_basename)); foreach my $droot (vinfo_subsection($svol, 'target')) { INFO "Checking for missing backups of subvolume \"$svol->{PRINT}\" in \"$droot->{PRINT}/\""; my @schedule; my $resume_total = 0; my $resume_success = 0; foreach my $snapshot (@related_snapshots) { if(get_receive_targets($droot, $snapshot, exact => 1, warn => 1)){ DEBUG "Found correlated target of: $snapshot->{PRINT}"; next; } DEBUG "Adding backup candidate: $snapshot->{PRINT}"; push(@schedule, { value => $snapshot, btrbk_date => $snapshot->{node}{BTRBK_DATE}, # not enforcing resuming of latest snapshot anymore (since v0.23.0) # preserve => $snapshot->{node}{FORCE_PRESERVE}, }); } if(scalar @schedule) { DEBUG "Checking schedule for backup candidates"; # add all present backups as informative_only: these are needed for correct results of schedule() foreach my $vol (@{vinfo_subvol_list($droot, btrbk_direct_leaf => $snapshot_basename)}) { push(@schedule, { informative_only => 1, value => $vol, btrbk_date => $vol->{node}{BTRBK_DATE}, }); } my ($preserve, undef) = schedule( schedule => \@schedule, preserve => config_preserve_hash($droot, "target"), ); my @resume = grep defined, @$preserve; # remove entries with no value from list (target subvolumes) $resume_total = scalar @resume; foreach my $snapshot (sort { $a->{node}{cgen} <=> $b->{node}{cgen} } @resume) { # Continue gracefully (skip instead of abort) on existing (possibly garbled) target if(my $err_vol = vinfo_subvol($droot, $snapshot->{NAME})) { my $err_msg = "Please delete stray subvolumes: \"btrbk clean $droot->{PRINT}\""; FIX_MANUALLY($droot, $err_msg); WARN "Target subvolume \"$err_vol->{PRINT}\" exists, but is not a receive target of \"$snapshot->{PRINT}\""; WARN $err_msg; WARN "Skipping backup of: $snapshot->{PRINT}"; $droot->{SUBVOL_RECEIVED} //= []; push(@{$droot->{SUBVOL_RECEIVED}}, { ERROR => 1, received_subvolume => $err_vol }); next; } my ($clone_src, $target_parent_node); my $parent = get_best_parent($snapshot, $snaproot, $droot, strict_related => ((config_key($droot, "incremental") // "") eq "strict"), clone_src => \$clone_src, target_parent_node => \$target_parent_node); if(macro_send_receive(source => $snapshot, target => $droot, parent => $parent, # this is if no suitable parent found clone_src => $clone_src, target_parent_node => $target_parent_node, )) { $resume_success++; } else { # note: ABORTED flag is already set by macro_send_receive() ERROR("Error while resuming backups, aborting"); last; } } } if($resume_total) { INFO "Created $resume_success/$resume_total missing backups"; } else { INFO "No missing backups found"; } } } } } # # remove backups following a preserve daily/weekly/monthly scheme # my $schedule_results; if($preserve_snapshots && $preserve_backups) { INFO "Preserving all snapshots and backups"; } else { $schedule_results = []; foreach my $sroot (vinfo_subsection($config, 'volume')) { foreach my $svol (vinfo_subsection($sroot, 'subvolume')) { my $snaproot = vinfo_snapshot_root($svol); my $snapshot_basename = config_key($svol, "snapshot_name") // die; my $target_aborted = 0; my @related_snapshots = sort({ cmp_date($b->{node}{BTRBK_DATE}, $a->{node}{BTRBK_DATE}) } # sort descending get_related_snapshots($snaproot, $svol, $snapshot_basename)); foreach my $droot (vinfo_subsection($svol, 'target', 1)) { if(IS_ABORTED($droot)) { if(IS_ABORTED($droot, "skip_cmdline_")) { $target_aborted ||= -1; } else { $target_aborted = 1; } next; } # preserve latest common snapshot/backup (for incremental targets) if(config_key($droot, "incremental")) { foreach my $snapshot (@related_snapshots) { my @receive_targets = get_receive_targets($droot, $snapshot, exact => 1); if(scalar(@receive_targets)) { DEBUG "Force preserve for latest common snapshot: $snapshot->{PRINT}"; $snapshot->{node}{FORCE_PRESERVE} = 'preserve forced: latest common snapshot'; foreach(@receive_targets) { DEBUG "Force preserve for latest common target: $_->{PRINT}"; $_->{node}{FORCE_PRESERVE} = 'preserve forced: latest common target'; } last; } } } if($preserve_backups) { INFO "Preserving all backups"; } else { # # delete backups # INFO "Cleaning backups of subvolume \"$svol->{PRINT}\": $droot->{PRINT}/$snapshot_basename.*"; unless(macro_delete($droot, $snapshot_basename, $droot, { preserve => config_preserve_hash($droot, "target"), results => $schedule_results, result_hints => { topic => "backup", root_path => $droot->{PATH} }, }, commit => config_key($droot, "btrfs_commit_delete"), type => "delete_target", qgroup => { destroy => config_key($droot, "target_qgroup_destroy"), type => "qgroup_destroy_target" }, )) { $target_aborted = 1; } } } # # delete snapshots # if($preserve_snapshots) { INFO "Preserving all snapshots"; } elsif($target_aborted) { if($target_aborted == -1) { INFO "Skipping cleanup of snapshots for subvolume \"$svol->{PRINT}\", as at least one target is skipped by command line argument"; } else { WARN "Skipping cleanup of snapshots for subvolume \"$svol->{PRINT}\", as at least one target aborted earlier"; } } else { INFO "Cleaning snapshots" . ($wipe_snapshots ? " (wipe)" : "") . ": $snaproot->{PRINT}/$snapshot_basename.*"; macro_delete($snaproot, $snapshot_basename, $svol, { preserve => config_preserve_hash($svol, "snapshot", wipe => $wipe_snapshots), results => $schedule_results, result_hints => { topic => "snapshot", root_path => $snaproot->{PATH} }, }, commit => config_key($svol, "btrfs_commit_delete"), type => "delete_snapshot", qgroup => { destroy => config_key($svol, "snapshot_qgroup_destroy"), type => "qgroup_destroy_snapshot" }, ); } } } } my $exit_status = exit_status($config); my $time_elapsed = time - $start_time; INFO "Completed within: ${time_elapsed}s (" . localtime(time) . ")"; action("finished", status => $exit_status ? "partial" : "success", duration => $time_elapsed, message => $exit_status ? "At least one backup task aborted" : undef, ); close_transaction_log(); unless($quiet) { # # print scheduling results # if($print_schedule && $schedule_results) { my @data = map { { %$_, vinfo_prefixed_keys("", $_->{value}) }; } @$schedule_results; my @data_snapshot = grep { $_->{topic} eq "snapshot" } @data; my @data_backup = grep { $_->{topic} eq "backup" } @data; if(scalar(@data_snapshot)) { print_formatted("schedule", \@data_snapshot, title => "SNAPSHOT SCHEDULE", paragraph => 1); } if(scalar(@data_backup)) { print_formatted("schedule", \@data_backup, title => "BACKUP SCHEDULE", paragraph => 1); } } # # print summary # $output_format ||= "custom"; if($output_format eq "custom") { my @out; foreach my $sroot (vinfo_subsection($config, 'volume', 1)) { foreach my $svol (vinfo_subsection($sroot, 'subvolume', 1)) { my @subvol_out; if($svol->{SNAPSHOT_UP_TO_DATE}) { push @subvol_out, "=== $svol->{SNAPSHOT_UP_TO_DATE}->{PRINT}"; } if($svol->{SNAPSHOT_CREATED}) { push @subvol_out, "+++ $svol->{SNAPSHOT_CREATED}->{PRINT}"; } foreach(@{$svol->{SUBVOL_DELETED} // []}) { push @subvol_out, "--- $_->{PRINT}"; } foreach my $droot (vinfo_subsection($svol, 'target', 1)) { foreach(@{$droot->{SUBVOL_RECEIVED} // []}) { my $create_mode = "***"; $create_mode = ">>>" if($_->{parent}); # substr($create_mode, 0, 1, '%') if($_->{resume}); $create_mode = "!!!" if($_->{ERROR}); push @subvol_out, "$create_mode $_->{received_subvolume}->{PRINT}"; } foreach(@{$droot->{SUBVOL_DELETED} // []}) { push @subvol_out, "--- $_->{PRINT}"; } if(IS_ABORTED($droot, "abort_")) { push @subvol_out, "!!! Target \"$droot->{PRINT}\" aborted: " . ABORTED_TEXT($droot); } } if(IS_ABORTED($sroot, "abort_")) { # repeat volume errors in subvolume context push @subvol_out, "!!! Volume \"$sroot->{PRINT}\" aborted: " . ABORTED_TEXT($sroot); } if(IS_ABORTED($svol, "abort_")) { # don't print "" on skip_cmdline or skip_noauto push @subvol_out, "!!! Aborted: " . ABORTED_TEXT($svol); } # print "" for subvolume, unless aborted by "skip_" unless(scalar(@subvol_out) || IS_ABORTED($sroot, "skip_") || IS_ABORTED($svol, "skip_")) { @subvol_out = ""; } if(@subvol_out) { push @out, "$svol->{PRINT}", @subvol_out, ""; } } } my @cmdline_options = map { "exclude: $_" } @exclude_cmdline; push @cmdline_options, "$skip_snapshots: No snapshots created" if($skip_snapshots); push @cmdline_options, "$skip_backups: No backups created" if($skip_backups); push @cmdline_options, "$preserve_snapshots: Preserved all snapshots" if($preserve_snapshots); push @cmdline_options, "$preserve_backups: Preserved all backups" if($preserve_backups); print_header(title => "Backup Summary", config => $config, time => $start_time, options => \@cmdline_options, legend => [ "=== up-to-date subvolume (source snapshot)", "+++ created subvolume (source snapshot)", "--- deleted subvolume", "*** received subvolume (non-incremental)", ">>> received subvolume (incremental)", ], ); print join("\n", @out); print_footer($config, $exit_status); } else { # print action log (without transaction start messages) my @data = grep { $_->{status} !~ /starting$/ } @transaction_log; print_formatted("transaction", \@data, title => "TRANSACTION LOG"); } } exit $exit_status if($exit_status); } } 1; btrbk-0.32.5/btrbk.conf.example000066400000000000000000000144211432521455000163150ustar00rootroot00000000000000# # Example btrbk configuration file # # # Please refer to the btrbk.conf(5) man-page for a complete # description of all configuration options. # For more examples, see README.md included with this package. # # btrbk.conf(5): # README.md: # # Note that the options can be overridden per volume/subvolume/target # in the corresponding sections. # # Enable transaction log transaction_log /var/log/btrbk.log # Specify SSH private key for remote connections ssh_identity /etc/btrbk/ssh/id_ed25519 ssh_user root # Use sudo if btrbk or lsbtr is run by regular user backend_local_user btrfs-progs-sudo # Enable stream buffer. Adding a buffer between the sending and # receiving side is generally a good idea. # NOTE: If enabled, make sure to install the "mbuffer" package! stream_buffer 256m # Directory in which the btrfs snapshots are created. Relative to # of the volume section. # If not set, the snapshots are created in . # # If you want to set a custom name for the snapshot (and backups), # use the "snapshot_name" option within the subvolume section. # # NOTE: btrbk does not automatically create this directory, and the # snapshot creation will fail if it is not present. # snapshot_dir _btrbk_snap # Always create snapshots. Set this to "ondemand" to only create # snapshots if the target volume is reachable. Set this to "no" if # snapshot creation is done by another instance of btrbk. #snapshot_create always # Perform incremental backups (set to "strict" if you want to prevent # creation of non-incremental backups if no parent is found). #incremental yes # Specify after what time (in full hours after midnight) backups/ # snapshots are considered as a daily backup/snapshot #preserve_hour_of_day 0 # Specify on which day of week weekly/monthly backups are to be # preserved. #preserve_day_of_week sunday # Preserve all snapshots for a minimum period of time. #snapshot_preserve_min 1d # Retention policy for the source snapshots. #snapshot_preserve h d w m y # Preserve all backup targets for a minimum period of time. #target_preserve_min no # Retention policy for backup targets: #target_preserve h d w m y # Retention policy for archives ("btrbk archive" command): #archive_preserve_min no #archive_preserve h d w m y # Enable compression for remote btrfs send/receive operations: #stream_compress no #stream_compress_level default #stream_compress_threads default # Enable lock file support: Ensures that only one instance of btrbk # can be run at a time. #lockfile /var/lock/btrbk.lock # Don't wait for transaction commit on deletion. Enable this to make # sure the deletion of subvolumes is committed to disk when btrbk # terminates. #btrfs_commit_delete no # # Volume section (optional): "volume " # # Base path within a btrfs filesystem # containing the subvolumes to be backuped # (usually the mount-point of a btrfs filesystem # mounted with subvolid=5 option). # # Subvolume section: "subvolume " # # Subvolume to be backuped, relative to # in volume section. # # Target section: "target " # # (optional) type, defaults to "send-receive". # Directory within a btrfs filesystem # receiving the backups. # # NOTE: The parser does not care about indentation, this is only for # human readability. All options apply to the last section # encountered, overriding the corresponding option of the upper # section. This means that the global options must be set on top, # before any "volume", "subvolume" or "target section. # # # Example retention policy: # snapshot_preserve_min 2d snapshot_preserve 14d target_preserve_min no target_preserve 20d 10w *m # # Simple setup: Backup root and home to external disk # snapshot_dir /btrbk_snapshots target /mnt/btr_backup subvolume / subvolume /home # # Complex setup # # In order to keep things organized, it is recommended to use "volume" # sections and mount the top-level subvolume (subvolid=5): # # $ mount -o subvolid=5 /dev/sda1 /mnt/btr_pool # # Backup to external disk mounted on /mnt/btr_backup volume /mnt/btr_pool # Create snapshots in /mnt/btr_pool/btrbk_snapshots snapshot_dir btrbk_snapshots # Target for all subvolume sections: target /mnt/btr_backup # Some default btrfs installations (e.g. Ubuntu) use "@" for rootfs # (mounted at "/") and "@home" (mounted at "/home"). Note that this # is only a naming convention. #subvolume @ subvolume root subvolume home subvolume kvm # Use different retention policy for kvm backups: target_preserve 7d 4w # Backup data to external disk as well as remote host volume /mnt/btr_data subvolume data # Always create snapshot, even if targets are unreachable snapshot_create always target /mnt/btr_backup target ssh://backup.my-remote-host.com/mnt/btr_backup # Backup from remote host, with different naming volume ssh://my-remote-host.com/mnt/btr_pool subvolume data_0 snapshot_dir snapshots/btrbk snapshot_name data_main target /mnt/btr_backup/my-remote-host.com # Backup on demand (noauto) to remote host running busybox, login as # regular user using ssh-agent with current user name (ssh_user no) # and default credentials (ssh_identity no). volume /home noauto yes compat busybox backend_remote btrfs-progs-sudo ssh_user no ssh_identity no target ssh://my-user-host.com/mnt/btr_backup/home subvolume alice subvolume bob # Resume backups from remote host which runs its own btrbk instance # creating snapshots for "home" in "/mnt/btr_pool/btrbk_snapshots". volume ssh://my-remote-host.com/mnt/btr_pool snapshot_dir btrbk_snapshots snapshot_create no snapshot_preserve_min all subvolume home target /mnt/btr_backup/my-remote-host.com btrbk-0.32.5/contrib/000077500000000000000000000000001432521455000143465ustar00rootroot00000000000000btrbk-0.32.5/contrib/bash/000077500000000000000000000000001432521455000152635ustar00rootroot00000000000000btrbk-0.32.5/contrib/bash/completion.bash000066400000000000000000000055041432521455000202770ustar00rootroot00000000000000_btrbk_init_cmds() { # set $cmds to an array of the commands so far # # for example, for this command: # # btrbk -v --override warn_unknown_targets=yes list config --long # # then $cmds is: # # cmds=(list config) # cmds=() local i for ((i = 1; i < cword; i++)); do case "${words[i-1]}" in '-c' | '--config' | '--exclude' | '-l' | '--loglevel' | '--format' | '--lockfile' | '--override') continue ;; esac [[ ${words[i]} != -* ]] && cmds+=(${words[i]}) done return 0 } _btrbk() { local cur prev words cword split cmds _init_completion -s || return _btrbk_init_cmds || return case "$prev" in '-c' | '--config') _filedir return ;; '--exclude') return ;; '-l' | '--loglevel') COMPREPLY=($(compgen -W 'error warn info debug trace' -- "$cur")) return ;; '--format') COMPREPLY=($(compgen -W 'table long raw' -- "$cur")) return ;; '--lockfile') _filedir return ;; '--override') return ;; esac $split && return if [[ $cur == -* ]]; then COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur")) [[ $COMPREPLY == *= ]] && compopt -o nospace else if [[ ! -v 'cmds[0]' ]]; then COMPREPLY=($(compgen -W 'run dryrun snapshot resume prune archive clean stats list usage origin diff extents ls' -- "$cur")) fi fi case "${cmds[0]}" in 'archive') # if [[ ! -v 'cmds[1]' ]]; then _filedir -d # elif [[ ! -v 'cmds[2]' ]]; then _filedir -d # [--raw] elif [[ $cur == -* ]]; then COMPREPLY+=($(compgen -W '--raw' -- "$cur")) fi ;; 'list') if [[ ! -v 'cmds[1]' ]]; then COMPREPLY=($(compgen -W 'all snapshots backups latest config source volume target' -- "$cur")) fi ;; 'origin') # if [[ ! -v 'cmds[1]' ]]; then _filedir -d fi ;; 'ls') # |... _filedir -d ;; 'extents') # [diff] ... [exclusive] ... if [[ ! -v 'cmds[1]' ]]; then COMPREPLY+=($(compgen -W 'diff' -- "$cur")) elif [[ ! ${cmds[*]} =~ (^|[[:space:]])"exclusive"($|[[:space:]]) ]]; then COMPREPLY+=($(compgen -W 'exclusive' -- "$cur")) fi _filedir -d ;; esac } && complete -F _btrbk btrbk _lsbtr() { local cur prev words cword split _init_completion -s || return case "$prev" in '-c' | '--config') _filedir ;; '--override') ;; esac $split && return if [[ $cur == -* ]]; then COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur")) [[ $COMPREPLY == *= ]] && compopt -o nospace else # |... _filedir -d fi } && complete -F _lsbtr lsbtr # ex: filetype=bash btrbk-0.32.5/contrib/cron/000077500000000000000000000000001432521455000153075ustar00rootroot00000000000000btrbk-0.32.5/contrib/cron/btrbk-mail000077500000000000000000000171131432521455000172640ustar00rootroot00000000000000#!/bin/bash ## Wrapper script running "btrbk" and sending email with results now=$(date +%Y%m%d) ##### start config section ##### # Email recipients, separated by whitespace: mailto=${MAILTO:-root} # Email subject: mail_subject_prefix="btrbk <${HOSTNAME:-localhost}>" # Add summary and/or detail (rsync/btrbk command output) to mail body. # If both are not set, a mail is only sent on errors. mail_summary=yes mail_detail=no # List of mountpoints to be mounted/unmounted (whitespace-separated) # mount_targets="/mnt/btr_pool /mnt/backup" mount_targets= # rsync declarations (repeat complete block for more declarations): rsync_src[example_data]="user@example.com:/data/" rsync_dst[example_data]="/mnt/backup/example.com/data/" rsync_log[example_data]="/mnt/backup/example.com/data-${now}.log" rsync_rsh[example_data]="ssh -i /mnt/backup/ssh_keys/id_rsa" rsync_opt[example_data]="-az --delete --inplace --numeric-ids --acls --xattrs" # If set, add "rsync_dst" to "sync_fs" (see below) if rsync reports files transferred #sync_fs_onchange[example_data]=yes # Enabled rsync declarations (whitespace-separated list) #rsync_enable="example_data" rsync_enable= # If set, do not run btrbk if rsync reports no changes. # If set to "quiet", do not send mail. #skip_btrbk_if_unchanged=quiet # Array of directories to sync(1) prior to running btrbk. This is # useful for source subvolumes having "snapshot_create ondemand" # configured in btrbk.conf. #sync_fs=("/mnt/btr_data" "/mnt/btr_pool") # btrbk command / options: btrbk_command="run" btrbk_opts="-c /etc/btrbk/btrbk.conf" ### Layout options: # Prefix command output: useful when using mail clients displaying # btrbk summary lines starting with ">>>" as quotations. #mail_cmd_block_prefix='\\u200B' # zero-width whitespace #mail_cmd_block_prefix=". " ##### end config section ##### check_options() { [[ -n "$btrbk_command" ]] || die "btrbk_command is not set" for key in $rsync_enable; do [[ -n "${rsync_src[$key]}" ]] || die "rsync_src is not set for \"$key\"" [[ -n "${rsync_dst[$key]}" ]] || die "rsync_dst is not set for \"$key\"" [[ -n "${rsync_opt[$key]}" ]] || die "rsync_opt is not set for \"$key\"" done } send_mail() { # assemble mail subject local subject="$mail_subject_prefix" [[ -n "$has_errors" ]] && subject+=" ERROR"; [[ -n "$status" ]] && subject+=" - $status"; [[ -n "$xstatus" ]] && subject+=" (${xstatus:2})"; # assemble mail body local body= if [[ -n "$info" ]] && [[ -n "$has_errors" ]] || [[ "${mail_summary:-no}" = "yes" ]]; then body+="$info" fi if [[ -n "$detail" ]] && [[ -n "$has_errors" ]] || [[ "${mail_detail:-no}" = "yes" ]]; then [[ -n "$body" ]] && body+="\n\nDETAIL:\n" body+="$detail" fi # skip sending mail on empty body if [[ -z "$body" ]] && [[ -n "$has_errors" ]]; then body+="FATAL: something went wrong (errors present but empty mail body)\n" fi [[ -z "$body" ]] && exit 0 # send mail echo -e "$body" | mail -s "$subject" $mailto if [[ $? -ne 0 ]]; then echo -e "$0: Failed to send btrbk mail to \"$mailto\", dumping mail:\n" 1>&2 echo -e "$subject\n\n$body" 1>&2 fi } einfo() { info+="$1\n" } ebegin() { ebtext=$1 detail+="\n### $1\n" } eend() { if [[ $1 -eq 0 ]]; then eetext=${3-success} detail+="\n" else has_errors=1 eetext="ERROR (code=$1)" [[ -n "$2" ]] && eetext+=": $2" detail+="\n### $eetext\n" fi info+="$ebtext: $eetext\n" return $1 } die() { einfo "FATAL: ${1}, exiting" has_errors=1 send_mail exit 1 } run_cmd() { cmd_out=$("$@" 2>&1) local ret=$? detail+="++ ${@@Q}\n" if [[ -n "${mail_cmd_block_prefix:-}" ]] && [[ -n "$cmd_out" ]]; then detail+=$(echo -n "$cmd_out" | sed "s/^/${mail_cmd_block_prefix}/") detail+="\n" else detail+=$cmd_out fi return $ret } mount_all() { # mount all mountpoints listed in $mount_targets mounted="" for mountpoint in $mount_targets; do ebegin "Mounting $mountpoint" run_cmd findmnt -n $mountpoint if [[ $? -eq 0 ]]; then eend -1 "already mounted" else detail+="\n" run_cmd mount --target $mountpoint eend $? && mounted+=" $mountpoint" fi done } umount_mounted() { for mountpoint in $mounted; do ebegin "Unmounting $mountpoint" run_cmd umount $mountpoint eend $? done } check_options mount_all # # run rsync for all $rsync_enable # for key in $rsync_enable; do ebegin "Running rsync[$key]" if [[ -d "${rsync_dst[$key]}" ]]; then # There is no proper way to get a proper machine readable # output of "rsync did not touch anything at destination", so # we add "--info=stats2" and parse the output. # NOTE: This also appends the stats to the log file (rsync_log). # Another approach to count the files would be something like: # "rsync --out-format='' | wc -l" run_cmd rsync ${rsync_opt[$key]} \ --info=stats2 \ ${rsync_log[$key]:+--log-file="${rsync_log[$key]}"} \ ${rsync_rsh[$key]:+-e "${rsync_rsh[$key]}"} \ "${rsync_src[$key]}" \ "${rsync_dst[$key]}" exitcode=$? # parse stats2 (count created/deleted/transferred files) REGEXP=$'\n''Number of created files: ([0-9]+)' REGEXP+='.*'$'\n''Number of deleted files: ([0-9]+)' REGEXP+='.*'$'\n''Number of regular files transferred: ([0-9]+)' if [[ $cmd_out =~ $REGEXP ]]; then rsync_stats="${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}" rsync_stats_long="${BASH_REMATCH[1]} created, ${BASH_REMATCH[2]} deleted, ${BASH_REMATCH[3]} transferred" nfiles=$(( ${BASH_REMATCH[1]} + ${BASH_REMATCH[2]} + ${BASH_REMATCH[3]} )) else rsync_stats_long="failed to parse stats, assuming files transferred" rsync_stats="-1/-1/-1" nfiles=-1 fi eend $exitcode "$rsync_stats_long" "$rsync_stats_long" xstatus+=", rsync[$key]=$rsync_stats" if [[ $nfiles -ne 0 ]]; then # NOTE: on error, we assume files are transferred rsync_files_transferred=1 [[ -n "${sync_fs_onchange[$key]}" ]] && sync_fs+=("${rsync_dst[$key]}") fi else eend -1 "Destination directory not found, skipping: ${rsync_dst[$key]}" fi done # honor skip_btrbk_if_unchanged (only if rsync is enabled and no files were transferred) if [[ -n "$rsync_enable" ]] && [[ -n "$skip_btrbk_if_unchanged" ]] && [[ -z "$rsync_files_transferred" ]]; then einfo "No files transferred, exiting" status="No files transferred" umount_mounted if [[ "$skip_btrbk_if_unchanged" != "quiet" ]] || [[ -n "$has_errors" ]]; then send_mail fi exit 0 fi # # sync filesystems in sync_fs # if [[ ${#sync_fs[@]} -gt 0 ]]; then ebegin "Syncing filesystems at ${sync_fs[@]}" run_cmd sync -f "${sync_fs[@]}" eend $? fi # # run btrbk # ebegin "Running btrbk" run_cmd btrbk ${btrbk_opts:-} ${btrbk_command} exitcode=$? case $exitcode in 0) status="All backups successful" ;; 3) status="Another instance of btrbk is running, no backup tasks performed!" ;; 10) status="At least one backup task aborted!" ;; *) status="btrbk failed with error code $exitcode" ;; esac eend $exitcode "$status" umount_mounted send_mail btrbk-0.32.5/contrib/cron/btrbk-verify000077500000000000000000000330111432521455000176410ustar00rootroot00000000000000#!/bin/bash # # NAME # # btrbk-verify - check latest btrbk snapshot/backup pairs # # # SYNOPSIS # # btrbk-verify [options] [filter...] # # # DESCRIPTION # # Compare btrbk backups. Reads all files and attributes, and # compares checksums of source and target. Uses rsync(1) as backend, # in dry-run mode with all preserve options enabled. # # Resolves snapshot/backup pairs by evaluating the output of # "btrbk list latest [filter...]". The filter argument is passed # directly to btrbk, see btrbk(1) FILTER STATEMENTS. # # Restrictions: # - ".d..t...... ./" lines are ignored by default: # Root folder timestamp always differ. # - "cd+++++++++ .*" lines are ignored by default: # Nested subvolumes appear as new empty directories. # - btrbk raw targets are skipped # - rsync needs root in most cases (see --ssh-* options) # # NOTE: Depending on your setup (hardware, btrfs mount options), # btrbk-verify may eat all your CPU power and use high bandwidth! # Consider nice(1), ionice(1). # Incomplete resource eater list: # - rsync: checksums, heavy disk I/O # - btrfs: decompression, encryption # - ssh: compression, encryption # # # EXAMPLES # # btrbk-verify latest /mnt/btr_pool # # Verify latest backups from targets configured in # /etc/btrbk/btrbk.conf, matching the "/mnt/btr_pool" filter. # # btrbk-verify all # # Verify ALL backups from targets in /etc/btrbk/btrbk.conf. # NOTE: This really re-checksums ALL files FOR EACH BACKUP, # even if they were not touched between backups! # # btrbk-verify latest -n -v -v # # Print detailed log as well as command executed by this script, # without actually executing rsync commands (-n, --dry-run). # # btrbk-verify --ssh-agent --ssh-user root --ssh-identity /etc/btrbk/ssh/id_ed25519 # # Use "ssh -i /etc/btrbk/ssh/id_ed25519 -l root" for rsync rsh # (override settings from btrbk.conf), start an ssh-agent(1) for # this session and verify all latest snapshot / backups. # # # SEE ALSO # # btrbk(1), btrbk.conf(5), rsync(1), nice(1), ionice(1) # # # AUTHOR # # Axel Burri # set -u set -e set -o pipefail btrbk_version_min='0.32.0' # defaults: ignore subvol dirs and root folder timestamp change ignore_nested_subvolume_dir=1 ignore_root_folder_timestamp=1 ssh_identity= ssh_user= ssh_start_agent= verbose=0 stats_enabled= dryrun= print_usage() { #80----------------------------------------------------------------------------- cat 1>&2 < [btrbk-list-options...] [filter...] options: -h, --help display this help message -c, --config FILE specify btrbk configuration file -n, --dry-run perform a trial run without verifying subvolumes -v, --verbose be verbose (set twice for debug loglevel) --stats print rsync stats to stderr (--info=stats2) --strict treat all rsync diffs as errors --ignore-acls ignore acls when verifying subvolumes --ignore-xattrs ignore xattrs when verifying subvolumes --ssh-identity FILE override ssh_identity from btrbk.conf(5) with FILE, and clear all other ssh_* options (use with --ssh-user) --ssh-user USER override ssh_user from btrbk.conf(5) with USER, and clear all other ssh_* options(use with --ssh-identity) --ssh-agent start ssh-agent(1) and add identity commands: latest verify most recent snapshots and backups (btrbk list latest) all verify all snapshots and backups (btrbk list backups) For additional information, see EOF #80----------------------------------------------------------------------------- exit ${1:-0} } list_subcommand= btrbk_args=() rsync_args=(-n --itemize-changes --checksum -a --delete --numeric-ids --hard-links --acls --xattrs --devices --specials) while [[ "$#" -ge 1 ]]; do key="$1" case $key in latest) [[ -n "$list_subcommand" ]] && print_usage 2; list_subcommand="latest" ;; all) [[ -n "$list_subcommand" ]] && print_usage 2; list_subcommand="backups" ;; -n|--dry-run) dryrun=1 ;; --stats) # enable rsync stats2 (transfer statistics) rsync_args+=(--info=stats2) stats_enabled=1 ;; --strict) # treat all rsync diffs as errors: # - empty directories (nested subvolumes) # - root folder timestamp mismatch ignore_nested_subvolume_dir= ignore_root_folder_timestamp= ;; --ignore-*) # --ignore-acls, --ignore-xattrs, --ignore-device, ... # remove "--xxx" flag from rsync_args for --ignore-xxx rsync_args=(${rsync_args[@]/"--"${key#"--ignore-"}}) ;; --ssh-identity) # use different ssh identity (-i option) for rsync rsh. # NOTE: this overrides all btrbk ssh_* options ssh_identity="$2" shift ;; --ssh-user) # use different ssh user (-l option) for rsync rsh # NOTE: this overrides all btrbk ssh_* options ssh_user="$2" shift ;; --ssh-agent) ssh_start_agent=1 ;; -v|--verbose) verbose=$((verbose+1)) btrbk_args+=("-v") ;; -h|--help) print_usage 0 ;; *) # all other args are passed to btrbk (filter, -c,--config=FILE) btrbk_args+=("$key") ;; esac shift done log_line() { echo "$@" 1>&2 } log_stats () { [[ -n "$stats_enabled" ]] && log_line "$@" ; return 0; } log_verbose() { [[ $verbose -ge 1 ]] && log_line "$@" ; return 0; } log_debug() { [[ $verbose -ge 2 ]] && log_line "$@" ; return 0; } log_cmd() { local prefix="" [[ -n "$dryrun" ]] && prefix="(dryrun) " log_debug "### ${prefix}$@" } tlog() { # same output as btrbk transaction log local status=$1 local comment=${2:-} [[ -n "$dryrun" ]] && [[ "$status" == "starting" ]] && status="dryrun_starting" local line="$(date --iso-8601=seconds) verify-rsync ${status} ${target} ${source} - -" [[ -n "$comment" ]] && line="$line # $comment"; tlog_text+="$line\n" log_debug "$line" } tlog_print() { # tlog goes to stdout echo -e "\nTRANSACTION LOG\n---------------\n${tlog_text:-}" } # parse "rsync -i,--itemize-changes" output. # prints ndiffs to stdout, and detailed log messages to stderr count_rsync_diffs() { local nn=0 local rsync_line_match='^(...........) (.*)$' local dump_stats_mode= # unset IFS: no word splitting, trimming (read literal line) while IFS= read -r rsync_line; do local postfix_txt="" if [[ -n "$dump_stats_mode" ]]; then # dump_stats_mode enabled, echo to stderr log_stats "${rsync_line}" elif [[ "$rsync_line" == "" ]]; then # empty line denotes start of --info=stats, enable dump_stats_mode dump_stats_mode=1 log_stats "--- BEGIN rsync stats2 dump ---" elif [[ "$rsync_line" =~ $rsync_line_match ]]; then rl_flags="${BASH_REMATCH[1]}" rl_path="${BASH_REMATCH[2]}" if [[ -n "$ignore_root_folder_timestamp" ]] && [[ "$rsync_line" == ".d..t...... ./" ]]; then # ignore timestamp on root folder, for some reason this does not match postfix_txt=" # IGNORE reason=ignore_root_folder_timestamp" elif [[ -n "$ignore_nested_subvolume_dir" ]] && [[ "$rl_flags" == "cd+++++++++" ]]; then # nested subvolumes appear as new empty directories ("cd+++++++++") in rsync (btrfs bug?) postfix_txt=" # IGNORE reason=ignore_nested_subvolume_dir" else nn=$((nn+1)) postfix_txt=" # FAIL ndiffs=$nn" fi log_verbose "[rsync] ${rsync_line}${postfix_txt}" else nn=$((nn+1)) log_line "btrbk-verify: ERROR: failed to parse rsync line: ${rsync_line}" fi done [[ -n "$dump_stats_mode" ]] && log_stats "--- END rsync stats2 dump ---" echo $nn return 0 } rsync_rsh() { # btrbk v0.27.0 sets source_rsh="ssh [flags...] ssh_user@ssh_host" # this returns "ssh [flags...] -l ssh_user" local rsh=$1 local rsh_match="(.*) ([a-z0-9_-]+)@([a-zA-Z0-9.-]+)$" if [[ -z "$rsh" ]]; then return elif [[ -n "$ssh_user" ]] || [[ -n "$ssh_identity" ]]; then # override btrbk.conf from command line arguments log_debug "Overriding all ssh_* options from btrbk.conf" local cmd="ssh -q" [[ -n "$ssh_identity" ]] && cmd="$cmd -i '$ssh_identity'" [[ -n "$ssh_user" ]] && cmd="$cmd -l '$ssh_user'" echo "$cmd" elif [[ $rsh =~ $rsh_match ]]; then echo "${BASH_REMATCH[1]} -l ${BASH_REMATCH[2]}" else log_line "btrbk-verify: ERROR: failed to parse source_rsh: $rsh" exit 1 fi } kill_ssh_agent() { echo "Stopping SSH agent" eval `ssh-agent -k` } start_ssh_agent() { if [[ -z "$ssh_identity" ]]; then log_line "btrbk-verify: ERROR: no SSH identity specified for agent" print_usage 2 fi echo "Starting SSH agent" eval `ssh-agent -s` ssh_agent_running=1 trap 'exit_trap_action' EXIT ssh-add "$ssh_identity" } eval_btrbk_resolved_line() { local line=" $1" local prefix=$2 local required_keys=$3 # reset all variables first for vv in $required_keys; do eval "${prefix}${vv}=" done for vv in $required_keys; do # basic input validation, set prefixed variable (eval) local match=" ${vv}='([^']*('\\\\''[^']*)*)'" if [[ $line =~ $match ]] ; then eval "${prefix}${vv}='${BASH_REMATCH[1]}'" || return 1 else log_line "btrbk-verify: ERROR: Missing variable \"${vv}\"" return 1 fi done } exit_trap_action() { [[ -n "${ssh_agent_running:-}" ]] && kill_ssh_agent [[ $verbose -gt 0 ]] && tlog_print } # start ssh-agent(1) [[ -n "$ssh_start_agent" ]] && start_ssh_agent # run "btrbk list" [[ -z "$list_subcommand" ]] && print_usage 2 log_verbose "Resolving btrbk $list_subcommand" btrbk_cmd=("btrbk" "list" "$list_subcommand" "--format=raw" "-q" "${btrbk_args[@]}") log_debug "### ${btrbk_cmd[@]}" btrbk_list=$("${btrbk_cmd[@]}") btrbk_list_exitstatus=$? if [[ $btrbk_list_exitstatus -ne 0 ]]; then log_line "btrbk-verify: ERROR: Command execution failed (status=$btrbk_list_exitstatus): ${btrbk_cmd[@]}" exit 1 fi log_debug "--- BEGIN btrbk list $list_subcommand ---" log_debug "$btrbk_list" log_debug "--- END btrbk list $list_subcommand ---" tlog_text="" exitstatus=0 # trap on EXIT (includes all signals) trap 'exit_trap_action' EXIT while read -r btrbk_list_line; do # set R_xxx variables from format=raw line (table format "resolved") log_debug "Evaluating [btrbk list] line: $btrbk_list_line" [[ -z "$btrbk_list_line" ]] && continue if ! eval_btrbk_resolved_line "$btrbk_list_line" \ "R_" "snapshot_subvolume target_subvolume source_host target_host target_type source_rsh target_rsh" then log_line "btrbk-verify: ERROR: Parse error of command output: ${btrbk_cmd[@]}" log_line "Make sure to have >=btrbk-${btrbk_version_min} installed!" exitstatus=1 break fi source="${R_snapshot_subvolume}/" target="${R_target_subvolume}/" [[ -n "$R_source_host" ]] && source="${R_source_host}:${source}" [[ -n "$R_target_host" ]] && target="${R_target_host}:${target}" if [[ -z "$R_snapshot_subvolume" ]]; then log_line "WARNING: Skipping task (missing snapshot): target=$target" elif [[ -z "$R_target_subvolume" ]]; then log_line "Skipping task (no target): source=$source" elif [[ "$R_target_type" != "send-receive" ]]; then log_line "Skipping task (target_type=$R_target_type): source=$source, target=$target" elif [[ -n "$R_source_rsh" ]] && [[ -n "$R_target_rsh" ]]; then log_line "WARNING: Skipping task (SSH for both source and target is not supported): target=$target" else log_line "Comparing [rsync] $source $target" # rsync rsh is either source_rsh or target_rsh or empty eff_rsh="$R_source_rsh" [[ -z "$eff_rsh" ]] && eff_rsh="$R_target_rsh" eff_rsh=$(rsync_rsh "$eff_rsh") rsync_cmd=("rsync" "${rsync_args[@]}") [[ -n "$eff_rsh" ]] && rsync_cmd+=("-e" "$eff_rsh") rsync_cmd+=("${source}" "${target}") log_cmd "${rsync_cmd[@]}" [[ -n "$dryrun" ]] && rsync_cmd=("cat" "/dev/null") #rsync_cmd=("echo" '........... SHOULD/FAIL/'); # simulate failure #rsync_cmd=("echo" 'cd+++++++++ SHOULD/IGNORE/'); # simulate ignored # execute rsync tlog "starting" set +e ndiffs=$("${rsync_cmd[@]}" | count_rsync_diffs) rsync_exitstatus=$? set -e if [[ $rsync_exitstatus -ne 0 ]] || [[ -z "$ndiffs" ]]; then log_line "btrbk-verify: ERROR: Command execution failed (status=$rsync_exitstatus): ${rsync_cmd[@]}" tlog "ERROR" exitstatus=10 elif [[ $ndiffs -gt 0 ]]; then log_line "VERIFY FAIL (ndiffs=$ndiffs): ${source} ${target}" tlog "fail" "ndiffs=$ndiffs" exitstatus=10 else log_verbose "Compare success (ndiffs=$ndiffs)" tlog "success" fi fi done <<< "$btrbk_list" #done < <(echo "$btrbk_list") # more posix'ish # NOTE: this triggers exit_trap_action() exit $exitstatus btrbk-0.32.5/contrib/crypt/000077500000000000000000000000001432521455000155075ustar00rootroot00000000000000btrbk-0.32.5/contrib/crypt/kdf_pbkdf2.py000077500000000000000000000035771432521455000200740ustar00rootroot00000000000000#!/usr/bin/env python3 # # kdf_pbkdf2.py - (kdf_backend for btrbk) # # Copyright (c) 2017 Axel Burri # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # --------------------------------------------------------------------- # The official btrbk website is located at: # https://digint.ch/btrbk/ # # Author: # Axel Burri # --------------------------------------------------------------------- import sys import os import getpass import hashlib def passprompt(): pprompt = lambda: (getpass.getpass("Passphrase: "), getpass.getpass("Retype passphrase: ")) p1, p2 = pprompt() while p1 != p2: print("No match, please try again", file=sys.stderr) p1, p2 = pprompt() return p1 if len(sys.argv) <= 1: print("Usage: {} ".format(sys.argv[0]), file=sys.stderr) sys.exit(1) hash_name = "sha256" iterations = 300000 dklen = int(sys.argv[1]) salt = os.urandom(16) password = passprompt().encode("utf-8") dk = hashlib.pbkdf2_hmac(hash_name=hash_name, password=password, salt=salt, iterations=iterations, dklen=dklen) salt_hex = "".join(["{:02x}".format(x) for x in salt]) dk_hex = "".join(["{:02x}".format(x) for x in dk]) print("KEY=" + dk_hex); print("algoritm=pbkdf2_hmac"); print("hash_name=" + hash_name); print("salt=" + salt_hex); print("iterations=" + str(iterations)); btrbk-0.32.5/contrib/migration/000077500000000000000000000000001432521455000163375ustar00rootroot00000000000000btrbk-0.32.5/contrib/migration/raw_suffix2sidecar000077500000000000000000000144731432521455000220700ustar00rootroot00000000000000#!/usr/bin/perl # # raw_suffix2sidecar - migrate to btrbk raw target sidecar files # # Copyright (C) 2017 Axel Burri # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # --------------------------------------------------------------------- # The official btrbk website is located at: # https://digint.ch/btrbk/ # # Author: # Axel Burri # --------------------------------------------------------------------- # Create raw sidecar ".info" files from uuid-suffixed raw backup files # generated by btrbk < v0.26.0. use strict; use warnings FATAL => qw( all ); use Getopt::Long qw(GetOptions); our $VERSION = '0.26.0'; # match btrbk version our $AUTHOR = 'Axel Burri '; our $PROJECT_HOME = ''; my $VERSION_INFO = "raw_suffix2sidecar (btrbk migration script), version $VERSION"; my $compress_format_alt = 'gz|bz2|xz|lzo|lz4'; my $file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/; # note: ubuntu uses '@' in the subvolume layout: my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/; my $timestamp_postfix_match = qr/\.(?[0-9]{4})(?[0-9]{2})(?
[0-9]{2})(T(?[0-9]{2})(?[0-9]{2})((?[0-9]{2})(?(Z|[+-][0-9]{4})))?)?(_(?[0-9]+))?/; # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]" my $raw_postfix_match = qr/--(?$uuid_match)(\@(?$uuid_match))?\.btrfs?(\.(?($compress_format_alt)))?(\.(?gpg))?(\.(?split_aa))?(\.(?part))?/; # matches ".btrfs_[@][.gz|bz2|xz][.gpg][.split_aa][.part]" my $dryrun; my %raw_info_sort = ( TYPE => 1, FILE => 2, RECEIVED_UUID => 3, RECEIVED_PARENT_UUID => 4, INCOMPLETE => 5, compress => 9, split => 10, encrypt => 11, ); sub VERSION_MESSAGE { print STDERR $VERSION_INFO . "\n\n"; } sub HELP_MESSAGE { print STDERR "usage: raw_suffix2sidecar ...\n"; print STDERR "\n"; print STDERR "options:\n"; # "--------------------------------------------------------------------------------"; # 80 print STDERR " -h, --help display this help message\n"; print STDERR " --version display version information\n"; print STDERR " -n, --dry-run perform a trial run with no changes made\n"; print STDERR "\n"; print STDERR "For additional information, see $PROJECT_HOME\n"; } sub write_raw_info($$) { my $file = shift // die; my $raw_info = shift // die; my $info_file = $file . '.info'; my @line; push @line, "#raw_suffix2sidecar-v$VERSION"; push @line, "# Do not edit this file"; # sort by %raw_info_sort, then by key foreach(sort { (($raw_info_sort{$a} || 99) <=> ($raw_info_sort{$b} || 99)) || ($a cmp $b) } keys %$raw_info) { push @line, ($_ . '=' . $raw_info->{$_}) if($raw_info->{$_}); } print "Creating info file: $info_file\n"; unless($dryrun) { open (INFOFILE, ">> $info_file") || die "Failed to open $info_file"; print INFOFILE join("\n", @line) . "\n"; close(INFOFILE); } return $info_file; } MAIN: { Getopt::Long::Configure qw(gnu_getopt); unless(GetOptions( 'help|h' => sub { VERSION_MESSAGE(); HELP_MESSAGE(0); exit 0; }, 'version' => sub { VERSION_MESSAGE(); exit 0; }, 'dry-run|n' => \$dryrun, )) { VERSION_MESSAGE(); HELP_MESSAGE(0); exit 2; } unless(@ARGV) { VERSION_MESSAGE(); HELP_MESSAGE(); exit 1; } foreach my $target_dir (@ARGV) { $target_dir =~ s/\/+$//; print "Processing directory: $target_dir/\n"; opendir(my($dh), $target_dir) || die "Failed to open directory '$target_dir': $!"; my @files = readdir($dh); closedir $dh; my @splitfiles = @files; foreach my $file (@files) { if($file =~ /^(?$file_match$timestamp_postfix_match)$raw_postfix_match$/) { print "\nProcessing raw backup: $file\n"; my $newname = $+{basename} || die; my %raw_info = ( TYPE => 'raw', RECEIVED_UUID => $+{received_uuid}, RECEIVED_PARENT_UUID => $+{parent_uuid}, INCOMPLETE => $+{incomplete} ? 1 : 0, compress => $+{compress}, split => ($+{split} ? (-s $file) : undef), # file size encrypt => $+{encrypt}, ); die "Missing received uuid in file: $file" unless $raw_info{RECEIVED_UUID}; $newname .= '.btrfs'; $newname .= '.' . $raw_info{compress} if($raw_info{compress}); $newname .= '.' . $raw_info{encrypt} if($raw_info{encrypt}); $raw_info{FILE} = $newname; write_raw_info("$target_dir/$newname", \%raw_info); if($raw_info{split}) { my $sfile = $file; $sfile =~ s/_aa$//; # we match on ".split_aa" above foreach my $splitfile (@splitfiles) { if($splitfile =~ /^${sfile}(_[a-z]+)$/) { my $suffix = $1 // die; print "Renaming file: $target_dir/$splitfile -> $target_dir/$newname.split$suffix\n"; unless($dryrun) { rename("$target_dir/$splitfile", "$target_dir/$newname.split$suffix") || die "Failed to rename file: $target_dir/$splitfile -> $target_dir/${newname}.split$suffix: $!"; } } } } else { print "Renaming file: $target_dir/$file -> $target_dir/$newname\n"; unless($dryrun) { rename("$target_dir/$file", "$target_dir/$newname") || die "Failed to rename file: $target_dir/$file -> $target_dir/$newname"; } } } } } if($dryrun) { print "\nNOTE: Dryrun was active, none of the operations above were actually executed!\n"; } } 1; btrbk-0.32.5/contrib/systemd/000077500000000000000000000000001432521455000160365ustar00rootroot00000000000000btrbk-0.32.5/contrib/systemd/btrbk.service.in000066400000000000000000000001601432521455000211260ustar00rootroot00000000000000[Unit] Description=btrbk backup Documentation=man:btrbk(1) [Service] Type=oneshot ExecStart=@BINDIR@/btrbk run btrbk-0.32.5/contrib/systemd/btrbk.timer.in000066400000000000000000000002041432521455000206050ustar00rootroot00000000000000[Unit] Description=btrbk daily backup [Timer] OnCalendar=daily AccuracySec=10min Persistent=true [Install] WantedBy=timers.target btrbk-0.32.5/contrib/tools/000077500000000000000000000000001432521455000155065ustar00rootroot00000000000000btrbk-0.32.5/contrib/tools/btrbk_restore_raw.py000077500000000000000000000164571432521455000216200ustar00rootroot00000000000000#!/usr/bin/env python3 import os import logging import subprocess import argparse logger = logging.getLogger(__name__) class TransformProcess: def run(self, bfile, options, **kw): return subprocess.Popen(self.get_cmd(bfile, options), **kw) def get_cmd(self, bfile, options): raise NotImplementedError() @classmethod def add_parser_options(cls, parser): pass class TransformOpensslDecrypt(TransformProcess): @staticmethod def get_cmd(bfile, options): return [ 'openssl', 'enc', '-d', '-' + bfile.info['cipher'], '-K', open(options.openssl_keyfile, 'r').read(), '-iv', bfile.info['iv'] ] @staticmethod def add_parser_options(parser): parser.add_argument('--openssl-keyfile', help="path to private encryption key file") class TransformDecompress(TransformProcess): def __init__(self, program): self.p = program def get_cmd(self, bfile, options): return [self.p, '-d'] class TransformBtrfsReceive(TransformProcess): @classmethod def run(cls, bfile, options, **kw): return subprocess.Popen(cls.get_cmd(bfile, options), **kw) @staticmethod def get_cmd(bfile, options): return ['btrfs', 'receive', options.restore_dir] TRANSFORMERS = ( TransformOpensslDecrypt, TransformDecompress, TransformBtrfsReceive ) class BtrfsPipeline: def __init__(self, bfile): self.bfile = bfile self.processors = [] def append(self, transformer): self.processors.append(transformer) def run(self, options): processes = [] with open(self.bfile.data_file, 'rb') as next_input: for transformer in self.processors: process = transformer.run( self.bfile, options, stdin=next_input, stdout=subprocess.PIPE, stderr=subprocess.PIPE) next_input = process.stdout processes.append(process) btrfs_process = TransformBtrfsReceive.run( self.bfile, options, stdin=next_input, stderr=subprocess.PIPE, stdout=subprocess.DEVNULL) processes.append(btrfs_process) # warning: the code below is pretty ugly and hacky terminated = 0 while terminated < len(processes): for p in processes: if p.returncode is not None: continue msg = None try: p.wait(timeout=1) except subprocess.TimeoutExpired as e: pass except Exception as e: msg = e else: msg = p.stderr.read().decode('utf-8').strip() finally: if p.returncode is not None: terminated += 1 if p.returncode != 0: for p_other in processes: p_other.terminate() terminated += 1 if msg: logger.error(f"error running {p.args}: {msg}") def get_cmd(self, options): command_pipe = [['cat', self.bfile.data_file]] for transformer in self.processors: command_pipe.append(transformer.get_cmd(self.bfile, options)) command_pipe.append(TransformBtrfsReceive.get_cmd(self.bfile, options)) return ' | '.join(' '.join(x) for x in command_pipe) class BackupFile: def __init__(self, path): assert path.endswith('.info') self.info_file = path self.info = self._parse_info() self.uuid = self.info['RECEIVED_UUID'] self.data_file = os.path.join(os.path.dirname(path), self.info['FILE']) self.parent = self.info.get('RECEIVED_PARENT_UUID') self.is_restored = False def _parse_info(self): config = {} with open(self.info_file, 'r') as fh: # skip command option line for line in fh.readlines(): if '=' not in line: continue key, val = line.strip().split('=', maxsplit=1) config[key] = val return config def get_transformers(self): if 'encrypt' in self.info: if self.info['encrypt'] == 'gpg': raise NotImplementedError('gpg encryption') elif self.info['encrypt'] == 'openssl_enc': yield TransformOpensslDecrypt() else: raise Exception(f'unknown encryption type: "{self.info["encrypt"]}"') if 'compress' in self.info: yield TransformDecompress(self.info['compress']) def restore_file(self, options): assert self.info.get('TYPE') == 'raw' assert not self.info.get('INCOMPLETE') pipeline = BtrfsPipeline(self) for transformer in self.get_transformers(): pipeline.append(transformer) if options.dry_run: print(pipeline.get_cmd(options)) else: logger.info(f"restoring backup {os.path.basename(self.data_file)}") pipeline.run(options) self.is_restored = True def restore_from_path(backup, options): path = os.path.dirname(backup) info_files = {} backup_file = BackupFile(backup + '.info') restored_files = set() for entry in os.scandir(path): if entry.is_file() and entry.name.endswith('.info'): info = BackupFile(entry.path) info_files[info.uuid] = info restored_files.update(restore_backup(backup_file, info_files, options)) logger.info(f"finished; restored {len(restored_files)} backup files") def restore_backup(bfile, parents, options): if bfile.is_restored: return if bfile.parent: parent = parents.get(bfile.parent) if not parent: msg = (f"missing parent {bfile.parent} for" f"'{os.path.basename(bfile.info_file)}'") if options.ignore_missing: logger.warning(msg) else: raise Exception(msg) else: yield from restore_backup(parent, parents, options) bfile.restore_file(options) yield bfile.uuid def main(): parser = argparse.ArgumentParser(description="restore btrbk raw backup") parser.add_argument('backup', help="backup file to restore; for incremental" " backups the parent files must be in the same directory") parser.add_argument('restore_dir', help="target directory for restored subvolumes" " (path argument for \"btrfs receive\")") parser.add_argument('-n', '--dry-run', action='store_true', help="print commmands that would be executed") parser.add_argument('--ignore-missing', action='store_true', help="do not fail on missing parent snapshots") for transformer in TRANSFORMERS: transformer.add_parser_options(parser) args = parser.parse_args() if args.dry_run: logger.setLevel('ERROR') restore_from_path(args.backup, args) if __name__ == '__main__': logger.setLevel('INFO') logging.basicConfig(format='%(asctime)s %(levelname)s - %(message)s') main() btrbk-0.32.5/doc/000077500000000000000000000000001432521455000134535ustar00rootroot00000000000000btrbk-0.32.5/doc/FAQ.md000066400000000000000000000235761432521455000144210ustar00rootroot00000000000000btrbk FAQ ========= How can I auto-mount btrfs filesystems used by btrbk? ----------------------------------------------------- Given that the "volume" lines in the btrbk configuration file are valid mount-points, you can loop through the configuration and mount the volumes like this: #!/bin/sh btrbk list volume --format=raw | while read line; do eval $line $volume_rsh mount $volume_path done Note that the `btrbk list` command accepts filters (see [btrbk(1)], FILTER STATEMENTS), which means you can e.g. add "group automount" tags in your configuration and dump only the volumes of this group: `btrbk list volume automount`. [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html How can I setup a debian pre-install hook? ------------------------------------------ Create a file `/etc/apt/apt.conf.d/70btrbk`, e.g.: // create a btrfs snapshot before (un)installing packages Dpkg::Pre-Invoke {"/usr/bin/btrbk run /mnt/btr_pool/rootfs";}; In order to make sure that the snapshots are always generated and nothing is deleted, add the btrbk command line options `--preserve --override=snapshot_create=always`. Why is "subvolume ." configuration not recommended? --------------------------------------------------- Referring to a btrbk configuration like this: volume / subvolume . snapshot_name rootfs Btrbk is designed to operate on the subvolumes *within* a root subvolume. In the config above, the btrbk snapshots would be created *inside* the source subvolume, altering it (from user perspective). From btrfs perspective this is not a problem, as the snapshots are separate subvolumes referring to the source subvolume and mapped into the file system tree below the source subvolume. The recommended way is to split your data into subvolumes, e.g.: # btrfs subvolume create /mnt/data/www # btrfs subvolume create /mnt/data/mysql # btrfs subvolume create /mnt/data/projectx This way you make full advantage of the btrfs filesystem, as all your data now has a name, which helps organizing things a lot. This gets even more important as soon as you start snapshotting and send/receiving. The btrbk configuration for this would be: volume /mnt/data subvolume www [...] subvolume mysql [...] subvolume projectx [...] How should I organize my btrfs filesystem? ------------------------------------------ There's lots of ways to do this, and each one of them has its reason to exist. Make sure to read the [btrfs SysadminGuide on kernel.org](https://btrfs.wiki.kernel.org/index.php/SysadminGuide) as a good entry point. ### btrfs root If your linux root filesystem is btrfs, I recommend booting linux from a btrfs subvolume, and use the btrfs root only as a container for subvolumes (i.e. NOT booting from "subvolid=5"). This has the big advantage that you can choose the subvolume in which to boot by simply switching the `rootflags=subvol=` kernel boot option. Example (/boot/grub/grub.cfg): menuentry 'Linux' { linux /boot/vmlinuz root=/dev/sdb3 ro rootflags=subvol=rootfs quiet } menuentry 'Linux (testing)' { linux /boot/vmlinuz root=/dev/sdb3 ro rootflags=subvol=rootfs_testing } Note that btrbk snapshots and backups are read-only, this means you have to create a run-time (rw) snapshot before booting into it: # btrfs subvolume snapshot /mnt/btr_pool/backup/btrbk/rootfs-20150101 /mnt/btr_pool/rootfs_testing How do I convert '/' (subvolid=5) into a subvolume? --------------------------------------------------- There's several ways to achieve this, the solution described below guarantees not to create new files (extents) on disk. ### Step 1: make a snapshot of your root filesystem Assuming that '/' is mounted with `subvolid=5`: # btrfs subvolume snapshot / /rootfs Note that this command does NOT make any physical copy of the files of your subvolumes within "/", it will only add some metadata. ### Step 2: (optional) add the toplevel subvolume to fstab Add mount point for subvolid=5 to fstab, something like this: /rootfs/etc/fstab: /dev/sda1 /mnt/btr_pool btrfs subvolid=5,noatime 0 0 > This step is not critical for a proper root change, but will save > your time by preventing further configurations/reboots and manually > mounting the toplevel subvolume. ### Step 3: boot from the new subvolume "rootfs". Either add `rootflags=subvol=rootfs` to grub.cfg, or set subvolume "rootfs" as default: # btrfs subvolume set-default / You can obtain `` via `btrfs subvolume show /rootfs | grep "Subvolume ID"` > Editing grub.cfg manually may lead you some troubles if you perform > some actions that will fire `grub-mkconfig`. ### Step 4: after reboot, check if everything went fine: First check your **system log** for btrfs errors: cat /var/log/messages | grep -i btrfs | grep -i error then check if current `/` is our new subvolume: # btrfs subvolume show / Name: rootfs ... Great, this tells us that we just booted into our new snapshot! # btrfs subvolume show /mnt/btr_pool /mnt/btr_pool is toplevel subvolume This means that the root volume (subvolid=5) is correctly mounted. ### Step 5: delete old (duplicate) files Carefully delete all old files from `/mnt/btr_pool`, except "rootfs" and any other subvolumes within "/mnt/btr_pool". In other words, delete any folders that are NOT LISTED by `btrfs subvolume list -a /mnt/btr_pool`: # cd /mnt/btr_pool # mkdir TO_BE_REMOVED # mv bin sbin usr lib var ... TO_BE_REMOVED Then reboot. If everything went fine, remove the directory: # cd /mnt/btr_pool # rm -rf TO_BE_REMOVED What is the most efficient way to clone btrfs storage? ------------------------------------------------------ It is very common (and avisable!) to keep backups on a separate location. In some situations, it is also required to transport the data physically, either to the datacenter or to your safe in the basement. ### Answer 1: Use "btrbk archive" A robust approach is to use external disks as archives (secondary backups), and regularly run "btrbk archive" on them. As a nice side effect, this also detects possible read-errors on your backup targets (Note that a "btrfs scrub" is still more effective for that purpose). See **btrbk archive** command in [btrbk(1)] for more details. ### Answer 2: Use external storage as "stream-fifo" This example uses a USB disk as "stream-fifo" for transferring (cloning) of btrfs subvolumes: 1. For all source subvolumes (in order of generation): `btrfs send /source/subvolX -p PARENT > /usbdisk/streamN` 2. At the target location, restore the streams (in order of generation): `cat /usbdisk/streamN | btrfs receive /target` This approach has the advantage that you don't need to reformat your USB disk. This works fine, but be aware that you may run into trouble if a single stream gets corrupted, making all subsequent streams unusable. ### Warning: Avoid using "dd" on btrfs filesystems! If you use `dd` (e.g. in order to clone a partition), make sure you don't mount the cloned filesystem at the same time as the original one. You will end up having multiple filesystems **sharing identical UUID**, which will break things. If you _really_ want to do this, make sure to run: btrfstune -u /dev/sdaX which changes the UUID of the given device. Note that the btrfs subvolumes still share identical UUID's, but at least the kernel can cope with it (see [this post on stackexchange](https://unix.stackexchange.com/questions/246976/btrfs-subvolume-uuid-clash) ). Btrbk on the other hand relies on subvolume UUID's being *universally unique*, and uses them as hash keys for identifying and caching filesystem and subvolume trees, which leads to undefined behavior if multiple identical UUID's are processed. I'm getting an error: Aborted: "Received UUID" is set ----------------------------------------------------- You probably restored a backup with send-receive, and made it read/write using `btrfs property set`. This is bad, as all snapshots and backups will inherit this identical "Received UUID", which results in all these subvolumes will be treated as "containing same data". To fix this, create a "proper" snapshot: # cd /mnt/btr_pool # mv mysubvolume mysubvolume.broken # btrfs subvolume snapshot mysubvolume.broken mysubvolume Now, `mysubvolume` should have an empty "Received UUID". Note that in order to have a clean environment, you also need to fix all subvolumes (snapshots as well as backups) that you created with the broken subvolume. Check if there are more broken subvolumes: # btrfs subvolume show mysubvolume.broken # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above> # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above> Either delete them (they won't be used for incremental send-receive anyways), or clean them as follows: # btrfs subvolume snapshot listed_ro_subvol listed_ro_subvol.rw # btrfs subvolume delete listed_ro_subvol # btrfs subvolume snapshot -r listed_ro_subvol.rw listed_ro_subvol # btrfs subvolume delete listed_ro_subvol.rw Finally, don't forget to delete the broken source subvolume: # btrfs subvolume delete mysubvolume.broken You should now have a clean environment, and btrbk will not complain any more. I'm getting an error: Aborted: subvolume has no UUID ---------------------------------------------------- If your file system was created with btrfs-progs < 4.16, the btrfs root subvolume (id=5) has no UUID. You can check this by calling: # btrfs subvolume show /mnt/btr_pool / Name: UUID: - [...] Without a UUID, the snapshots would get no parent_uuid, leaving btrbk unable to track parent/child relationships. In this case, btrbk refuses to create snapshots and backups. btrbk-0.32.5/doc/Makefile000066400000000000000000000031421432521455000151130ustar00rootroot00000000000000DOCS = FAQ.md \ upgrade_to_v0.23.0.md MAN_MAN1 = btrbk.1 \ lsbtr.1 \ ssh_filter_btrbk.1 MAN_MAN5 = btrbk.conf.5 PN = btrbk PREFIX ?= /usr DOCDIR = $(PREFIX)/share/doc/$(PN) MAN1DIR = $(PREFIX)/share/man/man1 MAN5DIR = $(PREFIX)/share/man/man5 ifeq ($(COMPRESS), yes) DOCS := $(addsuffix .gz,$(DOCS)) MAN_MAN1 := $(addsuffix .gz,$(MAN_MAN1)) MAN_MAN5 := $(addsuffix .gz,$(MAN_MAN5)) endif # convert using "asciidoctor": ASCIIDOCTOR_MANPAGE = asciidoctor -d manpage -b manpage ASCIIDOCTOR_HTML = asciidoctor -b html5 -d article # reproducible builds: reference date is ":date:" attribute from asciidoc source date_attr = $(shell sed -rn 's/:date:\s*//p' $(1)) source_date_epoch = $(shell date +%s -u -d $(call date_attr,$(1))) all: man man: man1 man5 man1: $(MAN_MAN1) man5: $(MAN_MAN5) install: install-man install-doc install-man: man install -d -m 755 "$(DESTDIR)$(MAN1DIR)" install -d -m 755 "$(DESTDIR)$(MAN5DIR)" install -p -m 644 $(MAN_MAN1) "$(DESTDIR)$(MAN1DIR)" install -p -m 644 $(MAN_MAN5) "$(DESTDIR)$(MAN5DIR)" install-doc: $(DOCS) install -d -m 755 "$(DESTDIR)$(DOCDIR)" install -p -m 644 $(DOCS) "$(DESTDIR)$(DOCDIR)" clean: rm -f *.md.gz *.[15] *.[15].{gz,html} %.gz : % gzip -9 -n -c $< > $@ %.1 : %.1.asciidoc SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_MANPAGE) -o $@ $< %.5 : %.5.asciidoc SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_MANPAGE) -o $@ $< %.html : %.asciidoc SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_HTML) -o $@ $< btrbk-0.32.5/doc/btrbk.1.asciidoc000066400000000000000000000444771432521455000164360ustar00rootroot00000000000000btrbk(1) ======== :date: 2022-10-23 :release-version: 0.32.5 :man manual: Btrbk Manual :man source: Btrbk {release-version} NAME ---- btrbk - backup tool for btrfs subvolumes SYNOPSIS -------- [verse] btrbk [-h|--help] [--version] [-c|--config ] [-n|--dry-run] [--exclude ] [-p|--preserve] [--preserve-snapshots] [--preserve-backups] [-v|--verbose] [-q|--quiet] [-l|--loglevel ] [-t|--table] [-L|--long] [-1|--single-column] [--format ] [--pretty] [-S|--print-schedule] [--progress] [--lockfile ] [--override =] [[--] ...] DESCRIPTION ----------- *btrbk* is a backup tool for btrfs subvolumes, taking advantage of btrfs specific capabilities to create atomic snapshots and transfer them incrementally to a target btrfs filesystem. It is able to perform backups from one source to multiple destinations. For most operations, *btrbk* requires 'root privileges' to run correctly. Alternatively, consider using "btrfs-progs-sudo" or "btrfs-progs-btrbk" backends, both of which allows you to run btrbk as a regular user. Refer to configuration option 'backend' in btrbk.conf(5) for more details. === Snapshots and Backups Snapshots as well as backup subvolumes are created in the form: .[_N] Where '' is identical to the source subvolume name, unless the configuration option 'snapshot_name' is set. '' is a timestamp describing the creation time (local time of the host running btrbk) of the snapshot/backup. The format can be configured using the 'timestamp_format' option, refer to btrbk.conf(5) for details. If multiple snapshots/backups are created on the same date/time, 'N' will be incremented on each snapshot, starting at 1. If a snapshot or backup does not match the naming scheme above (e.g. if it has been renamed manually), btrbk will leave it untouched. Note that in btrfs terminology, a 'snapshot' is a ``subvolume with a given initial content of the original subvolume'' (showing a parent-uuid, see btrfs-subvolume(8)), and they can be read-write (default) or read-only. In btrbk terminology, 'snapshot' means ``read-only btrfs snapshot'', and 'backup' means ``read-only subvolume created with send/receive'' (showing a received-uuid). OPTIONS ------- -h, --help:: Prints the synopsis and a list of the commands. --version:: Prints the btrbk version. -c, --config :: Read the configuration from . -n, --dry-run:: Don't run anything that would alter the filesystem, just show the snapshots and backup subvolumes that would be created/deleted by the *run*, *snapshot*, *resume*, *prune*, *archive* and *clean* commands. Use in conjunction with '-l debug' to see the btrfs commands that would be executed. --exclude :: Exclude configured sections matching ''. See <<_filter_statements,FILTER STATEMENTS>> below. -p, --preserve:: Preserve all snapshots and backups. Skips deletion of any snapshots and backups, even if specified in the configuration file (shortcut for "--preserve-snapshots --preserve-backups"). --preserve-snapshots:: Preserve all snapshots. Skips deletion of any snapshots, even if specified in the configuration file. --preserve-backups:: Preserve all backups. Skips deletion of any backups, even if specified in the configuration file. --wipe:: Ignore configured snapshot retention policy, delete all but the latest snapshots instead. All snapshots needed for incremental backup (latest common) are also preserved. Useful if you are getting low on disk space (ENOSPC). -v, --verbose:: Increase the logging level, see "--loglevel". -q, --quiet:: Quiet operation. If set, btrbk does not print the summary after executing the *run*, *snapshot*, *resume*, *prune*, or *archive* commands. -l, --loglevel :: Set the level of verbosity for the stderr logging. Accepted levels are: error, warn, info, debug, and trace. Default is info. -t, --table:: Print output in table format (shortcut for "--format=table"). -L, --long:: Print output in long table format (shortcut for "--format=long"). -1, --single-column:: Print output as single column (not available for all commands). --format table|long|raw|col:[h:]:: Print output in specified format. If set to "raw", prints space-separated, quoted key=value pairs (machine readable). + If set to "col:", prints only the specified (comma-separated list). Header lines are ommitted if the "h:" modifier is present. Columns prefixed with "-" are collapsed if empty. Columns postfixed with ":RALIGN" are right-aligned. --pretty:: Print table output with lowercase, underlined column headings (instead of single-line uppercase headings). -S, --print-schedule:: Print detailed scheduler information on *run*, *snapshot*, *resume*, *prune* and *archive* commands. Use the '--format' command line option to switch between different output formats. --progress:: Show progress bar on send-receive operation. Requires "mbuffer" command (version >= 20180505) installed on the host running btrbk. --lockfile :: Create lockfile on startup; checks lockfile before running any btrfs commands (using perl "flock"), and exits if the lock is held by another btrbk instance. Overrides configuration option "lockfile". Ignored on dryrun ('-n', '--dry-run'). --override =:: Override a configuration option with . Globally, for ALL contexts. Use with care! COMMANDS -------- === Actions The following commands are used to create snapshots and/or backups. All actions can operate in dry-run mode ('-n', '--dry-run'). Use the '--format' command line option to switch between different output formats. See section RETENTION POLICY in *btrbk.conf*(5) for information on configuring the retention policy. *run* [filter...]:: Perform snapshot and backup operations as specified in the configuration file. If the optional [filter...] arguments are present, snapshots and backups are only performed for the subvolumes/targets matching a filter statement (see <<_filter_statements,FILTER STATEMENTS>> below). + *Step 0: Read Data*;; Read information from the source and target btrfs filesystems in order to perform sanity checks and identify parent/child and received-from relationships. + *Step 1: Create Snapshots*;; If the checks succeed, btrbk creates snapshots for the source subvolumes specified in the configuration file, according to the 'snapshot_create' option. + *Step 2: Create Backups*;; For each specified target, btrbk creates the backups as follows: After comparing the backups to the source snapshots, btrbk transfers all missing snapshots needed to satisfy the configured target retention policy, incrementally from the latest common parent subvolume found. If no common parent subvolume is found (or if the 'incremental' option is set to ``no''), a full (non-incremental) backup is created. + *Step 3: Delete Backups*;; Unless the -p, --preserve or --preserve-backups option is set, backup subvolumes that are not preserved by their configured retention policy will be deleted. Note that the latest snapshot/backup pair are always preserved, regardless of the retention policy. + *Step 4: Delete Snapshots*;; Unless the -p, --preserve or --preserve-snapshots option is set, snapshots that are not preserved by their configured retention policy will be deleted. Note that the latest snapshot (the one created in step 1) as well as the latest snapshot/backup pair are always preserved, regardless of the retention policy. If any target is unreachable or has errors, all snapshots are preserved in order not to break the incremental chain. *dryrun* [filter...]:: Don't run any btrfs commands that would alter the filesystem, just show the snapshots and backup subvolumes that would be created/deleted by the *run* command. Use in conjunction with '-l debug' to see the btrfs commands that would be executed. *snapshot* [filter...]:: Snapshot only: skips backup creation and deletion (steps 2 and 3). Use in conjunction with -p, --preserve (or --preserve-snapshots) if you also want to skip snapshot deletion (step 4). + Note that snapshot deletion is skipped if the target is not accessible, as it is still required in order to determine the latest snapshot/backup pair (which is always preserved, regardless of the retention policy). *resume* [filter...]:: Resume backups: skips snapshot creation (step 1), transfers and deletes snapshots/backups in order to satisfy their configured retention policy. Use in conjunction with -p, --preserve, --preserve-backups, --preserve-snapshots if you want to skip backup and/or snapshot deletion (steps 3, 4). *prune* [filter...]:: Prune snapshots and backups: skips snapshot and backup creation (steps 1, 2), only deletes snapshots and backups in order to satisfy their configured retention policy. Useful for cleaning the disk after changing the retention policy. Use in conjunction with --preserve-backups, --preserve-snapshots if you want to skip backup or snapshot deletion (steps 3, 4). + Note that deletion is skipped if source or target is not accessible, as it is still required in order to determine the latest snapshot/backup pair (which is always preserved, regardless of the retention policy). *archive* [--raw]:: Recursively copy all subvolumes created by btrbk from to directory, optionally rescheduled using 'archive_preserve_*' configuration options. Also creates directory tree on . Useful for creating extra archive copies (clones) from your backup disks. Note that you can continue using btrbk after swapping your backup disk with the archive disk. + If you want to use nested subvolumes on the target filesystem, you need to create them by hand (e.g. by running "btrfs subvolume create /dir"). Check the output of --dry-run if unsure. + Note that this feature needs a *linux kernel >=4.4* to work correctly! + If '--raw' option is set, creates raw targets (experimental, see btrbk.conf(5), TARGET TYPES). *clean* [filter...]:: Delete incomplete (garbled) backups. Incomplete backups can be left behind on network errors or kill signals while a send/receive operation is ongoing, and are identified by the "received_uuid" flag not being set on a target (backup) subvolume. The following table gives a quick overview of the action commands and resulting snapshot creation (S+), backup creation (B+), snapshot deletion (S-), and backup deletion (B-): ifdef::backend-docbook,backend-manpage[] .... Command Option S+ B+ S- B- -------------------------------------------- run x x x x run --preserve x x run --preserve-snapshots x x x run --preserve-backups x x x snapshot x x snapshot --preserve x resume x x x resume --preserve x resume --preserve-snapshots x x resume --preserve-backups x x prune x x prune --preserve-snapshots x prune --preserve-backups x .... endif::backend-docbook,backend-manpage[] ifndef::backend-docbook,backend-manpage[] [cols="2*> below). *list* [filter...]:: Print information defined by in a tabular form. Optionally filtered by [filter...] arguments (see <<_filter_statements,FILTER STATEMENTS>> below). + Available subcommands (default ``all''): + -- ifndef::backend-docbook,backend-manpage[] [horizontal] endif::backend-docbook,backend-manpage[] *all*;; List all snapshots and backups created by btrbk. *snapshots*;; List all snapshots created by btrbk. *backups*;; List all backups (and correlated snapshots) created by btrbk. *latest*;; List most recent common snapshot/backup pair, or most recent snapshot if no common found. *config*;; List configured source/snapshot/target relations. *source*;; List configured source/snapshot relations. *volume*;; List configured volume sections. *target*;; List configured targets. -- + Use the '--format' command line option to switch between different output formats. *usage* [filter...]:: Print filesystem usage information for all source/target volumes, optionally filtered by [filter...] arguments (see <<_filter_statements,FILTER STATEMENTS>> below). Note that the "free" value is an estimate of the amount of data that can still be written to the file system. *origin* :: Print the subvolume origin tree: Shows the parent-child relationships as well as the received-from information. Use the '--format' command line option to switch between different output formats. *diff* :: List the modified files since generation (transid) of subvolume in subvolume . Columns: + ------------ SIZE file was modified for a total of SIZE bytes COUNT file was modified in COUNT generations FLAGS "+" file accessed at offset 0 (at least once) "c" COMPRESS flag is set (at least once) "i" INLINE flag is set (at least once) ------------ *extents* [diff] ... [exclusive ...]:: Print accurate disk space usage and diff based on extent data (FIEMAP ioctl, slow!). + -- Subvolumes following the 'exclusive' keyword are added to a separate set, and additional set-exclusive data is printed at the end of the list. This gives a hint of how much data will be freed if deleting all subvolumes in the set. Example: btrbk extents diff /backup/data.* exclusive /backup/data.2010* The EXCLUSIVE column shows the set-exclusive data of all other listed (!) subvolumes (relative complement of block regions). Provided that all related subvolumes (holding references to extents) are also listed, this amount of disk space would be freed when deleting the subvolume. The DIFF column shows the data added to the previous subvolume (relative complement of block regions). If called with the '--related' option, btrbk also lists all related subvolumes. This is not recommended for backups, as parent-uuid relations break for received subvolumes as soon as an intermediate subvolume is deleted. Note that reading all extents is a disk-intensive task, expect long execution times and high ram usage. Consider setting 'cache_dir'. -- *ls* |...:: List all btrfs subvolumes below . Use the '--format' command line option to switch between different output formats. See lsbtr(1). *config* print|print-all:: Prints the parsed configuration file. FILTER STATEMENTS ----------------- Filter arguments are accepted in form: :: Matches the 'group' configuration option of 'volume', 'subvolume' or 'target' sections. [:]:: Matches the 'hostname' portion from '' of 'volume' or 'target' sections. |:: Matches 'volume', 'subvolume' or 'target' sections by either relative or absolute path (if starting with "/" or "ssh://" or ":/"), accepting wildcard character "*". Relative paths are matched against the end of the pathname. Either: + -- :: Matches 'volume' sections. /:: Matches 'subvolume' sections. //:: Matches 'subvolume' sections defining snapshots with the configured 'snapshot_dir' and 'snapshot_name'. :: Matches 'target' sections. /:: Matches 'target' sections within 'subvolume' sections defining snapshots with the configured 'snapshot_name'. Accepted formats for '' are: ssh://[:]/ : -- Note that for *run* and *snapshot* commands, a filter matching a 'target' configuration section also enables snapshot creation of the surrounding 'subvolume' section. If this is not desired, consider running *snapshot* and *resume* commands separately. Filter statements can match multiple times (e.g. on group as well as host name). In such a case, all matches are processed. FILES ----- +/etc/btrbk.conf+:: +/etc/btrbk/btrbk.conf+:: Default configuration file. The file format and configuration options are described in *btrbk.conf*(5). EXIT STATUS ----------- *btrbk* returns the following error codes: ifndef::backend-docbook,backend-manpage[] [horizontal] endif::backend-docbook,backend-manpage[] 0:: No problems occurred. 1:: Generic error code. 2:: Parse error: when parsing command-line options or configuration file. 3:: Lockfile error: if lockfile is present on startup. 10:: Backup abort: At least one backup task aborted. 255:: Script error. AVAILABILITY ------------ Please refer to the btrbk project page ** for further details. SEE ALSO -------- *btrbk.conf*(5), *btrfs*(8) For more information about btrfs and incremental backups, see the web site at https://btrfs.wiki.kernel.org/index.php/Incremental_Backup AUTHOR ------ Axel Burri btrbk-0.32.5/doc/btrbk.conf.5.asciidoc000066400000000000000000000615511432521455000173560ustar00rootroot00000000000000btrbk.conf(5) ============= :date: 2022-10-23 :release-version: 0.32.5 :man manual: Btrbk Manual :man source: Btrbk {release-version} NAME ---- btrbk.conf - btrbk configuration file SYNOPSIS -------- [verse] /etc/btrbk.conf /etc/btrbk/btrbk.conf DESCRIPTION ----------- The btrbk configuration file specifies which btrfs subvolumes on the filesystem are to be processed, what target subvolumes should be used to create the backups, and where the snapshots should be generated. The retention policy, as well as most other options can be defined either globally or within a section. The options specified always apply to the last section encountered, superseding the values set in upper-level sections. This means that global options must be set before any sections are defined. Blank lines are ignored. A hash character (#) starts a comment extending until end of line. SECTIONS -------- *volume* | (optional):: Absolute path pointing to a btrfs file system containing the source subvolume(s) to be backed up. Usually the mount point of a btrfs filesystem mounted with the 'subvolid=5' option. *subvolume* :: Subvolume to be backed up, relative to the '' of the 'volume' section, or absolute if the 'volume' section is omitted. Accepts wildcard character "*". + -- Note that if this subvolume is btrfs root (id=5), it needs to have a valid UUID, which is not the case for file systems created with btrfs-progs < 4.16. -- *target* [send-receive|raw] |:: Target directory where the backup subvolumes are to be created. The optional target type defaults to ``send-receive'', see <<_target_types,TARGET TYPES>> below for details. + -- Multiple 'target' sections are allowed, in any context: a 'target' defined in 'volume' or global context will be used for all underlying 'subvolume' sections (hint: run "btrbk list" or "btrbk config print" to see the resulting configuration). -- If a '' is specified, btrbk actions (shell commands) are executed remotely via ssh, using the <<_ssh_options,SSH Options>> described below. Accepted formats are: ssh://[:]/ : Where '' is either a host name, an IPv4 address in dotted-decimal form, or an IP literal encapsulated within square brackets (e.g. "[2001:db8::7]"). If you are connecting to virtual machines, consider configuring several 'volume' sections for a '', with distinct '' numbers for each machine. OPTIONS ------- The options described here can be specified in 'global context' as well as 'volume', 'subvolume' and 'target' sections, unless stated otherwise. === Basic Options *timestamp_format* short|long|long-iso:: Timestamp format used as postfix for new snapshot subvolume names. Defaults to ``long''. + -- ifndef::backend-docbook,backend-manpage[] [horizontal] endif::backend-docbook,backend-manpage[] *short*;; +YYYYMMDD[_N]+ (e.g. "20150825", "20150825_1") *long*;; +YYYYMMDDhhmm[_N]+ (e.g. "20150825T1531") *long-iso*;; +YYYYMMDDhhmmss±hhmm[_N]+ (e.g. "20150825T153123+0200") -- + Note that a postfix "_N" is appended to the timestamp if a snapshot or backup already exists with the timestamp of current date/time. + Use ``long-iso'' if you want to make sure that btrbk never creates ambiguous time stamps (which can happen if multiple snapshots are created during a daylight saving time clock change). + Note that using ``long-iso'' has implications on the scheduling, see <<_reference_time,Reference Time>> below. *snapshot_dir* :: Directory in which the btrfs snapshots are created, relative to '' of the 'volume' section, or absolute if the 'volume' section is omitted. Note that btrbk does not automatically create this directory, and the snapshot creation will fail if it is not present. *snapshot_name* :: Base name of the created snapshot (and backup). This option is only valid in the 'subvolume' section. Defaults to ''. *snapshot_create* always|onchange|ondemand|no:: If set to ``always'', snapshots are always created. If set to ``onchange'', snapshots are only created if the last snapshot is not up-to-date, i.e. the source subvolume has changed (more precisely: the btrfs generation has been increased) since the last snapshot was created. If set to ``ondemand'', snapshots are only created if at least one target subvolume is reachable (useful if you are tight on disk space and you only need btrbk for backups to an external disk which is not always connected). If set to ``no'', the snapshots are never created (useful if another instance of btrbk is taking care of snapshot creation). Defaults to ``always''. *incremental* yes|no|strict:: If set, incremental backups are created. If set to ``strict'', non-incremental (initial) backups are never created, and incremental backups are restricted to 'related parents' (by parent-uuid relationship). Defaults to ``yes''. + -- Note that even if the parent-uuid chain is broken, snapshots and backups can still share data (which is especially true for backups created with 'incremental' option enabled), and are perfectly suitable as parents for incremental send-receive operations. But as btrbk can not be certain about this, such operations are disallowed in "incremental strict" mode. -- *noauto* yes|no:: If set, the context is skipped by all btrbk actions unless explicitly enabled by a matching btrbk '' command line argument (e.g. "btrbk run myfilter"). === Grouping Options *group* []...:: Add the current section (volume, subvolume or target) to user-defined groups, which can be used as filter for most btrbk commands (see btrbk(1) FILTER STATEMENTS). This option can be set multiple times within the same context. === Retention Policy Options *preserve_day_of_week* monday|tuesday|...|sunday:: Defines on what day a snapshot/backup is considered to be a "weekly" backup. Weekly, monthly and yearly backups are preserved on this day of week (see <<_retention_policy,RETENTION POLICY>> below). Defaults to ``sunday''. *preserve_hour_of_day* [0..23]:: Defines after what time (in full hours since midnight) a snapshot/backup is considered to be a "daily" backup. Daily, weekly, monthly and yearly backups are preserved on this hour (see <<_retention_policy,RETENTION POLICY>> below). Ignored on snapshots or backups without time information ('timestamp_format short'). Defaults to ``0''. *snapshot_preserve* no|:: Set retention policy for snapshots (see <<_retention_policy,RETENTION POLICY>> below). If set to ``no'', preserve snapshots according to 'snapshot_preserve_min' only. Defaults to ``no''. + -- Note that 'snapshot_preserve' has no effect if 'snapshot_preserve_min' is set to ``all'' (the default). -- *snapshot_preserve_min* all|latest|{h,d,w,m,y}:: Preserve all snapshots for a minimum amount of hours (h), days (d), weeks (w), months (m) or years (y), regardless of how many there are. If set to ``all'', preserve all snapshots forever. If set to ``latest'', preserve latest snapshot. Defaults to ``all''. *target_preserve* no|:: Set retention policy for backups (see <<_retention_policy,RETENTION POLICY>> below). If set to ``no'', preserve backups according to 'target_preserve_min' only. Defaults to ``no''. + -- Note that 'target_preserve' has no effect if 'target_preserve_min' is set to ``all'' (the default). -- *target_preserve_min* all|latest|no|{h,d,w,m,y}:: Preserve all backups for a minimum amount of hours (h), days (d), weeks (w), months (m) or years (y), regardless of how many there are. If set to ``all'', preserve all backups forever. If set to ``latest'', always preserve the latest backup (useful in conjunction with "target_preserve no", if you want to keep the latest backup only). If set to ``no'', only the backups following the 'target_preserve' policy are created. Defaults to ``all''. *archive_preserve* no|:: Set retention policy for archives ("btrbk archive" command), with same semantics as 'target_preserve'. *archive_preserve_min* all|latest|no|{h,d,w,m,y}:: Set retention policy for archives ("btrbk archive" command), with same semantics as 'target_preserve_min'. *archive_exclude* :: Exclude subvolumes matching from archiving. The pattern accepts wildcard character "*", and is matched against the end of the pathname. === SSH Options *ssh_identity* |no:: Absolute path to a ssh identity file (private key). If not set, the ssh default is used (see ssh(1), "-i identity_file"). Note that if the identity key is password protected and no authentication agent is used, btrbk will prompt for user input on every connection attempt. *ssh_user* |no:: Remote username for ssh. Defaults to ``root''. Make sure the remote user is able to run "btrfs" with root privileges (see option 'backend' for details). If set to ``no'', the ssh default is used. *ssh_compression* yes|no:: Enables or disables the compression of ssh connections. Defaults to ``no''. Note that if *stream_compress* is enabled, ssh compression will always be disabled for send/receive operations. *ssh_cipher_spec* default|:: Selects the cipher specification for encrypting the session (comma-separated list of ciphers in order of preference). See the "-c cipher_spec" option in ssh(1) for more information. Defaults to ``default'' (the ciphers specified in ssh_config(5)). === Data Stream Options *stream_compress* |no:: Compress the btrfs send stream before transferring it from/to remote locations. Defaults to ``no''. If enabled, make sure that '' is available on the source and target hosts. Supported '': gzip, pigz, bzip2, pbzip2, xz, lzo, lz4, zstd. *stream_compress_level* default|:: Compression level for the specified ''. Refer to the related man-page for details (usually [1..9], where 1 means fastest compression). Defaults to ``default'' (the default compression level of ''). *stream_compress_long* default|:: Enable long distance matching for the specified ''. Refer to the related man-page for details. Only supported for "zstd". *stream_compress_threads* default|:: Number of threads to use for . Only supported for "pigz", "pbzip2", "zstd" and recent versions of "xz". *stream_compress_adapt* default|:: Enable adaptive compression for . Only supported for "zstd" (version >= 1.3.6). *stream_buffer* |no:: Add a buffer to the btrfs send stream (locally, on uncompressed data), with a maximum size of ''. This can give a speed improvement (measured up to 20%) on both local or remote operations, but also increases system load. A suffix of "k", "m", "g", or "%" can be added to '' to denote kilobytes (*1024), megabytes, gigabytes, or a percentage of total physical memory. Defaults to ``no''. + -- If enabled, make sure that the "mbuffer" command (at least version 20180505) is available on the host running btrbk. As of btrbk-0.29.0, mbuffer(1) is used for both 'rate_limit' and 'stream_buffer' options: mbuffer [-m ] [-r ] Note that mbuffer(1) always reads defaults from "`/etc/mbuffer.rc"` and "`~/.mbuffer.rc`". Leave this option disabled if your main concern is a stable backup process: while recent versions of mbuffer have proven reliable, it is often desirable to keep things simple rather than adding an additional, multi-threaded process to the command pipe. -- *stream_buffer_remote* |no:: Add a buffer on remote hosts (either source or target). Defaults to ``no''. + -- Enable this if you prefer buffering on the remote side, or even on both sides: reasons for this depend on available memory, disk and cpu performance (btrfs send/receive, compression), as well as networking constraints. -- *rate_limit* |no:: Limit the read rate of the btrfs send stream to '' bytes per second (locally, on uncompressed send stream). A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on. Defaults to ``no''. Note that 'rate_limit' implicitly adds a stream buffer (see 'stream_buffer' option above). *rate_limit_remote* |no:: Add rate limit on remote hosts (either source or target). Defaults to ``no''. Note that it usually does not make much sense to enable both 'rate_limit' and 'rate_limit_remote'. === System Options *transaction_log* |no:: If set, all transactions (snapshot create, subvolume send-receive, subvolume delete) as well as abort messages are logged to , in a space-separated table format: "localtime type status target_url source_url parent_url message". *transaction_syslog* |no:: If set, all transactions (as described in 'transaction_log' above) are logged to syslog. The program name used in the messages is "btrbk". Accepted parameters for '': user, mail, daemon, auth, lpr, news, cron, authpriv, local0..local7. *lockfile* |no:: Create lockfile on startup; checks lockfile before running any btrfs commands (using perl "flock"), and exits if the lock is held by another btrbk instance. Ignored on dryrun ('-n', '--dry-run'). See also '--lockfile' command-line option. *backend* :: Backend filesystem utilities to be used for btrfs specific operations. Available backends: + -- *btrfs-progs*:: Default backend, btrfs commands are called as specified in btrfs(8) (e.g. "btrfs subvolume show"). *btrfs-progs-btrbk*:: btrfs commands are separated by a dash instead of a whitespace (e.g. "btrfs-subvolume-show" instead of "btrfs subvolume show"). Useful for setting suid or file capabilities (setcap) on specific btrfs commands, as implemented in . *btrfs-progs-sudo*:: btrfs commands are prefixed with "sudo -n" (e.g. "sudo -n btrfs subvolume show" instead of "btrfs subvolume show"). Make sure to have appropriate (root) permissions for the "btrfs" command groups as well as the "readlink" and "test" commands in /etc/sudoers. *btrfs-progs-doas*:: Similar to btrfs-progs-sudo, using prefix "doas -n". If you want to set this option for local or remote hosts only, set *backend_local* or *backend_remote* (e.g. "backend_remote btrfs-progs-btrbk"). If you want to set this option for regular (non-root) user only, set *backend_local_user*. -- *compat* ...:: Enable compatibility options. Available 'compat-option': + -- *busybox*:: Use busybox compatible commands, at the expense of slight overhead while reading filesystem information. *ignore_receive_errors* _*experimental*_:: Tell btrfs-receive(8) to not terminate on errors by setting "--max-errors=0" option. Print warnings instead. + A known use case for this are target hosts lacking xattr support (e.g. some Synology NAS), while the send-stream contains "lsetxattr" commands. Another case is targets failing to set otime, complaining with "ERROR: attribute 12 requested but not present". + Note that there is *no guarantee that backups created with this option enabled can be restored at all*. If you want to set this option for local or remote hosts only, set *compat_local* or *compat_remote* (e.g. "compat_remote busybox"). -- *cache_dir* :: If set, cache extent maps for the "btrbk extents" command. === Btrfs Specific Options *incremental_prefs* [:]...:: Specify the preferences to determine the best common (correlated) parent and clone sources for incremental backups, by choosing from predefined candidate lists. + -- The 'list-spec' defines from what candidate list the next parent/clone-src should be appended to a result list; 'amount' defines how many (e.g. "sro:1 sro:1" is identical to "sro:2"), or all if omitted. Any candidate which is already in the results is dropped. The resulting list of subvolumes is then used as parameters for the btrfs-send(8) command: the first for "-p ", all others for "-c ". Available 'list-spec' (candidate lists = filtered subsets of correlated subvolumes): *sro*,*srn*:: All from 'snapshot_dir' matching 'snapshot_name', with parent_uuid relationship, sorted by btrbk timestamp (o=older n=newer). *sao*,*san*:: All from 'snapshot_dir' matching 'snapshot_name', sorted by btrbk timestamp (o=older n=newer). *aro*,*arn*:: All from 'incremental_resolve', with parent_uuid relationship, sorted by cgen (o=older n=newer). Defaults to "sro:1 srn:1 sao:1 san:1 aro:1 arn:1". Note that for most operations the default resolves a single parent, as there usually are no newer snapshots, and all "sro:1 sao:1 aro:1" resolve to the same snapshot. Example: "defaults,sao,san,aro,arn" takes the defaults, and adds clone sources for all (!) known candidates on the filesystem. -- *incremental_clones* yes|no:: If enabled, btrbk adds "-c " to the btrfs-send(8) command for all correlated subvolumes resolved by 'incremental_prefs'. If disabled, only "-p " is used. Defaults to ``yes''. *incremental_resolve* mountpoint|directory:: Specifies where to search for the best common parent for incremental backups. If set to ``mountpoint'', use parents in the filesystem tree below the mount point of the snapshot and target directory. If set to ``directory'', use parents strictly below snapshot/target directories. Set this to ``directory'' if you get access problems (when not running btrbk as root). Defaults to ``mountpoint''. *btrfs_commit_delete* yes|no:: If set, wait for the transaction commit at the end of each snapshot or backup deletion (sets '--commit-each' option for "btrfs subvolume delete"). Defaults to ``no''. *snapshot_qgroup_destroy* yes|no _*experimental*_:: {blank} *target_qgroup_destroy* yes|no _*experimental*_:: {blank} *archive_qgroup_destroy* yes|no _*experimental*_:: Whenever a subvolume is deleted, also destroy corresponding default qgroup "+0/+". Only useful if you have enabled btrfs quota support. See also: === Informative Options *warn_unknown_targets* yes|no:: If set, prints a warning if btrbk encounters a target subvolume at a unknown location (i.e. not following btrbk naming scheme, or outside the target directory). Defaults to ``no''. RETENTION POLICY ---------------- Retention policies are defined individually for snapshots, backups and archives (summarized as "backups" in the following text), using a combination of: **_preserve_min* all|latest|no|{h,d,w,m,y}:: Amount of time (duration) in which all backups are preserved. **_preserve* no|:: Schedule (single points in time) for which individual backups are preserved. Note that if "preserve_min" is set to ``all'' (the default), any setting of "preserve" obviously has no effect. The format for '' is: [h] [d] [w] [m] [y] *hourly*:: Defines how many hours back hourly backups should be preserved. The first backup of an hour is considered an hourly backup. *daily*:: Defines how many days back daily backups should be preserved. The first backup of a day (starting at 'preserve_hour_of_day') is considered a daily backup. *weekly*:: Defines how many weeks back weekly backups should be preserved. The first daily backup created at 'preserve_day_of_week' (or the first backup in this week if none was made on the exact day) is considered as a weekly backup. *monthly*:: Defines how many months back monthly backups should be preserved. Every first weekly backup in a month is considered a monthly backup. *yearly*:: Defines for how many years back yearly backups should be preserved. Every first monthly backup in a year is considered a yearly backup. Use an asterisk for ``all'' (e.g. "target_preserve 60d *m" states: "preserve daily backups for 60 days back, and all monthly backups"). Hint: Run btrbk with the '-S', '--print-schedule' option to get a comprehensive output of the scheduler results. === Reference Time The local time on the host running btrbk defines the reference time for all date/time calculations, especially for "beginning of a day", and as a consequence for the first daily, weekly, monthly or yearly backups. The local time on remote hosts (ssh source/target) is never used. Unless "timestamp_format long-iso" is set, daily backups are preserved at "preserve_hour_of_day" (defaults to midnight) of the respective time zone (and not for "00:00 UTC", which would be "14:00" in Honolulu). This becomes relevant for setups with multiple btrbk instances, e.g. many snapshot-only instances (spread around the world), and a fetch-only instance on the backup server. Caveat: * If "timestamp_format long-iso" is set, each btrbk instance on has a different interpretation of "first in day". Make sure to run btrbk with the same time zone on every host, e.g. by setting the TZ environment variable (see tzset(3)). TARGET TYPES ------------ *send-receive*:: Backup to a btrfs filesystem, using "btrfs send/receive". This is the recommended (standard) target type. The '' must be an absolute path and point to a subvolume or directory within a btrfs file system. See btrfs-send(8), btrfs-receive(8). *raw* _*experimental*_:: Backup to a raw (filesystem independent) file from the output of btrfs-send(8), with optional compression and encryption. + -- Note that the target preserve mechanism is currently disabled for incremental raw backups (btrbk does not delete any incremental raw files)! Raw backups consist of two files: the main data file containing the btrfs send stream, and a sidecar file ".info" containing metadata: .[_N].btrfs[.gz|.bz2|.xz][.gpg] .[_N].btrfs[.gz|.bz2|.xz][.gpg].info For 'incremental' backups ("incremental yes"), please note that: * As soon as a single 'incremental' backup file is lost or corrupted, all later incremental backups become invalid, as there is no common parent for the subsequent incremental images anymore. This might be a good compromise for a vacation backup plan, but for the long term make sure that a non-incremental backup is triggered from time to time. * There is currently no support for rotation of incremental backups: if 'incremental' is set, a full backup must be triggered manually from time to time in order to be able to delete old backups. Additional options for raw targets: *raw_target_compress* |no:: Compression algorithm to use for raw backup target. Supported '': gzip, pigz, bzip2, pbzip2, xz, lzo, lz4, zstd. *raw_target_compress_level* default|:: Compression level for the specified . *raw_target_compress_long* default|:: Enable long distance matching for the specified ''. *raw_target_compress_threads* default|:: Number of threads to use for . *raw_target_split* |no:: Split the raw backup file into pieces of size ''. *raw_target_block_size* :: Block size to use for writing the raw backup file. Defaults to ``128K''. *raw_target_encrypt* gpg|openssl_enc|no:: If enabled, encrypt the target raw file using gpg or openssl_enc. Additional options for "raw_target_encrypt gpg": *gpg_keyring* :: Keyring to use for gpg, e.g. "`/etc/btrbk/gpg/pubring.kbx`". *gpg_recipient* ...:: Encrypt for user id '' (email address). Additional options for "raw_target_encrypt openssl_enc" ('very experimental'): *openssl_ciphername*{nbsp}:: Defaults to ``aes-256-cbc''. *openssl_iv_size* |no:: Depends on selected cipher. *openssl_keyfile* |no:: Point to a key file in hex (absolute path). Example key file creation (256bit key): + ------------ # dd if=/dev/urandom bs=1 count=32 \ | od -x -A n \ | tr -d "[:space:]" > /path/to/keyfile ------------ *kdf_backend* |no:: KDF backend to be executed, e.g. "`/usr/share/btrbk/scripts/kdf_pbkdf2.py`". *kdf_keysize* :: Defaults to ``32''. *kdf_keygen* once|each:: Defaults to ``once''. -- AVAILABILITY ------------ Please refer to the btrbk project page ** for further details. SEE ALSO -------- *btrbk*(1) AUTHOR ------ Axel Burri btrbk-0.32.5/doc/install.md000066400000000000000000000021061432521455000154420ustar00rootroot00000000000000Installation ============ Btrbk is a single perl script, and does not require any special installation procedures or libraries. In order to install the btrbk executable along with the documentation and an example configuration file, choose one of the following methods: ### Generic Linux System Download and unpack the latest [btrbk source tarball] and type: sudo make install ### Gentoo Linux btrbk is in portage: emerge app-backup/btrbk ### Debian Based Distros btrbk is in debian stable (utils): https://packages.debian.org/stable/utils/btrbk Packages are also available via NeuroDebian: http://neuro.debian.net/pkgs/btrbk.html ### Fedora Linux btrbk is in the official Fedora repos: https://apps.fedoraproject.org/packages/btrbk sudo dnf install btrbk ### Arch Linux btrbk is in AUR: https://aur.archlinux.org/packages/btrbk/ ### Alpine Linux btrbk is in the community repository apk add btrbk ### Void Linux btrbk is in Void's `current` repository xbps-install -S btrbk [btrbk source tarball]: https://digint.ch/download/btrbk/releases/ btrbk-0.32.5/doc/lsbtr.1.asciidoc000066400000000000000000000052631432521455000164460ustar00rootroot00000000000000lsbtr(1) ======== :date: 2022-10-23 :release-version: 0.32.5 :man manual: Btrbk Manual :man source: Btrbk {release-version} NAME ---- lsbtr - list btrfs subvolumes SYNOPSIS -------- [verse] lsbtr [-h|--help] [--version] [-l|--long] [-u|--uuid] [-1|--single-column] [--raw] [--format ] [-v|--verbose] [-c|--config ] [--override =] [[--] |...] DESCRIPTION ----------- List btrfs subvolumes and their mount points visible by the file system below ''. *lsbtr* is part of *btrbk* (basically a shortcut for "btrbk ls"), and takes some global configuration options from btrbk.conf(5) if present. *lsbtr* requires 'root privileges' to run correctly. Alternatively, consider using "btrfs-progs-sudo" or "btrfs-progs-btrbk" backends, both of which allows you to run lsbtr as a regular user. Refer to configuration option 'backend' in btrbk.conf(5) for more details. OPTIONS ------- -h, --help:: Prints the synopsis and a list of the commands. --version:: Prints the btrbk version. -l, --long:: Print output in long table format (additionally print subvolume path). -u, --uuid:: Print UUID table (parent/received relations). -1, --single-column:: Print path column only (delimited by newline). --raw:: Print space-separated key="value" pairs (machine readable). --format table|long|raw|col:[h:]:: Print output in specified format. If set to "raw", prints space-separated key="value" pairs (machine readable). + If set to "col:", prints only the specified (comma-separated list). Header lines are ommitted if the "h:" modifier is present. Columns prefixed with "-" are collapsed if empty. Columns postfixed with ":RALIGN" are right-aligned. -v, --verbose:: Increase the level of verbosity. -c, --config :: Read the configuration from . --override =:: Override a configuration option with . FILES ----- +/etc/btrbk.conf+:: +/etc/btrbk/btrbk.conf+:: Default configuration file. The file format and configuration options are described in *btrbk.conf*(5). EXIT STATUS ----------- *lsbtr* returns the following error codes: ifndef::backend-docbook,backend-manpage[] [horizontal] endif::backend-docbook,backend-manpage[] 0:: No problems occurred. 1:: Generic error code. 2:: Parse error: when parsing command-line options or configuration file. 255:: Script error. AVAILABILITY ------------ Please refer to the btrbk project page ** for further details. SEE ALSO -------- *btrbk*(1), *btrbk.conf*(5), *btrfs*(8) AUTHOR ------ Axel Burri btrbk-0.32.5/doc/ssh_filter_btrbk.1.asciidoc000066400000000000000000000054201432521455000206410ustar00rootroot00000000000000ssh_filter_btrbk(1) =================== :date: 2022-10-23 :release-version: 0.32.5 :man manual: Btrbk Manual :man source: Btrbk {release-version} NAME ---- ssh_filter_btrbk - ssh command filter script for btrbk SYNOPSIS -------- [verse] ssh_filter_btrbk.sh [-s|--source] [-t|--target] [-d|--delete] [-i|--info] [--snapshot] [--send] [--receive] [-p|--restrict-path ] [-l|--log] [--sudo] DESCRIPTION ----------- *ssh_filter_btrbk.sh* restricts SSH commands to commands used by 'btrbk'. It examines the SSH_ORIGINAL_COMMAND environment variable (set by sshd) and executes it only if it contains commands used by 'btrbk'. The accepted commands are specified by the "--source", "--target", "--delete" and "--info" options. The following commands are always allowed: - "btrfs subvolume show" (not affected by "--restrict-path") - "btrfs subvolume list" (not affected by "--restrict-path") - "readlink" - "test -d" (only if "compat busybox" configuration option is set) - "cat /proc/self/mountinfo" - pipes through "gzip", "pigz", "bzip2", "pbzip2", "xz", "lzop", "lz4", "zstd" (stream_compress) - pipes through "mbuffer" (stream_buffer, rate_limit) Example line in /root/.ssh/authorized_keys on a backup target host: command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup" ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@mydomain.com OPTIONS ------- -s, --source:: Allow commands for backup source: "btrfs subvolume snapshot", "btrfs send". Equivalent to "--snapshot --send". -t, --target:: Allow commands for backup and archive target: "btrfs receive", "mkdir". -d, --delete:: Allow commands for subvolume deletion: "btrfs subvolume delete". This is used for backup source if 'snapshot_preserve_daily' is not set to ``all'', and for backup targets if 'target_preserve_daily' is not set to ``all''. -i, --info:: Allow informative commands: "btrfs subvolume find-new", "btrfs filesystem usage". This is used by btrbk 'info' and 'diff' commands. --snapshot:: Allow btrfs snapshot command: "btrfs subvolume snapshot". --send:: Allow btrfs send command: "btrfs send". --receive:: Allow btrfs receive command: "btrfs receive". -p, --restrict-path :: Restrict commands to . Note that "btrfs subvolume show", "btrfs subvolume list" are NOT affected by this option. -l, --log:: Log ACCEPT and REJECT messages to the system log. --sudo:: Allow btrfs commands to be called via sudo. Enable this if you have "backend btrfs-progs-sudo" in your btrbk configuration file. AVAILABILITY ------------ Please refer to the btrbk project page ** for further details. SEE ALSO -------- *btrbk*(1), *btrbk.conf*(5), *btrfs*(8) AUTHOR ------ Axel Burri btrbk-0.32.5/doc/upgrade_to_v0.23.0.md000066400000000000000000000056601432521455000171230ustar00rootroot00000000000000Upgrading to btrbk-v0.23.0 ========================== In order to keep btrbk simple and intuitive while adding new features, it became inevitable to change the semantics of the "retention policy" related configuration options. What has changed? ----------------- ### Preserve *first* instead of *last* snapshot/backup btrbk used to *always* transfer the latest snapshot to the target location, while considering the *last* snapshot/backup of a day as a daily backup (and also the last weekly as a monthly). This made it very cumbersome when running btrbk in a cron job as well as manually, because the last manually created snapshot was immediately transferred on every run, and used as the daily backup (instead of the one created periodically by the cron job). The new semantics are to consider the *first* (instead of *last*) snapshot of a hour/day/week/month as the one to be preserved, while only transferring the snapshots needed to satisfy the target retention policy. ### Preserve snapshots for a minimum amount of time In order to specify a minimum amount of time in which *all* snapshots should be preserved, the new "snapshot_preserve_min" and "target_preserve_min" configuration options were introduced. This was previously covered by "snapshot_preserve_daily", which caused a lot of confusion among users. Upgrading the configuration file: /etc/btrbk/btrbk.conf ------------------------------------------------------- Please read the description of the "run" command in [btrbk(1)], as well as the "RETENTION POLICY" section in [btrbk.conf(5)] for a detailed description. Make sure to understand the new concept, and run `btrbk --print-schedule dryrun` after updating the configuration. ### Upgrade retention policy If you want the same behaviour as before: # replace this: snapshot_preserve_daily snapshot_preserve_weekly snapshot_preserve_monthly # with: snapshot_preserve_min d snapshot_preserve w m # ... do the same with "target_preserve_*" options But what you probably want is something like: snapshot_preserve_min 5d snapshot_preserve d w m target_preserve_min no target_preserve d w m *y This states: * Keep all snapshots for five days (no matter how many there are) * Transfer only the first snapshot of a day to the target * Keep all "first snapshots of a day" for `` days, etc. ### Upgrade "resume_missing" If you have a line: "resume_missing yes" somwhere in your config, simply remove it. btrbk always resumes missing backups. If you have "resume_missing no", you can imitate this behaviour by setting: target_preserve_min latest target_preserve no This states: "always transfer the latest snapshot to the target". [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html [btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html btrbk-0.32.5/lsbtr000077700000000000000000000000001432521455000150012btrbkustar00rootroot00000000000000btrbk-0.32.5/ssh_filter_btrbk.sh000077500000000000000000000137501432521455000166010ustar00rootroot00000000000000#!/bin/bash set -e set -u export PATH=/sbin:/bin:/usr/sbin:/usr/bin enable_log= restrict_path_list= allow_list= allow_exact_list= allow_rate_limit=1 allow_stream_buffer=1 allow_compress=1 compress_list="gzip|pigz|bzip2|pbzip2|xz|lzop|lz4|zstd" # note that the backslash is NOT a metacharacter in a POSIX bracket expression! option_match='-[a-zA-Z0-9=-]+' # matches short as well as long options file_match_sane='/[0-9a-zA-Z_@+./-]*' # matches file path (equal to $file_match in btrbk < 0.32.0) file_match="/[^']*" # btrbk >= 0.32.0 quotes file arguments: match all but single quote file_arg_match="('${file_match}'|${file_match_sane})" # support btrbk < 0.32.0 log_cmd() { if [[ -n "$enable_log" ]]; then logger -p $1 -t ssh_filter_btrbk.sh "$2 (Name: ${LOGNAME:-}; Remote: ${SSH_CLIENT:-})${3:+: $3}: $SSH_ORIGINAL_COMMAND" fi } allow_cmd() { allow_list="${allow_list}|$1" } allow_exact_cmd() { allow_exact_list="${allow_exact_list}|$1" } reject_and_die() { local reason=$1 log_cmd "auth.err" "btrbk REJECT" "$reason" echo "ERROR: ssh_filter_btrbk.sh: ssh command rejected: $reason: $SSH_ORIGINAL_COMMAND" 1>&2 exit 255 } run_cmd() { log_cmd "auth.info" "btrbk ACCEPT" eval " $SSH_ORIGINAL_COMMAND" } reject_filtered_cmd() { if [[ -n "$restrict_path_list" ]]; then # match any of restrict_path_list, # or any file/directory (matching file_match) below restrict_path path_match="'(${restrict_path_list})(${file_match})?'" path_match_legacy="(${restrict_path_list})(${file_match_sane})?" else # match any absolute file/directory (matching file_match) path_match="'${file_match}'" path_match_legacy="${file_match_sane}" fi # btrbk >= 0.32.0 quotes files, allow both (legacy) path_match="(${path_match}|${path_match_legacy})" if [[ -n "$allow_compress" ]]; then decompress_match="(${compress_list}) -d -c( -[pT][0-9]+)?" compress_match="(${compress_list}) -c( -[0-9])?( -[pT][0-9]+)?" else decompress_match= compress_match= fi # rate_limit_remote and stream_buffer_remote use combined # "mbuffer" as of btrbk-0.29.0 if [[ -n "$allow_stream_buffer" ]] || [[ -n "$allow_rate_limit" ]]; then mbuffer_match="mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?" else mbuffer_match= fi # allow multiple paths (e.g. "btrfs subvolume snapshot ") allow_cmd_match="(${allow_list})( ${option_match})*( ${path_match})+" stream_in_match="(${decompress_match} \| )?(${mbuffer_match} \| )?" stream_out_match="( \| ${mbuffer_match})?( \| ${compress_match}$)?" allow_stream_match="^${stream_in_match}${allow_cmd_match}${stream_out_match}" if [[ $SSH_ORIGINAL_COMMAND =~ $allow_stream_match ]] ; then return 0 fi exact_cmd_match="^(${allow_exact_list})$"; if [[ $SSH_ORIGINAL_COMMAND =~ $exact_cmd_match ]] ; then return 0 fi reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${restrict_path_list//|/\", \"}\")}" } # check for "--sudo" option before processing other options sudo_prefix= for key; do [[ "$key" == "--sudo" ]] && sudo_prefix="sudo -n " [[ "$key" == "--doas" ]] && sudo_prefix="doas -n " done while [[ "$#" -ge 1 ]]; do key="$1" case $key in -l|--log) enable_log=1 ;; --sudo|--doas) # already processed above ;; -p|--restrict-path) restrict_path_list="${restrict_path_list}|${2%/}" # add to list while removing trailing slash shift # past argument ;; -s|--source) allow_cmd "${sudo_prefix}btrfs subvolume snapshot" allow_cmd "${sudo_prefix}btrfs send" ;; -t|--target) allow_cmd "${sudo_prefix}btrfs receive" allow_cmd "${sudo_prefix}mkdir" ;; -c|--compress) # deprecated option, compression is always allowed ;; -d|--delete) allow_cmd "${sudo_prefix}btrfs subvolume delete" ;; -i|--info) allow_cmd "${sudo_prefix}btrfs subvolume find-new" allow_cmd "${sudo_prefix}btrfs filesystem usage" ;; --snapshot) allow_cmd "${sudo_prefix}btrfs subvolume snapshot" ;; --send) allow_cmd "${sudo_prefix}btrfs send" ;; --receive) allow_cmd "${sudo_prefix}btrfs receive" ;; *) echo "ERROR: ssh_filter_btrbk.sh: failed to parse command line option: $key" 1>&2 exit 255 ;; esac shift done # NOTE: subvolume queries are NOT affected by "--restrict-path": # btrbk also calls show/list on the mount point of the subvolume allow_exact_cmd "${sudo_prefix}btrfs subvolume (show|list)( ${option_match})* ${file_arg_match}"; allow_cmd "${sudo_prefix}readlink" # resolve symlink allow_exact_cmd "${sudo_prefix}test -d ${file_arg_match}" # check directory (only for compat=busybox) allow_exact_cmd "cat /proc/self/mountinfo" # resolve mountpoints allow_exact_cmd "cat /proc/self/mounts" # legacy, for btrbk < 0.27.0 # remove leading "|" on alternation lists allow_list=${allow_list#\|} allow_exact_list=${allow_exact_list#\|} restrict_path_list=${restrict_path_list#\|} case "$SSH_ORIGINAL_COMMAND" in *\.\./*) reject_and_die 'directory traversal' ;; *\$*) reject_and_die 'unsafe character "$"' ;; *\&*) reject_and_die 'unsafe character "&"' ;; *\(*) reject_and_die 'unsafe character "("' ;; *\{*) reject_and_die 'unsafe character "{"' ;; *\;*) reject_and_die 'unsafe character ";"' ;; *\<*) reject_and_die 'unsafe character "<"' ;; *\>*) reject_and_die 'unsafe character ">"' ;; *\`*) reject_and_die 'unsafe character "`"' ;; *\|*) [[ -n "$allow_compress" ]] || [[ -n "$allow_rate_limit" ]] || [[ -n "$allow_stream_buffer" ]] || reject_and_die 'unsafe character "|"' ;; esac reject_filtered_cmd run_cmd