pax_global_header00006660000000000000000000000064141760432260014517gustar00rootroot0000000000000052 comment=95c9736bf95ad1b457e531cd1dc1664ed2667378 plocate-1.1.15/000077500000000000000000000000001417604322600132335ustar00rootroot00000000000000plocate-1.1.15/.clang-format000066400000000000000000000041271417604322600156120ustar00rootroot00000000000000AlignAfterOpenBracket: Align AlignConsecutiveAssignments: false AlignConsecutiveDeclarations: false AlignEscapedNewlines: DontAlign AlignOperands: false AlignTrailingComments: false AllowAllParametersOfDeclarationOnNextLine: true AllowShortBlocksOnASingleLine: false AllowShortCaseLabelsOnASingleLine: false AllowShortFunctionsOnASingleLine: Inline AllowShortIfStatementsOnASingleLine: false AllowShortLoopsOnASingleLine: false AlwaysBreakAfterReturnType: None AlwaysBreakBeforeMultilineStrings: false AlwaysBreakTemplateDeclarations: true BinPackArguments: true BinPackParameters: true BreakBeforeBinaryOperators: None BreakBeforeBraces: WebKit BreakBeforeTernaryOperators: true BreakConstructorInitializers: BeforeColon #BreakInheritanceList: BeforeColon BreakStringLiterals: false CompactNamespaces: false ConstructorInitializerAllOnOneLineOrOnePerLine: false Cpp11BracedListStyle: false DerivePointerAlignment: false FixNamespaceComments: false IncludeBlocks: Regroup IndentCaseLabels: false IndentPPDirectives: None IndentWrappedFunctionNames: false KeepEmptyLinesAtTheStartOfBlocks: false Language: Cpp MaxEmptyLinesToKeep: 1 NamespaceIndentation: None PointerAlignment: Right ReflowComments: true SortIncludes: true SortUsingDeclarations: true SpaceAfterCStyleCast: false SpaceAfterTemplateKeyword: false SpaceBeforeAssignmentOperators: true #SpaceBeforeCpp11BracedList: false SpaceBeforeCtorInitializerColon: true SpaceBeforeInheritanceColon: true SpaceBeforeParens: ControlStatements SpaceBeforeRangeBasedForLoopColon: true SpaceInEmptyParentheses: false SpacesBeforeTrailingComments: 2 SpacesInAngles: false SpacesInCStyleCastParentheses: false SpacesInContainerLiterals: false SpacesInParentheses: false SpacesInSquareBrackets: false Standard: Cpp11 # HACK: TabWidth 200 makes sure we use tabs for single-tab indentation only, # never for alignment. This makes the code readable at any tab width. # It works only because we have ColumnLimit 0. ColumnLimit: 0 UseTab: ForContinuationAndIndentation AccessModifierOffset: -200 ConstructorInitializerIndentWidth: 200 ContinuationIndentWidth: 200 IndentWidth: 200 TabWidth: 200 plocate-1.1.15/.gitignore000066400000000000000000000000321417604322600152160ustar00rootroot00000000000000*.o plocate plocate-build plocate-1.1.15/.gitmodules000066400000000000000000000000001417604322600153760ustar00rootroot00000000000000plocate-1.1.15/COPYING000066400000000000000000000432541417604322600142760ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. plocate-1.1.15/NEWS000066400000000000000000000120661417604322600137370ustar00rootroot00000000000000plocate 1.1.15, January 31st, 2022 - Various bugfixes. plocate 1.1.14, December 27th, 2021 - Add an option --add-single-prunepath. This makes it possible to prune a path with spaces in the file name, which was previously impossible. - Documentation updates. plocate 1.1.13, November 6th, 2021 - Various bugfixes. plocate 1.1.12, October 5th, 2021 - Implement the -N/--literal option. This matches the GNU coreutils option to turn off the automatic quoting of special characters when printing to a tty. - Various bugfixes, including from Arsen Arsenović. plocate 1.1.11, September 5th, 2021 - Fix a crash with --existing if scanning all blocks (ie., very short patterns, or --regex), by turning off asynchronous stat() in this case. Reported by Michael Arndt. plocate 1.1.10, August 23rd, 2021 - Various bugfixes, including from the Void Linux repository and from Arsen Arsenović. plocate 1.1.9, August 18th, 2021 - Support the -e (--existing) option from mlocate. - Make it possible to set systemunitdir explicitly. Based on a patch by Zbigniew Jędrzejewski-Szmek. plocate 1.1.8, June 12th, 2021 - Various bugfixes. plocate 1.1.7, April 7th, 2021 - Correct randomization of updatedb systemd unit start time. Patch by Vladimir Lomov. plocate 1.1.6, April 2nd, 2021 - Support searching multiple plocate databases, including the LOCATE_PATH environment variable. See the plocate(1) man page for more information. - Fix an issue where updatedb would not recurse into directories on certain filesystems, in particular the deprecated XFS V4. - Randomize updatedb systemd unit start time. Suggested by Calum McConnell. plocate 1.1.5, February 24th, 2021 - Various bugfixes. plocate 1.1.4, February 14th, 2021 - updatedb now uses ~15% less CPU time. - Installs a file CACHEDIR.tag into /var/lib/plocate, to mark the directory as autogenerated. Suggested by Marco d'Itri. - Manpage fixes; patch by Jakub Wilk. plocate 1.1.3, December 19th, 2020 - Various portability fixes. plocate 1.1.2, December 10th, 2020 - Various bugfixes. plocate 1.1.1, December 5th, 2020 - Fix an issue where the database could be built with the wrong check_visibility flag. plocate 1.1.0, December 2nd, 2020 - Include a native updatedb, based on code and man pages from mlocate's updatedb (but heavily tweaked and reworked). This removes the dependency on mlocate's database, at the expense of 1% larger plocate database. plocate-build does not write the needed timestamps, so the first upgrade from mlocate to native plocate requires a full rescan. Subsequent rescans will be about as fast as updatedb.mlocate + plocate-build. plocate-build is now deprecated. - The path name now defaults to /var/lib/plocate, and the group name now defaults to plocate. Both can be changed at configure time. - Databases are now written near-atomically. An aborted updatedb or plocate-build will never leave a stray file. - Escape filenames with backticks in them. plocate 1.0.7, October 31st, 2020 - Fix an infinite loop when encountering invalid UTF-8 in file names. Bug report and patch by Leah Neukirchen. - Typo fixes in man pages; patch by Leah Neukirchen. - Portability fixes. plocate 1.0.6, October 30th, 2020 - Escape unprintable characters, quotes, newlines and the likes when outputting file names to a terminal, similar to GNU ls. This is to reduce the security impact of malicious filenames (e.g. containing color or more nefarious escape codes). It is not active when using -0, or when piping to another program. - Support building databases from plaintext files (plocate-build -p). - Portability fixes. plocate 1.0.5, October 17th, 2020 - Implement the -b/--basename option. - Various bugfixes. plocate 1.0.4, October 16th, 2020 - Fix a bug where plocate.db would be evicted from the OS cache (the behavior was only meant for benchmarking). - On fallback linear scans, such as regexes or very short patterns, use multithreading to speed up the search. plocate 1.0.3, October 15th, 2020 - Portability fixes; plocate will now compile and run on non-x86 architectures (although without SIMD optimizations). Note that plocate.db is not portable across endians, but this shouldn't be a big problem in practice. - Filenames are now compressed using a shared zstd dictionary, which makes plocate.db ~7% smaller, and linear scans ~20% faster. This makes plocate-build ~20% slower, though, since it needs to scan through mlocate.db in a special pre-pass to sample filenames for the dictionary. This changes the dictionary format (from version 0 to version 1), but the new plocate can still read version 0 dictionaries, so the transition should be seamless. - Various bugfixes. plocate 1.0.2, October 12th, 2020 - Various bugfixes. plocate 1.0.1, October 11th, 2020 - Unbreak linear scanning (for very short patterns, or regexes). plocate 1.0.0, October 11th, 2020 - Initial release. plocate-1.1.15/README000066400000000000000000000035721417604322600141220ustar00rootroot00000000000000plocate is a locate based on posting lists. Compared to mlocate, it is much faster, and its index is much smaller. updatedb speed is similar (or you can convert mlocate's index to plocate format using plocate-build). It supports most mlocate options; see --help or the man page (man -l plocate.1) for more information. The file format may still change (if so, plocate will notify you itself that you need to rerun plocate-build or updatedb). To build and install, you will need a C++17 compiler and a development package for Zstd (https://facebook.github.io/zstd/). liburing (https://github.com/axboe/liburing) and a kernel supporting io_uring (Linux 5.1 or newer) is optional, but strongly recommended for best performance, especially if you do not have an SSD. Installation is run as: meson obj cd obj ninja sudo addgroup --system plocate sudo ninja install sudo systemctl enable plocate-updatedb.timer To build the database for the first time after install, you can do sudo updatedb The initial run of updatedb needs to scan the entire filesystem. Subsequent runs of updatedb will be much faster, and create a slightly (~5%) smaller database. If you wish to run some tests of the TurboPFor implementation against the reference implementation, you can check it out and run as follows: git clone https://github.com/powturbo/TurboPFor-Integer-Compression ( cd TurboPFor-Integer-Compression && make -j8 ) cd obj ninja reconfigure ninja bench plocate (except updatedb), and the plocate-specific changes to updatedb, is Copyright 2020 Steinar H. Gunderson . Licensed under the GNU General Public License, either version 2, or (at your option) any later version. See the included file COPYING. updatedb is Copyright (C) 2005, 2007 Red Hat, Inc. All rights reserved. Licensed under the GNU General Public License, version 2. See the included file COPYING. plocate-1.1.15/access_rx_cache.cpp000066400000000000000000000045341417604322600170420ustar00rootroot00000000000000#include "access_rx_cache.h" #include "io_uring_engine.h" #include #include #include #include using namespace std; void AccessRXCache::check_access(const char *filename, bool allow_async, function cb) { if (!check_visibility) { cb(true); return; } lock_guard lock(mu); if (engine == nullptr || !engine->get_supports_stat()) { allow_async = false; } for (const char *end = strchr(filename + 1, '/'); end != nullptr; end = strchr(end + 1, '/')) { string parent_path(filename, end - filename); // string_view from C++20. auto cache_it = cache.find(parent_path); if (cache_it != cache.end()) { // Found in the cache. if (!cache_it->second) { cb(false); return; } continue; } if (!allow_async) { bool ok = access(parent_path.c_str(), R_OK | X_OK) == 0; cache.emplace(parent_path, ok); if (!ok) { cb(false); return; } continue; } // We want to call access(), but it could block on I/O. io_uring doesn't support // access(), but we can do a dummy asynchonous statx() to populate the kernel's cache, // which nearly always makes the next access() instantaneous. // See if there's already a pending stat that matches this, // or is a subdirectory. auto it = pending_stats.lower_bound(parent_path); if (it != pending_stats.end() && it->first.size() >= parent_path.size() && it->first.compare(0, parent_path.size(), parent_path) == 0) { it->second.emplace_back(PendingStat{ filename, move(cb) }); } else { it = pending_stats.emplace(filename, vector{}).first; engine->submit_stat(filename, [this, it, filename{ strdup(filename) }, cb{ move(cb) }](bool) { // The stat returned, so now do the actual access() calls. // All of them should be in cache, so don't fire off new statx() // calls during that check. check_access(filename, /*allow_async=*/false, move(cb)); free(filename); // Call all others that waited for the same stat() to finish. // They may fire off new stat() calls if needed. vector pending = move(it->second); pending_stats.erase(it); for (PendingStat &ps : pending) { check_access(ps.filename.c_str(), /*allow_async=*/true, move(ps.cb)); } }); } return; // The rest will happen in async context. } // Passed all checks. cb(true); } plocate-1.1.15/access_rx_cache.h000066400000000000000000000013501417604322600165000ustar00rootroot00000000000000#ifndef _ACCESS_RX_CACHE_H #define _ACCESS_RX_CACHE_H 1 #include #include #include #include #include #include class IOUringEngine; class AccessRXCache { public: AccessRXCache(IOUringEngine *engine, bool check_visibility) : engine(engine), check_visibility(check_visibility) {} void check_access(const char *filename, bool allow_async, std::function cb); private: std::unordered_map cache; struct PendingStat { std::string filename; std::function cb; }; std::map> pending_stats; IOUringEngine *engine; std::mutex mu; bool check_visibility; }; #endif // !defined(_ACCESS_RX_CACHE_H) plocate-1.1.15/bench.cpp000066400000000000000000000133401417604322600150170ustar00rootroot00000000000000#include #include #include #include #include #include #define dprintf(...) //#define dprintf(...) fprintf(stderr, __VA_ARGS__); #include "complete_pread.h" #include "db.h" #include "io_uring_engine.h" #include "turbopfor-encode.h" #include "turbopfor.h" #include "vp4.h" using namespace std; using namespace std::chrono; bool use_debug = false; int main(void) { int fd = open("plocate.db", O_RDONLY); if (fd == -1) { perror("plocate.db"); exit(1); } Header hdr; complete_pread(fd, &hdr, sizeof(hdr), /*offset=*/0); unique_ptr ht(new Trigram[hdr.hashtable_size + hdr.extra_ht_slots + 1]); complete_pread(fd, ht.get(), (hdr.hashtable_size + hdr.extra_ht_slots + 1) * sizeof(Trigram), hdr.hash_table_offset_bytes); size_t posting_list_bytes = 0, own_posting_list_bytes = 0, total_elements = 0, most_bytes_pl = 0; uint32_t longest_pl = 0; vector> posting_lists; for (unsigned i = 0; i < hdr.hashtable_size + hdr.extra_ht_slots; ++i) { if (ht[i].num_docids == 0) { continue; } size_t len = ht[i + 1].offset - ht[i].offset; string str; str.resize(len); complete_pread(fd, &str[0], len, ht[i].offset); posting_lists.emplace_back(move(str), ht[i].num_docids); longest_pl = std::max(ht[i].num_docids, longest_pl); most_bytes_pl = std::max(len, most_bytes_pl); posting_list_bytes += len; total_elements += ht[i].num_docids; } ht.reset(); fprintf(stderr, "Read %zu posting lists.\n", posting_lists.size()); string encoded_pl; encoded_pl.resize(longest_pl * 2 + 16384); // Lots of margin. size_t num_decode_errors = 0, num_encode_errors = 0; for (auto &[pl, num_docids] : posting_lists) { //fprintf(stderr, "%zu bytes, %u docids\n", pl.size(), num_docids); vector out1, out2; out1.resize(num_docids + 128); out2.resize(num_docids + 128); unsigned char *pldata = reinterpret_cast(&pl[0]); p4nd1dec128v32(pldata, num_docids, &out1[0]); decode_pfor_delta1_128(pldata, num_docids, /*interleaved=*/true, &out2[0]); for (unsigned i = 0; i < num_docids; ++i) { if (out1[i] != out2[i]) { if (++num_decode_errors < 10) { fprintf(stderr, "Decode error:\n"); for (unsigned j = 0; j < num_docids; ++j) { fprintf(stderr, "%3u: reference=%u ours=%u (diff=%d)\n", j, out1[j], out2[j], out1[j] - out2[j]); } } break; } } // Test encoding, by encoding with out own implementation // and checking that decoding with the reference gives // the same result. We do not measure performance (we're slow). uint32_t deltas[128]; unsigned char *ptr = reinterpret_cast(&encoded_pl[0]); ptr = write_baseval(out1[0], ptr); for (unsigned i = 1; i < num_docids; i += 128) { unsigned num_docids_this_block = std::min(num_docids - i, 128u); for (unsigned j = 0; j < num_docids_this_block; ++j) { deltas[j] = out1[i + j] - out1[i + j - 1] - 1; } bool interleaved = (num_docids_this_block == 128); ptr = encode_pfor_single_block<128>(deltas, num_docids_this_block, interleaved, ptr); } own_posting_list_bytes += ptr - reinterpret_cast(&encoded_pl[0]); pldata = reinterpret_cast(&encoded_pl[0]); p4nd1dec128v32(pldata, num_docids, &out2[0]); for (unsigned i = 0; i < num_docids; ++i) { if (out1[i] != out2[i]) { if (++num_encode_errors < 10) { fprintf(stderr, "Encode error:\n"); for (unsigned j = 0; j < num_docids; ++j) { fprintf(stderr, "%3u: reference=%u ours=%u (diff=%d)\n", j, out1[j], out2[j], out1[j] - out2[j]); } } break; } } } fprintf(stderr, "%zu/%zu posting lists had errors in decoding.\n", num_decode_errors, posting_lists.size()); fprintf(stderr, "%zu/%zu posting lists had errors in encoding.\n", num_encode_errors, posting_lists.size()); // Benchmark. vector dummy; dummy.resize(longest_pl + 128); steady_clock::time_point start = steady_clock::now(); for (auto &[pl, num_docids] : posting_lists) { unsigned char *pldata = reinterpret_cast(&pl[0]); p4nd1dec128v32(pldata, num_docids, &dummy[0]); } steady_clock::time_point end = steady_clock::now(); double reference_sec = duration(end - start).count(); fprintf(stderr, "Decoding with reference implementation: %.1f ms\n", 1e3 * reference_sec); start = steady_clock::now(); for (auto &[pl, num_docids] : posting_lists) { unsigned char *pldata = reinterpret_cast(&pl[0]); decode_pfor_delta1_128(pldata, num_docids, /*interleaved=*/true, &dummy[0]); } end = steady_clock::now(); double own_sec = duration(end - start).count(); fprintf(stderr, "Decoding with own implementation: %.3f ms (%.2f%% speed)\n", 1e3 * own_sec, 100.0 * reference_sec / own_sec); fprintf(stderr, "Size with own implementation: %.1f MB (%.2f%% of reference, %+d bytes)\n", own_posting_list_bytes / 1048576.0, 100.0 * own_posting_list_bytes / posting_list_bytes, int(own_posting_list_bytes) - int(posting_list_bytes)); // Three numbers giving rules of thumb for judging our own implementation: // // - Compressed speed is easy to compare to disk I/O, to see the relative importance // - Uncompressed speed is easy to compare to intersection speeds and memory bandwidth // (also very roughly comparable to the benchmark numbers in the TurboPFor README) // - ns/element gives an absolute measure for plocate (e.g. if we can decompress at // 1 ns/element, a 10k-element posting list goes by in 0.01 ms, which is way beyond // instantaneous in practice). fprintf(stderr, "%.1f MB/sec (compressed), %.1f MB/sec (uncompressed), %.1f ns/element\n", posting_list_bytes / own_sec / 1048576.0, (total_elements * sizeof(uint32_t)) / own_sec / 1048576.0, 1e9 * own_sec / total_elements); } plocate-1.1.15/bind-mount.cpp000066400000000000000000000210751417604322600160200ustar00rootroot00000000000000/* Bind mount detection. Note: if you change this, change tmpwatch as well. Copyright (C) 2005, 2007, 2008, 2012 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #include "bind-mount.h" #include "conf.h" #include "lib.h" #include #include #include #include #include #include #include #include #include #include #include #include #include using namespace std; /* mountinfo handling */ /* A single mountinfo entry */ struct mount { int id, parent_id; unsigned dev_major, dev_minor; string root; string mount_point; string fs_type; string source; }; /* Path to mountinfo */ static const char *mountinfo_path; atomic mountinfo_updated{ false }; multimap, mount> mount_entries; // Keyed by device major/minor. /* Read a line from F. Return a string, or empty string on error. */ static string read_mount_line(FILE *f) { string line; for (;;) { char buf[LINE_MAX]; if (fgets(buf, sizeof(buf), f) == nullptr) { if (feof(f)) break; return ""; } size_t chunk_length = strlen(buf); if (chunk_length > 0 && buf[chunk_length - 1] == '\n') { line.append(buf, chunk_length - 1); break; } line.append(buf, chunk_length); } return line; } /* Parse a space-delimited entry in STR, decode octal escapes, write it to DEST (allocated from mount_string_obstack) if it is not nullptr. Return 0 if OK, -1 on error. */ static int parse_mount_string(string *dest, const char **str) { const char *src = *str; while (*src == ' ' || *src == '\t') { src++; } if (*src == 0) { return -1; } string mount_string; for (;;) { char c = *src; switch (c) { case 0: case ' ': case '\t': goto done; case '\\': if (src[1] >= '0' && src[1] <= '7' && src[2] >= '0' && src[2] <= '7' && src[3] >= '0' && src[3] <= '7') { unsigned v; v = ((src[1] - '0') << 6) | ((src[2] - '0') << 3) | (src[3] - '0'); if (v <= UCHAR_MAX) { mount_string.push_back(v); src += 4; break; } } /* Else fall through */ default: mount_string.push_back(c); src++; } } done: *str = src; if (dest != nullptr) { *dest = move(mount_string); } return 0; } /* Read a single entry from F. Return true if succesful. */ static bool read_mount_entry(FILE *f, mount *me) { string line = read_mount_line(f); if (line.empty()) { return false; } size_t offset; if (sscanf(line.c_str(), "%d %d %u:%u%zn", &me->id, &me->parent_id, &me->dev_major, &me->dev_minor, &offset) != 4) { return false; } const char *ptr = line.c_str() + offset; if (parse_mount_string(&me->root, &ptr) != 0 || parse_mount_string(&me->mount_point, &ptr) != 0 || parse_mount_string(nullptr, &ptr) != 0) { return false; } bool separator_found; do { string option; if (parse_mount_string(&option, &ptr) != 0) { return false; } separator_found = strcmp(option.c_str(), "-") == 0; } while (!separator_found); if (parse_mount_string(&me->fs_type, &ptr) != 0 || parse_mount_string(&me->source, &ptr) != 0 || parse_mount_string(nullptr, &ptr) != 0) { return false; } return true; } /* Read mount information from mountinfo_path, update mount_entries and num_mount_entries. Return 0 if OK, -1 on error. */ static int read_mount_entries(void) { FILE *f = fopen(mountinfo_path, "r"); if (f == nullptr) { return -1; } mount_entries.clear(); mount me; while (read_mount_entry(f, &me)) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, " `%s' (%d on %d) is `%s' of `%s' (%u:%u), type `%s'\n", me.mount_point.c_str(), me.id, me.parent_id, me.root.c_str(), me.source.c_str(), me.dev_major, me.dev_minor, me.fs_type.c_str()); } mount_entries.emplace(make_pair(me.dev_major, me.dev_minor), me); } fclose(f); return 0; } /* Bind mount path list maintenace and top-level interface. */ /* mountinfo_path file descriptor, or -1 */ static int mountinfo_fd; /* Known bind mount paths */ static struct vector bind_mount_paths; /* = { 0, }; */ /* Next bind_mount_paths entry */ static size_t bind_mount_paths_index; /* = 0; */ /* Rebuild bind_mount_paths */ static void rebuild_bind_mount_paths(void) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "Rebuilding bind_mount_paths:\n"); } if (read_mount_entries() != 0) { return; } if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "Matching bind_mount_paths:\n"); } bind_mount_paths.clear(); for (const auto &[dev_id, me] : mount_entries) { const auto &[first, second] = mount_entries.equal_range(make_pair(me.dev_major, me.dev_minor)); for (auto it = first; it != second; ++it) { const mount &other = it->second; if (other.id == me.id) { // Don't compare an element to itself. continue; } // We have two mounts from the same device. Is one a prefix of the other? // If there are two that are equal, prefer the one with lowest ID. if (me.root.size() > other.root.size() && me.root.find(other.root) == 0) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, " => adding `%s' (root `%s' is a child of `%s', mounted on `%s')\n", me.mount_point.c_str(), me.root.c_str(), other.root.c_str(), other.mount_point.c_str()); } bind_mount_paths.push_back(me.mount_point); break; } if (me.root == other.root && me.id > other.id) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, " => adding `%s' (duplicate of mount point `%s')\n", me.mount_point.c_str(), other.mount_point.c_str()); } bind_mount_paths.push_back(me.mount_point); break; } } } if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "...done\n"); } string_list_dir_path_sort(&bind_mount_paths); } /* Return true if PATH is a destination of a bind mount. (Bind mounts "to self" are ignored.) */ bool is_bind_mount(const char *path) { if (mountinfo_updated.exchange(false)) { // Atomic test-and-clear. rebuild_bind_mount_paths(); bind_mount_paths_index = 0; } return string_list_contains_dir_path(&bind_mount_paths, &bind_mount_paths_index, path); } /* Initialize state for is_bind_mount(), to read data from MOUNTINFO. */ void bind_mount_init(const char *mountinfo) { mountinfo_path = mountinfo; mountinfo_fd = open(mountinfo_path, O_RDONLY); if (mountinfo_fd == -1) return; rebuild_bind_mount_paths(); // mlocate re-polls this for each and every directory it wants to check, // for unclear reasons; it's possible that it's worried about a new recursive // bind mount being made while updatedb is running, causing an infinite loop? // Since it's probably for some good reason, we do the same, but we don't // want the barrage of syscalls. It's not synchronous, but the poll signal // isn't either; there's a slight race condition, but one that could only // be exploited by root. // // The thread is forcibly terminated on exit(), so we just let it loop forever. thread poll_thread([&] { for (;;) { struct pollfd pfd; /* Unfortunately (mount --bind $path $path/subdir) would leave st_dev unchanged between $path and $path/subdir, so we must keep reparsing mountinfo_path each time it changes. */ pfd.fd = mountinfo_fd; pfd.events = POLLPRI; if (poll(&pfd, 1, /*timeout=*/-1) == -1) { perror("poll()"); exit(1); } if ((pfd.revents & POLLPRI) != 0) { mountinfo_updated = true; } } }); poll_thread.detach(); } plocate-1.1.15/bind-mount.h000066400000000000000000000024621417604322600154640ustar00rootroot00000000000000/* Bind mount detection. Copyright (C) 2005, 2007, 2008 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #ifndef BIND_MOUNT_H__ #define BIND_MOUNT_H__ /* System mount information file */ #define MOUNTINFO_PATH "/proc/self/mountinfo" /* Return true if PATH is a destination of a bind mount. (Bind mounts "to self" are ignored.) */ extern bool is_bind_mount(const char *path); /* Initialize state for is_bind_mount(), to read data from MOUNTINFO. */ extern void bind_mount_init(const char *mountinfo); #endif plocate-1.1.15/complete_pread.cpp000066400000000000000000000012141417604322600167200ustar00rootroot00000000000000#include #include #include #include bool try_complete_pread(int fd, void *ptr, size_t len, off_t offset) { while (len > 0) { ssize_t ret = pread(fd, ptr, len, offset); if (ret == -1 && errno == EINTR) { continue; } if (ret <= 0) { return false; } ptr = reinterpret_cast(ptr) + ret; len -= ret; offset += ret; } return true; } void complete_pread(int fd, void *ptr, size_t len, off_t offset) { if (!try_complete_pread(fd, ptr, len, offset)) { if (errno == 0) { fprintf(stderr, "pread: Short read (file corrupted?)\n"); } else { perror("pread"); } exit(1); } } plocate-1.1.15/complete_pread.h000066400000000000000000000007031417604322600163670ustar00rootroot00000000000000#ifndef COMPLETE_PREAD_H #define COMPLETE_PREAD_H 1 #include // A wrapper around pread() that retries on short reads and EINTR, // so you never need to call it twice. Always synchronous (no io_uring). bool try_complete_pread(int fd, void *ptr, size_t len, off_t offset); // Same, but exit on failure, so never returns a short read. void complete_pread(int fd, void *ptr, size_t len, off_t offset); #endif // !defined(COMPLETE_PREAD_H) plocate-1.1.15/conf.cpp000066400000000000000000000425711417604322600146750ustar00rootroot00000000000000/* updatedb configuration. Copyright (C) 2005, 2007, 2008 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #include "conf.h" #include "error.h" #include "lib.h" #include #include #include #include #include #include #include #include #include #include using namespace std; /* true if locate(1) should check whether files are visible before reporting them */ bool conf_check_visibility = true; /* Filesystems to skip, converted to uppercase and sorted by name */ vector conf_prunefs; /* Directory names to skip, sorted by name */ vector conf_prunenames; /* Paths to skip, sorted by name using dir_path_cmp () */ vector conf_prunepaths; /* true if bind mounts should be skipped */ bool conf_prune_bind_mounts; /* = false; */ /* true if pruning debug output was requested */ bool conf_debug_pruning; /* = false; */ /* Root of the directory tree to store in the database (canonical) */ char *conf_scan_root; /* = NULL; */ /* Absolute (not necessarily canonical) path to the database */ string conf_output; /* 1 if file names should be written to stdout as they are found */ bool conf_verbose; /* = false; */ /* Configuration representation for the database configuration block */ string conf_block; int conf_block_size = 32; bool use_debug = false; /* Parse a STR, store the parsed boolean value to DEST; return 0 if OK, -1 on error. */ static int parse_bool(bool *dest, const char *str) { if (strcmp(str, "0") == 0 || strcmp(str, "no") == 0) { *dest = false; return 0; } if (strcmp(str, "1") == 0 || strcmp(str, "yes") == 0) { *dest = true; return 0; } return -1; } /* String list handling */ /* Add values from space-separated VAL to VAR and LIST */ static void var_add_values(vector *list, const char *val) { for (;;) { const char *start; while (isspace((unsigned char)*val)) val++; if (*val == 0) break; start = val; do val++; while (*val != 0 && !isspace((unsigned char)*val)); list->emplace_back(start, val - start); } } /* Finish variable LIST, sort its contents, remove duplicates */ static void var_finish(vector *list) { sort(list->begin(), list->end()); auto new_end = unique(list->begin(), list->end()); list->erase(new_end, list->end()); } /* UPDATEDB_CONF parsing */ /* UPDATEDB_CONF (locked) */ static FILE *uc_file; /* Line number at token start; type matches error_at_line () */ static unsigned uc_line; /* Current line number; type matches error_at_line () */ static unsigned uc_current_line; /* Last string returned by uc_lex */ static string uc_lex_buf; /* Token types */ enum { UCT_EOF, UCT_EOL, UCT_IDENTIFIER, UCT_EQUAL, UCT_QUOTED, UCT_OTHER, UCT_PRUNE_BIND_MOUNTS, UCT_PRUNEFS, UCT_PRUNENAMES, UCT_PRUNEPATHS }; /* Return next token from uc_file; for UCT_IDENTIFIER, UCT_QUOTED or keywords, store the data to uc_lex_buf (valid until next call). */ static int uc_lex(void) { int c; uc_lex_buf.clear(); uc_line = uc_current_line; do { c = getc_unlocked(uc_file); if (c == EOF) return UCT_EOF; } while (c != '\n' && isspace((unsigned char)c)); switch (c) { case '#': do { c = getc_unlocked(uc_file); if (c == EOF) return UCT_EOF; } while (c != '\n'); /* Fall through */ case '\n': uc_current_line++; if (uc_current_line == 0) { error_at_line(0, 0, UPDATEDB_CONF, uc_current_line - 1, _("warning: Line number overflow")); error_message_count--; /* Don't count as an error */ } return UCT_EOL; case '=': return UCT_EQUAL; case '"': { while ((c = getc_unlocked(uc_file)) != '"') { if (c == EOF || c == '\n') { error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("missing closing `\"'")); ungetc(c, uc_file); break; } uc_lex_buf.push_back(c); } return UCT_QUOTED; } default: { if (!isalpha((unsigned char)c) && c != '_') return UCT_OTHER; do { uc_lex_buf.push_back(c); c = getc_unlocked(uc_file); } while (c != EOF && (isalnum((unsigned char)c) || c == '_')); ungetc(c, uc_file); if (uc_lex_buf == "PRUNE_BIND_MOUNTS") return UCT_PRUNE_BIND_MOUNTS; if (uc_lex_buf == "PRUNEFS") return UCT_PRUNEFS; if (uc_lex_buf == "PRUNENAMES") return UCT_PRUNENAMES; if (uc_lex_buf == "PRUNEPATHS") return UCT_PRUNEPATHS; return UCT_IDENTIFIER; } } } /* Parse /etc/updatedb.conf. Exit on I/O or syntax error. */ static void parse_updatedb_conf(void) { int old_error_one_per_line; unsigned old_error_message_count; bool had_prune_bind_mounts, had_prunefs, had_prunenames, had_prunepaths; uc_file = fopen(UPDATEDB_CONF, "r"); if (uc_file == NULL) { if (errno != ENOENT) error(EXIT_FAILURE, errno, _("can not open `%s'"), UPDATEDB_CONF); goto err; } flockfile(uc_file); uc_current_line = 1; old_error_message_count = error_message_count; old_error_one_per_line = error_one_per_line; error_one_per_line = 1; had_prune_bind_mounts = false; had_prunefs = false; had_prunenames = false; had_prunepaths = false; for (;;) { bool *had_var; int var_token, token; token = uc_lex(); switch (token) { case UCT_EOF: goto eof; case UCT_EOL: continue; case UCT_PRUNE_BIND_MOUNTS: had_var = &had_prune_bind_mounts; break; case UCT_PRUNEFS: had_var = &had_prunefs; break; case UCT_PRUNENAMES: had_var = &had_prunenames; break; case UCT_PRUNEPATHS: had_var = &had_prunepaths; break; case UCT_IDENTIFIER: error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("unknown variable `%s'"), uc_lex_buf.c_str()); goto skip_to_eol; default: error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("variable name expected")); goto skip_to_eol; } if (*had_var != false) { error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("variable `%s' was already defined"), uc_lex_buf.c_str()); goto skip_to_eol; } *had_var = true; var_token = token; token = uc_lex(); if (token != UCT_EQUAL) { error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("`=' expected after variable name")); goto skip_to_eol; } token = uc_lex(); if (token != UCT_QUOTED) { error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("value in quotes expected after `='")); goto skip_to_eol; } if (var_token == UCT_PRUNE_BIND_MOUNTS) { if (parse_bool(&conf_prune_bind_mounts, uc_lex_buf.c_str()) != 0) { error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("invalid value `%s' of PRUNE_BIND_MOUNTS"), uc_lex_buf.c_str()); goto skip_to_eol; } } else if (var_token == UCT_PRUNEFS) var_add_values(&conf_prunefs, uc_lex_buf.c_str()); else if (var_token == UCT_PRUNENAMES) var_add_values(&conf_prunenames, uc_lex_buf.c_str()); else if (var_token == UCT_PRUNEPATHS) var_add_values(&conf_prunepaths, uc_lex_buf.c_str()); else abort(); token = uc_lex(); if (token != UCT_EOL && token != UCT_EOF) { error_at_line(0, 0, UPDATEDB_CONF, uc_line, _("unexpected data after variable value")); goto skip_to_eol; } /* Fall through */ skip_to_eol: while (token != UCT_EOL) { if (token == UCT_EOF) goto eof; token = uc_lex(); } } eof: if (ferror(uc_file)) error(EXIT_FAILURE, 0, _("I/O error reading `%s'"), UPDATEDB_CONF); error_one_per_line = old_error_one_per_line; funlockfile(uc_file); fclose(uc_file); if (error_message_count != old_error_message_count) exit(EXIT_FAILURE); err:; } /* Command-line argument parsing */ /* Output --help text */ static void help(void) { printf(_("Usage: updatedb [OPTION]...\n" "Update a plocate database.\n" "\n" " -f, --add-prunefs FS omit also FS (space-separated)\n" " -n, --add-prunenames NAMES omit also NAMES (space-separated)\n" " -e, --add-prunepaths PATHS omit also PATHS (space-separated)\n" " --add-single-prunepath PATH omit also PATH\n" " -U, --database-root PATH the subtree to store in " "database (default \"/\")\n" " -h, --help print this help\n" " -o, --output FILE database to update (default\n" " `%s')\n" " -b, --block-size SIZE number of filenames to store\n" " in each block (default 32)\n" " --prune-bind-mounts FLAG omit bind mounts (default " "\"no\")\n" " --prunefs FS filesystems to omit from " "database\n" " --prunenames NAMES directory names to omit from " "database\n" " --prunepaths PATHS paths to omit from database\n" " -l, --require-visibility FLAG check visibility before " "reporting files\n" " (default \"yes\")\n" " -v, --verbose print paths of files as they " "are found\n" " -V, --version print version information\n" "\n" "The configuration defaults to values read from\n" "`%s'.\n"), DBFILE, UPDATEDB_CONF); printf(_("\n" "Report bugs to %s.\n"), PACKAGE_BUGREPORT); } /* Prepend current working directory to PATH; return resulting path */ static string prepend_cwd(const string &path) { const char *res; string buf; buf.resize(BUFSIZ); /* Not PATH_MAX because it is not defined on some platforms. */ do buf.resize(buf.size() * 1.5); while ((res = getcwd(buf.data(), buf.size())) == NULL && errno == ERANGE); if (res == NULL) error(EXIT_FAILURE, errno, _("can not get current working directory")); buf.resize(strlen(buf.data())); return buf + '/' + path; } /* Parse ARGC, ARGV. Exit on error or --help, --version. */ static void parse_arguments(int argc, char *argv[]) { enum { OPT_DEBUG_PRUNING = CHAR_MAX + 1, OPT_ADD_SINGLE_PRUNEPATH = CHAR_MAX + 2 }; static const struct option options[] = { { "add-prunefs", required_argument, NULL, 'f' }, { "add-prunenames", required_argument, NULL, 'n' }, { "add-prunepaths", required_argument, NULL, 'e' }, { "add-single-prunepath", required_argument, NULL, OPT_ADD_SINGLE_PRUNEPATH }, { "database-root", required_argument, NULL, 'U' }, { "debug-pruning", no_argument, NULL, OPT_DEBUG_PRUNING }, { "help", no_argument, NULL, 'h' }, { "output", required_argument, NULL, 'o' }, { "prune-bind-mounts", required_argument, NULL, 'B' }, { "prunefs", required_argument, NULL, 'F' }, { "prunenames", required_argument, NULL, 'N' }, { "prunepaths", required_argument, NULL, 'P' }, { "require-visibility", required_argument, NULL, 'l' }, { "verbose", no_argument, NULL, 'v' }, { "version", no_argument, NULL, 'V' }, { "block-size", required_argument, 0, 'b' }, { "debug", no_argument, 0, 'D' }, // Not documented. { NULL, 0, NULL, 0 } }; bool prunefs_changed, prunenames_changed, prunepaths_changed; bool got_prune_bind_mounts, got_visibility; prunefs_changed = false; prunenames_changed = false; prunepaths_changed = false; got_prune_bind_mounts = false; got_visibility = false; for (;;) { int opt, idx; opt = getopt_long(argc, argv, "U:Ve:f:hl:n:o:vb:D", options, &idx); switch (opt) { case -1: goto options_done; case '?': exit(EXIT_FAILURE); case 'B': if (got_prune_bind_mounts != false) error(EXIT_FAILURE, 0, _("--%s would override earlier command-line argument"), "prune-bind-mounts"); got_prune_bind_mounts = true; if (parse_bool(&conf_prune_bind_mounts, optarg) != 0) error(EXIT_FAILURE, 0, _("invalid value `%s' of --%s"), optarg, "prune-bind-mounts"); break; case 'F': if (prunefs_changed != false) error(EXIT_FAILURE, 0, _("--%s would override earlier command-line argument"), "prunefs"); prunefs_changed = true; conf_prunefs.clear(); var_add_values(&conf_prunefs, optarg); break; case 'N': if (prunenames_changed != false) error(EXIT_FAILURE, 0, _("--%s would override earlier command-line argument"), "prunenames"); prunenames_changed = true; conf_prunenames.clear(); var_add_values(&conf_prunenames, optarg); break; case 'P': if (prunepaths_changed != false) error(EXIT_FAILURE, 0, _("--%s would override earlier command-line argument"), "prunepaths"); prunepaths_changed = true; conf_prunepaths.clear(); var_add_values(&conf_prunepaths, optarg); break; case 'U': if (conf_scan_root != NULL) error(EXIT_FAILURE, 0, _("--%s specified twice"), "database-root"); conf_scan_root = realpath(optarg, nullptr); if (conf_scan_root == NULL) error(EXIT_FAILURE, errno, _("invalid value `%s' of --%s"), optarg, "database-root"); break; case 'V': puts("updatedb (" PACKAGE_NAME ") " PACKAGE_VERSION); puts(_("Copyright (C) 2007 Red Hat, Inc. All rights reserved.\n" "This software is distributed under the GPL v.2.\n" "\n" "This program is provided with NO WARRANTY, to the extent " "permitted by law.")); exit(EXIT_SUCCESS); case 'e': prunepaths_changed = true; var_add_values(&conf_prunepaths, optarg); break; case OPT_ADD_SINGLE_PRUNEPATH: prunepaths_changed = true; conf_prunepaths.push_back(optarg); break; case 'f': prunefs_changed = true; var_add_values(&conf_prunefs, optarg); break; case 'h': help(); exit(EXIT_SUCCESS); case 'l': if (got_visibility != false) error(EXIT_FAILURE, 0, _("--%s specified twice"), "require-visibility"); got_visibility = true; if (parse_bool(&conf_check_visibility, optarg) != 0) error(EXIT_FAILURE, 0, _("invalid value `%s' of --%s"), optarg, "require-visibility"); break; case 'n': prunenames_changed = true; var_add_values(&conf_prunenames, optarg); break; case 'o': if (!conf_output.empty()) error(EXIT_FAILURE, 0, _("--%s specified twice"), "output"); conf_output = optarg; break; case 'v': conf_verbose = true; break; case 'b': conf_block_size = atoi(optarg); break; case 'D': use_debug = true; break; case OPT_DEBUG_PRUNING: conf_debug_pruning = true; break; default: abort(); } } options_done: if (optind != argc) error(EXIT_FAILURE, 0, _("unexpected operand on command line")); if (conf_scan_root == NULL) { static char root[] = "/"; conf_scan_root = root; } if (conf_output.empty()) conf_output = DBFILE; if (conf_output[0] != '/') conf_output = prepend_cwd(conf_output); } /* Conversion of configuration for main code */ /* Store a string list to OBSTACK */ static void gen_conf_block_string_list(string *obstack, const vector *strings) { for (const string &str : *strings) { *obstack += str; *obstack += '\0'; } *obstack += '\0'; } /* Generate conf_block */ static void gen_conf_block(void) { conf_block.clear(); #define CONST(S) conf_block.append(S, sizeof(S)) /* conf_check_visibility value is stored in the header */ CONST("prune_bind_mounts"); /* Add two NUL bytes after the value */ conf_block.append(conf_prune_bind_mounts != false ? "1\0" : "0\0", 3); CONST("prunefs"); gen_conf_block_string_list(&conf_block, &conf_prunefs); CONST("prunenames"); gen_conf_block_string_list(&conf_block, &conf_prunenames); CONST("prunepaths"); gen_conf_block_string_list(&conf_block, &conf_prunepaths); /* scan_root is contained directly in the header */ /* conf_output, conf_verbose are not relevant */ #undef CONST } /* Parse /etc/updatedb.conf and command-line arguments ARGC, ARGV. Exit on error or --help, --version. */ void conf_prepare(int argc, char *argv[]) { parse_updatedb_conf(); parse_arguments(argc, argv); for (string &str : conf_prunefs) { /* Assuming filesystem names are ASCII-only */ for (char &c : str) c = toupper(c); } /* Finish the variable only after converting filesystem names to upper case to avoid keeping duplicates that originally differed in case and to sort them correctly. */ var_finish(&conf_prunefs); var_finish(&conf_prunenames); var_finish(&conf_prunepaths); gen_conf_block(); string_list_dir_path_sort(&conf_prunepaths); if (conf_debug_pruning) { /* This is debuging output, don't mark anything for translation */ fprintf(stderr, "conf_block:\n"); for (char c : conf_block) { if (isascii((unsigned char)c) && isprint((unsigned char)c) && c != '\\') putc(c, stderr); else { fprintf(stderr, "\\%03o", (unsigned)(unsigned char)c); if (c == 0) putc('\n', stderr); } } fprintf(stderr, "\n-----------------------\n"); } } plocate-1.1.15/conf.h000066400000000000000000000043001417604322600143260ustar00rootroot00000000000000/* updatedb configuration. Copyright (C) 2005, 2007, 2008 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #ifndef CONF_H__ #define CONF_H__ #include #include #include /* true if locate(1) should check whether files are visible before reporting them */ extern bool conf_check_visibility; /* Filesystems to skip, converted to uppercase and sorted by name */ extern std::vector conf_prunefs; /* Directory names to skip, sorted by name */ extern std::vector conf_prunenames; /* Paths to skip, sorted by name using dir_path_cmp () */ extern std::vector conf_prunepaths; /* true if bind mounts should be skipped */ extern bool conf_prune_bind_mounts; /* true if pruning debug output was requested */ extern bool conf_debug_pruning; /* Root of the directory tree to store in the database (canonical) */ extern char *conf_scan_root; /* Absolute (not necessarily canonical) path to the database */ extern std::string conf_output; /* true if file names should be written to stdout as they are found */ extern bool conf_verbose; /* Configuration representation for the database configuration block */ extern std::string conf_block; /* Parse /etc/updatedb.conf and command-line arguments ARGC, ARGV. Exit on error or --help, --version. */ extern void conf_prepare(int argc, char *argv[]); extern int conf_block_size; extern bool use_debug; #endif plocate-1.1.15/database-builder.cpp000066400000000000000000000476361417604322600171470ustar00rootroot00000000000000#include "database-builder.h" #include "dprintf.h" #include "turbopfor-encode.h" #include #include #ifdef HAS_ENDIAN_H #include #endif #include #include #include #include #include #include #include #include #include #define P4NENC_BOUND(n) ((n + 127) / 128 + (n + 32) * sizeof(uint32_t)) #define NUM_TRIGRAMS 16777216 using namespace std; using namespace std::chrono; constexpr unsigned num_overflow_slots = 16; string zstd_compress(const string &src, ZSTD_CDict *cdict, string *tempbuf); class PostingListBuilder { public: inline void add_docid(uint32_t docid); inline void add_first_docid(uint32_t docid); void finish(); vector encoded; size_t get_num_docids() const { // Updated only when we flush, so check that we're finished. assert(pending_deltas.empty()); return num_docids; } private: void write_header(uint32_t docid); void append_block(); vector pending_deltas; uint32_t num_docids = 0; // Should be size_t, except the format only supports 2^32 docids per posting list anyway. uint32_t last_docid = -1; }; void PostingListBuilder::add_docid(uint32_t docid) { // Deduplicate against the last inserted value, if any. if (docid == last_docid) { return; } pending_deltas.push_back(docid - last_docid - 1); last_docid = docid; if (pending_deltas.size() == 128) { append_block(); pending_deltas.clear(); num_docids += 128; } } void PostingListBuilder::add_first_docid(uint32_t docid) { write_header(docid); ++num_docids; last_docid = docid; } void PostingListBuilder::finish() { if (pending_deltas.empty()) { return; } assert(!encoded.empty()); // write_header() should already have run. // No interleaving for partial blocks. unsigned char buf[P4NENC_BOUND(128)]; unsigned char *end = encode_pfor_single_block<128>(pending_deltas.data(), pending_deltas.size(), /*interleaved=*/false, buf); encoded.insert(encoded.end(), buf, end); num_docids += pending_deltas.size(); pending_deltas.clear(); } void PostingListBuilder::append_block() { unsigned char buf[P4NENC_BOUND(128)]; assert(pending_deltas.size() == 128); unsigned char *end = encode_pfor_single_block<128>(pending_deltas.data(), 128, /*interleaved=*/true, buf); encoded.insert(encoded.end(), buf, end); } void PostingListBuilder::write_header(uint32_t docid) { unsigned char buf[P4NENC_BOUND(1)]; unsigned char *end = write_baseval(docid, buf); encoded.insert(encoded.end(), buf, end); } void DictionaryBuilder::add_file(string filename, dir_time) { if (keep_current_block) { // Only bother saving the filenames if we're actually keeping the block. if (!current_block.empty()) { current_block.push_back('\0'); } current_block += filename; } if (++num_files_in_block == block_size) { flush_block(); } } void DictionaryBuilder::flush_block() { if (keep_current_block) { if (slot_for_current_block == -1) { lengths.push_back(current_block.size()); sampled_blocks.push_back(move(current_block)); } else { lengths[slot_for_current_block] = current_block.size(); sampled_blocks[slot_for_current_block] = move(current_block); } } current_block.clear(); num_files_in_block = 0; ++block_num; if (block_num < blocks_to_keep) { keep_current_block = true; slot_for_current_block = -1; } else { // Keep every block with equal probability (reservoir sampling). uint64_t idx = uniform_int_distribution(0, block_num)(reservoir_rand); keep_current_block = (idx < blocks_to_keep); slot_for_current_block = idx; } } string DictionaryBuilder::train(size_t buf_size) { string dictionary_buf; sort(sampled_blocks.begin(), sampled_blocks.end()); // Seemingly important for decompression speed. for (const string &block : sampled_blocks) { dictionary_buf += block; } string buf; buf.resize(buf_size); size_t ret = ZDICT_trainFromBuffer(&buf[0], buf_size, dictionary_buf.data(), lengths.data(), lengths.size()); if (ZDICT_isError(ret)) { return ""; } dprintf("Sampled %zu bytes in %zu blocks, built a dictionary of size %zu\n", dictionary_buf.size(), lengths.size(), ret); buf.resize(ret); sampled_blocks.clear(); lengths.clear(); return buf; } class EncodingCorpus : public DatabaseReceiver { public: EncodingCorpus(FILE *outfp, size_t block_size, ZSTD_CDict *cdict, bool store_dir_times); ~EncodingCorpus(); void add_file(std::string filename, dir_time dt) override; void flush_block() override; void finish() override; std::vector filename_blocks; size_t num_files = 0, num_files_in_block = 0, num_blocks = 0; bool seen_trigram(uint32_t trgm) { return invindex[trgm] != nullptr; } size_t num_files_seen() const override { return num_files; } PostingListBuilder &get_pl_builder(uint32_t trgm) { return *invindex[trgm]; } void add_docid(uint32_t trgm, uint32_t docid) { if (invindex[trgm] == nullptr) { invindex[trgm] = new PostingListBuilder; invindex[trgm]->add_first_docid(docid); } else { invindex[trgm]->add_docid(docid); } } size_t num_trigrams() const; std::string get_compressed_dir_times(); private: void compress_dir_times(size_t allowed_slop); std::unique_ptr invindex; FILE *outfp; off_t outfp_pos; // Cheaper than calling ftell(outfp) all the time. std::string current_block; std::string tempbuf; const size_t block_size; const bool store_dir_times; ZSTD_CDict *cdict; ZSTD_CStream *dir_time_ctx = nullptr; std::string dir_times; // Buffer of still-uncompressed data. std::string dir_times_compressed; }; EncodingCorpus::EncodingCorpus(FILE *outfp, size_t block_size, ZSTD_CDict *cdict, bool store_dir_times) : invindex(new PostingListBuilder *[NUM_TRIGRAMS]), outfp(outfp), outfp_pos(ftell(outfp)), block_size(block_size), store_dir_times(store_dir_times), cdict(cdict) { fill(invindex.get(), invindex.get() + NUM_TRIGRAMS, nullptr); if (store_dir_times) { dir_time_ctx = ZSTD_createCStream(); ZSTD_initCStream(dir_time_ctx, /*level=*/6); } } EncodingCorpus::~EncodingCorpus() { for (unsigned i = 0; i < NUM_TRIGRAMS; ++i) { delete invindex[i]; } } void EncodingCorpus::add_file(string filename, dir_time dt) { ++num_files; if (!current_block.empty()) { current_block.push_back('\0'); } current_block += filename; if (++num_files_in_block == block_size) { flush_block(); } if (store_dir_times) { if (dt.sec == -1) { // Not a directory. dir_times.push_back('\0'); } else { dir_times.push_back('\1'); dir_times.append(reinterpret_cast(&dt.sec), sizeof(dt.sec)); dir_times.append(reinterpret_cast(&dt.nsec), sizeof(dt.nsec)); } compress_dir_times(/*allowed_slop=*/4096); } } void EncodingCorpus::compress_dir_times(size_t allowed_slop) { while (dir_times.size() >= allowed_slop) { size_t old_size = dir_times_compressed.size(); dir_times_compressed.resize(old_size + 4096); ZSTD_outBuffer outbuf; outbuf.dst = dir_times_compressed.data() + old_size; outbuf.size = 4096; outbuf.pos = 0; ZSTD_inBuffer inbuf; inbuf.src = dir_times.data(); inbuf.size = dir_times.size(); inbuf.pos = 0; int ret = ZSTD_compressStream(dir_time_ctx, &outbuf, &inbuf); if (ret < 0) { fprintf(stderr, "ZSTD_compressStream() failed\n"); exit(1); } dir_times_compressed.resize(old_size + outbuf.pos); dir_times.erase(dir_times.begin(), dir_times.begin() + inbuf.pos); if (outbuf.pos == 0 && inbuf.pos == 0) { // Nothing happened (not enough data?), try again later. return; } } } void EncodingCorpus::flush_block() { if (current_block.empty()) { return; } uint32_t docid = num_blocks; // Create trigrams. const char *ptr = current_block.c_str(); const char *end = ptr + current_block.size(); while (ptr < end - 3) { // Must be at least one filename left, that's at least three bytes. if (ptr[0] == '\0') { // This filename is zero bytes, so skip it (and the zero terminator). ++ptr; continue; } else if (ptr[1] == '\0') { // This filename is one byte, so skip it (and the zero terminator). ptr += 2; continue; } else if (ptr[2] == '\0') { // This filename is two bytes, so skip it (and the zero terminator). ptr += 3; continue; } for (;;) { // NOTE: Will read one byte past the end of the trigram, but it's OK, // since we always call it from contexts where there's a terminating zero byte. uint32_t trgm; memcpy(&trgm, ptr, sizeof(trgm)); ++ptr; trgm = le32toh(trgm); add_docid(trgm & 0xffffff, docid); if (trgm <= 0xffffff) { // Terminating zero byte, so we're done with this filename. // Skip the remaining two bytes, and the zero terminator. ptr += 3; break; } } } // Compress and add the filename block. filename_blocks.push_back(outfp_pos); string compressed = zstd_compress(current_block, cdict, &tempbuf); if (fwrite(compressed.data(), compressed.size(), 1, outfp) != 1) { perror("fwrite()"); exit(1); } outfp_pos += compressed.size(); current_block.clear(); num_files_in_block = 0; ++num_blocks; } void EncodingCorpus::finish() { flush_block(); } size_t EncodingCorpus::num_trigrams() const { size_t num = 0; for (unsigned trgm = 0; trgm < NUM_TRIGRAMS; ++trgm) { if (invindex[trgm] != nullptr) { ++num; } } return num; } string EncodingCorpus::get_compressed_dir_times() { if (!store_dir_times) { return ""; } compress_dir_times(/*allowed_slop=*/0); assert(dir_times.empty()); for (;;) { size_t old_size = dir_times_compressed.size(); dir_times_compressed.resize(old_size + 4096); ZSTD_outBuffer outbuf; outbuf.dst = dir_times_compressed.data() + old_size; outbuf.size = 4096; outbuf.pos = 0; int ret = ZSTD_endStream(dir_time_ctx, &outbuf); if (ret < 0) { fprintf(stderr, "ZSTD_compressStream() failed\n"); exit(1); } dir_times_compressed.resize(old_size + outbuf.pos); if (ret == 0) { // All done. break; } } return dir_times_compressed; } string zstd_compress(const string &src, ZSTD_CDict *cdict, string *tempbuf) { static ZSTD_CCtx *ctx = nullptr; if (ctx == nullptr) { ctx = ZSTD_createCCtx(); } size_t max_size = ZSTD_compressBound(src.size()); if (tempbuf->size() < max_size) { tempbuf->resize(max_size); } size_t size; if (cdict == nullptr) { size = ZSTD_compressCCtx(ctx, &(*tempbuf)[0], max_size, src.data(), src.size(), /*level=*/6); } else { size = ZSTD_compress_usingCDict(ctx, &(*tempbuf)[0], max_size, src.data(), src.size(), cdict); } return string(tempbuf->data(), size); } bool is_prime(uint32_t x) { if ((x % 2) == 0 || (x % 3) == 0) { return false; } uint32_t limit = ceil(sqrt(x)); for (uint32_t factor = 5; factor <= limit; ++factor) { if ((x % factor) == 0) { return false; } } return true; } uint32_t next_prime(uint32_t x) { if ((x % 2) == 0) { ++x; } while (!is_prime(x)) { x += 2; } return x; } unique_ptr create_hashtable(EncodingCorpus &corpus, const vector &all_trigrams, uint32_t ht_size, uint32_t num_overflow_slots) { unique_ptr ht(new Trigram[ht_size + num_overflow_slots + 1]); // 1 for the sentinel element at the end. for (unsigned i = 0; i < ht_size + num_overflow_slots + 1; ++i) { ht[i].trgm = uint32_t(-1); ht[i].num_docids = 0; ht[i].offset = 0; } for (uint32_t trgm : all_trigrams) { // We don't know offset yet, so set it to zero. Trigram to_insert{ trgm, uint32_t(corpus.get_pl_builder(trgm).get_num_docids()), 0 }; uint32_t bucket = hash_trigram(trgm, ht_size); unsigned distance = 0; while (ht[bucket].num_docids != 0) { // Robin Hood hashing; reduces the longest distance by a lot. unsigned other_distance = bucket - hash_trigram(ht[bucket].trgm, ht_size); if (distance > other_distance) { swap(to_insert, ht[bucket]); distance = other_distance; } ++bucket, ++distance; if (distance > num_overflow_slots) { return nullptr; } } ht[bucket] = to_insert; } return ht; } DatabaseBuilder::DatabaseBuilder(const char *outfile, gid_t owner, int block_size, string dictionary, bool check_visibility) : outfile(outfile), block_size(block_size) { umask(0027); string path = outfile; path.resize(path.find_last_of('/') + 1); if (path.empty()) { path = "."; } int fd = -1; #ifdef O_TMPFILE fd = open(path.c_str(), O_WRONLY | O_TMPFILE, 0640); if (fd == -1 && errno != EOPNOTSUPP && errno != EISDIR) { perror(path.c_str()); exit(1); } #endif if (fd == -1) { temp_filename = string(outfile) + ".XXXXXX"; fd = mkstemp(&temp_filename[0]); if (fd == -1) { perror(temp_filename.c_str()); exit(1); } if (fchmod(fd, 0640) == -1) { perror("fchmod"); exit(1); } } if (owner != (gid_t)-1) { if (fchown(fd, (uid_t)-1, owner) == -1) { perror("fchown"); exit(1); } } outfp = fdopen(fd, "wb"); if (outfp == nullptr) { perror(outfile); exit(1); } // Write the header. memcpy(hdr.magic, "\0plocate", 8); hdr.version = -1; // Mark as broken. hdr.hashtable_size = 0; // Not known yet. hdr.extra_ht_slots = num_overflow_slots; hdr.num_docids = 0; hdr.hash_table_offset_bytes = -1; // We don't know these offsets yet. hdr.max_version = 2; hdr.filename_index_offset_bytes = -1; hdr.zstd_dictionary_length_bytes = -1; hdr.check_visibility = check_visibility; fwrite(&hdr, sizeof(hdr), 1, outfp); if (dictionary.empty()) { hdr.zstd_dictionary_offset_bytes = 0; hdr.zstd_dictionary_length_bytes = 0; } else { hdr.zstd_dictionary_offset_bytes = ftell(outfp); fwrite(dictionary.data(), dictionary.size(), 1, outfp); hdr.zstd_dictionary_length_bytes = dictionary.size(); cdict = ZSTD_createCDict(dictionary.data(), dictionary.size(), /*level=*/6); } hdr.directory_data_length_bytes = 0; hdr.directory_data_offset_bytes = 0; hdr.next_zstd_dictionary_length_bytes = 0; hdr.next_zstd_dictionary_offset_bytes = 0; hdr.conf_block_length_bytes = 0; hdr.conf_block_offset_bytes = 0; } DatabaseReceiver *DatabaseBuilder::start_corpus(bool store_dir_times) { corpus_start = steady_clock::now(); corpus = new EncodingCorpus(outfp, block_size, cdict, store_dir_times); return corpus; } void DatabaseBuilder::set_next_dictionary(std::string next_dictionary) { this->next_dictionary = move(next_dictionary); } void DatabaseBuilder::set_conf_block(std::string conf_block) { this->conf_block = move(conf_block); } void DatabaseBuilder::finish_corpus() { corpus->finish(); hdr.num_docids = corpus->filename_blocks.size(); // Stick an empty block at the end as sentinel. corpus->filename_blocks.push_back(ftell(outfp)); const size_t bytes_for_filenames = corpus->filename_blocks.back() - corpus->filename_blocks.front(); // Write the offsets to the filenames. hdr.filename_index_offset_bytes = ftell(outfp); const size_t bytes_for_filename_index = corpus->filename_blocks.size() * sizeof(uint64_t); fwrite(corpus->filename_blocks.data(), corpus->filename_blocks.size(), sizeof(uint64_t), outfp); corpus->filename_blocks.clear(); corpus->filename_blocks.shrink_to_fit(); // Finish up encoding the posting lists. size_t trigrams = 0, longest_posting_list = 0; size_t bytes_for_posting_lists = 0; for (unsigned trgm = 0; trgm < NUM_TRIGRAMS; ++trgm) { if (!corpus->seen_trigram(trgm)) continue; PostingListBuilder &pl_builder = corpus->get_pl_builder(trgm); pl_builder.finish(); longest_posting_list = max(longest_posting_list, pl_builder.get_num_docids()); trigrams += pl_builder.get_num_docids(); bytes_for_posting_lists += pl_builder.encoded.size(); } size_t num_trigrams = corpus->num_trigrams(); dprintf("%zu files, %zu different trigrams, %zu entries, avg len %.2f, longest %zu\n", corpus->num_files, num_trigrams, trigrams, double(trigrams) / num_trigrams, longest_posting_list); dprintf("%zu bytes used for posting lists (%.2f bits/entry)\n", bytes_for_posting_lists, 8 * bytes_for_posting_lists / double(trigrams)); dprintf("Building posting lists took %.1f ms.\n\n", 1e3 * duration(steady_clock::now() - corpus_start).count()); // Find the used trigrams. vector all_trigrams; for (unsigned trgm = 0; trgm < NUM_TRIGRAMS; ++trgm) { if (corpus->seen_trigram(trgm)) { all_trigrams.push_back(trgm); } } // Create the hash table. unique_ptr hashtable; uint32_t ht_size = next_prime(all_trigrams.size()); for (;;) { hashtable = create_hashtable(*corpus, all_trigrams, ht_size, num_overflow_slots); if (hashtable == nullptr) { dprintf("Failed creating hash table of size %u, increasing by 5%% and trying again.\n", ht_size); ht_size = next_prime(ht_size * 1.05); } else { dprintf("Created hash table of size %u.\n\n", ht_size); break; } } // Find the offsets for each posting list. size_t bytes_for_hashtable = (ht_size + num_overflow_slots + 1) * sizeof(Trigram); uint64_t offset = ftell(outfp) + bytes_for_hashtable; for (unsigned i = 0; i < ht_size + num_overflow_slots + 1; ++i) { hashtable[i].offset = offset; // Needs to be there even for empty slots. if (hashtable[i].num_docids == 0) { continue; } const vector &encoded = corpus->get_pl_builder(hashtable[i].trgm).encoded; offset += encoded.size(); } // Write the hash table. hdr.hash_table_offset_bytes = ftell(outfp); hdr.hashtable_size = ht_size; fwrite(hashtable.get(), ht_size + num_overflow_slots + 1, sizeof(Trigram), outfp); // Write the actual posting lists. for (unsigned i = 0; i < ht_size + num_overflow_slots + 1; ++i) { if (hashtable[i].num_docids == 0) { continue; } const vector &encoded = corpus->get_pl_builder(hashtable[i].trgm).encoded; fwrite(encoded.data(), encoded.size(), 1, outfp); } // Finally, write the directory times (for updatedb). string compressed_dir_times = corpus->get_compressed_dir_times(); size_t bytes_for_compressed_dir_times = 0; if (!compressed_dir_times.empty()) { hdr.directory_data_offset_bytes = ftell(outfp); hdr.directory_data_length_bytes = compressed_dir_times.size(); fwrite(compressed_dir_times.data(), compressed_dir_times.size(), 1, outfp); bytes_for_compressed_dir_times = compressed_dir_times.size(); compressed_dir_times.clear(); } // Write the recommended dictionary for next update. if (!next_dictionary.empty()) { hdr.next_zstd_dictionary_offset_bytes = ftell(outfp); hdr.next_zstd_dictionary_length_bytes = next_dictionary.size(); fwrite(next_dictionary.data(), next_dictionary.size(), 1, outfp); } // And the configuration block. if (!conf_block.empty()) { hdr.conf_block_offset_bytes = ftell(outfp); hdr.conf_block_length_bytes = conf_block.size(); fwrite(conf_block.data(), conf_block.size(), 1, outfp); } // Rewind, and write the updated header. hdr.version = 1; fseek(outfp, 0, SEEK_SET); fwrite(&hdr, sizeof(hdr), 1, outfp); if (!temp_filename.empty()) { if (rename(temp_filename.c_str(), outfile.c_str()) == -1) { perror("rename"); exit(1); } } else { #ifdef O_TMPFILE // Give the file a proper name, making it visible in the file system. // TODO: It would be nice to be able to do this atomically, like with rename. unlink(outfile.c_str()); char procpath[256]; snprintf(procpath, sizeof(procpath), "/proc/self/fd/%d", fileno(outfp)); if (linkat(AT_FDCWD, procpath, AT_FDCWD, outfile.c_str(), AT_SYMLINK_FOLLOW) == -1) { perror("linkat"); exit(1); } #endif } fclose(outfp); size_t total_bytes = (bytes_for_hashtable + bytes_for_posting_lists + bytes_for_filename_index + bytes_for_filenames + bytes_for_compressed_dir_times); dprintf("Block size: %7d files\n", block_size); dprintf("Dictionary: %'7.1f MB\n", hdr.zstd_dictionary_length_bytes / 1048576.0); dprintf("Hash table: %'7.1f MB\n", bytes_for_hashtable / 1048576.0); dprintf("Posting lists: %'7.1f MB\n", bytes_for_posting_lists / 1048576.0); dprintf("Filename index: %'7.1f MB\n", bytes_for_filename_index / 1048576.0); dprintf("Filenames: %'7.1f MB\n", bytes_for_filenames / 1048576.0); if (bytes_for_compressed_dir_times != 0) { dprintf("Modify times: %'7.1f MB\n", bytes_for_compressed_dir_times / 1048576.0); } dprintf("Total: %'7.1f MB\n", total_bytes / 1048576.0); dprintf("\n"); } plocate-1.1.15/database-builder.h000066400000000000000000000045001417604322600165730ustar00rootroot00000000000000#ifndef _DATABASE_BUILDER_H #define _DATABASE_BUILDER_H 1 #include "db.h" #include #include #include #include #include #include #include #include #include #include class PostingListBuilder; // {0,0} means unknown or so current that it should never match. // {-1,0} means it's not a directory. struct dir_time { int64_t sec; int32_t nsec; bool operator<(const dir_time &other) const { if (sec != other.sec) return sec < other.sec; return nsec < other.nsec; } bool operator>=(const dir_time &other) const { return !(other < *this); } }; constexpr dir_time unknown_dir_time{ 0, 0 }; constexpr dir_time not_a_dir{ -1, 0 }; class DatabaseReceiver { public: virtual ~DatabaseReceiver() = default; virtual void add_file(std::string filename, dir_time dt) = 0; virtual void flush_block() = 0; virtual void finish() { flush_block(); } // EncodingCorpus only. virtual size_t num_files_seen() const { return -1; } }; class DictionaryBuilder : public DatabaseReceiver { public: DictionaryBuilder(size_t blocks_to_keep, size_t block_size) : blocks_to_keep(blocks_to_keep), block_size(block_size) {} void add_file(std::string filename, dir_time dt) override; void flush_block() override; std::string train(size_t buf_size); private: const size_t blocks_to_keep, block_size; std::string current_block; uint64_t block_num = 0; size_t num_files_in_block = 0; std::mt19937 reservoir_rand{ 1234 }; // Fixed seed for reproducibility. bool keep_current_block = true; int64_t slot_for_current_block = -1; std::vector sampled_blocks; std::vector lengths; }; class EncodingCorpus; class DatabaseBuilder { public: DatabaseBuilder(const char *outfile, gid_t owner, int block_size, std::string dictionary, bool check_visibility); DatabaseReceiver *start_corpus(bool store_dir_times); void set_next_dictionary(std::string next_dictionary); void set_conf_block(std::string conf_block); void finish_corpus(); private: FILE *outfp; std::string outfile; std::string temp_filename; Header hdr; const int block_size; std::chrono::steady_clock::time_point corpus_start; EncodingCorpus *corpus = nullptr; ZSTD_CDict *cdict = nullptr; std::string next_dictionary, conf_block; }; #endif // !defined(_DATABASE_BUILDER_H) plocate-1.1.15/db.h000066400000000000000000000025751417604322600140020ustar00rootroot00000000000000#ifndef DB_H #define DB_H 1 #include struct Header { char magic[8]; // "\0plocate"; uint32_t version; // 2 is the current version. uint32_t hashtable_size; uint32_t extra_ht_slots; uint32_t num_docids; uint64_t hash_table_offset_bytes; uint64_t filename_index_offset_bytes; // Version 1 and up only. uint32_t max_version; // Nominally 1 or 2, but can be increased if more features are added in a backward-compatible way. uint32_t zstd_dictionary_length_bytes; uint64_t zstd_dictionary_offset_bytes; // Only if max_version >= 2, and only relevant for updatedb. uint64_t directory_data_length_bytes; uint64_t directory_data_offset_bytes; uint64_t next_zstd_dictionary_length_bytes; uint64_t next_zstd_dictionary_offset_bytes; uint64_t conf_block_length_bytes; uint64_t conf_block_offset_bytes; // Only if max_version >= 2. bool check_visibility; }; struct Trigram { uint32_t trgm; uint32_t num_docids; uint64_t offset; bool operator==(const Trigram &other) const { return trgm == other.trgm; } bool operator<(const Trigram &other) const { return trgm < other.trgm; } }; inline uint32_t hash_trigram(uint32_t trgm, uint32_t ht_size) { // CRC-like computation. uint32_t crc = trgm; for (int i = 0; i < 32; i++) { bool bit = crc & 0x80000000; crc <<= 1; if (bit) { crc ^= 0x1edc6f41; } } return crc % ht_size; } #endif // !defined(DB_H) plocate-1.1.15/dprintf.h000066400000000000000000000003651417604322600150560ustar00rootroot00000000000000#ifndef _DPRINTF_H #define _DPRINTF_H 1 #include extern bool use_debug; // Debug printf. #define dprintf(...) \ do { \ if (use_debug) { \ fprintf(stderr, __VA_ARGS__); \ } \ } while (false) #endif // !defined(_DPRINTF_H) plocate-1.1.15/io_uring_engine.cpp000066400000000000000000000157051417604322600171070ustar00rootroot00000000000000#include #include #include #include #include #include #ifndef WITHOUT_URING #include #endif #include "complete_pread.h" #include "dprintf.h" #include "io_uring_engine.h" #include #include #include #include #include #include using namespace std; IOUringEngine::IOUringEngine(size_t slop_bytes) : slop_bytes(slop_bytes) { #ifdef WITHOUT_URING int ret = -1; dprintf("Compiled without liburing support; not using io_uring.\n"); #else int ret = io_uring_queue_init(queue_depth, &ring, 0); if (ret < 0) { dprintf("io_uring_queue_init() failed; not using io_uring.\n"); } #endif using_uring = (ret >= 0); #ifndef WITHOUT_URING if (using_uring) { io_uring_probe *probe = io_uring_get_probe_ring(&ring); supports_stat = (probe != nullptr && io_uring_opcode_supported(probe, IORING_OP_STATX)); if (!supports_stat) { dprintf("io_uring on this kernel does not support statx(); will do synchronous access checking.\n"); } free(probe); } #endif } void IOUringEngine::submit_stat(const char *path [[maybe_unused]], std::function cb [[maybe_unused]]) { assert(supports_stat); #ifndef WITHOUT_URING if (pending_reads < queue_depth) { io_uring_sqe *sqe = io_uring_get_sqe(&ring); if (sqe == nullptr) { fprintf(stderr, "io_uring_get_sqe: %s\n", strerror(errno)); exit(1); } submit_stat_internal(sqe, strdup(path), move(cb)); } else { QueuedStat qs; qs.cb = move(cb); qs.pathname = strdup(path); queued_stats.push(move(qs)); } #endif } void IOUringEngine::submit_read(int fd, size_t len, off_t offset, function cb) { if (!using_uring) { // Synchronous read. string s; s.resize(len + slop_bytes); complete_pread(fd, &s[0], len, offset); cb(string_view(s.data(), len)); return; } #ifndef WITHOUT_URING if (pending_reads < queue_depth) { io_uring_sqe *sqe = io_uring_get_sqe(&ring); if (sqe == nullptr) { fprintf(stderr, "io_uring_get_sqe: %s\n", strerror(errno)); exit(1); } submit_read_internal(sqe, fd, len, offset, move(cb)); } else { queued_reads.push(QueuedRead{ fd, len, offset, move(cb) }); } #endif } #ifndef WITHOUT_URING void IOUringEngine::submit_read_internal(io_uring_sqe *sqe, int fd, size_t len, off_t offset, function cb) { void *buf; if (posix_memalign(&buf, /*alignment=*/4096, len + slop_bytes)) { fprintf(stderr, "Couldn't allocate %zu bytes: %s\n", len, strerror(errno)); exit(1); } PendingRead *pending = new PendingRead; pending->op = OP_READ; pending->read_cb = move(cb); pending->read.buf = buf; pending->read.len = len; pending->read.fd = fd; pending->read.offset = offset; pending->read.iov = iovec{ buf, len }; io_uring_prep_readv(sqe, fd, &pending->read.iov, 1, offset); io_uring_sqe_set_data(sqe, pending); ++pending_reads; } void IOUringEngine::submit_stat_internal(io_uring_sqe *sqe, char *path, std::function cb) { PendingRead *pending = new PendingRead; pending->op = OP_STAT; pending->stat_cb = move(cb); pending->stat.pathname = path; pending->stat.buf = new struct statx; io_uring_prep_statx(sqe, /*fd=*/-1, pending->stat.pathname, AT_STATX_SYNC_AS_STAT | AT_SYMLINK_NOFOLLOW, STATX_MODE, pending->stat.buf); io_uring_sqe_set_data(sqe, pending); ++pending_reads; } #endif void IOUringEngine::finish() { if (!using_uring) { return; } #ifndef WITHOUT_URING bool anything_to_submit = true; while (pending_reads > 0) { io_uring_cqe *cqe; if (io_uring_peek_cqe(&ring, &cqe) != 0) { if (anything_to_submit) { // Nothing ready, so submit whatever is pending and then do a blocking wait. int ret = io_uring_submit_and_wait(&ring, 1); if (ret < 0) { fprintf(stderr, "io_uring_submit(queued): %s\n", strerror(-ret)); exit(1); } anything_to_submit = false; } else { int ret = io_uring_wait_cqe(&ring, &cqe); if (ret < 0) { fprintf(stderr, "io_uring_wait_cqe: %s\n", strerror(-ret)); exit(1); } } } unsigned head; io_uring_for_each_cqe(&ring, head, cqe) { PendingRead *pending = reinterpret_cast(cqe->user_data); if (pending->op == OP_STAT) { io_uring_cqe_seen(&ring, cqe); --pending_reads; size_t old_pending_reads = pending_reads; pending->stat_cb(cqe->res == 0); free(pending->stat.pathname); delete pending->stat.buf; delete pending; if (pending_reads != old_pending_reads) { // A new read was made in the callback (and not queued), // so we need to re-submit. anything_to_submit = true; } } else { if (cqe->res <= 0) { fprintf(stderr, "async read failed: %s\n", strerror(-cqe->res)); exit(1); } if (size_t(cqe->res) < pending->read.iov.iov_len) { // Incomplete read, so resubmit it. pending->read.iov.iov_base = (char *)pending->read.iov.iov_base + cqe->res; pending->read.iov.iov_len -= cqe->res; pending->read.offset += cqe->res; io_uring_cqe_seen(&ring, cqe); io_uring_sqe *sqe = io_uring_get_sqe(&ring); if (sqe == nullptr) { fprintf(stderr, "No free SQE for resubmit; this shouldn't happen.\n"); exit(1); } io_uring_prep_readv(sqe, pending->read.fd, &pending->read.iov, 1, pending->read.offset); io_uring_sqe_set_data(sqe, pending); anything_to_submit = true; } else { io_uring_cqe_seen(&ring, cqe); --pending_reads; size_t old_pending_reads = pending_reads; pending->read_cb(string_view(reinterpret_cast(pending->read.buf), pending->read.len)); free(pending->read.buf); delete pending; if (pending_reads != old_pending_reads) { // A new read was made in the callback (and not queued), // so we need to re-submit. anything_to_submit = true; } } } } // See if there are any queued stats we can submit now. // Running a stat means we're very close to printing out a match, // which is more important than reading more blocks from disk. // (Even if those blocks returned early, they would only generate // more matches that would be blocked by this one in Serializer.) // Thus, prioritize stats. while (!queued_stats.empty() && pending_reads < queue_depth) { io_uring_sqe *sqe = io_uring_get_sqe(&ring); if (sqe == nullptr) { fprintf(stderr, "io_uring_get_sqe: %s\n", strerror(errno)); exit(1); } QueuedStat &qs = queued_stats.front(); submit_stat_internal(sqe, qs.pathname, move(qs.cb)); queued_stats.pop(); anything_to_submit = true; } // See if there are any queued reads we can submit now. while (!queued_reads.empty() && pending_reads < queue_depth) { io_uring_sqe *sqe = io_uring_get_sqe(&ring); if (sqe == nullptr) { fprintf(stderr, "io_uring_get_sqe: %s\n", strerror(errno)); exit(1); } QueuedRead &qr = queued_reads.front(); submit_read_internal(sqe, qr.fd, qr.len, qr.offset, move(qr.cb)); queued_reads.pop(); anything_to_submit = true; } } #endif } plocate-1.1.15/io_uring_engine.h000066400000000000000000000040021417604322600165400ustar00rootroot00000000000000#ifndef IO_URING_ENGINE_H #define IO_URING_ENGINE_H 1 #include #include #include #include #include #include struct io_uring_sqe; #ifndef WITHOUT_URING #include #endif class IOUringEngine { public: IOUringEngine(size_t slop_bytes); void submit_read(int fd, size_t len, off_t offset, std::function cb); // NOTE: We just do the stat() to get the data into the dentry cache for fast access, // or to check whether the file exists. Thus, the callback has only an OK/not OK boolean. void submit_stat(const char *path, std::function cb); bool get_supports_stat() { return supports_stat; } void finish(); size_t get_waiting_reads() const { return pending_reads + queued_reads.size(); } private: #ifndef WITHOUT_URING void submit_read_internal(io_uring_sqe *sqe, int fd, size_t len, off_t offset, std::function cb); void submit_stat_internal(io_uring_sqe *sqe, char *path, std::function cb); io_uring ring; #endif size_t pending_reads = 0; // Number of requests we have going in the ring. bool using_uring, supports_stat = false; const size_t slop_bytes; struct QueuedRead { int fd; size_t len; off_t offset; std::function cb; }; std::queue queued_reads; struct QueuedStat { char *pathname; // Owned by us. std::function cb; }; std::queue queued_stats; enum Op { OP_READ, OP_STAT }; struct PendingRead { Op op; std::function read_cb; std::function stat_cb; union { struct { void *buf; size_t len; // For re-submission. int fd; off_t offset; iovec iov; } read; struct { char *pathname; struct statx *buf; } stat; }; }; // 256 simultaneous requests should be ample, for slow and fast media alike. static constexpr size_t queue_depth = 256; }; #endif // !defined(IO_URING_ENGINE_H) plocate-1.1.15/lib.cpp000066400000000000000000000047031417604322600145110ustar00rootroot00000000000000/* Common functions. Copyright (C) 2005, 2007 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #include "lib.h" #include "db.h" #include #include #include #include #include #include #include #include #include #include #include using namespace std; /* Compare two path names using the database directory order. This is not exactly strcmp () order: "a" < "a.b", so "a/z" < "a.b". */ int dir_path_cmp(const string &a, const string &b) { auto [ai, bi] = mismatch(a.begin(), a.end(), b.begin(), b.end()); if (ai == a.end() && bi == b.end()) { return 0; } if (ai == a.end()) { return -1; } if (bi == b.end()) { return 1; } if (*ai == *bi) { return 0; } if (*ai == '/') { return -1; } if (*bi == '/') { return 1; } return int((unsigned char)*ai) - int((unsigned char)*bi); } /* Sort LIST using dir_path_cmp () */ void string_list_dir_path_sort(vector *list) { sort(list->begin(), list->end(), [](const string &a, const string &b) { return dir_path_cmp(a, b) < 0; }); } /* Is PATH included in LIST? Update *IDX to move within LIST. LIST is assumed to be sorted using dir_path_cmp (), successive calls to this function are assumed to use PATH values increasing in dir_path_cmp (). */ bool string_list_contains_dir_path(const vector *list, size_t *idx, const string &path) { int cmp = 0; while (*idx < list->size() && (cmp = dir_path_cmp((*list)[*idx], path)) < 0) { (*idx)++; } if (*idx < list->size() && cmp == 0) { (*idx)++; return true; } return false; } plocate-1.1.15/lib.h000066400000000000000000000033741417604322600141610ustar00rootroot00000000000000/* Common functions. Copyright (C) 2005, 2007 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #ifndef LIB_H__ #define LIB_H__ #include #include #include #include #include #include #define _(X) (X) /* Compare two path names using the database directory order. This is not exactly strcmp () order: "a" < "a.b", so "a/z" < "a.b". */ extern int dir_path_cmp(const std::string &a, const std::string &b); /* Sort LIST using dir_path_cmp () */ extern void string_list_dir_path_sort(std::vector *list); /* Is PATH included in LIST? Update *IDX to move within LIST. LIST is assumed to be sorted using dir_path_cmp (), successive calls to this function are assumed to use PATH values increasing in dir_path_cmp (). */ extern bool string_list_contains_dir_path(const std::vector *list, size_t *idx, const std::string &path); #endif plocate-1.1.15/meson.build000066400000000000000000000110101417604322600153660ustar00rootroot00000000000000project('plocate', 'cpp', default_options: ['buildtype=debugoptimized','cpp_std=c++17'], version: '1.1.15') add_project_arguments('-DGROUPNAME="' + get_option('locategroup') + '"', language: 'cpp') add_project_arguments('-DUPDATEDB_CONF="/etc/updatedb.conf"', language: 'cpp') dbfile = join_paths(get_option('sharedstatedir'), get_option('dbpath')) add_project_arguments('-DDBFILE="' + dbfile + '"', language: 'cpp') add_project_arguments('-DPACKAGE_NAME="plocate"', language: 'cpp') add_project_arguments('-DPACKAGE_VERSION="' + meson.project_version() + '"', language: 'cpp') add_project_arguments('-DPACKAGE_BUGREPORT="steinar+plocate@gunderson.no"', language: 'cpp') cxx = meson.get_compiler('cpp') uringdep = dependency('liburing', required: false) zstddep = dependency('libzstd') threaddep = dependency('threads') atomicdep = cxx.find_library('atomic', required: false) if not uringdep.found() add_project_arguments('-DWITHOUT_URING', language: 'cpp') endif if cxx.has_header('endian.h') add_project_arguments('-DHAS_ENDIAN_H', language: 'cpp') endif # Function multiversioning is x86-only _and_ not available on certain targets # (they need “ifunc”, indirect functions, a GNU extension). code = '''__attribute__((target("default"))) void foo(); __attribute__((target("sse2"))) void foo(); void bar() { foo(); }''' if cxx.compiles(code, name: 'function multiversioning') add_project_arguments('-DHAS_FUNCTION_MULTIVERSIONING', language: 'cpp') endif executable('plocate', ['plocate.cpp', 'io_uring_engine.cpp', 'turbopfor.cpp', 'parse_trigrams.cpp', 'serializer.cpp', 'access_rx_cache.cpp', 'needle.cpp', 'complete_pread.cpp'], dependencies: [uringdep, zstddep, threaddep, atomicdep], install: true, install_mode: ['rwxr-sr-x', 'root', get_option('locategroup')]) executable('plocate-build', ['plocate-build.cpp', 'database-builder.cpp'], dependencies: [zstddep], install: true, install_dir: get_option('sbindir')) updatedb_progname = get_option('updatedb_progname') executable(updatedb_progname, ['updatedb.cpp', 'database-builder.cpp', 'conf.cpp', 'lib.cpp', 'bind-mount.cpp', 'complete_pread.cpp'], dependencies: [zstddep, threaddep], install: true, install_dir: get_option('sbindir')) conf_data = configuration_data() conf_data.set('PROCESSED_BY_MESON', '1') conf_data.set('sbindir', join_paths(get_option('prefix'), get_option('sbindir'))) conf_data.set('groupname', get_option('locategroup')) conf_data.set('dbfile', dbfile) conf_data.set('updatedb_conf', '/etc/updatedb.conf') conf_data.set('updatedb_progname', updatedb_progname) update_script = configure_file(input: 'update-plocate.sh', output: 'update-plocate.sh', configuration: conf_data) fs = import('fs') meson.add_install_script('mkdir.sh', fs.parent(dbfile)) if get_option('install_cron') install_data(update_script, install_dir: '/etc/cron.daily', rename: 'plocate') endif install_man('plocate.1') install_man('plocate-build.8') updatedb_man = configure_file(input: 'updatedb.8.in', output: updatedb_progname + '.8', configuration: conf_data) install_man(updatedb_man) updatedb_conf_man = configure_file(input: 'updatedb.conf.5.in', output: 'updatedb.conf.5', configuration: conf_data) install_man(updatedb_conf_man) if get_option('install_systemd') unitdir = get_option('systemunitdir') if unitdir == '' systemd = dependency('systemd', required: false) if systemd.found() unitdir = systemd.get_pkgconfig_variable('systemdsystemunitdir') endif endif if unitdir != '' updatedb_service = configure_file(input: 'plocate-updatedb.service.in', output: 'plocate-updatedb.service', configuration: conf_data) install_data(updatedb_service, install_dir: unitdir) install_data('plocate-updatedb.timer', install_dir: unitdir) endif endif # Requires having TurboPFor checked out, so not built by default. # Unless you have a recent Meson, there's no apparently good way # of calling into that directory to run make, and we're not # replicating their build system, so it has to be manual. pfordir = meson.source_root() + '/TurboPFor-Integer-Compression' if run_command('[', '-r', pfordir + '/libic.a', ']').returncode() == 0 turbopfordep = declare_dependency( include_directories: include_directories('TurboPFor-Integer-Compression'), dependencies: meson.get_compiler('cpp').find_library('ic', dirs: pfordir)) executable('bench', ['bench.cpp', 'io_uring_engine.cpp', 'turbopfor.cpp', 'complete_pread.cpp'], dependencies: [uringdep, turbopfordep], build_by_default: false, install: false) endif plocate-1.1.15/meson_options.txt000066400000000000000000000013501417604322600166670ustar00rootroot00000000000000option('install_cron', type: 'boolean', value: false, description: 'Install mlocate conversion script into /etc/cron.daily') option('install_systemd', type: 'boolean', value: true, description: 'Install systemd service and timer for updatedb, if systemd is installed') option('systemunitdir', type: 'string', description: 'Where to install systemd units to (default: autodetect)') option('locategroup', type: 'string', value: 'plocate', description: 'Group that the install script will use for the .db file') option('updatedb_progname', type: 'string', value: 'updatedb', description: 'Binary name of updatedb') option('dbpath', type: 'string', value: 'plocate/plocate.db', description: 'Path to plocate database relative to "sharedstatedir"') plocate-1.1.15/mkdir.sh000077500000000000000000000004101417604322600146730ustar00rootroot00000000000000#! /bin/sh mkdir -p ${DESTDIR}/"$1" cat <${DESTDIR}/"$1"/CACHEDIR.TAG Signature: 8a477f597d28d172789f06886806bc55 # This file is a cache directory tag created by plocate. # For information about cache directory tags, see: # https://bford.info/cachedir/ EOF plocate-1.1.15/needle.cpp000066400000000000000000000027671417604322600152070ustar00rootroot00000000000000#include "needle.h" #include "options.h" #include "parse_trigrams.h" #include #include #include #include #include #include #include using namespace std; bool matches(const Needle &needle, const char *haystack) { if (needle.type == Needle::STRSTR) { return strstr(haystack, needle.str.c_str()) != nullptr; } else if (needle.type == Needle::GLOB) { int flags = ignore_case ? FNM_CASEFOLD : 0; return fnmatch(needle.str.c_str(), haystack, flags) == 0; } else { assert(needle.type == Needle::REGEX); return regexec(&needle.re, haystack, /*nmatch=*/0, /*pmatch=*/nullptr, /*flags=*/0) == 0; } } string unescape_glob_to_plain_string(const string &needle) { string unescaped; for (size_t i = 0; i < needle.size(); i += read_unigram(needle, i).second) { uint32_t ch = read_unigram(needle, i).first; assert(ch != WILDCARD_UNIGRAM); if (ch == PREMATURE_END_UNIGRAM) { fprintf(stderr, "Pattern '%s' ended prematurely\n", needle.c_str()); exit(1); } unescaped.push_back(ch); } return unescaped; } regex_t compile_regex(const string &needle) { regex_t re; int flags = REG_NOSUB; if (ignore_case) { flags |= REG_ICASE; } if (use_extended_regex) { flags |= REG_EXTENDED; } int err = regcomp(&re, needle.c_str(), flags); if (err != 0) { char errbuf[256]; regerror(err, &re, errbuf, sizeof(errbuf)); fprintf(stderr, "Error when compiling regex '%s': %s\n", needle.c_str(), errbuf); exit(1); } return re; } plocate-1.1.15/needle.h000066400000000000000000000006631417604322600146450ustar00rootroot00000000000000#ifndef _NEEDLE_H #define _NEEDLE_H 1 #include #include struct Needle { enum { STRSTR, REGEX, GLOB } type; std::string str; // Filled in no matter what. regex_t re; // For REGEX. }; bool matches(const Needle &needle, const char *haystack); std::string unescape_glob_to_plain_string(const std::string &needle); regex_t compile_regex(const std::string &needle); #endif // !defined(_NEEDLE_H) plocate-1.1.15/options.h000066400000000000000000000007121417604322600150770ustar00rootroot00000000000000#ifndef _OPTIONS_H #define _OPTIONS_H #include extern bool ignore_case; extern bool only_count; extern bool print_nul; extern bool use_debug; extern bool flush_cache; extern bool patterns_are_regex; extern bool use_extended_regex; extern bool check_existence; extern int64_t limit_matches; extern int64_t limit_left; // Not strictly an option. extern bool stdout_is_tty; // Same. extern bool literal_printing; #endif // !defined(_OPTIONS_H) plocate-1.1.15/parse_trigrams.cpp000066400000000000000000000220341417604322600167620ustar00rootroot00000000000000#include "parse_trigrams.h" #include "unique_sort.h" #include #include #include #include using namespace std; string print_td(const TrigramDisjunction &td) { if (td.read_trigrams.size() == 0) { // Before we've done hash lookups (or none matched), so print all alternatives. if (td.trigram_alternatives.size() == 1) { return print_trigram(td.trigram_alternatives[0]); } else { string ret; ret = "("; bool first = true; for (uint32_t trgm : td.trigram_alternatives) { if (!first) ret += " OR "; ret += print_trigram(trgm); first = false; } return ret + ")"; } } else { // Print only those that we actually have in the index. if (td.read_trigrams.size() == 1) { return print_trigram(td.read_trigrams[0].first.trgm); } else { string ret; ret = "("; bool first = true; for (auto &[trgmptr, len] : td.read_trigrams) { if (!first) ret += " OR "; ret += print_trigram(trgmptr.trgm); first = false; } return ret + ")"; } } } string print_trigram(uint32_t trgm) { char ch[3] = { char(trgm & 0xff), char((trgm >> 8) & 0xff), char((trgm >> 16) & 0xff) }; string str = "'"; for (unsigned i = 0; i < 3;) { if (ch[i] == '\\') { str.push_back('\\'); str.push_back(ch[i]); ++i; } else if (int(ch[i]) >= 32 && int(ch[i]) <= 127) { // Holds no matter whether char is signed or unsigned. str.push_back(ch[i]); ++i; } else { // See if we have an entire UTF-8 codepoint, and that it's reasonably printable. mbtowc(nullptr, 0, 0); wchar_t pwc; int ret = mbtowc(&pwc, ch + i, 3 - i); if (ret >= 1 && pwc >= 32) { str.append(ch + i, ret); i += ret; } else { char buf[16]; snprintf(buf, sizeof(buf), "\\x{%02x}", (unsigned char)ch[i]); str += buf; ++i; } } } str += "'"; return str; } pair read_unigram(const string &s, size_t start) { if (start >= s.size()) { return { PREMATURE_END_UNIGRAM, 0 }; } if (s[start] == '\\') { // Escaped character. if (start + 1 >= s.size()) { return { PREMATURE_END_UNIGRAM, 1 }; } else { return { (unsigned char)s[start + 1], 2 }; } } if (s[start] == '*' || s[start] == '?') { // Wildcard. return { WILDCARD_UNIGRAM, 1 }; } if (s[start] == '[') { // Character class; search to find the end. size_t len = 1; if (start + len >= s.size()) { return { PREMATURE_END_UNIGRAM, len }; } if (s[start + len] == '!') { ++len; } if (start + len >= s.size()) { return { PREMATURE_END_UNIGRAM, len }; } if (s[start + len] == ']') { ++len; } for (;;) { if (start + len >= s.size()) { return { PREMATURE_END_UNIGRAM, len }; } if (s[start + len] == ']') { return { WILDCARD_UNIGRAM, len + 1 }; } ++len; } } // Regular letter. return { (unsigned char)s[start], 1 }; } uint32_t read_trigram(const string &s, size_t start) { pair u1 = read_unigram(s, start); if (u1.first == WILDCARD_UNIGRAM || u1.first == PREMATURE_END_UNIGRAM) { return u1.first; } pair u2 = read_unigram(s, start + u1.second); if (u2.first == WILDCARD_UNIGRAM || u2.first == PREMATURE_END_UNIGRAM) { return u2.first; } pair u3 = read_unigram(s, start + u1.second + u2.second); if (u3.first == WILDCARD_UNIGRAM || u3.first == PREMATURE_END_UNIGRAM) { return u3.first; } return u1.first | (u2.first << 8) | (u3.first << 16); } struct TrigramState { string buffered; unsigned next_codepoint; bool operator<(const TrigramState &other) const { if (next_codepoint != other.next_codepoint) return next_codepoint < other.next_codepoint; return buffered < other.buffered; } bool operator==(const TrigramState &other) const { return next_codepoint == other.next_codepoint && buffered == other.buffered; } }; void parse_trigrams_ignore_case(const string &needle, vector *trigram_groups) { vector> alternatives_for_cp; // Parse the needle into Unicode code points, and do inverse case folding // on each to find legal alternatives. This is far from perfect (e.g. ß // will not become ss), but it's generally the best we can do without // involving ICU or the likes. mbtowc(nullptr, 0, 0); const char *ptr = needle.c_str(); unique_ptr buf(new char[MB_CUR_MAX]); while (*ptr != '\0') { wchar_t ch; int ret = mbtowc(&ch, ptr, strlen(ptr)); if (ret == -1) { perror(ptr); exit(1); } vector alt; alt.push_back(string(ptr, ret)); ptr += ret; if (towlower(ch) != wint_t(ch)) { ret = wctomb(buf.get(), towlower(ch)); alt.push_back(string(buf.get(), ret)); } if (towupper(ch) != wint_t(ch) && towupper(ch) != towlower(ch)) { ret = wctomb(buf.get(), towupper(ch)); alt.push_back(string(buf.get(), ret)); } alternatives_for_cp.push_back(move(alt)); } // Now generate all possible byte strings from those code points in order; // e.g., from abc, we'd create a and A, then extend those to ab aB Ab AB, // then abc abC aBc aBC and so on. Since we don't want to have 2^n // (or even 3^n) strings, we only extend them far enough to cover at // least three bytes; this will give us a set of candidate trigrams // (the filename must have at least one of those), and then we can // chop off the first byte, deduplicate states and continue extending // and generating trigram sets. // // There are a few special cases, notably the dotted i (İ), where the // UTF-8 versions of upper and lower case have different number of bytes. // If this happens, we can have combinatorial explosion and get many more // than the normal 8 states. We detect this and simply bomb out; it will // never really happen in real strings, and stopping trigram generation // really only means our pruning of candidates will be less effective. vector states; states.push_back(TrigramState{ "", 0 }); for (;;) { // Extend every state so that it has buffered at least three bytes. // If this isn't possible, we are done with the string (can generate // no more trigrams). bool need_another_pass; do { need_another_pass = false; vector new_states; for (const TrigramState &state : states) { if (read_trigram(state.buffered, 0) != PREMATURE_END_UNIGRAM) { // No need to extend this further. new_states.push_back(state); continue; } if (state.next_codepoint == alternatives_for_cp.size()) { // We can't form a complete trigram from this alternative, // so we're done. return; } for (const string &rune : alternatives_for_cp[state.next_codepoint]) { TrigramState new_state{ state.buffered + rune, state.next_codepoint + 1 }; if (read_trigram(state.buffered, 0) == PREMATURE_END_UNIGRAM) { need_another_pass = true; } new_states.push_back(move(new_state)); } } states = move(new_states); } while (need_another_pass); // OK, so now we have a bunch of states, and all of them are at least // three bytes long. This means we have a complete set of trigrams, // and the destination filename must contain at least one of them. // Output those trigrams, cut out the first byte and then deduplicate // the states before we continue. bool any_wildcard = false; vector trigram_alternatives; for (TrigramState &state : states) { trigram_alternatives.push_back(read_trigram(state.buffered, 0)); state.buffered.erase(0, read_unigram(state.buffered, 0).second); assert(trigram_alternatives.back() != PREMATURE_END_UNIGRAM); if (trigram_alternatives.back() == WILDCARD_UNIGRAM) { // If any of the candidates are wildcards, we need to drop the entire OR group. // (Most likely, all of them would be anyway.) We need to keep stripping out // the first unigram from each state. any_wildcard = true; } } unique_sort(&trigram_alternatives); // Could have duplicates, although it's rare. unique_sort(&states); if (!any_wildcard) { TrigramDisjunction new_pt; new_pt.remaining_trigrams_to_read = trigram_alternatives.size(); new_pt.trigram_alternatives = move(trigram_alternatives); new_pt.max_num_docids = 0; trigram_groups->push_back(move(new_pt)); } if (states.size() > 100) { // A completely crazy pattern with lots of those special characters. // We just give up; this isn't a realistic scenario anyway. // We already have lots of trigrams that should reduce the amount of // candidates. return; } } } void parse_trigrams(const string &needle, bool ignore_case, vector *trigram_groups) { if (ignore_case) { parse_trigrams_ignore_case(needle, trigram_groups); return; } // The case-sensitive case is straightforward. for (size_t i = 0; i < needle.size(); i += read_unigram(needle, i).second) { uint32_t trgm = read_trigram(needle, i); if (trgm == WILDCARD_UNIGRAM || trgm == PREMATURE_END_UNIGRAM) { // Invalid trigram, so skip. continue; } TrigramDisjunction new_pt; new_pt.remaining_trigrams_to_read = 1; new_pt.trigram_alternatives.push_back(trgm); new_pt.max_num_docids = 0; trigram_groups->push_back(move(new_pt)); } } plocate-1.1.15/parse_trigrams.h000066400000000000000000000066561417604322600164430ustar00rootroot00000000000000#ifndef _PARSE_TRIGRAMS_H #define _PARSE_TRIGRAMS_H 1 #include "db.h" #include #include #include #include // One or more trigrams, with an implicit OR between them. For case-sensitive searches, // this is just e.g. “abc”, but for case-insensitive, it would be “abc OR abC or aBc ...” etc. struct TrigramDisjunction { unsigned index = -1; // For debugging only. // The alternatives as determined by parse_trigrams(). std::vector trigram_alternatives; // Like trigram_alternatives, but only the ones we've actually read from the // hash table (the non-existent ones are filtered out). The second member is // the length in bytes. Incomplete if remaining_trigrams_to_read > 0. std::vector> read_trigrams; // Sum of num_docids in all trigrams. This is usually a fairly good indicator // of the real number of docids, since there are few files that would have e.g. // both abc and abC in them (but of course, with multiple files in the same // docid block, it is far from unheard of). uint32_t max_num_docids; // While reading posting lists: Holds the union of the posting lists read // so far. Once remaining_trigrams_to_read == 0 (all are read), will be taken // out and used for intersections against the other disjunctions. std::vector docids; // While looking up in the hash table (filling out read_trigrams): Number of // lookups in the hash table remaining. While reading actual posting lists // (filling out docids): Number of posting lists left to read. unsigned remaining_trigrams_to_read; }; // Take the given needle (search string) and break it down into a set of trigrams // (or trigram alternatives; see TrigramDisjunction) that must be present for the // string to match. (Note: They are not _sufficient_ for the string to match; // false positives might very well occur and must be weeded out later.) // // For the case-sensitive case, this is straightforward; just take every trigram // present in the needle and add them (e.g. abcd -> abc AND bcd). // For case-insensitivity, it's trickier; see the comments in the function. // // Note that our trigrams are on the basis of bytes, not Unicode code points. // This both simplifies table structure (everything is the same length), and // guards us against trigram explosion (imagine every combination of CJK characters // getting their own trigram). void parse_trigrams(const std::string &needle, bool ignore_case, std::vector *trigram_groups); static constexpr uint32_t WILDCARD_UNIGRAM = 0xFF000000; static constexpr uint32_t PREMATURE_END_UNIGRAM = 0xFF000001; // Reads a unigram, taking into account escaping (\ becomes ). // Returns WILDCARD_UNIGRAM if there's an invalid unigram, ie., we found // a glob character (?, * or a [] group). Returns EOS_UNIGRAM if we went // past the end of the string, e.g., a string that ends in a backslash. // The second element is always the length. std::pair read_unigram(const std::string &s, size_t start); // Reads a trigram, ie., three calls to read_unigram(). Needs to start on a valid unigram. // Returns WILDCARD_UNIGRAM or PREMATURE_END_UNIGRAM of either of those occurred // during reading of the string. uint32_t read_trigram(const std::string &s, size_t start); // For debugging. std::string print_td(const TrigramDisjunction &td); std::string print_trigram(uint32_t trgm); #endif // !defined(_PARSE_TRIGRAMS_H) plocate-1.1.15/plocate-build.8000066400000000000000000000036151417604322600160550ustar00rootroot00000000000000.TH plocate-build 8 "Oct 2020" plocate-build .SH NAME plocate-build \- generate index for plocate .SH SYNOPSIS .B plocate-build .I "[OPTION]..." .I "MLOCATE_DB" .I "PLOCATE_DB" .SH DESCRIPTION .B plocate-build creates an index from .BR plocate (1) from an index earlier generated by .BR updatedb (8) from the .BR mlocate (1) package. Most users would rather want to use plocate's own .BR updatedb (8), which is more direct. If .I PLOCATE_DB already exists, it will be overwritten (non-atomically). The file is created such that it is not world-readable, as the final access check is done by .BR plocate (1) during the search. .SH OPTIONS .TP \fB\-b\fR, \fB\-\-block\-size\fR \fISIZE\fR Create blocks containing .I SIZE filenames each, compress them together, and treat them as the same element in the posting lists. This makes the index smaller (because the compression algorithm gets more context to work with, and because there are fewer elements in each posting list), but also makes posting lists less precise, moving more work to weeding out false positives after posting list intersection. Making this number larger will make linear search (for \fB\-\-regex\fR, or very short patterns) faster, with diminishing returns around 256 filenames per block. However, making it too large will cause more false positives, reducing overall performance for typical queries. The default value is 32. Setting it to 1 makes one (compressed) block per filename. .TP \fB\-p\fR, \fB\-\-plaintext\fR Treat the input as plain text, with entries delimited by newlines, instead of an mlocate database. .TP .B \-\-help Print out usage information, then exit successfully. .TP .B \-\-version Print out version information, then exit successfully. .SH AUTHOR Steinar H. Gunderson .SH SEE ALSO \fBplocate\fP(1), \fB/etc/cron.daily/plocate\fR (which is called \fBupdate-plocate.sh\fR in the source distribution) plocate-1.1.15/plocate-build.cpp000066400000000000000000000126351417604322600164720ustar00rootroot00000000000000#include "database-builder.h" #include "db.h" #include "dprintf.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include using namespace std; using namespace std::chrono; bool use_debug = false; enum { DBE_NORMAL = 0, /* A non-directory file */ DBE_DIRECTORY = 1, /* A directory */ DBE_END = 2 /* End of directory contents; contains no name */ }; // From mlocate. struct db_header { uint8_t magic[8]; uint32_t conf_size; uint8_t version; uint8_t check_visibility; uint8_t pad[2]; }; // From mlocate. struct db_directory { uint64_t time_sec; uint32_t time_nsec; uint8_t pad[4]; }; string read_cstr(FILE *fp) { string ret; for (;;) { int ch = getc(fp); if (ch == -1) { perror("getc"); exit(1); } if (ch == 0) { return ret; } ret.push_back(ch); } } void handle_directory(FILE *fp, DatabaseReceiver *receiver) { db_directory dummy; if (fread(&dummy, sizeof(dummy), 1, fp) != 1) { if (feof(fp)) { return; } else { perror("fread"); } } string dir_path = read_cstr(fp); if (dir_path == "/") { dir_path = ""; } for (;;) { int type = getc(fp); if (type == DBE_NORMAL) { string filename = read_cstr(fp); receiver->add_file(dir_path + "/" + filename, unknown_dir_time); } else if (type == DBE_DIRECTORY) { string dirname = read_cstr(fp); receiver->add_file(dir_path + "/" + dirname, unknown_dir_time); } else { return; // Probably end. } } } void read_plaintext(FILE *fp, DatabaseReceiver *receiver) { if (fseek(fp, 0, SEEK_SET) != 0) { perror("fseek"); exit(1); } while (!feof(fp)) { char buf[1024]; if (fgets(buf, sizeof(buf), fp) == nullptr) { break; } string s(buf); assert(!s.empty()); while (s.back() != '\n' && !feof(fp)) { // The string was longer than the buffer, so read again. if (fgets(buf, sizeof(buf), fp) == nullptr) { break; } s += buf; } if (!s.empty() && s.back() == '\n') s.pop_back(); receiver->add_file(move(s), unknown_dir_time); } } void read_mlocate(FILE *fp, DatabaseReceiver *receiver) { if (fseek(fp, 0, SEEK_SET) != 0) { perror("fseek"); exit(1); } db_header hdr; if (fread(&hdr, sizeof(hdr), 1, fp) != 1) { perror("short read"); exit(1); } // TODO: Care about the base path. string path = read_cstr(fp); if (fseek(fp, ntohl(hdr.conf_size), SEEK_CUR) != 0) { perror("skip conf block"); exit(1); } while (!feof(fp)) { handle_directory(fp, receiver); } } void do_build(const char *infile, const char *outfile, int block_size, bool plaintext) { FILE *infp = fopen(infile, "rb"); if (infp == nullptr) { perror(infile); exit(1); } // Train the dictionary by sampling real blocks. // The documentation for ZDICT_trainFromBuffer() claims that a reasonable // dictionary size is ~100 kB, but 1 kB seems to actually compress better for us, // and decompress just as fast. DictionaryBuilder builder(/*blocks_to_keep=*/1000, block_size); if (plaintext) { read_plaintext(infp, &builder); } else { read_mlocate(infp, &builder); } string dictionary = builder.train(1024); DatabaseBuilder db(outfile, /*owner=*/-1, block_size, dictionary, /*check_visibility=*/true); DatabaseReceiver *corpus = db.start_corpus(/*store_dir_times=*/false); if (plaintext) { read_plaintext(infp, corpus); } else { read_mlocate(infp, corpus); } fclose(infp); dprintf("Read %zu files from %s\n", corpus->num_files_seen(), infile); db.finish_corpus(); } void usage() { printf( "Usage: plocate-build MLOCATE_DB PLOCATE_DB\n" "\n" "Generate plocate index from mlocate.db, typically /var/lib/mlocate/mlocate.db.\n" "Normally, the destination should be /var/lib/mlocate/plocate.db.\n" "\n" " -b, --block-size SIZE number of filenames to store in each block (default 32)\n" " -p, --plaintext input is a plaintext file, not an mlocate database\n" " --help print this help\n" " --version print version information\n"); } void version() { printf("plocate-build %s\n", PACKAGE_VERSION); printf("Copyright 2020 Steinar H. Gunderson\n"); printf("License GPLv2+: GNU GPL version 2 or later .\n"); printf("This is free software: you are free to change and redistribute it.\n"); printf("There is NO WARRANTY, to the extent permitted by law.\n"); } int main(int argc, char **argv) { static const struct option long_options[] = { { "block-size", required_argument, 0, 'b' }, { "plaintext", no_argument, 0, 'p' }, { "help", no_argument, 0, 'h' }, { "version", no_argument, 0, 'V' }, { "debug", no_argument, 0, 'D' }, // Not documented. { 0, 0, 0, 0 } }; int block_size = 32; bool plaintext = false; setlocale(LC_ALL, ""); for (;;) { int option_index = 0; int c = getopt_long(argc, argv, "b:hpVD", long_options, &option_index); if (c == -1) { break; } switch (c) { case 'b': block_size = atoi(optarg); break; case 'p': plaintext = true; break; case 'h': usage(); exit(0); case 'V': version(); exit(0); case 'D': use_debug = true; break; default: exit(1); } } if (argc - optind != 2) { usage(); exit(1); } do_build(argv[optind], argv[optind + 1], block_size, plaintext); exit(EXIT_SUCCESS); } plocate-1.1.15/plocate-updatedb.service.in000066400000000000000000000003501417604322600204350ustar00rootroot00000000000000[Unit] Description=Update the plocate database ConditionACPower=true [Service] Type=oneshot ExecStart=@sbindir@/@updatedb_progname@ LimitNOFILE=131072 IOSchedulingClass=idle PrivateTmp=true PrivateDevices=true PrivateNetwork=true plocate-1.1.15/plocate-updatedb.timer000066400000000000000000000002521417604322600175110ustar00rootroot00000000000000[Unit] Description=Update the plocate database daily [Timer] OnCalendar=daily RandomizedDelaySec=12h AccuracySec=20min Persistent=true [Install] WantedBy=timers.target plocate-1.1.15/plocate.1000066400000000000000000000123251417604322600147470ustar00rootroot00000000000000.TH locate 1 "Oct 2020" plocate .SH NAME plocate \- find files by name, quickly .SH SYNOPSIS .B plocate .I "[OPTION]..." .I "PATTERN..." .SH DESCRIPTION .B plocate finds all files on the system matching the given pattern (or all of the patterns if multiple are given). It does this by means of an index made by .BR updatedb (8) or (less commonly) converted from another index by .BR plocate-build (8). plocate is largely argument-compatible with .BR mlocate (1), but is significantly faster. In particular, it rarely needs to scan through its entire database, unless the pattern is very short (less than three bytes) or you want to search for a regular expression. It does not try to maintain compatibility with BSD locate, or non-UTF-8 filenames and locales. Most I/O is done asynchronously, but the results are synchronized so that output comes in the same order every time. When multiple patterns are given, .B plocate will search for files that match .I all of them. This is the main incompatibility with .BR mlocate (1), which searches for files that match one or more patterns, unless the \-A option is given. By default, patterns are taken to be substrings to search for. If at least one non-escaped globbing metacharacter (*, ? or []) is given, that pattern is instead taken to be a glob pattern (which means it needs to start and end in * for a substring match). If .B --regexp is given, patterns are instead taken to be (non-anchored) POSIX basic regular expressions, and if .B --regex is given, patterns are taken to be POSIX extended regular expressions. All of this matches .BR mlocate (1) behavior. Like .BR mlocate (1), .B plocate shows all files visible to the calling user (by virtue of having read and execute permissions on all parent directories), and none that are not, by means of running with the setgid bit set to access the index (which is built as root), but by testing visibility as the calling user. .SH OPTIONS .TP \fB\-A\fR, \fB\-\-all\fR Ignored for compatibility with .BR mlocate (1). .TP \fB\-b\fR, \fB\-\-basename\fR Match only against the file name portion of the path name, ie., the directory names will be excluded from the match (but still printed). This does not speed up the search, but can suppress uninteresting matches. .TP \fB\-c\fR, \fB\-\-count\fR Do not print each match. Instead, count them, and print out a total number at the end. .TP \fB\-d\fR, \fB\-\-database\fR \fIDBPATH\fR Find matches in the given database, instead of \fB/var/lib/plocate/plocate.db\fR. This argument can be given multiple times, to search multiple databases. It is also possible to give multiple databases in one argument, separated by .BR : . (Any character, including : and \\, can be escaped by prepending a \\.) .TP \fB\-e\fR, \fB\-\-existing\fR Print only entries that refer to files existing at the time .B locate is run. Note that unlike .BR mlocate (1), symlinks are not followed by default (and indeed, there is no option to change this). .TP \fB\-i\fR, \fB\-\-ignore\-case\fR Do a case-insensitive match as given by the current locale (default is case-sensitive, byte-by-byte match). Note that .B plocate does not support the full range of Unicode case folding rules; in particular, searching for \fIß\fR will not give you matches on \fIss\fR even in a German locale. Also note that this option will be somewhat slower than a case-sensitive match, since it needs to generate more candidates for searching the index. .TP \fB\-l\fR, \fB\-\-limit\fR \fILIMIT\fR Stop searching after .I LIMIT matches have been found. If .B \-\-count is given, the number printed out will be at most \fILIMIT\fR. .TP \fB\-N\fR, \fB\-\-literal\fR Print entry names without quoting. Normally, .B plocate will escape special characters in filenames, so that they are safe for consumption by typical shells (similar to the GNU coreutils .I shell-escape-always quoting style), unless printing to a pipe, but this options will turn off such quoting. .TP \fB\-0\fR, \fB\-\-null\fR Instead of writing a newline after every match, write a NUL (ASCII 0). This is useful for creating unambiguous output when it is to be processed by other tools (like \fBxargs\fP(1)), as filenames are allowed to contain embedded newlines. .TP \fB\-r\fR, \fB\-\-regexp\fR Patterns are taken to be POSIX basic regular expressions. See .BR regex (7) for more information. Note that this forces a linear scan through the entire database, which is slow. .TP .B \-\-regex Like \fB\-\-regexp\fR, but patterns are instead taken to be POSIX .I extended regular expressions. .TP \fB\-w\fR, \fB\-\-wholename\fR Match against the entire path name. This is the default, so unless \fB-b\fR is given first (see above), it will not do anything. This option thus exists only as compatibility with .BR mlocate (1). .TP .B \-\-help Print out usage information, then exit successfully. .TP .B \-\-version Print out version information, then exit successfully. .SH ENVIRONMENT .TP \fBLOCATE_PATH\fR If given, appended after the list of \fB\-\-database\fR paths (whether an explicit is given or the default is used). Colon-delimiting and character escaping follows the same rules as for \fB\-\-database\fR. .SH AUTHOR Steinar H. Gunderson .SH SEE ALSO \fBplocate-build\fP(8), \fBmlocate\fP(1), \fBupdatedb\fP(8) plocate-1.1.15/plocate.cpp000066400000000000000000000763431417604322600154030ustar00rootroot00000000000000#include "access_rx_cache.h" #include "complete_pread.h" #include "db.h" #include "dprintf.h" #include "io_uring_engine.h" #include "needle.h" #include "parse_trigrams.h" #include "serializer.h" #include "turbopfor.h" #include "unique_sort.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include using namespace std; using namespace std::chrono; bool ignore_case = false; bool only_count = false; bool print_nul = false; bool use_debug = false; bool flush_cache = false; bool patterns_are_regex = false; bool use_extended_regex = false; bool match_basename = false; bool check_existence = false; int64_t limit_matches = numeric_limits::max(); int64_t limit_left = numeric_limits::max(); bool stdout_is_tty = false; bool literal_printing = false; static bool in_forked_child = false; steady_clock::time_point start; ZSTD_DDict *ddict = nullptr; class Corpus { public: Corpus(int fd, const char *filename_for_errors, IOUringEngine *engine); ~Corpus(); void find_trigram(uint32_t trgm, function cb); void get_compressed_filename_block(uint32_t docid, function cb) const; size_t get_num_filename_blocks() const; off_t offset_for_block(uint32_t docid) const { return hdr.filename_index_offset_bytes + docid * sizeof(uint64_t); } const Header &get_hdr() const { return hdr; } public: const int fd; IOUringEngine *const engine; Header hdr; }; Corpus::Corpus(int fd, const char *filename_for_errors, IOUringEngine *engine) : fd(fd), engine(engine) { if (flush_cache) { off_t len = lseek(fd, 0, SEEK_END); if (len == -1) { perror("lseek"); exit(1); } posix_fadvise(fd, 0, len, POSIX_FADV_DONTNEED); } complete_pread(fd, &hdr, sizeof(hdr), /*offset=*/0); if (memcmp(hdr.magic, "\0plocate", 8) != 0) { fprintf(stderr, "%s: database is corrupt or not a plocate database; please rebuild it.\n", filename_for_errors); exit(1); } if (hdr.version != 0 && hdr.version != 1) { fprintf(stderr, "%s: has version %u, expected 0 or 1; please rebuild it.\n", filename_for_errors, hdr.version); exit(1); } if (hdr.version == 0) { // These will be junk data. hdr.zstd_dictionary_offset_bytes = 0; hdr.zstd_dictionary_length_bytes = 0; } if (hdr.max_version < 2) { // This too. (We ignore the other max_version 2 fields.) hdr.check_visibility = true; } } Corpus::~Corpus() { close(fd); } void Corpus::find_trigram(uint32_t trgm, function cb) { uint32_t bucket = hash_trigram(trgm, hdr.hashtable_size); engine->submit_read(fd, sizeof(Trigram) * (hdr.extra_ht_slots + 2), hdr.hash_table_offset_bytes + sizeof(Trigram) * bucket, [this, trgm, cb{ move(cb) }](string_view s) { const Trigram *trgmptr = reinterpret_cast(s.data()); for (unsigned i = 0; i < hdr.extra_ht_slots + 1; ++i) { if (trgmptr[i].trgm == trgm) { cb(trgmptr + i, trgmptr[i + 1].offset - trgmptr[i].offset); return; } } // Not found. cb(nullptr, 0); }); } void Corpus::get_compressed_filename_block(uint32_t docid, function cb) const { // Read the file offset from this docid and the next one. // This is always allowed, since we have a sentinel block at the end. engine->submit_read(fd, sizeof(uint64_t) * 2, offset_for_block(docid), [this, cb{ move(cb) }](string_view s) { const uint64_t *ptr = reinterpret_cast(s.data()); off_t offset = ptr[0]; size_t len = ptr[1] - ptr[0]; engine->submit_read(fd, len, offset, cb); }); } size_t Corpus::get_num_filename_blocks() const { return hdr.num_docids; } template void stat_if_needed(const char *filename, bool access_ok, IOUringEngine *engine, T cb) { if (!access_ok || !check_existence) { // Doesn't have access or doesn't care about existence, so no need to stat. cb(access_ok); } else if (engine == nullptr || !engine->get_supports_stat()) { // Do a synchronous stat. struct stat buf; bool ok = lstat(filename, &buf) == 0; cb(ok); } else { engine->submit_stat(filename, cb); } } void scan_file_block(const vector &needles, string_view compressed, IOUringEngine *engine, AccessRXCache *access_rx_cache, uint64_t seq, ResultReceiver *serializer, atomic *matched) { unsigned long long uncompressed_len = ZSTD_getFrameContentSize(compressed.data(), compressed.size()); if (uncompressed_len == ZSTD_CONTENTSIZE_UNKNOWN || uncompressed_len == ZSTD_CONTENTSIZE_ERROR) { fprintf(stderr, "ZSTD_getFrameContentSize() failed\n"); exit(1); } string block; block.resize(uncompressed_len + 1); static thread_local ZSTD_DCtx *ctx = ZSTD_createDCtx(); // Reused across calls. size_t err; if (ddict != nullptr) { err = ZSTD_decompress_usingDDict(ctx, &block[0], block.size(), compressed.data(), compressed.size(), ddict); } else { err = ZSTD_decompressDCtx(ctx, &block[0], block.size(), compressed.data(), compressed.size()); } if (ZSTD_isError(err)) { fprintf(stderr, "ZSTD_decompress(): %s\n", ZSTD_getErrorName(err)); exit(1); } block[block.size() - 1] = '\0'; auto test_candidate = [&](const char *filename, uint64_t local_seq, uint64_t next_seq) { access_rx_cache->check_access(filename, /*allow_async=*/true, [matched, engine, serializer, local_seq, next_seq, filename{ strdup(filename) }](bool ok) { stat_if_needed(filename, ok, engine, [matched, serializer, local_seq, next_seq, filename](bool ok) { if (ok) { ++*matched; serializer->print(local_seq, next_seq - local_seq, filename); } else { serializer->print(local_seq, next_seq - local_seq, ""); } free(filename); }); }); }; // We need to know the next sequence number before inserting into Serializer, // so always buffer one candidate. const char *pending_candidate = nullptr; uint64_t local_seq = seq << 32; for (const char *filename = block.data(); filename != block.data() + block.size(); filename += strlen(filename) + 1) { const char *haystack = filename; if (match_basename) { haystack = strrchr(filename, '/'); if (haystack == nullptr) { haystack = filename; } else { ++haystack; } } bool found = true; for (const Needle &needle : needles) { if (!matches(needle, haystack)) { found = false; break; } } if (found) { if (pending_candidate != nullptr) { test_candidate(pending_candidate, local_seq, local_seq + 1); ++local_seq; } pending_candidate = filename; } } if (pending_candidate == nullptr) { serializer->print(seq << 32, 1ULL << 32, ""); } else { test_candidate(pending_candidate, local_seq, (seq + 1) << 32); } } size_t scan_docids(const vector &needles, const vector &docids, const Corpus &corpus, IOUringEngine *engine) { Serializer docids_in_order; AccessRXCache access_rx_cache(engine, corpus.get_hdr().check_visibility); atomic matched{ 0 }; for (size_t i = 0; i < docids.size(); ++i) { uint32_t docid = docids[i]; corpus.get_compressed_filename_block(docid, [i, &matched, &needles, &access_rx_cache, engine, &docids_in_order](string_view compressed) { scan_file_block(needles, compressed, engine, &access_rx_cache, i, &docids_in_order, &matched); }); } engine->finish(); return matched; } struct WorkerThread { thread t; // We use a result queue instead of synchronizing Serializer, // since a lock on it becomes a huge choke point if there are // lots of threads. mutex result_mu; struct Result { uint64_t seq; uint64_t skip; string msg; }; vector results; }; class WorkerThreadReceiver : public ResultReceiver { public: WorkerThreadReceiver(WorkerThread *wt) : wt(wt) {} void print(uint64_t seq, uint64_t skip, const string msg) override { lock_guard lock(wt->result_mu); if (msg.empty() && !wt->results.empty() && wt->results.back().seq + wt->results.back().skip == seq) { wt->results.back().skip += skip; } else { wt->results.emplace_back(WorkerThread::Result{ seq, skip, move(msg) }); } } private: WorkerThread *wt; }; void deliver_results(WorkerThread *wt, Serializer *serializer) { vector results; { lock_guard lock(wt->result_mu); results = move(wt->results); } for (const WorkerThread::Result &result : results) { serializer->print(result.seq, result.skip, move(result.msg)); } } // We do this sequentially, as it's faster than scattering // a lot of I/O through io_uring and hoping the kernel will // coalesce it plus readahead for us. Since we assume that // we will primarily be CPU-bound, we'll be firing up one // worker thread for each spare core (the last one will // only be doing I/O). access() is still synchronous. uint64_t scan_all_docids(const vector &needles, int fd, const Corpus &corpus) { { const Header &hdr = corpus.get_hdr(); if (hdr.zstd_dictionary_length_bytes > 0) { string dictionary; dictionary.resize(hdr.zstd_dictionary_length_bytes); complete_pread(fd, &dictionary[0], hdr.zstd_dictionary_length_bytes, hdr.zstd_dictionary_offset_bytes); ddict = ZSTD_createDDict(dictionary.data(), dictionary.size()); } } AccessRXCache access_rx_cache(nullptr, corpus.get_hdr().check_visibility); Serializer serializer; uint32_t num_blocks = corpus.get_num_filename_blocks(); unique_ptr offsets(new uint64_t[num_blocks + 1]); complete_pread(fd, offsets.get(), (num_blocks + 1) * sizeof(uint64_t), corpus.offset_for_block(0)); atomic matched{ 0 }; mutex mu; condition_variable queue_added, queue_removed; deque> work_queue; // Under mu. bool done = false; // Under mu. unsigned num_threads = max(sysconf(_SC_NPROCESSORS_ONLN) - 1, 1); dprintf("Using %u worker threads for linear scan.\n", num_threads); unique_ptr threads(new WorkerThread[num_threads]); for (unsigned i = 0; i < num_threads; ++i) { threads[i].t = thread([&threads, &mu, &queue_added, &queue_removed, &work_queue, &done, &offsets, &needles, &access_rx_cache, &matched, i] { // regcomp() takes a lock on the regex, so each thread will need its own. const vector *use_needles = &needles; vector recompiled_needles; if (i != 0 && patterns_are_regex) { recompiled_needles = needles; for (Needle &needle : recompiled_needles) { needle.re = compile_regex(needle.str); } use_needles = &recompiled_needles; } WorkerThreadReceiver receiver(&threads[i]); for (;;) { uint32_t io_docid, last_docid; string compressed; { unique_lock lock(mu); queue_added.wait(lock, [&work_queue, &done] { return !work_queue.empty() || done; }); if (done && work_queue.empty()) { return; } tie(io_docid, last_docid, compressed) = move(work_queue.front()); work_queue.pop_front(); queue_removed.notify_all(); } for (uint32_t docid = io_docid; docid < last_docid; ++docid) { size_t relative_offset = offsets[docid] - offsets[io_docid]; size_t len = offsets[docid + 1] - offsets[docid]; // IOUringEngine isn't thread-safe, so we do any needed stat()s synchronously (nullptr engine). scan_file_block(*use_needles, { &compressed[relative_offset], len }, /*engine=*/nullptr, &access_rx_cache, docid, &receiver, &matched); } } }); } string compressed; for (uint32_t io_docid = 0; io_docid < num_blocks; io_docid += 32) { uint32_t last_docid = std::min(io_docid + 32, num_blocks); size_t io_len = offsets[last_docid] - offsets[io_docid]; if (compressed.size() < io_len) { compressed.resize(io_len); } complete_pread(fd, &compressed[0], io_len, offsets[io_docid]); { unique_lock lock(mu); queue_removed.wait(lock, [&work_queue] { return work_queue.size() < 256; }); // Allow ~2MB of data queued up. work_queue.emplace_back(io_docid, last_docid, move(compressed)); queue_added.notify_one(); // Avoid the thundering herd. } // Pick up some results, so that we are sure that we won't just overload. // (Seemingly, going through all of these causes slowness with many threads, // but taking only one is OK.) unsigned i = io_docid / 32; deliver_results(&threads[i % num_threads], &serializer); } { lock_guard lock(mu); done = true; queue_added.notify_all(); } for (unsigned i = 0; i < num_threads; ++i) { threads[i].t.join(); deliver_results(&threads[i], &serializer); } return matched; } // Takes the given posting list, unions it into the parts of the trigram disjunction // already read; if the list is complete, intersects with “cur_candidates”. // // Returns true if the search should be aborted (we are done). bool new_posting_list_read(TrigramDisjunction *td, vector decoded, vector *cur_candidates, vector *tmp) { if (td->docids.empty()) { td->docids = move(decoded); } else { tmp->clear(); set_union(decoded.begin(), decoded.end(), td->docids.begin(), td->docids.end(), back_inserter(*tmp)); swap(*tmp, td->docids); } if (--td->remaining_trigrams_to_read > 0) { // Need to wait for more. if (ignore_case) { dprintf(" ... %u reads left in OR group %u (%zu docids in list)\n", td->remaining_trigrams_to_read, td->index, td->docids.size()); } return false; } if (cur_candidates->empty()) { if (ignore_case) { dprintf(" ... all reads done for OR group %u (%zu docids)\n", td->index, td->docids.size()); } *cur_candidates = move(td->docids); } else { tmp->clear(); set_intersection(cur_candidates->begin(), cur_candidates->end(), td->docids.begin(), td->docids.end(), back_inserter(*tmp)); swap(*cur_candidates, *tmp); if (ignore_case) { if (cur_candidates->empty()) { dprintf(" ... all reads done for OR group %u (%zu docids), intersected (none left, search is done)\n", td->index, td->docids.size()); return true; } else { dprintf(" ... all reads done for OR group %u (%zu docids), intersected (%zu left)\n", td->index, td->docids.size(), cur_candidates->size()); } } } return false; } uint64_t do_search_file(const vector &needles, const std::string &filename) { int fd = open(filename.c_str(), O_RDONLY); if (fd == -1) { perror(filename.c_str()); exit(1); } // Drop privileges. if (setgid(getgid()) != 0) { perror("setgid"); exit(EXIT_FAILURE); } start = steady_clock::now(); if (access("/", R_OK | X_OK)) { // We can't find anything, no need to bother... return 0; } IOUringEngine engine(/*slop_bytes=*/16); // 16 slop bytes as described in turbopfor.h. Corpus corpus(fd, filename.c_str(), &engine); dprintf("Corpus init done after %.1f ms.\n", 1e3 * duration(steady_clock::now() - start).count()); vector trigram_groups; if (patterns_are_regex) { // We could parse the regex to find trigrams that have to be there // (there are actually known algorithms to deal with disjunctions // and such, too), but for now, we just go brute force. // Using locate with regexes is pretty niche. } else { for (const Needle &needle : needles) { parse_trigrams(needle.str, ignore_case, &trigram_groups); } } unique_sort( &trigram_groups, [](const TrigramDisjunction &a, const TrigramDisjunction &b) { return a.trigram_alternatives < b.trigram_alternatives; }, [](const TrigramDisjunction &a, const TrigramDisjunction &b) { return a.trigram_alternatives == b.trigram_alternatives; }); // Give them names for debugging. unsigned td_index = 0; for (TrigramDisjunction &td : trigram_groups) { td.index = td_index++; } // Collect which trigrams we need to look up in the hash table. unordered_map> trigrams_to_lookup; for (TrigramDisjunction &td : trigram_groups) { for (uint32_t trgm : td.trigram_alternatives) { trigrams_to_lookup[trgm].push_back(&td); } } if (trigrams_to_lookup.empty()) { // Too short for trigram matching. Apply brute force. // (We could have searched through all trigrams that matched // the pattern and done a union of them, but that's a lot of // work for fairly unclear gain.) uint64_t matched = scan_all_docids(needles, fd, corpus); dprintf("Done in %.1f ms, found %" PRId64 " matches.\n", 1e3 * duration(steady_clock::now() - start).count(), matched); return matched; } // Sneak in fetching the dictionary, if present. It's not necessarily clear // exactly where it would be cheapest to get it, but it needs to be present // before we can decode any of the posting lists. Most likely, it's // in the same filesystem block as the header anyway, so it should be // present in the cache. { const Header &hdr = corpus.get_hdr(); if (hdr.zstd_dictionary_length_bytes > 0) { engine.submit_read(fd, hdr.zstd_dictionary_length_bytes, hdr.zstd_dictionary_offset_bytes, [](string_view s) { ddict = ZSTD_createDDict(s.data(), s.size()); dprintf("Dictionary initialized after %.1f ms.\n", 1e3 * duration(steady_clock::now() - start).count()); }); } } // Look them all up on disk. bool should_early_exit = false; for (auto &[trgm, trigram_groups] : trigrams_to_lookup) { corpus.find_trigram(trgm, [trgm{ trgm }, trigram_groups{ &trigram_groups }, &should_early_exit](const Trigram *trgmptr, size_t len) { if (trgmptr == nullptr) { dprintf("trigram %s isn't found\n", print_trigram(trgm).c_str()); for (TrigramDisjunction *td : *trigram_groups) { --td->remaining_trigrams_to_read; // If we now know this trigram group doesn't match anything at all, // we can do early exit; however, if we're in a forked child, // that would confuse the parent process (since we don't write // our count to the pipe), so we wait until we're back in to the // regular (non-async) context. This is a fairly rare case anyway, // and the gains from dropping the remaining trigram reads are limited. if (td->remaining_trigrams_to_read == 0 && td->read_trigrams.empty()) { if (in_forked_child) { should_early_exit = true; } else { dprintf("zero matches in %s, so we are done\n", print_td(*td).c_str()); if (only_count) { printf("0\n"); } exit(0); } } } return; } for (TrigramDisjunction *td : *trigram_groups) { --td->remaining_trigrams_to_read; td->max_num_docids += trgmptr->num_docids; td->read_trigrams.emplace_back(*trgmptr, len); } }); } engine.finish(); dprintf("Hashtable lookups done after %.1f ms.\n", 1e3 * duration(steady_clock::now() - start).count()); if (should_early_exit) { return 0; } for (TrigramDisjunction &td : trigram_groups) { // Reset for reads. td.remaining_trigrams_to_read = td.read_trigrams.size(); if (ignore_case) { // If case-sensitive, they'll all be pretty obvious single-entry groups. dprintf("OR group %u (max_num_docids=%u): %s\n", td.index, td.max_num_docids, print_td(td).c_str()); } } // TODO: For case-insensitive (ie. more than one alternative in each), // prioritize the ones with fewer seeks? sort(trigram_groups.begin(), trigram_groups.end(), [&](const TrigramDisjunction &a, const TrigramDisjunction &b) { return a.max_num_docids < b.max_num_docids; }); unordered_map> uses_trigram; for (TrigramDisjunction &td : trigram_groups) { for (uint32_t trgm : td.trigram_alternatives) { uses_trigram[trgm].push_back(&td); } } unordered_set trigrams_submitted_read; vector cur_candidates, tmp, decoded; bool done = false; for (TrigramDisjunction &td : trigram_groups) { if (!cur_candidates.empty() && td.max_num_docids > cur_candidates.size() * 100) { dprintf("%s has up to %u entries, ignoring the rest (will " "weed out false positives later)\n", print_td(td).c_str(), td.max_num_docids); break; } for (auto &[trgmptr, len] : td.read_trigrams) { if (trigrams_submitted_read.count(trgmptr.trgm) != 0) { continue; } trigrams_submitted_read.insert(trgmptr.trgm); // Only stay a certain amount ahead, so that we don't spend I/O // on reading the latter, large posting lists. We are unlikely // to need them anyway, even if they should come in first. if (engine.get_waiting_reads() >= 5) { engine.finish(); if (done) break; } engine.submit_read(fd, len, trgmptr.offset, [trgmptr{ trgmptr }, len{ len }, &done, &cur_candidates, &tmp, &decoded, &uses_trigram](string_view s) { if (done) return; uint32_t trgm = trgmptr.trgm; const unsigned char *pldata = reinterpret_cast(s.data()); size_t num = trgmptr.num_docids; decoded.resize(num); decode_pfor_delta1_128(pldata, num, /*interleaved=*/true, &decoded[0]); assert(uses_trigram.count(trgm) != 0); bool was_empty = cur_candidates.empty(); if (ignore_case) { dprintf("trigram %s (%zu bytes) decoded to %zu entries\n", print_trigram(trgm).c_str(), len, num); } for (TrigramDisjunction *td : uses_trigram[trgm]) { done |= new_posting_list_read(td, decoded, &cur_candidates, &tmp); if (done) break; } if (!ignore_case) { if (was_empty) { dprintf("trigram %s (%zu bytes) decoded to %zu entries\n", print_trigram(trgm).c_str(), len, num); } else if (cur_candidates.empty()) { dprintf("trigram %s (%zu bytes) decoded to %zu entries (none left, search is done)\n", print_trigram(trgm).c_str(), len, num); } else { dprintf("trigram %s (%zu bytes) decoded to %zu entries (%zu left)\n", print_trigram(trgm).c_str(), len, num, cur_candidates.size()); } } }); } } engine.finish(); if (done) { return 0; } dprintf("Intersection done after %.1f ms. Doing final verification and printing:\n", 1e3 * duration(steady_clock::now() - start).count()); uint64_t matched = scan_docids(needles, cur_candidates, corpus, &engine); dprintf("Done in %.1f ms, found %" PRId64 " matches.\n", 1e3 * duration(steady_clock::now() - start).count(), matched); return matched; } // Run do_search_file() in a child process. // // The reason for this is that we're not robust against malicious input, so we need // to drop privileges after opening the file. (Otherwise, we could fall prey to an attack // where a user does locate -d badfile.db:/var/lib/plocate/plocate.db, badfile.db contains // a buffer overflow that takes over the process, and then uses the elevated privileges // to print out otherwise inaccessible paths.) We solve this by forking and treating the // child process as untrusted after it has dropped its privileges (which it does before // reading any data from the file); it returns a single 64-bit number over a pipe, // and that's it. The parent keeps its privileges, and can then fork out new children // without fear of being taken over. (The child keeps stdout for outputting results.) // // The count is returned over the pipe, because it's needed both for --limit and --count. uint64_t do_search_file_in_child(const vector &needles, const std::string &filename) { int pipefd[2]; if (pipe(pipefd) == -1) { perror("pipe"); exit(EXIT_FAILURE); } pid_t child_pid = fork(); switch (child_pid) { case 0: { // Child. close(pipefd[0]); in_forked_child = true; uint64_t matched = do_search_file(needles, filename); int ret; do { ret = write(pipefd[1], &matched, sizeof(matched)); } while (ret == -1 && errno == EINTR); if (ret != sizeof(matched)) { perror("write"); _exit(EXIT_FAILURE); } fflush(stdout); _exit(EXIT_SUCCESS); } case -1: // Error. perror("fork"); exit(EXIT_FAILURE); default: // Parent. close(pipefd[1]); break; } // Wait for the child to finish. int wstatus; pid_t err; do { err = waitpid(child_pid, &wstatus, 0); } while (err == -1 && errno == EINTR); if (err == -1) { perror("waitpid"); exit(EXIT_FAILURE); } if (WIFEXITED(wstatus)) { if (WEXITSTATUS(wstatus) != 0) { // The child has probably already printed out its error, so just propagate the exit status. exit(WEXITSTATUS(wstatus)); } // Success! } else if (!WIFEXITED(wstatus)) { fprintf(stderr, "FATAL: Child died unexpectedly while processing %s\n", filename.c_str()); exit(1); } // Now get the number of matches from the child. uint64_t matched; int ret; do { ret = read(pipefd[0], &matched, sizeof(matched)); } while (ret == -1 && errno == EINTR); if (ret == -1) { perror("read"); exit(EXIT_FAILURE); } else if (ret != sizeof(matched)) { fprintf(stderr, "FATAL: Short read through pipe (got %d bytes)\n", ret); exit(EXIT_FAILURE); } close(pipefd[0]); return matched; } // Parses a colon-separated list of strings and appends them onto the given vector. // Backslash escapes whatever comes after it. void parse_dbpaths(const char *ptr, vector *output) { string str; while (*ptr != '\0') { if (*ptr == '\\') { if (ptr[1] == '\0') { fprintf(stderr, "ERROR: Escape character at the end of string\n"); exit(EXIT_FAILURE); } // Escape. str.push_back(ptr[1]); ptr += 2; continue; } if (*ptr == ':') { // Separator. output->push_back(move(str)); ++ptr; continue; } str.push_back(*ptr++); } output->push_back(move(str)); } void usage() { printf( "Usage: plocate [OPTION]... PATTERN...\n" "\n" " -b, --basename search only the file name portion of path names\n" " -c, --count print number of matches instead of the matches\n" " -d, --database DBPATH search for files in DBPATH\n" " (default is " DBFILE ")\n" " -i, --ignore-case search case-insensitively\n" " -l, --limit LIMIT stop after LIMIT matches\n" " -0, --null delimit matches by NUL instead of newline\n" " -N, --literal do not quote filenames, even if printing to a tty\n" " -r, --regexp interpret patterns as basic regexps (slow)\n" " --regex interpret patterns as extended regexps (slow)\n" " -w, --wholename search the entire path name (default; see -b)\n" " --help print this help\n" " --version print version information\n"); } void version() { printf("%s %s\n", PACKAGE_NAME, PACKAGE_VERSION); printf("Copyright 2020 Steinar H. Gunderson\n"); printf("License GPLv2+: GNU GPL version 2 or later .\n"); printf("This is free software: you are free to change and redistribute it.\n"); printf("There is NO WARRANTY, to the extent permitted by law.\n"); exit(0); } int main(int argc, char **argv) { vector dbpaths; constexpr int EXTENDED_REGEX = 1000; constexpr int FLUSH_CACHE = 1001; static const struct option long_options[] = { { "help", no_argument, 0, 'h' }, { "count", no_argument, 0, 'c' }, { "all", no_argument, 0, 'A' }, { "basename", no_argument, 0, 'b' }, { "database", required_argument, 0, 'd' }, { "existing", no_argument, 0, 'e' }, { "ignore-case", no_argument, 0, 'i' }, { "limit", required_argument, 0, 'l' }, { "literal", no_argument, 0, 'N' }, { "null", no_argument, 0, '0' }, { "version", no_argument, 0, 'V' }, { "regexp", no_argument, 0, 'r' }, { "regex", no_argument, 0, EXTENDED_REGEX }, { "wholename", no_argument, 0, 'w' }, { "debug", no_argument, 0, 'D' }, // Not documented. // Enable to test cold-cache behavior (except for access()). Not documented. { "flush-cache", no_argument, 0, FLUSH_CACHE }, { 0, 0, 0, 0 } }; setlocale(LC_ALL, ""); for (;;) { int option_index = 0; int c = getopt_long(argc, argv, "Abcd:ehil:n:N0rwVD", long_options, &option_index); if (c == -1) { break; } switch (c) { case 'A': // Ignored. break; case 'b': match_basename = true; break; case 'c': only_count = true; break; case 'd': parse_dbpaths(optarg, &dbpaths); break; case 'e': check_existence = true; break; case 'h': usage(); exit(0); case 'i': ignore_case = true; break; case 'l': case 'n': limit_matches = limit_left = atoll(optarg); if (limit_matches <= 0) { fprintf(stderr, "Error: limit must be a strictly positive number.\n"); exit(1); } break; case 'N': literal_printing = true; break; case '0': print_nul = true; break; case 'r': patterns_are_regex = true; break; case EXTENDED_REGEX: patterns_are_regex = true; use_extended_regex = true; break; case 'w': match_basename = false; // No-op unless -b is given first. break; case 'D': use_debug = true; break; case FLUSH_CACHE: flush_cache = true; break; case 'V': version(); break; default: exit(1); } } if (use_debug || flush_cache) { // Debug information would leak information about which files exist, // so drop setgid before we open the file; one would either need to run // as root, or use a locally-built file. Doing the same thing for // flush_cache is mostly paranoia, in an attempt to prevent random users // from making plocate slow for everyone else. if (setgid(getgid()) != 0) { perror("setgid"); exit(EXIT_FAILURE); } } if (!print_nul) { stdout_is_tty = isatty(1); } vector needles; for (int i = optind; i < argc; ++i) { Needle needle; needle.str = argv[i]; // See if there are any wildcard characters, which indicates we should treat it // as an (anchored) glob. bool any_wildcard = false; for (size_t i = 0; i < needle.str.size(); i += read_unigram(needle.str, i).second) { if (read_unigram(needle.str, i).first == WILDCARD_UNIGRAM) { any_wildcard = true; break; } } if (patterns_are_regex) { needle.type = Needle::REGEX; needle.re = compile_regex(needle.str); } else if (any_wildcard) { needle.type = Needle::GLOB; } else if (ignore_case) { // strcasestr() doesn't handle locales correctly (even though LSB // claims it should), but somehow, fnmatch() does, and it's about // the same speed as using a regex. needle.type = Needle::GLOB; needle.str = "*" + needle.str + "*"; } else { needle.type = Needle::STRSTR; needle.str = unescape_glob_to_plain_string(needle.str); } needles.push_back(move(needle)); } if (needles.empty()) { fprintf(stderr, "plocate: no pattern to search for specified\n"); exit(0); } if (dbpaths.empty()) { // No -d given, so use our default. Note that this happens // even if LOCATE_PATH exists, to match mlocate behavior. dbpaths.push_back(DBFILE); } const char *locate_path = getenv("LOCATE_PATH"); if (locate_path != nullptr) { parse_dbpaths(locate_path, &dbpaths); } uint64_t matched = 0; for (size_t i = 0; i < dbpaths.size(); ++i) { uint64_t this_matched; if (i != dbpaths.size() - 1) { this_matched = do_search_file_in_child(needles, dbpaths[i]); } else { this_matched = do_search_file(needles, dbpaths[i]); } matched += this_matched; limit_left -= this_matched; } if (only_count) { printf("%" PRId64 "\n", matched); } } plocate-1.1.15/serializer.cpp000066400000000000000000000072061417604322600161150ustar00rootroot00000000000000#include "serializer.h" #include "dprintf.h" #include #include #include #include #include #include using namespace std; using namespace std::chrono; extern steady_clock::time_point start; void apply_limit() { if (--limit_left > 0) { return; } dprintf("Done in %.1f ms, found %" PRId64 " matches.\n", 1e3 * duration(steady_clock::now() - start).count(), limit_matches); if (only_count) { printf("%" PRId64 "\n", limit_matches); } exit(0); } void print_possibly_escaped(const string &str) { if (print_nul) { printf("%s%c", str.c_str(), 0); return; } else if (literal_printing || !stdout_is_tty) { printf("%s\n", str.c_str()); return; } // stdout is a terminal, so we should protect the user against // escapes, stray newlines and the likes. First of all, check if // all the characters are safe; we consider everything safe that // isn't a control character, ', " or \. People could make // filenames like "$(rm -rf)", but that's out-of-scope. const char *ptr = str.data(); size_t len = str.size(); mbtowc(nullptr, 0, 0); wchar_t pwc; bool all_safe = true; do { int ret = mbtowc(&pwc, ptr, len); if (ret == -1) { all_safe = false; // Malformed data. } else if (ret == 0) { break; // EOF. } else if (pwc < 32 || pwc == '\'' || pwc == '"' || pwc == '\\') { all_safe = false; } else if (pwc == '`') { // A rather odd case; ls quotes this but does not escape it. all_safe = false; } else { ptr += ret; len -= ret; } } while (all_safe && *ptr != '\0'); if (all_safe) { printf("%s\n", str.c_str()); return; } // Print escaped, but in such a way that the user can easily take the // escaped output and paste into the shell. We print much like GNU ls does, // ie., using the shell $'foo' construct whenever we need to print something // escaped. bool in_escaped_mode = false; printf("'"); mbtowc(nullptr, 0, 0); ptr = str.data(); len = str.size(); while (*ptr != '\0') { int ret = mbtowc(nullptr, ptr, len); if (ret == -1) { // Malformed data. printf("?"); ++ptr; --len; continue; } else if (ret == 0) { break; // EOF. } if ((unsigned char)*ptr < 32 || *ptr == '\'' || *ptr == '"' || *ptr == '\\') { if (!in_escaped_mode) { printf("'$'"); in_escaped_mode = true; } // The list of allowed escapes is from bash(1). switch (*ptr) { case '\a': printf("\\a"); break; case '\b': printf("\\b"); break; case '\f': printf("\\f"); break; case '\n': printf("\\n"); break; case '\r': printf("\\r"); break; case '\t': printf("\\t"); break; case '\v': printf("\\v"); break; case '\\': printf("\\\\"); break; case '\'': printf("\\'"); break; case '"': printf("\\\""); break; default: printf("\\%03o", *ptr); break; } } else { if (in_escaped_mode) { printf("''"); in_escaped_mode = false; } fwrite(ptr, ret, 1, stdout); } ptr += ret; len -= ret; } printf("'\n"); } void Serializer::print(uint64_t seq, uint64_t skip, const string msg) { if (only_count) { if (!msg.empty()) { apply_limit(); } return; } if (next_seq != seq) { pending.push(Element{ seq, skip, move(msg) }); return; } if (!msg.empty()) { print_possibly_escaped(msg); apply_limit(); } next_seq += skip; // See if any delayed prints can now be dealt with. while (!pending.empty() && pending.top().seq == next_seq) { if (!pending.top().msg.empty()) { print_possibly_escaped(pending.top().msg); apply_limit(); } next_seq += pending.top().skip; pending.pop(); } } plocate-1.1.15/serializer.h000066400000000000000000000013431417604322600155560ustar00rootroot00000000000000#ifndef _SERIALIZER_H #define _SERIALIZER_H 1 #include "options.h" #include #include #include #include class ResultReceiver { public: virtual ~ResultReceiver() = default; virtual void print(uint64_t seq, uint64_t skip, const std::string msg) = 0; }; class Serializer : public ResultReceiver { public: ~Serializer() { assert(limit_left <= 0 || pending.empty()); } void print(uint64_t seq, uint64_t skip, const std::string msg) override; private: uint64_t next_seq = 0; struct Element { uint64_t seq, skip; std::string msg; bool operator<(const Element &other) const { return seq > other.seq; } }; std::priority_queue pending; }; #endif // !defined(_SERIALIZER_H) plocate-1.1.15/turbopfor-common.h000066400000000000000000000013061417604322600167140ustar00rootroot00000000000000#ifndef _TURBOPFOR_COMMON_H #define _TURBOPFOR_COMMON_H 1 // Common definitions and utilities between turbopfor.h (decode) // and turbopfor-encode.h (encode). #include enum BlockType { FOR = 0, PFOR_VB = 1, PFOR_BITMAP = 2, CONSTANT = 3 }; // Does not properly account for overflow. inline unsigned div_round_up(unsigned val, unsigned div) { return (val + div - 1) / div; } inline unsigned bytes_for_packed_bits(unsigned num, unsigned bit_width) { return div_round_up(num * bit_width, CHAR_BIT); } constexpr uint32_t mask_for_bits(unsigned bit_width) { if (bit_width == 32) { return 0xFFFFFFFF; } else { return (1U << bit_width) - 1; } } #endif // !defined(_TURBOPFOR_COMMON_H) plocate-1.1.15/turbopfor-encode.h000066400000000000000000000260611417604322600166660ustar00rootroot00000000000000#ifndef _TURBOPFOR_ENCODE_H #define _TURBOPFOR_ENCODE_H // Much like turbopfor.h (and shares all of the same caveats), except this is // for encoding. It is _much_ slower than the reference implementation, but we // encode only during build, and most time in build is spent in other things // than encoding posting lists, so it only costs ~5-10% overall. Does not use // any special instruction sets, and generally isn't optimized at all. // // It encodes roughly about as dense as the reference encoder. #include "turbopfor-common.h" #include #include #ifdef HAS_ENDIAN_H #include #endif #include #include #include template void write_le(Docid val, void *out) { if constexpr (sizeof(Docid) == 8) { val = htole64(val); } else if constexpr (sizeof(Docid) == 4) { val = htole32(val); } else if constexpr (sizeof(Docid) == 2) { val = htole16(val); } else if constexpr (sizeof(Docid) == 1) { // No change. } else { assert(false); } memcpy(out, &val, sizeof(val)); } // Corresponds to read_baseval. template unsigned char *write_baseval(Docid in, unsigned char *out) { if (in < 128) { *out = in; return out + 1; } else if (in < 0x4000) { out[0] = (in >> 8) | 0x80; out[1] = in & 0xff; return out + 2; } else if (in < 0x200000) { out[0] = (in >> 16) | 0xc0; out[1] = in & 0xff; out[2] = (in >> 8) & 0xff; return out + 3; } else if (in < 0x10000000) { out[0] = (in >> 24) | 0xe0; out[1] = (in >> 16) & 0xff; out[2] = (in >> 8) & 0xff; out[3] = in & 0xff; return out + 4; } else { assert(false); // Not implemented. } } // Writes a varbyte-encoded exception. template unsigned char *write_vb(Docid val, unsigned char *out) { if (val <= 176) { *out++ = val; return out; } else if (val <= 16560) { val -= 177; *out++ = (val >> 8) + 177; *out++ = val & 0xff; return out; } else if (val <= 540848) { val -= 16561; *out = (val >> 16) + 241; write_le(val & 0xffff, out + 1); return out + 3; } else if (val <= 16777215) { *out = 249; write_le(val, out + 1); return out + 4; } else { *out = 250; write_le(val, out + 1); return out + 5; } } template inline unsigned num_bits(Docid x) { #ifdef __GNUC__ if (x == 0) { return 0; } else { return sizeof(Docid) * CHAR_BIT - __builtin_clz(x); } #else for (int i = sizeof(Docid) * CHAR_BIT; i-- > 0;) { if (x & (Docid{ 1 } << i)) { return i; } } return 0; #endif } struct BitWriter { public: BitWriter(unsigned char *out, unsigned bits) : out(out), bits(bits) {} void write(uint32_t val) { cur_val |= val << bits_used; write_le(cur_val, out); bits_used += bits; cur_val >>= (bits_used / 8) * 8; out += bits_used / 8; bits_used %= 8; } private: unsigned char *out; const unsigned bits; unsigned bits_used = 0; unsigned cur_val = 0; }; template struct InterleavedBitWriter { public: InterleavedBitWriter(unsigned char *out, unsigned bits) : out(out), bits(bits) {} void write(uint32_t val) { cur_val |= uint64_t(val) << bits_used; if (bits_used + bits >= 32) { write_le(cur_val & 0xffffffff, out); out += Stride; cur_val >>= 32; bits_used -= 32; // Underflow, but will be fixed below. } write_le(cur_val, out); bits_used += bits; } private: static constexpr unsigned Stride = NumStreams * sizeof(uint32_t); unsigned char *out; const unsigned bits; unsigned bits_used = 0; uint64_t cur_val = 0; }; // Bitpacks a set of values (making sure the top bits are lopped off). // If interleaved is set, makes SSE2-compatible interleaving (this is // only allowed for full blocks). template unsigned char *encode_bitmap(const Docid *in, unsigned num, unsigned bit_width, bool interleaved, unsigned char *out) { unsigned mask = mask_for_bits(bit_width); if (interleaved) { InterleavedBitWriter<4> bs0(out + 0 * sizeof(uint32_t), bit_width); InterleavedBitWriter<4> bs1(out + 1 * sizeof(uint32_t), bit_width); InterleavedBitWriter<4> bs2(out + 2 * sizeof(uint32_t), bit_width); InterleavedBitWriter<4> bs3(out + 3 * sizeof(uint32_t), bit_width); assert(num % 4 == 0); for (unsigned i = 0; i < num / 4; ++i) { bs0.write(in[i * 4 + 0] & mask); bs1.write(in[i * 4 + 1] & mask); bs2.write(in[i * 4 + 2] & mask); bs3.write(in[i * 4 + 3] & mask); } } else { BitWriter bs(out, bit_width); for (unsigned i = 0; i < num; ++i) { bs.write(in[i] & mask); } } return out + bytes_for_packed_bits(num, bit_width); } // See decode_for() for the format. template unsigned char *encode_for(const Docid *in, unsigned num, unsigned bit_width, bool interleaved, unsigned char *out) { return encode_bitmap(in, num, bit_width, interleaved, out); } // See decode_pfor_bitmap() for the format. template unsigned char *encode_pfor_bitmap(const Docid *in, unsigned num, unsigned bit_width, unsigned exception_bit_width, bool interleaved, unsigned char *out) { *out++ = exception_bit_width; // Bitmap of exceptions. { BitWriter bs(out, 1); for (unsigned i = 0; i < num; ++i) { bs.write((in[i] >> bit_width) != 0); } out += bytes_for_packed_bits(num, 1); } // Exceptions. { BitWriter bs(out, exception_bit_width); unsigned num_exceptions = 0; for (unsigned i = 0; i < num; ++i) { if ((in[i] >> bit_width) != 0) { bs.write(in[i] >> bit_width); ++num_exceptions; } } out += bytes_for_packed_bits(num_exceptions, exception_bit_width); } // Base values. out = encode_bitmap(in, num, bit_width, interleaved, out); return out; } // See decode_pfor_vb() for the format. template unsigned char *encode_pfor_vb(const Docid *in, unsigned num, unsigned bit_width, bool interleaved, unsigned char *out) { unsigned num_exceptions = 0; for (unsigned i = 0; i < num; ++i) { if ((in[i] >> bit_width) != 0) { ++num_exceptions; } } *out++ = num_exceptions; // Base values. out = encode_bitmap(in, num, bit_width, interleaved, out); // Exceptions. for (unsigned i = 0; i < num; ++i) { unsigned val = in[i] >> bit_width; if (val != 0) { out = write_vb(val, out); } } // Exception indexes. for (unsigned i = 0; i < num; ++i) { unsigned val = in[i] >> bit_width; if (val != 0) { *out++ = i; } } return out; } // Find out which block type would be the smallest for the given data. template BlockType decide_block_type(const Docid *in, unsigned num, unsigned *bit_width, unsigned *exception_bit_width) { // Check if the block is constant. bool constant = true; for (unsigned i = 1; i < num; ++i) { if (in[i] != in[0]) { constant = false; break; } } if (constant) { *bit_width = num_bits(in[0]); return BlockType::CONSTANT; } // Build up a histogram of bit sizes. unsigned histogram[sizeof(Docid) * CHAR_BIT + 1] = { 0 }; unsigned max_bits = 0; for (unsigned i = 0; i < num; ++i) { unsigned bits = num_bits(in[i]); ++histogram[bits]; max_bits = std::max(max_bits, bits); } // Straight-up FOR. unsigned best_cost = bytes_for_packed_bits(num, max_bits); unsigned best_bit_width = max_bits; // Try PFOR with bitmap exceptions. const unsigned bitmap_cost = bytes_for_packed_bits(num, 1); unsigned num_exceptions = 0; for (unsigned exception_bit_width = 1; exception_bit_width <= max_bits; ++exception_bit_width) { unsigned test_bit_width = max_bits - exception_bit_width; num_exceptions += histogram[test_bit_width + 1]; // 1 byte for signaling exception bit width, then the bitmap, // then the base values, then the exceptions. unsigned cost = 1 + bitmap_cost + bytes_for_packed_bits(num, test_bit_width) + bytes_for_packed_bits(num_exceptions, exception_bit_width); if (cost < best_cost) { best_cost = cost; best_bit_width = test_bit_width; } } // Make the histogram cumulative; histogram[n] now means the the number of // elements with n bits or more. for (unsigned i = max_bits; i > 0; --i) { histogram[i - 1] += histogram[i]; } // Try PFOR with varbyte exceptions. bool best_is_varbyte = false; for (unsigned test_bit_width = 0; test_bit_width < max_bits; ++test_bit_width) { // 1 byte for signaling number of exceptions, plus the base values, // and then we estimate up the varbytes and indexes using histogram // indexes. This isn't exact, but it only helps ~0.1% on the total // plocate.db size. unsigned cost = 1 + bytes_for_packed_bits(num, test_bit_width); if (cost >= best_cost) { break; } if (test_bit_width + 1 <= max_bits) { cost += 2 * histogram[test_bit_width + 1]; // > 0. if (test_bit_width + 7 <= max_bits) { cost += histogram[test_bit_width + 7]; // > 176, very roughly. if (test_bit_width + 14 <= max_bits) { cost += histogram[test_bit_width + 14]; // > 16560, very roughly. if (test_bit_width + 19 <= max_bits) { cost += histogram[test_bit_width + 19]; // > 540848, very roughly. if (test_bit_width + 24 <= max_bits) { cost += histogram[test_bit_width + 24]; // > 16777215, very roughly. } } } } } if (cost < best_cost) { best_cost = cost; best_bit_width = test_bit_width; best_is_varbyte = true; } } // TODO: Consider the last-resort option of just raw storage (255). if (best_is_varbyte) { *bit_width = best_bit_width; return BlockType::PFOR_VB; } else if (best_bit_width == max_bits) { *bit_width = max_bits; return BlockType::FOR; } else { *bit_width = best_bit_width; *exception_bit_width = max_bits - best_bit_width; return BlockType::PFOR_BITMAP; } } // The basic entry point. Takes one block of integers (which already must // be delta-minus-1-encoded) and packs it into TurboPFor format. // interleaved corresponds to the interleaved parameter in decode_pfor_delta1() // or the “128v” infix in the reference code's function names; such formats // are much faster to decode, so for full blocks, you probably want it. // The interleaved flag isn't stored anywhere; it's implicit whether you // want to use it for full blocks or not. // // The first value must already be written using write_baseval() (so the delta // coding starts from the second value). Returns the end of the string. // May write 4 bytes past the end. template unsigned char *encode_pfor_single_block(const Docid *in, unsigned num, bool interleaved, unsigned char *out) { assert(num > 0); if (interleaved) { assert(num == BlockSize); } unsigned bit_width, exception_bit_width; BlockType block_type = decide_block_type(in, num, &bit_width, &exception_bit_width); *out++ = (block_type << 6) | bit_width; switch (block_type) { case BlockType::CONSTANT: { unsigned bit_width = num_bits(in[0]); write_le(in[0], out); return out + div_round_up(bit_width, 8); } case BlockType::FOR: return encode_for(in, num, bit_width, interleaved, out); case BlockType::PFOR_BITMAP: return encode_pfor_bitmap(in, num, bit_width, exception_bit_width, interleaved, out); case BlockType::PFOR_VB: return encode_pfor_vb(in, num, bit_width, interleaved, out); default: assert(false); } } #endif // !defined(_TURBOPFOR_ENCODE_H) plocate-1.1.15/turbopfor.cpp000066400000000000000000000630221417604322600157640ustar00rootroot00000000000000#include #include #ifdef HAS_ENDIAN_H #include #endif #include #include #include // This is a mess. :-/ Maybe it would be good just to drop support for // multiversioning; the only platform it really helps is 32-bit x86. // This may change if we decide to use AVX or similar in the future, though. #if defined(__i386__) || defined(__x86_64__) #ifdef __SSE2__ #define COULD_HAVE_SSE2 #define SUPPRESS_DEFAULT #include #define TARGET_SSE2 #elif defined(HAS_FUNCTION_MULTIVERSIONING) #define COULD_HAVE_SSE2 #include #define TARGET_SSE2 __attribute__((target("sse2"))) #define TARGET_DEFAULT __attribute__((target("default"))) #else #define TARGET_DEFAULT #endif #else // Function multiversioning is x86-only. #define TARGET_DEFAULT #endif #include "turbopfor-common.h" #define dprintf(...) //#define dprintf(...) fprintf(stderr, __VA_ARGS__); #ifndef SUPPRESS_DEFAULT // Forward declarations to declare to the template code below that they exist. // (These must seemingly be non-templates for function multiversioning to work.) TARGET_DEFAULT const unsigned char * decode_for_interleaved_128_32(const unsigned char *in, uint32_t *out); TARGET_DEFAULT const unsigned char * decode_pfor_bitmap_interleaved_128_32(const unsigned char *in, uint32_t *out); TARGET_DEFAULT const unsigned char * decode_pfor_vb_interleaved_128_32(const unsigned char *in, uint32_t *out); #endif #ifdef COULD_HAVE_SSE2 TARGET_SSE2 const unsigned char * decode_for_interleaved_128_32(const unsigned char *in, uint32_t *out); TARGET_SSE2 const unsigned char * decode_pfor_bitmap_interleaved_128_32(const unsigned char *in, uint32_t *out); TARGET_SSE2 const unsigned char * decode_pfor_vb_interleaved_128_32(const unsigned char *in, uint32_t *out); #endif template Docid read_le(const void *in) { Docid val; memcpy(&val, in, sizeof(val)); if constexpr (sizeof(Docid) == 8) { return le64toh(val); } else if constexpr (sizeof(Docid) == 4) { return le32toh(val); } else if constexpr (sizeof(Docid) == 2) { return le16toh(val); } else if constexpr (sizeof(Docid) == 1) { return val; } else { assert(false); } } // Reads a single value with an encoding that looks a bit like PrefixVarint. // It's unclear why this doesn't use the varbyte encoding. template const unsigned char *read_baseval(const unsigned char *in, Docid *out) { //fprintf(stderr, "baseval: 0x%02x 0x%02x 0x%02x 0x%02x\n", in[0], in[1], in[2], in[3]); if (*in < 128) { *out = *in; return in + 1; } else if (*in < 192) { *out = ((uint32_t(in[0]) << 8) | uint32_t(in[1])) & 0x3fff; return in + 2; } else if (*in < 224) { *out = ((uint32_t(in[0]) << 16) | (uint32_t(in[2]) << 8) | (uint32_t(in[1]))) & 0x1fffff; return in + 3; } else if (*in < 240) { *out = ((uint32_t(in[0]) << 24) | (uint32_t(in[1]) << 16) | (uint32_t(in[2]) << 8) | (uint32_t(in[3]))) & 0xfffffff; return in + 4; } else { assert(false); // Not implemented. } } // Does not read past the end of the input. template const unsigned char *read_vb(const unsigned char *in, Docid *out) { if (*in <= 176) { *out = *in; return in + 1; } else if (*in <= 240) { *out = ((uint32_t(in[0] - 177) << 8) | uint32_t(in[1])) + 177; return in + 2; } else if (*in <= 248) { *out = ((uint32_t(in[0] - 241) << 16) | read_le(in + 1)) + 16561; return in + 3; } else if (*in == 249) { *out = (uint32_t(in[1])) | (uint32_t(in[2]) << 8) | (uint32_t(in[3]) << 16); return in + 4; } else if (*in == 250) { *out = read_le(in + 1); return in + 5; } else { assert(false); } } struct BitReader { public: BitReader(const unsigned char *in, unsigned bits) : in(in), bits(bits), mask(mask_for_bits(bits)) {} // Can read 4 bytes past the end of the input (if bits_used == 0). uint32_t read() { uint32_t val = (read_le(in) >> bits_used) & mask; bits_used += bits; in += bits_used / 8; bits_used %= 8; return val; } private: const unsigned char *in; const unsigned bits; const unsigned mask; unsigned bits_used = 0; }; template struct InterleavedBitReader { public: InterleavedBitReader(const unsigned char *in, unsigned bits) : in(in), bits(bits), mask(mask_for_bits(bits)) {} // Can read 4 bytes past the end of the input (if bit_width == 0). uint32_t read() { uint32_t val; if (bits_used + bits > 32) { val = (read_le(in) >> bits_used) | (read_le(in + Stride) << (32 - bits_used)); } else { val = (read_le(in) >> bits_used); } bits_used += bits; in += Stride * (bits_used / 32); bits_used %= 32; return val & mask; } private: static constexpr unsigned Stride = NumStreams * sizeof(uint32_t); const unsigned char *in; const unsigned bits; const unsigned mask; unsigned bits_used = 0; }; #ifdef COULD_HAVE_SSE2 struct InterleavedBitReaderSSE2 { public: TARGET_SSE2 InterleavedBitReaderSSE2(const unsigned char *in, unsigned bits) : in(reinterpret_cast(in)), bits(bits), mask(_mm_set1_epi32(mask_for_bits(bits))) {} // Can read 16 bytes past the end of the input (if bit_width == 0). TARGET_SSE2 __m128i read() { __m128i val = _mm_srli_epi32(_mm_loadu_si128(in), bits_used); if (bits_used + bits > 32) { __m128i val_upper = _mm_slli_epi32(_mm_loadu_si128(in + 1), 32 - bits_used); val = _mm_or_si128(val, val_upper); } val = _mm_and_si128(val, mask); bits_used += bits; in += bits_used / 32; bits_used %= 32; return val; } private: const __m128i *in; const unsigned bits; const __m128i mask; unsigned bits_used = 0; }; #endif // Constant block. Layout: // // - Bit width (6 bits) | type << 6 // - Base values ( bits, rounded up to nearest byte) // // Can read 4 bytes past the end of the input (if bit_width == 0). template const unsigned char *decode_constant(const unsigned char *in, unsigned num, Docid *out) { const unsigned bit_width = *in++ & 0x3f; Docid val = read_le(in); if (bit_width < sizeof(Docid) * 8) { val &= mask_for_bits(bit_width); } Docid prev_val = out[-1]; for (unsigned i = 0; i < num; ++i) { out[i] = prev_val = val + prev_val + 1; } return in + div_round_up(bit_width, 8); } // FOR block (ie., PFor without exceptions). Layout: // // - Bit width (6 bits) | type << 6 // - Base values ( values of bits, rounded up to a multiple of 32 values) // // Can read 4 bytes past the end of the input (inherit from BitReader). template const unsigned char *decode_for(const unsigned char *in, unsigned num, Docid *out) { const unsigned bit_width = *in++ & 0x3f; Docid prev_val = out[-1]; BitReader bs(in, bit_width); for (unsigned i = 0; i < num; ++i) { prev_val = out[i] = bs.read() + prev_val + 1; } return in + bytes_for_packed_bits(num, bit_width); } #ifdef COULD_HAVE_SSE2 class DeltaDecoderSSE2 { public: TARGET_SSE2 DeltaDecoderSSE2(uint32_t prev_val) : prev_val(_mm_set1_epi32(prev_val)) {} TARGET_SSE2 __m128i decode(__m128i val) { val = _mm_add_epi32(val, _mm_slli_si128(val, 4)); val = _mm_add_epi32(val, _mm_slli_si128(val, 8)); val = _mm_add_epi32(val, _mm_add_epi32(prev_val, delta)); prev_val = _mm_shuffle_epi32(val, _MM_SHUFFLE(3, 3, 3, 3)); return val; } private: // Use 4/3/2/1 as delta instead of fixed 1, so that we can do the prev_val + delta // in parallel with something else. const __m128i delta = _mm_set_epi32(4, 3, 2, 1); __m128i prev_val; }; template TARGET_SSE2 inline void delta_decode_sse2(uint32_t *out) { DeltaDecoderSSE2 delta(out[-1]); __m128i *outvec = reinterpret_cast<__m128i *>(out); for (unsigned i = 0; i < BlockSize / 4; ++i) { __m128i val = _mm_loadu_si128(outvec + i); _mm_storeu_si128(outvec + i, delta.decode(val)); } } // Can read 16 bytes past the end of its input (inherit from InterleavedBitReaderSSE2). template TARGET_SSE2 const unsigned char * decode_bitmap_sse2_unrolled(const unsigned char *in, uint32_t *out) { __m128i *outvec = reinterpret_cast<__m128i *>(out); DeltaDecoderSSE2 delta(out[-1]); InterleavedBitReaderSSE2 bs(in, bit_width); #pragma GCC unroll 32 for (unsigned i = 0; i < BlockSize / 4; ++i) { __m128i val = bs.read(); if constexpr (OrWithExisting) { val = _mm_or_si128(val, _mm_slli_epi32(_mm_loadu_si128(outvec + i), bit_width)); } if constexpr (DeltaDecode) { val = delta.decode(val); } _mm_storeu_si128(outvec + i, val); } in += bytes_for_packed_bits(BlockSize, bit_width); return in; } // Can read 16 bytes past the end of its input (inherit from InterleavedBitReaderSSE2). template TARGET_SSE2 const unsigned char * decode_bitmap_sse2(const unsigned char *in, unsigned bit_width, uint32_t *out) { switch (bit_width) { case 0: return decode_bitmap_sse2_unrolled(in, out); case 1: return decode_bitmap_sse2_unrolled(in, out); case 2: return decode_bitmap_sse2_unrolled(in, out); case 3: return decode_bitmap_sse2_unrolled(in, out); case 4: return decode_bitmap_sse2_unrolled(in, out); case 5: return decode_bitmap_sse2_unrolled(in, out); case 6: return decode_bitmap_sse2_unrolled(in, out); case 7: return decode_bitmap_sse2_unrolled(in, out); case 8: return decode_bitmap_sse2_unrolled(in, out); case 9: return decode_bitmap_sse2_unrolled(in, out); case 10: return decode_bitmap_sse2_unrolled(in, out); case 11: return decode_bitmap_sse2_unrolled(in, out); case 12: return decode_bitmap_sse2_unrolled(in, out); case 13: return decode_bitmap_sse2_unrolled(in, out); case 14: return decode_bitmap_sse2_unrolled(in, out); case 15: return decode_bitmap_sse2_unrolled(in, out); case 16: return decode_bitmap_sse2_unrolled(in, out); case 17: return decode_bitmap_sse2_unrolled(in, out); case 18: return decode_bitmap_sse2_unrolled(in, out); case 19: return decode_bitmap_sse2_unrolled(in, out); case 20: return decode_bitmap_sse2_unrolled(in, out); case 21: return decode_bitmap_sse2_unrolled(in, out); case 22: return decode_bitmap_sse2_unrolled(in, out); case 23: return decode_bitmap_sse2_unrolled(in, out); case 24: return decode_bitmap_sse2_unrolled(in, out); case 25: return decode_bitmap_sse2_unrolled(in, out); case 26: return decode_bitmap_sse2_unrolled(in, out); case 27: return decode_bitmap_sse2_unrolled(in, out); case 28: return decode_bitmap_sse2_unrolled(in, out); case 29: return decode_bitmap_sse2_unrolled(in, out); case 30: return decode_bitmap_sse2_unrolled(in, out); case 31: return decode_bitmap_sse2_unrolled(in, out); case 32: return decode_bitmap_sse2_unrolled(in, out); } assert(false); } #endif // Like decode_for(), but the values are organized in four independent streams, // for SIMD (presumably SSE2). Supports a whole block only. // // Can read 16 bytes past the end of its input (inherit from InterleavedBitReader). template const unsigned char *decode_for_interleaved_generic(const unsigned char *in, Docid *out) { const unsigned bit_width = *in++ & 0x3f; InterleavedBitReader<4> bs0(in + 0 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs1(in + 1 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs2(in + 2 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs3(in + 3 * sizeof(uint32_t), bit_width); for (unsigned i = 0; i < BlockSize / 4; ++i) { out[i * 4 + 0] = bs0.read(); out[i * 4 + 1] = bs1.read(); out[i * 4 + 2] = bs2.read(); out[i * 4 + 3] = bs3.read(); } Docid prev_val = out[-1]; for (unsigned i = 0; i < BlockSize; ++i) { out[i] = prev_val = out[i] + prev_val + 1; } return in + bytes_for_packed_bits(BlockSize, bit_width); } // Does not read past the end of the input. template const unsigned char *decode_for_interleaved(const unsigned char *in, Docid *out) { if constexpr (BlockSize == 128 && sizeof(Docid) == sizeof(uint32_t)) { return decode_for_interleaved_128_32(in, out); } else { return decode_for_interleaved_generic(in, out); } } #ifndef SUPPRESS_DEFAULT // Does not read past the end of the input. TARGET_DEFAULT const unsigned char * decode_for_interleaved_128_32(const unsigned char *in, uint32_t *out) { return decode_for_interleaved_generic<128>(in, out); } #endif #ifdef COULD_HAVE_SSE2 // Specialized version for SSE2. // Can read 16 bytes past the end of the input (inherit from decode_bitmap_sse2()). TARGET_SSE2 const unsigned char * decode_for_interleaved_128_32(const unsigned char *in, uint32_t *out) { constexpr unsigned BlockSize = 128; const unsigned bit_width = *in++ & 0x3f; in = decode_bitmap_sse2(in, bit_width, out); return in; } #endif // Can read 4 bytes past the end of the input (inherit from BitReader). template const unsigned char *decode_pfor_bitmap_exceptions(const unsigned char *in, unsigned num, Docid *out) { const unsigned exception_bit_width = *in++; const uint64_t *exception_bitmap_ptr = reinterpret_cast(in); in += div_round_up(num, 8); int num_exceptions = 0; BitReader bs(in, exception_bit_width); for (unsigned i = 0; i < num; i += 64, ++exception_bitmap_ptr) { uint64_t exceptions = read_le(exception_bitmap_ptr); if (num - i < 64) { // We've read some bytes past the end, so clear out the junk bits. exceptions &= (1ULL << (num - i)) - 1; } for (; exceptions != 0; exceptions &= exceptions - 1, ++num_exceptions) { unsigned idx = (ffsll(exceptions) - 1) + i; out[idx] = bs.read(); } } in += bytes_for_packed_bits(num_exceptions, exception_bit_width); return in; } // PFor block with bitmap exceptions. Layout: // // - Bit width (6 bits) | type << 6 // - Exception bit width (8 bits) // - Bitmap of which values have exceptions ( bits, rounded up to a byte) // - Exceptions ( values of bits, rounded up to a byte) // - Base values ( values of bits, rounded up to a byte) // // Can read 4 bytes past the end of the input (inherit from BitReader). template const unsigned char *decode_pfor_bitmap(const unsigned char *in, unsigned num, Docid *out) { memset(out, 0, num * sizeof(Docid)); const unsigned bit_width = *in++ & 0x3f; in = decode_pfor_bitmap_exceptions(in, num, out); // Decode the base values, and delta-decode. Docid prev_val = out[-1]; BitReader bs(in, bit_width); for (unsigned i = 0; i < num; ++i) { out[i] = prev_val = ((out[i] << bit_width) | bs.read()) + prev_val + 1; } return in + bytes_for_packed_bits(num, bit_width); } // Like decode_pfor_bitmap(), but the base values are organized in four // independent streams, for SIMD (presumably SSE2). Supports a whole block only. // // Can read 16 bytes past the end of the input (inherit from InterleavedBitReader // and decode_pfor_bitmap_exceptions()). template const unsigned char *decode_pfor_bitmap_interleaved_generic(const unsigned char *in, Docid *out) { memset(out, 0, BlockSize * sizeof(Docid)); const unsigned bit_width = *in++ & 0x3f; in = decode_pfor_bitmap_exceptions(in, BlockSize, out); // Decode the base values. InterleavedBitReader<4> bs0(in + 0 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs1(in + 1 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs2(in + 2 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs3(in + 3 * sizeof(uint32_t), bit_width); for (unsigned i = 0; i < BlockSize / 4; ++i) { out[i * 4 + 0] = bs0.read() | (out[i * 4 + 0] << bit_width); out[i * 4 + 1] = bs1.read() | (out[i * 4 + 1] << bit_width); out[i * 4 + 2] = bs2.read() | (out[i * 4 + 2] << bit_width); out[i * 4 + 3] = bs3.read() | (out[i * 4 + 3] << bit_width); } // Delta-decode. Docid prev_val = out[-1]; for (unsigned i = 0; i < BlockSize; ++i) { out[i] = prev_val = out[i] + prev_val + 1; } return in + bytes_for_packed_bits(BlockSize, bit_width); } // Can read 16 bytes past the end of the input (inherit from decode_pfor_bitmap_interleaved_generic()). template const unsigned char *decode_pfor_bitmap_interleaved(const unsigned char *in, Docid *out) { if constexpr (BlockSize == 128 && sizeof(Docid) == sizeof(uint32_t)) { return decode_pfor_bitmap_interleaved_128_32(in, out); } else { return decode_pfor_bitmap_interleaved_generic(in, out); } } #ifndef SUPPRESS_DEFAULT TARGET_DEFAULT const unsigned char * decode_pfor_bitmap_interleaved_128_32(const unsigned char *in, uint32_t *out) { return decode_pfor_bitmap_interleaved_generic<128>(in, out); } #endif #ifdef COULD_HAVE_SSE2 // Specialized version for SSE2. // // Can read 16 bytes past the end of the input (inherit from InterleavedBitReaderSSE2 // and decode_pfor_bitmap_exceptions()). TARGET_SSE2 const unsigned char * decode_pfor_bitmap_interleaved_128_32(const unsigned char *in, uint32_t *out) { constexpr unsigned BlockSize = 128; // Set all output values to zero, before the exceptions are filled in. #pragma GCC unroll 4 for (unsigned i = 0; i < BlockSize / 4; ++i) { _mm_storeu_si128(reinterpret_cast<__m128i *>(out) + i, _mm_setzero_si128()); } const unsigned bit_width = *in++ & 0x3f; in = decode_pfor_bitmap_exceptions(in, BlockSize, out); in = decode_bitmap_sse2(in, bit_width, out); return in; } #endif // PFor block with variable-byte exceptions. Layout: // // - Bit width (6 bits) | type << 6 // - Number of exceptions (8 bits) // - Base values ( values of bits, rounded up to a byte) // - Exceptions: // - If first byte is 255, 32-bit values (does not include the 255 byte) // - Else, varbyte-encoded values (includes the non-255 byte) // - Indexes of exceptions ( bytes). // // Can read 4 bytes past the end of the input (inherit from BitReader, // assuming zero exceptions). template const unsigned char *decode_pfor_vb(const unsigned char *in, unsigned num, Docid *out) { //fprintf(stderr, "in=%p out=%p num=%u\n", in, out, num); const unsigned bit_width = *in++ & 0x3f; unsigned num_exceptions = *in++; // Decode the base values. BitReader bs(in, bit_width); for (unsigned i = 0; i < num; ++i) { out[i] = bs.read(); } in += bytes_for_packed_bits(num, bit_width); // Decode exceptions. Docid exceptions[BlockSize]; if (*in == 255) { ++in; for (unsigned i = 0; i < num_exceptions; ++i) { exceptions[i] = read_le(in); in += sizeof(Docid); } } else { for (unsigned i = 0; i < num_exceptions; ++i) { in = read_vb(in, &exceptions[i]); } } // Apply exceptions. for (unsigned i = 0; i < num_exceptions; ++i) { unsigned idx = *in++; out[idx] |= exceptions[i] << bit_width; } // Delta-decode. Docid prev_val = out[-1]; for (unsigned i = 0; i < num; ++i) { out[i] = prev_val = out[i] + prev_val + 1; } return in; } // Like decode_pfor_vb(), but the base values are organized in four // independent streams, for SIMD (presumably SSE2). Supports a whole block only. // Can read 16 bytes past the end of its input (inherit from InterleavedBitReader). template const unsigned char *decode_pfor_vb_interleaved_generic(const unsigned char *in, Docid *out) { const unsigned bit_width = *in++ & 0x3f; unsigned num_exceptions = *in++; // Decode the base values. InterleavedBitReader<4> bs0(in + 0 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs1(in + 1 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs2(in + 2 * sizeof(uint32_t), bit_width); InterleavedBitReader<4> bs3(in + 3 * sizeof(uint32_t), bit_width); for (unsigned i = 0; i < BlockSize / 4; ++i) { out[i * 4 + 0] = bs0.read(); out[i * 4 + 1] = bs1.read(); out[i * 4 + 2] = bs2.read(); out[i * 4 + 3] = bs3.read(); } in += bytes_for_packed_bits(BlockSize, bit_width); // Decode exceptions. Docid exceptions[BlockSize]; if (*in == 255) { ++in; for (unsigned i = 0; i < num_exceptions; ++i) { exceptions[i] = read_le(in); in += sizeof(Docid); } } else { for (unsigned i = 0; i < num_exceptions; ++i) { in = read_vb(in, &exceptions[i]); } } // Apply exceptions. for (unsigned i = 0; i < num_exceptions; ++i) { unsigned idx = *in++; out[idx] |= exceptions[i] << bit_width; } // Delta-decode. Docid prev_val = out[-1]; for (unsigned i = 0; i < BlockSize; ++i) { out[i] = prev_val = out[i] + prev_val + 1; } return in; } // Can read 16 bytes past the end of its input (inherit from decode_pfor_vb_interleaved_generic()). template const unsigned char *decode_pfor_vb_interleaved(const unsigned char *in, Docid *out) { if constexpr (BlockSize == 128 && sizeof(Docid) == sizeof(uint32_t)) { return decode_pfor_vb_interleaved_128_32(in, out); } else { return decode_pfor_vb_interleaved_generic(in, out); } } #ifndef SUPPRESS_DEFAULT TARGET_DEFAULT const unsigned char * decode_pfor_vb_interleaved_128_32(const unsigned char *in, uint32_t *out) { return decode_pfor_vb_interleaved_generic<128>(in, out); } #endif #ifdef COULD_HAVE_SSE2 // Specialized version for SSE2. // Can read 16 bytes past the end of the input (inherit from decode_bitmap_sse2()). TARGET_SSE2 const unsigned char * decode_pfor_vb_interleaved_128_32(const unsigned char *in, uint32_t *out) { constexpr unsigned BlockSize = 128; using Docid = uint32_t; const unsigned bit_width = *in++ & 0x3f; unsigned num_exceptions = *in++; // Decode the base values. in = decode_bitmap_sse2(in, bit_width, out); // Decode exceptions. Docid exceptions[BlockSize]; if (*in == 255) { ++in; for (unsigned i = 0; i < num_exceptions; ++i) { exceptions[i] = read_le(in); in += sizeof(Docid); } } else { for (unsigned i = 0; i < num_exceptions; ++i) { in = read_vb(in, &exceptions[i]); } } // Apply exceptions. for (unsigned i = 0; i < num_exceptions; ++i) { unsigned idx = *in++; out[idx] |= exceptions[i] << bit_width; } delta_decode_sse2(out); return in; } #endif // Can read 16 bytes past the end of the input (inherit from several functions). template const unsigned char *decode_pfor_delta1(const unsigned char *in, unsigned num, bool interleaved, Docid *out) { if (num == 0) { return in; } in = read_baseval(in, out++); for (unsigned i = 1; i < num; i += BlockSize, out += BlockSize) { const unsigned num_this_block = std::min(num - i, BlockSize); switch (in[0] >> 6) { case BlockType::FOR: if (interleaved && num_this_block == BlockSize) { dprintf("%d+%d: blocktype=%d (for, interleaved), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_for_interleaved(in, out); } else { dprintf("%d+%d: blocktype=%d (for), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_for(in, num_this_block, out); } break; case BlockType::PFOR_VB: if (interleaved && num_this_block == BlockSize) { dprintf("%d+%d: blocktype=%d (pfor + vb, interleaved), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_pfor_vb_interleaved(in, out); } else { dprintf("%d+%d: blocktype=%d (pfor + vb), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_pfor_vb(in, num_this_block, out); } break; case BlockType::PFOR_BITMAP: if (interleaved && num_this_block == BlockSize) { dprintf("%d+%d: blocktype=%d (pfor + bitmap, interleaved), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_pfor_bitmap_interleaved(in, out); } else { dprintf("%d+%d: blocktype=%d (pfor + bitmap), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_pfor_bitmap(in, num_this_block, out); } break; case BlockType::CONSTANT: dprintf("%d+%d: blocktype=%d (constant), bitwidth=%d\n", i, num_this_block, in[0] >> 6, in[0] & 0x3f); in = decode_constant(in, num_this_block, out); break; } } return in; } const unsigned char *decode_pfor_delta1_128(const unsigned char *in, unsigned num, bool interleaved, uint32_t *out) { return decode_pfor_delta1<128>(in, num, interleaved, out); } plocate-1.1.15/turbopfor.h000066400000000000000000000030171417604322600154270ustar00rootroot00000000000000#ifndef _TURBOPFOR_H #define _TURBOPFOR_H 1 // A reimplementation of parts of the TurboPFor codecs, using the same // storage format. These are not as fast as the reference implementation // (about 80% of the performance, averaged over a real plocate corpus), // and do not support the same breadth of codecs (in particular, only // delta-plus-1 is implemented, and only 32-bit docids are tested), // but aim to be more portable and (ideally) easier-to-understand. // In particular, they will compile on x86 without SSE4.1 or AVX support. // Unlike the reference code, only GCC and GCC-compatible compilers // (e.g. Clang) are supported. // // The main reference is https://michael.stapelberg.ch/posts/2019-02-05-turbopfor-analysis/, // although some implementation details have been worked out by studying the // TurboPFor code. // // The decoder, like the reference implementation, is not robust against // malicious of corrupted. Several functions (again like the reference // implementation) can read N bytes past the end, so you need to have some slop // in the input buffers; this is documented for each function (unlike // the reference implementation), but the documented slop assumes a // non-malicious encoder. // // Although all of these algorithms are templatized, we expose only // the single specialization that we need, in order to increase the // speed of incremental compilation. const unsigned char *decode_pfor_delta1_128(const unsigned char *in, unsigned num, bool interleaved, uint32_t *out); #endif // !defined(_TURBOPFOR_H) plocate-1.1.15/unique_sort.h000066400000000000000000000007251417604322600157650ustar00rootroot00000000000000#ifndef _UNIQUE_SORT_H #define _UNIQUE_SORT_H 1 #include template, class EqualTo = std::equal_to> void unique_sort(Container *c, const LessThan < = LessThan(), const EqualTo &eq = EqualTo()) { sort(c->begin(), c->end(), lt); auto new_end = unique(c->begin(), c->end(), eq); c->erase(new_end, c->end()); } #endif // !defined(_UNIQUE_SORT_H) plocate-1.1.15/update-plocate.sh000077500000000000000000000005111417604322600164760ustar00rootroot00000000000000#! /bin/bash set -e if [ @PROCESSED_BY_MESON@ = 1 ]; then SBINDIR=@sbindir@ LOCATEGROUP=@groupname@ DBFILE=@dbfile@ else SBINDIR=/usr/local/sbin LOCATEGROUP=plocate DBFILE=/var/lib/plocate/plocate.db fi $SBINDIR/plocate-build /var/lib/mlocate/mlocate.db $DBFILE.new chgrp $LOCATEGROUP $DBFILE.new mv $DBFILE.new $DBFILE plocate-1.1.15/updatedb.8.in000066400000000000000000000122201417604322600155160ustar00rootroot00000000000000.\" A man page for updatedb(8). -*- nroff -*- .\" .\" Copyright (C) 2005, 2007, 2008 Red Hat, Inc. All rights reserved. .\" .\" This copyrighted material is made available to anyone wishing to use, .\" modify, copy, or redistribute it subject to the terms and conditions of the .\" GNU General Public License v.2. .\" .\" This program is distributed in the hope that it will be useful, but WITHOUT .\" ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or .\" FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for .\" more details. .\" .\" You should have received a copy of the GNU General Public License along .\" with this program; if not, write to the Free Software Foundation, Inc., .\" 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. .\" .\" Author: Miloslav Trmac .TH updatedb 8 "Dec 2020" plocate .SH NAME updatedb \- update a database for plocate .SH SYNOPSIS \fBupdatedb\fR [\fIOPTION\fR]... .SH DESCRIPTION .B updatedb creates or updates a database used by .BR locate (1). If the database already exists, its data is reused to avoid rereading directories that have not changed. .B updatedb is usually run daily from a .BR systemd.timer (8) to update the default database. .SH EXIT STATUS .B updatedb returns with exit status 0 on success, 1 on error. .SH OPTIONS The \fBPRUNE_BIND_MOUNTS\fR, \fBPRUNEFS\fR, .B PRUNENAMES and .B PRUNEPATHS variables, which are modified by some of the options, are documented in detail in .BR updatedb.conf (5). .TP \fB\-f\fR, \fB\-\-add-prunefs\fB \fIFS\fR Add entries in white-space-separated list \fIFS\fR to \fBPRUNEFS\fR. .TP \fB\-n\fR, \fB\-\-add-prunenames\fB \fINAMES\fR Add entries in white-space-separated list \fINAMES\fR to \fBPRUNENAMES\fR. .TP \fB\-e\fR, \fB\-\-add-prunepaths\fB \fIPATHS\fR Add entries in white-space-separated list \fIPATHS\fR to \fBPRUNEPATHS\fR. .TP \fB\-e\fR, \fB\-\-add-single-prunepath\fB \fIPATH\fR Add \fIPATH\fR to \fBPRUNEPATHS\fR. Note that this is currently the only way to add a path with a space in it. .TP \fB\-U\fR, \fB\-\-database\-root\fR \fIPATH\fR Store only results of scanning the file system subtree rooted at \fIPATH\fR to the generated database. The whole file system is scanned by default. .BR locate (1) outputs entries as absolute path names which don't contain symbolic links, regardless of the form of \fIPATH\fR. .TP \fB\-\-debug\-pruning\fR Write debugging information about pruning decisions to standard error output. .TP \fB\-h\fR, \fB\-\-help\fR Write a summary of the available options to standard output and exit successfully. .TP \fB\-o\fR, \fB\-\-output\fR \fIFILE\fR Write the database to .I FILE instead of using the default database. .TP \fB\-\-prune\-bind\-mounts\fR \fIFLAG\fR Set .B PRUNE_BIND_MOUNTS to \fIFLAG\fR, overriding the configuration file. .TP \fB\-\-prunefs\fR \fIFS\fR Set \fBPRUNEFS\fR to \fIFS\fR, overriding the configuration file. .TP \fB\-\-prunenames\fR \fINAMES\fR Set \fBPRUNENAMES\fR to \fINAMES\fR, overriding the configuration file. .TP \fB\-\-prunepaths\fR \fIPATHS\fR Set \fBPRUNEPATHS\fR to \fIPATHS\fR, overriding the configuration file. .TP \fB\-l\fR, \fB\-\-require\-visibility\fR \fIFLAG\fR Set the \*(lqrequire file visibility before reporting it\*(rq flag in the generated database to \fIFLAG\fR. If .I FLAG is .B 0 or \fBno\fR, or if the database file is readable by "others" or it is not owned by \fB@groupname@\fR, .BR locate (1) outputs the database entries even if the user running .BR locate (1) could not have read the directory necessary to find out the file described by the database entry. If .I FLAG is .B 1 or .B yes (the default), .BR locate (1) checks the permissions of parent directories of each entry before reporting it to the invoking user. To make the file existence truly hidden from other users, the database group is set to .B @groupname@ and the database permissions prohibit reading the database by users using other means than .BR locate (1), which is set-gid \fB@groupname@\fR. Note that the visibility flag is checked only if the database is owned by .B @groupname@ and it is not readable by "others". .TP \fB\-v\fR, \fB\-\-verbose\fR Output path names of files to standard output, as soon as they are found. .TP \fB\-V\fR, \fB\-\-version\fR Write information about the version and license of .B locate on standard output and exit successfully. .SH EXAMPLES To create a private plocate database as a user other than \fBroot\fR, run .RS .B updatedb -l 0 \-o .I db_file .B \-U .I source_directory .RE Note that all users that can read .I db_file can get the complete list of files in the subtree of \fIsource_directory\fR. .SH FILES .TP \fB@updatedb_conf@\fR A configuration file. See .BR updatedb.conf (5). Uses exactly the same format as the one used by .BR mlocate (1)'s updatedb, so they can be shared. .TP \fB@dbfile@\fR The database updated by default. .SH SECURITY Databases built with .B \-\-require\-visibility no allow users to find names of files and directories of other users, which they would not otherwise be able to do. .SH AUTHOR Miloslav Trmac .P Steinar H. Gunderson .SH SEE ALSO .BR locate (1), .BR updatedb.conf (5) plocate-1.1.15/updatedb.conf.5.in000066400000000000000000000100031417604322600164340ustar00rootroot00000000000000.\" A man page for updatedb.conf. -*- nroff -*- .\" .\" Copyright (C) 2005, 2007, 2008 Red Hat, Inc. All rights reserved. .\" .\" This copyrighted material is made available to anyone wishing to use, .\" modify, copy, or redistribute it subject to the terms and conditions of the .\" GNU General Public License v.2. .\" .\" This program is distributed in the hope that it will be useful, but WITHOUT .\" ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or .\" FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for .\" more details. .\" .\" You should have received a copy of the GNU General Public License along .\" with this program; if not, write to the Free Software Foundation, Inc., .\" 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. .\" .\" Author: Miloslav Trmac .TH updatedb.conf 5 "Oct 2020" plocate .SH NAME @updatedb_conf@ \- a configuration file for updatedb(8) .SH DESCRIPTION .B @updatedb_conf@ is a text file. Blank lines are ignored. A .B # character outside of a quoted string starts a comment extending until end of line. Other lines must be of the following form: .RS .I VARIABLE .B = \fB"\fIVALUE\fB"\fR .RE White space between tokens is ignored. .I VARIABLE is an alphanumeric string which does not start with a digit. .I VALUE can contain any character except for \fB\(dq\fR. No escape mechanism is supported within .I VALUE and there is no way to write .I VALUE spanning more than one line. Unknown .I VARIABLE values are considered an error. The defined variables are: .TP \fBPRUNEFS\fR A whitespace-separated list of file system types (as used in \fB/etc/mtab\fR) which should not be scanned by .BR updatedb (8). The file system type matching is case-insensitive. By default, no file system types are skipped. When scanning a file system is skipped, all file systems mounted in the subtree are skipped too, even if their type does not match any entry in \fBPRUNEFS\fR. .TP \fBPRUNENAMES\fR A whitespace-separated list of directory names (without paths) which should not be scanned by .BR updatedb (8). By default, no directory names are skipped. Note that only directories can be specified, and no pattern mechanism (e.g. globbing) is used. .TP \fBPRUNEPATHS\fR A whitespace-separated list of path names of directories which should not be scanned by .BR updatedb (8). Each path name must be exactly in the form in which the directory would be reported by .BR locate (1). By default, no paths are skipped. .TP \fBPRUNE_BIND_MOUNTS\fR One of the strings \fB0\fR, \fBno\fR, \fB1\fR or \fByes\fR. If .B PRUNE_BIND_MOUNTS is \fB1\fR or \fByes\fR, bind mounts are not scanned by .BR updatedb (8). All file systems mounted in the subtree of a bind mount are skipped as well, even if they are not bind mounts. As an exception, bind mounts of a directory on itself are not skipped. Note that Btrfs subvolume mounts are handled internally in the kernel as bind mounts (see .BR btrfs-subvolume (8)), and thus, may get skipped if you have also mounted the filesystem root itself. To counteract this, make your root directory a Btrfs subvolume, too. By default, bind mounts are not skipped. .SH NOTES When a directory is matched by \fBPRUNEFS\fR, \fBPRUNENAMES\fR or \fBPRUNEPATHS\fR, .BR updatedb (8) does not scan the contents of the directory. The path of the directory itself is, however, entered in the created database. For example, if .I /tmp is in \fBPRUNEPATHS\fR, .BR locate (1) will not show any files stored in \fI/tmp\fR, but it can show the .I /tmp directory. This behavior differs from traditional .B locate implementations. In some .BR updatedb (8) implementations \fBPRUNEPATHS\fR can be used to exclude non-directory files. This is not the case in this implementation. .B @updatedb_conf@ is a shell script in some implementations, which allows much more flexibility in defining the variables. Equivalent functionality can be achieved by using the command-line options to .BR updatedb (8). .SH AUTHOR Miloslav Trmac .SH SEE ALSO .BR locate (1), .BR updatedb (8) plocate-1.1.15/updatedb.cpp000066400000000000000000000616161417604322600155410ustar00rootroot00000000000000/* updatedb(8). Copyright (C) 2005, 2007, 2008 Red Hat, Inc. All rights reserved. This copyrighted material is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License v.2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Author: Miloslav Trmac plocate modifications: Copyright (C) 2020 Steinar H. Gunderson. plocate parts and modifications are licensed under the GPLv2 or, at your option, any later version. */ #include "bind-mount.h" #include "complete_pread.h" #include "conf.h" #include "database-builder.h" #include "db.h" #include "dprintf.h" #include "io_uring_engine.h" #include "lib.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include using namespace std; using namespace std::chrono; /* Next conf_prunepaths entry */ static size_t conf_prunepaths_index; /* = 0; */ void usage() { printf( "Usage: updatedb PLOCATE_DB\n" "\n" "Generate plocate index from mlocate.db, typically /var/lib/mlocate/mlocate.db.\n" "Normally, the destination should be /var/lib/mlocate/plocate.db.\n" "\n" " -b, --block-size SIZE number of filenames to store in each block (default 32)\n" " -p, --plaintext input is a plaintext file, not an mlocate database\n" " --help print this help\n" " --version print version information\n"); } void version() { printf("updatedb %s\n", PACKAGE_VERSION); printf("Copyright (C) 2007 Red Hat, Inc. All rights reserved.\n"); printf("Copyright 2020 Steinar H. Gunderson\n"); printf("This software is distributed under the GPL v.2.\n"); printf("\n"); printf("This program is provided with NO WARRANTY, to the extent permitted by law.\n"); } int opendir_noatime(int dirfd, const char *path) { static bool noatime_failed = false; if (!noatime_failed) { #ifdef O_NOATIME int fd = openat(dirfd, path, O_RDONLY | O_DIRECTORY | O_NOATIME); #else int fd = openat(dirfd, path, O_RDONLY | O_DIRECTORY); #endif if (fd != -1) { return fd; } else if (errno == EPERM) { /* EPERM is fairly O_NOATIME-specific; missing access rights cause EACCES. */ noatime_failed = true; // Retry below. } else { return -1; } } return openat(dirfd, path, O_RDONLY | O_DIRECTORY); } bool time_is_current(const dir_time &t) { static dir_time cache{ 0, 0 }; /* This is more difficult than it should be because Linux uses a cheaper time source for filesystem timestamps than for gettimeofday() and they can get slightly out of sync, see https://bugzilla.redhat.com/show_bug.cgi?id=244697 . This affects even nanosecond timestamps (and don't forget that tv_nsec existence doesn't guarantee that the underlying filesystem has such resolution - it might be microseconds or even coarser). The worst case is probably FAT timestamps with 2-second resolution (although using such a filesystem violates POSIX file times requirements). So, to be on the safe side, require a >3.0 second difference (2 seconds to make sure the FAT timestamp changed, 1 more to account for the Linux timestamp races). This large margin might make updatedb marginally more expensive, but it only makes a difference if the directory was very recently updated _and_ is will not be updated again until the next updatedb run; this is not likely to happen for most directories. */ /* Cache gettimeofday () results to rule out obviously old time stamps; CACHE contains the earliest time we reject as too current. */ if (t < cache) { return false; } struct timeval tv; gettimeofday(&tv, nullptr); cache.sec = tv.tv_sec - 3; cache.nsec = tv.tv_usec * 1000; return t >= cache; } struct entry { string name; bool is_directory; // For directories only: int fd = -1; dir_time dt = unknown_dir_time; dir_time db_modified = unknown_dir_time; dev_t dev; }; bool filesystem_is_excluded(const char *path) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "Checking whether filesystem `%s' is excluded:\n", path); } FILE *f = setmntent("/proc/mounts", "r"); if (f == nullptr) { return false; } struct mntent *me; while ((me = getmntent(f)) != nullptr) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, " `%s', type `%s'\n", me->mnt_dir, me->mnt_type); } string type(me->mnt_type); for (char &p : type) { p = toupper(p); } if (find(conf_prunefs.begin(), conf_prunefs.end(), type) != conf_prunefs.end()) { /* Paths in /proc/self/mounts contain no symbolic links. Besides avoiding a few system calls, avoiding the realpath () avoids hangs if the filesystem is unavailable hard-mounted NFS. */ char *dir = me->mnt_dir; if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, " => type matches, dir `%s'\n", dir); } bool res = (strcmp(path, dir) == 0); if (dir != me->mnt_dir) free(dir); if (res) { endmntent(f); return true; } } } if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "...done\n"); } endmntent(f); return false; } dir_time get_dirtime_from_stat(const struct stat &buf) { dir_time ctime{ buf.st_ctim.tv_sec, int32_t(buf.st_ctim.tv_nsec) }; dir_time mtime{ buf.st_mtim.tv_sec, int32_t(buf.st_mtim.tv_nsec) }; dir_time dt = max(ctime, mtime); if (time_is_current(dt)) { /* The directory might be changing right now and we can't be sure the timestamp will be changed again if more changes happen very soon, mark the timestamp as invalid to force rescanning the directory next time updatedb is run. */ return unknown_dir_time; } else { return dt; } } // Represents the old database we are updating. class ExistingDB { public: explicit ExistingDB(int fd); ~ExistingDB(); pair read_next(); void unread(pair record) { unread_record = move(record); } string read_next_dictionary() const; bool get_error() const { return error; } private: const int fd; Header hdr; uint32_t current_docid = 0; string current_filename_block; const char *current_filename_ptr = nullptr, *current_filename_end = nullptr; off_t compressed_dir_time_pos; string compressed_dir_time; string current_dir_time_block; const char *current_dir_time_ptr = nullptr, *current_dir_time_end = nullptr; pair unread_record; // Used in one-shot mode, repeatedly. ZSTD_DCtx *ctx; // Used in streaming mode. ZSTD_DCtx *dir_time_ctx; ZSTD_DDict *ddict = nullptr; // If true, we've discovered an error or EOF, and will return only // empty data from here. bool eof = false, error = false; }; ExistingDB::ExistingDB(int fd) : fd(fd) { if (fd == -1) { error = true; return; } if (!try_complete_pread(fd, &hdr, sizeof(hdr), /*offset=*/0)) { if (conf_verbose) { perror("pread(header)"); } error = true; return; } if (memcmp(hdr.magic, "\0plocate", 8) != 0) { if (conf_verbose) { fprintf(stderr, "Old database had header mismatch, ignoring.\n"); } error = true; return; } if (hdr.version != 1 || hdr.max_version < 2) { if (conf_verbose) { fprintf(stderr, "Old database had version mismatch (version=%d max_version=%d), ignoring.\n", hdr.version, hdr.max_version); } error = true; return; } // Compare the configuration block with our current one. if (hdr.conf_block_length_bytes != conf_block.size()) { if (conf_verbose) { fprintf(stderr, "Old database had different configuration block (size mismatch), ignoring.\n"); } error = true; return; } string str; str.resize(hdr.conf_block_length_bytes); if (!try_complete_pread(fd, str.data(), hdr.conf_block_length_bytes, hdr.conf_block_offset_bytes)) { if (conf_verbose) { perror("pread(conf_block)"); } error = true; return; } if (str != conf_block) { if (conf_verbose) { fprintf(stderr, "Old database had different configuration block (contents mismatch), ignoring.\n"); } error = true; return; } // Read dictionary, if it exists. if (hdr.zstd_dictionary_length_bytes > 0) { string dictionary; dictionary.resize(hdr.zstd_dictionary_length_bytes); if (try_complete_pread(fd, &dictionary[0], hdr.zstd_dictionary_length_bytes, hdr.zstd_dictionary_offset_bytes)) { ddict = ZSTD_createDDict(dictionary.data(), dictionary.size()); } else { if (conf_verbose) { perror("pread(dictionary)"); } error = true; return; } } compressed_dir_time_pos = hdr.directory_data_offset_bytes; ctx = ZSTD_createDCtx(); dir_time_ctx = ZSTD_createDCtx(); } ExistingDB::~ExistingDB() { if (fd != -1) { close(fd); } } pair ExistingDB::read_next() { if (!unread_record.first.empty()) { auto ret = move(unread_record); unread_record.first.clear(); return ret; } if (eof || error) { return { "", not_a_dir }; } // See if we need to read a new filename block. if (current_filename_ptr == nullptr) { if (current_docid >= hdr.num_docids) { eof = true; return { "", not_a_dir }; } // Read the file offset from this docid and the next one. // This is always allowed, since we have a sentinel block at the end. off_t offset_for_block = hdr.filename_index_offset_bytes + current_docid * sizeof(uint64_t); uint64_t vals[2]; if (!try_complete_pread(fd, vals, sizeof(vals), offset_for_block)) { if (conf_verbose) { perror("pread(offset)"); } error = true; return { "", not_a_dir }; } off_t offset = vals[0]; size_t compressed_len = vals[1] - vals[0]; unique_ptr compressed(new char[compressed_len]); if (!try_complete_pread(fd, compressed.get(), compressed_len, offset)) { if (conf_verbose) { perror("pread(block)"); } error = true; return { "", not_a_dir }; } unsigned long long uncompressed_len = ZSTD_getFrameContentSize(compressed.get(), compressed_len); if (uncompressed_len == ZSTD_CONTENTSIZE_UNKNOWN || uncompressed_len == ZSTD_CONTENTSIZE_ERROR) { if (conf_verbose) { fprintf(stderr, "ZSTD_getFrameContentSize() failed\n"); } error = true; return { "", not_a_dir }; } string block; block.resize(uncompressed_len + 1); size_t err; if (ddict != nullptr) { err = ZSTD_decompress_usingDDict(ctx, &block[0], block.size(), compressed.get(), compressed_len, ddict); } else { err = ZSTD_decompressDCtx(ctx, &block[0], block.size(), compressed.get(), compressed_len); } if (ZSTD_isError(err)) { if (conf_verbose) { fprintf(stderr, "ZSTD_decompress(): %s\n", ZSTD_getErrorName(err)); } error = true; return { "", not_a_dir }; } block[block.size() - 1] = '\0'; current_filename_block = move(block); current_filename_ptr = current_filename_block.data(); current_filename_end = current_filename_block.data() + current_filename_block.size(); ++current_docid; } // See if we need to read more directory time data. while (current_dir_time_ptr == current_dir_time_end || (*current_dir_time_ptr != 0 && size_t(current_dir_time_end - current_dir_time_ptr) < sizeof(dir_time) + 1)) { if (current_dir_time_ptr != nullptr) { const size_t bytes_consumed = current_dir_time_ptr - current_dir_time_block.data(); current_dir_time_block.erase(current_dir_time_block.begin(), current_dir_time_block.begin() + bytes_consumed); } // See if we can get more data out without reading more. const size_t existing_data = current_dir_time_block.size(); current_dir_time_block.resize(existing_data + 4096); ZSTD_outBuffer outbuf; outbuf.dst = current_dir_time_block.data() + existing_data; outbuf.size = 4096; outbuf.pos = 0; ZSTD_inBuffer inbuf; inbuf.src = compressed_dir_time.data(); inbuf.size = compressed_dir_time.size(); inbuf.pos = 0; int err = ZSTD_decompressStream(dir_time_ctx, &outbuf, &inbuf); if (err < 0) { if (conf_verbose) { fprintf(stderr, "ZSTD_decompress(): %s\n", ZSTD_getErrorName(err)); } error = true; return { "", not_a_dir }; } compressed_dir_time.erase(compressed_dir_time.begin(), compressed_dir_time.begin() + inbuf.pos); current_dir_time_block.resize(existing_data + outbuf.pos); if (inbuf.pos == 0 && outbuf.pos == 0) { // No movement, we'll need to try to read more data. char buf[4096]; size_t bytes_to_read = min( hdr.directory_data_offset_bytes + hdr.directory_data_length_bytes - compressed_dir_time_pos, sizeof(buf)); if (bytes_to_read == 0) { error = true; return { "", not_a_dir }; } if (!try_complete_pread(fd, buf, bytes_to_read, compressed_dir_time_pos)) { if (conf_verbose) { perror("pread(dirtime)"); } error = true; return { "", not_a_dir }; } compressed_dir_time_pos += bytes_to_read; compressed_dir_time.insert(compressed_dir_time.end(), buf, buf + bytes_to_read); // Next iteration will now try decompressing more. } current_dir_time_ptr = current_dir_time_block.data(); current_dir_time_end = current_dir_time_block.data() + current_dir_time_block.size(); } string filename = current_filename_ptr; current_filename_ptr += filename.size() + 1; if (current_filename_ptr == current_filename_end) { // End of this block. current_filename_ptr = nullptr; } if (*current_dir_time_ptr == 0) { ++current_dir_time_ptr; return { move(filename), not_a_dir }; } else { ++current_dir_time_ptr; dir_time dt; memcpy(&dt.sec, current_dir_time_ptr, sizeof(dt.sec)); current_dir_time_ptr += sizeof(dt.sec); memcpy(&dt.nsec, current_dir_time_ptr, sizeof(dt.nsec)); current_dir_time_ptr += sizeof(dt.nsec); return { move(filename), dt }; } } string ExistingDB::read_next_dictionary() const { if (hdr.next_zstd_dictionary_length_bytes == 0 || hdr.next_zstd_dictionary_length_bytes > 1048576) { return ""; } string str; str.resize(hdr.next_zstd_dictionary_length_bytes); if (!try_complete_pread(fd, str.data(), hdr.next_zstd_dictionary_length_bytes, hdr.next_zstd_dictionary_offset_bytes)) { if (conf_verbose) { perror("pread(next_dictionary)"); } return ""; } return str; } // Scans the directory with absolute path “path”, which is opened as “fd”. // Uses relative paths and openat() only, evading any issues with PATH_MAX // and time-of-check-time-of-use race conditions. (mlocate's updatedb // does a much more complicated dance with changing the current working // directory, probably in the interest of portability to old platforms.) // “parent_dev” must be the device of the parent directory of “path”. // // Takes ownership of fd. int scan(const string &path, int fd, dev_t parent_dev, dir_time modified, dir_time db_modified, ExistingDB *existing_db, DatabaseReceiver *corpus, DictionaryBuilder *dict_builder) { if (string_list_contains_dir_path(&conf_prunepaths, &conf_prunepaths_index, path)) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "Skipping `%s': in prunepaths\n", path.c_str()); } close(fd); return 0; } if (conf_prune_bind_mounts && is_bind_mount(path.c_str())) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "Skipping `%s': bind mount\n", path.c_str()); } close(fd); return 0; } // We read in the old directory no matter whether it is current or not, // because even if we're not going to use it, we'll need the modification directory // of any subdirectories. // Skip over anything before this directory; it is stuff that we would have // consumed earlier if we wanted it. for (;;) { pair record = existing_db->read_next(); if (record.first.empty()) { break; } if (dir_path_cmp(path, record.first) <= 0) { existing_db->unread(move(record)); break; } } // Now read everything in this directory. vector db_entries; const string path_plus_slash = path.back() == '/' ? path : path + '/'; for (;;) { pair record = existing_db->read_next(); if (record.first.empty()) { break; } if (record.first.rfind(path_plus_slash, 0) != 0) { // No longer starts with path, so we're in a different directory. existing_db->unread(move(record)); break; } if (record.first.find_first_of('/', path_plus_slash.size()) != string::npos) { // Entered into a subdirectory of a subdirectory. // Due to our ordering, this also means we're done. existing_db->unread(move(record)); break; } entry e; e.name = record.first.substr(path_plus_slash.size()); e.is_directory = (record.second.sec >= 0); e.db_modified = record.second; db_entries.push_back(e); } DIR *dir = nullptr; vector entries; if (!existing_db->get_error() && db_modified.sec > 0 && modified.sec == db_modified.sec && modified.nsec == db_modified.nsec) { // Not changed since the last database, so we can replace the readdir() // by reading from the database. (We still need to open and stat everything, // though, but that happens in a later step.) entries = move(db_entries); if (conf_verbose) { for (const entry &e : entries) { printf("%s/%s\n", path.c_str(), e.name.c_str()); } } } else { dir = fdopendir(fd); // Takes over ownership of fd. if (dir == nullptr) { perror("fdopendir"); exit(1); } dirent *de; while ((de = readdir(dir)) != nullptr) { if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0) { continue; } if (strlen(de->d_name) == 0) { /* Unfortunately, this does happen, and mere assert() does not give users enough information to complain to the right people. */ fprintf(stderr, "file system error: zero-length file name in directory %s", path.c_str()); continue; } entry e; e.name = de->d_name; if (de->d_type == DT_UNKNOWN) { // Evidently some file systems, like older versions of XFS // (mkfs.xfs -m crc=0 -n ftype=0), can return this, // and we need a stat(). If we wanted to optimize for this, // we could probably defer it to later (we're stat-ing directories // when recursing), but this is rare, and not really worth it -- // the second stat() will be cached anyway. struct stat buf; if (fstatat(fd, de->d_name, &buf, AT_SYMLINK_NOFOLLOW) == 0 && S_ISDIR(buf.st_mode)) { e.is_directory = true; } else { e.is_directory = false; } } else { e.is_directory = (de->d_type == DT_DIR); } if (conf_verbose) { printf("%s/%s\n", path.c_str(), de->d_name); } entries.push_back(move(e)); } sort(entries.begin(), entries.end(), [](const entry &a, const entry &b) { return a.name < b.name; }); // Load directory modification times from the old database. auto db_it = db_entries.begin(); for (entry &e : entries) { for (; db_it != db_entries.end(); ++db_it) { if (e.name < db_it->name) { break; } if (e.name == db_it->name) { e.db_modified = db_it->db_modified; break; } } } } // For each entry, we want to add it to the database. but this includes the modification time // for directories, which means we need to open and stat it at this point. // // This means we may need to have many directories open at the same time, but it seems to be // the simplest (only?) way of being compatible with mlocate's notion of listing all contents // of a given directory before recursing, without buffering even more information. Hopefully, // we won't go out of file descriptors here (it could happen if someone has tens of thousands // of subdirectories in a single directory); if so, the admin will need to raise the limit. for (entry &e : entries) { if (!e.is_directory) { e.dt = not_a_dir; continue; } if (find(conf_prunenames.begin(), conf_prunenames.end(), e.name) != conf_prunenames.end()) { if (conf_debug_pruning) { /* This is debugging output, don't mark anything for translation */ fprintf(stderr, "Skipping `%s': in prunenames\n", e.name.c_str()); } continue; } e.fd = opendir_noatime(fd, e.name.c_str()); if (e.fd == -1) { if (errno == EMFILE || errno == ENFILE) { // The admin probably wants to know about this. perror((path_plus_slash + e.name).c_str()); rlimit rlim; if (getrlimit(RLIMIT_NOFILE, &rlim) == -1) { fprintf(stderr, "Hint: Try `ulimit -n 131072' or similar.\n"); } else { fprintf(stderr, "Hint: Try `ulimit -n %lu' or similar (current limit is %lu).\n", rlim.rlim_cur * 2, rlim.rlim_cur); } exit(1); } continue; } struct stat buf; if (fstat(e.fd, &buf) != 0) { // It's possible that this is a filesystem that's excluded // (and the failure is e.g. because the network is down). // As a last-ditch effort, we try to check that before dying, // i.e., duplicate the check from further down. // // It would be better to be able to run filesystem_is_excluded() // for cheap on everything and just avoid the stat, but it seems // hard to do that without any kind of raciness. if (filesystem_is_excluded((path_plus_slash + e.name).c_str())) { close(e.fd); e.fd = -1; continue; } perror((path_plus_slash + e.name).c_str()); exit(1); } e.dev = buf.st_dev; if (buf.st_dev != parent_dev) { if (filesystem_is_excluded((path_plus_slash + e.name).c_str())) { close(e.fd); e.fd = -1; continue; } } e.dt = get_dirtime_from_stat(buf); } // Actually add all the entries we figured out dates for above. for (const entry &e : entries) { corpus->add_file(path_plus_slash + e.name, e.dt); dict_builder->add_file(path_plus_slash + e.name, e.dt); } // Now scan subdirectories. for (const entry &e : entries) { if (e.is_directory && e.fd != -1) { int ret = scan(path_plus_slash + e.name, e.fd, e.dev, e.dt, e.db_modified, existing_db, corpus, dict_builder); if (ret == -1) { // TODO: The unscanned file descriptors will leak, but it doesn't really matter, // as we're about to exit. closedir(dir); return -1; } } } if (dir == nullptr) { close(fd); } else { closedir(dir); } return 0; } int main(int argc, char **argv) { // We want to bump the file limit; do it if we can (usually we are root // and can set whatever we want). 128k should be ample for most setups. rlimit rlim; if (getrlimit(RLIMIT_NOFILE, &rlim) != -1) { // Even root cannot increase rlim_cur beyond rlim_max, // so we need to try to increase rlim_max first. // Ignore errors, though. if (rlim.rlim_max < 131072) { rlim.rlim_max = 131072; setrlimit(RLIMIT_NOFILE, &rlim); getrlimit(RLIMIT_NOFILE, &rlim); } rlim_t wanted = std::max(rlim.rlim_cur, 131072); rlim.rlim_cur = std::min(wanted, rlim.rlim_max); setrlimit(RLIMIT_NOFILE, &rlim); // Ignore errors. } conf_prepare(argc, argv); if (conf_prune_bind_mounts) { bind_mount_init(MOUNTINFO_PATH); } int fd = open(conf_output.c_str(), O_RDONLY); ExistingDB existing_db(fd); DictionaryBuilder dict_builder(/*blocks_to_keep=*/1000, conf_block_size); gid_t owner = -1; if (conf_check_visibility) { group *grp = getgrnam(GROUPNAME); if (grp == nullptr) { fprintf(stderr, "Unknown group %s\n", GROUPNAME); exit(1); } owner = grp->gr_gid; } DatabaseBuilder db(conf_output.c_str(), owner, conf_block_size, existing_db.read_next_dictionary(), conf_check_visibility); db.set_conf_block(conf_block); DatabaseReceiver *corpus = db.start_corpus(/*store_dir_times=*/true); int root_fd = opendir_noatime(AT_FDCWD, conf_scan_root); if (root_fd == -1) { perror("."); exit(1); } struct stat buf; if (fstat(root_fd, &buf) == -1) { perror("."); exit(1); } scan(conf_scan_root, root_fd, buf.st_dev, get_dirtime_from_stat(buf), /*db_modified=*/unknown_dir_time, &existing_db, corpus, &dict_builder); // It's too late to use the dictionary for the data we already compressed, // unless we wanted to either scan the entire file system again (acceptable // for plocate-build where it's cheap, less so for us), or uncompressing // and recompressing. Instead, we store it for next time, assuming that the // data changes fairly little from time to time. string next_dictionary = dict_builder.train(1024); db.set_next_dictionary(next_dictionary); db.finish_corpus(); exit(EXIT_SUCCESS); }