pax_global_header00006660000000000000000000000064130625300230014504gustar00rootroot0000000000000052 comment=ce0df56c2666f599061a691e4dd8ddec523967a3 vmtouch-1.3.0/000077500000000000000000000000001306253002300131725ustar00rootroot00000000000000vmtouch-1.3.0/.gitignore000066400000000000000000000000221306253002300151540ustar00rootroot00000000000000vmtouch vmtouch.8 vmtouch-1.3.0/CHANGES000066400000000000000000000064101306253002300141660ustar00rootroot000000000000001.3.0 2017-03-16 * New switch: "-b ". This enables "batch mode" where the list of files to crawl is read from the specified file (Thanks Tw) * New switch: "-0". When this is enabled, the files in "batch mode" are separated by NUL bytes instead of newlines * New switch: "-F". Prevents vmtouch from traversing separate filesystems (requested by mailinglists35) * Lots of updates to the debian packaging (Thanks anarcat) * Use standard path for manpages (Thanks anarcat) * On linux, if an open fails due to EPERM then try again without O_NOATIME 1.2.0 2016-12-01 * -i feature which lets you ignore entire files and directories (Thanks Etienne Bruines) * -I feature which lets you only process filenames matching certain patterns. * Both -i and -I support wildcards * Specify C99 standard during compile (Thanks ecebuzz) 1.1.0 2016-08-08 * Better error checking for extremely large values to command line parameters (Thanks Matthew Fernandez) * Fix some boundary conditions in the range support added in 1.0.1 (Thanks Justas Lavišius) * On Linux, support touching/evicting/displaying block devices directly. This displays the underlying buffer cache, not the filesystem cache (Thanks to maq123 for the suggestion) * On Linux, open files with O_NOATIME so that we don't cause unnecessary disk activity recording access times (Thanks Mat R.) * Replaces a stat() call with an fstat() call which is slightly more efficient. * Skipped symlinks are no longer included in total file count * Closes file descriptors after locking memory since there is no need to keep them open. This makes it less likely you will hit the RLIMIT_NOFILE when using -l or -L. * TUNING.md file (Thanks to Artem Sheremet for the idea and to Vladimir Kotal for Solaris tuning info) 1.0.2 2015-12-04 * Clean-up Makefile and improve its portability. * Documentation updates. 1.0.1 2015-10-16 * Change default max file size (specified with -m) to SIZE_MAX instead of 500m. This is 4G on 32-bit systems and effectively unlimited on 64-bit systems. (Thanks Erik Garrison) * -p mode which lets you view/touch/evict/lock ranges of files instead of just whole files. (Thanks Justas Lavišius) * Update debian packaging files. (Thanks Luka Blaskovic) * Fix bug that prevented you from trying to crawl your entire filesystem from the root. * Fix to work with glibc 2.20. (Thanks Jim Garrison) * Fix format string warning when compiling with clang. (Thanks Mikolaj Golub) * Cleaner and better Makefile. (Thanks Emmanuel Kasper and Mikolaj Golub) * Don't double-count the same file that just happens to have multiple hard-links to it. (Thanks Carsten Otto) * Add option to wait for daemon mode to finish mlocking. (Thanks Kristofer Karlsson) 0.8.0 2012-12-04 * Patch from Marc Brooker: avoid overflowing stack when allocating array for mincore() output * Patch from Federico Lucifredi: Avoid ugly NaNs in output * Full OS X support: msync(2) evicts pages on this system * HP-UX support from Shane Seymour (thanks!) * Patch from soarpenguin: detect invalid size formats 0.7.3 * Solaris support for page eviction 0.7.2 * Portability fixes 0.7.1 * First public release vmtouch-1.3.0/Makefile000066400000000000000000000011161306253002300146310ustar00rootroot00000000000000PREFIX?= /usr/local BINDIR?= $(PREFIX)/bin MANDIR?= $(PREFIX)/share/man/man8 CC?= cc CFLAGS+= -Wall -O2 -g -std=c99 all: vmtouch vmtouch.8 .PHONY: all install clean uninstall vmtouch: vmtouch.c ${CC} ${CFLAGS} -o vmtouch vmtouch.c vmtouch.8: vmtouch.pod pod2man --section 8 --center "System Manager's Manual" --release " " vmtouch.pod > vmtouch.8 install: vmtouch vmtouch.8 mkdir -p $(BINDIR) $(MANDIR) install -m 0755 vmtouch $(BINDIR)/vmtouch install -m 0644 vmtouch.8 $(MANDIR)/vmtouch.8 clean: rm -f vmtouch vmtouch.8 uninstall: rm $(BINDIR)/vmtouch $(MANDIR)/vmtouch.8 vmtouch-1.3.0/README.md000066400000000000000000000017611306253002300144560ustar00rootroot00000000000000# vmtouch - the Virtual Memory Toucher ## Portable file system cache diagnostics and control vmtouch is a tool for learning about and controlling the file system cache of unix and unix-like systems. It is BSD licensed so you can basically do whatever you want with it. ### Quick install guide: $ git clone https://github.com/hoytech/vmtouch.git $ cd vmtouch $ make $ sudo make install ### More information * [Official Website](https://hoytech.com/vmtouch/) * [Online manual](https://github.com/hoytech/vmtouch/blob/master/vmtouch.pod) * [System tuning parameters](https://github.com/hoytech/vmtouch/blob/master/TUNING.md) ### Support To complement the open source community, Hoytech offers services related to vmtouch: * Advanced feature development * Support contracts * Training sessions Please contact Doug Hoyte for more information. ### Authors vmtouch is copyright (c) 2009-2016 Doug Hoyte and contributors. vmtouch-1.3.0/TODO000066400000000000000000000015541306253002300136670ustar00rootroot00000000000000* Machine readable input/output (JSON?) * Continually call mincore() every time a residency chart is drawn * Deprecate -w (it should be the default when combined with -d) * -a switch where you can specify flags to madvise() that should be applied to your mappings * Examples: -a sequential -a willneed -a noreserve * switch where you can specify flags to mmap() (would be nice to use -m but it's taken) * Examples: -m noreserve -m populate * Use cases: * Try opening large mappings with MAP_HUGETLB on linux 2.6.32+ * From Dennis (Apr 2012): "Additionally you might want to use MAP_NORESERVE for the mmap, this seems to speed things a bit up for multiple huge files." * Windows support some day? * https://msdn.microsoft.com/en-us/library/hh780543(v=vs.85).aspx * https://blogs.msdn.microsoft.com/oldnewthing/20170113-00/?p=95185 vmtouch-1.3.0/TUNING.md000066400000000000000000000040311306253002300145160ustar00rootroot00000000000000# System tuning parameters for vmtouch ## Limits on memory locking See your operating system's `mlock(2)` manual page since it probably describes the limitations on this system call which vmtouch uses. ### Linux * **locked memory rlimit**: Processes typically have a limit on the number of memory that is locked. This can be raised with `ulimit -l` (see `RLIMIT_MEMLOCK` in [setrlimit(2)](http://linux.die.net/man/2/setrlimit)) if you are the super-user. Processes with the `CAP_IPC_LOCK` are not affected by this limit, and it can be raised for unprivileged processes by editing [limits.conf](http://linux.die.net/man/5/limits.conf). * **vm.max_map_count**: This is a [sysctl](http://linux.die.net/man/8/sysctl) that controls the maximum number of VMAs (virtual memory areas) that a process can map. Since vmtouch needs a VMA for every file, this limits the number of files that can be locked by an individual vmtouch process. * Since Linux 2.6.9 (?) there is no system-wide limit on the amount of locked memory. ### FreeBSD * See [mlock(2)](https://www.freebsd.org/cgi/man.cgi?query=mlock&sektion=2&manpath=freebsd-release-ports) * **sysctls**: see `vm.max_wired` and `vm.stats.vm.v_wire_count` * **security.bsd.unprivileged_mlock**: Whether unprivileged users can lock memory ### OpenBSD * Has both a per-process resource limit and a system-wide limit on locked memory, see [mlock(2)](http://man.openbsd.org/mlock.2) ### Solaris On Solaris, the memory locked page limit can be set on per-project or per-zone basis, see resource-controls(5) man page for details. The limits can be manipulated and observed using prctl or zonecfg command, e.g.: $ prctl -n project.max-locked-memory -i process $$ process: 4690: bash NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT project.max-locked-memory usage 0B system 16.0EB max deny - Also, there is the PRIV_PROC_LOCK_MEMORY privilege which controls whether given process can lock memory. vmtouch-1.3.0/debian/000077500000000000000000000000001306253002300144145ustar00rootroot00000000000000vmtouch-1.3.0/debian/changelog000066400000000000000000000130711306253002300162700ustar00rootroot00000000000000vmtouch (1.3.0) unstable; urgency=medium [ doug ] * README * move some comments from .c into TODO and CHANGES * Patch from Marc Brooker: avoid overflowing stack when allocating array for mincore() output * Patch from Federico Lucifredi: Avoid ugly NaNs in output, Mac OS X report * gitignore, copyright update * Full OS X support: msync(2) evicts pages on this system * update * watcher script [ Alexey Ivanov ] * [debian] Added debian/ dir and Makefile [ soarpenguin ] * Update vmtouch.c * fix the error of *tp != '\0' [ Nick Barkas ] * change Debian arch from all to any * add Debian init script with default config [ doug ] * TODO update * HP-UX support from Shane Seymour (thanks!) * Note HP-UX support * HP-UX system notes from Shane Seymour * port man page to POD format * doc update * bump version * Deprecation notice for -p switch * tested CPUs note * S5 presentation: Intro to Virtual Memory * Presentation update * CHANGES notes, comment clean-up * tagging 0.8.0 [ Kristofer Karlsson ] * Add option to wait for daemon to mlock the files. [ doug ] * removing presentation since it now has its own repo [ Carsten Otto ] * ignore hardlink duplicates by default [ doug ] * Document Carsten Otto's -h switch in the man page * regenerate man page [ Emmanuel Kasper ] * Enhance Makefile: * generate manpage from build * add manpage to install target * add uninstall target * remove vmtouch.8 as it is now generated in build step [ doug ] * gitignore generated man-page [ Mikolaj Golub ] * Respect CC, CFLAGS, PREFIX, BINDIR, MANDIR variable * Fix "data argument not used by format string" warning. [ Jim Garrison ] * Fix to work with glibc 2.20 [ doug ] * make default CFLAGS a bit more user-friendly * bugfix for when trying to crawl entire filesystem [ Luka Blaskovic ] * Update debian packaging. [ Justas Lavišius ] * implement parse_range for -p * provide comments for parse_range * implement page mode (-p) for touch, evict and lock * provide better output for touch * document page mode (-p) * round up offset to nearest multiple of pagesize * remove unused variable pages_in_core in vmtouch_file * skip files smaller than offset provided in -p * Revert "provide better output for touch" [ doug ] * done by Justas -- thanks! * add some links to README * trailing whitespace * change default max file size to SIZE_MAX * releasing 1.0.1 * update copyright year * don't install watch-vmtouch script * fix dependencies in makefile, declare phony targets properly * bsd support in make install * proper headers in generated man-page * convert readme to mark-down * releasing 1.0.2 [ Patrik Modesto ] * fix debian packaging after README convert [ doug ] * bump copyright year * use https for vmtouch website links [ Matthew Fernandez ] * Fix: catch parsing double values that are too large. [ doug ] * Detect multiplication overflow in parse_size function * support block devices [ Justas Lavišius ] * Round range lower limit down, not up (Fixes #27) * Round down properly & allow 0 value in size/range [ Ancients ] * Added NO_ATIME open to prevent incrementing atime values when touching files. [ doug ] * Documentation on tuning system limits * note that professional support is available via hoytech software * Vladimir Kotal provided some details on the Solaris memory-locking limits * Forgot to add the note about PRIV_PROC_LOCK_MEMORY * make the O_NOATIME not interfere with compiling on systems that don't have this functionality * avoid uninitialized variable warnings on older compilers * releasing 1.1.0 [ Etienne Bruines ] * Added -i option to ignore files and directories. (#42) [ Doug Hoyte ] * some minor fixups for the -i feature * support wildcard matching in ignore list with fnmatch * -I feature which lets you process only files matching certain patterns * triage TODO a bit [ ecebuzz ] * Turn on C99 option in Makefile (#44) [ Doug Hoyte ] * releasing 1.2.0 * On linux, if an open fails due to EPERM then try again without O_NOATIME [ Doug ] * notes for possible windows support [ Antoine Beaupré ] * add myself to uploaders to sponsor this in debian * update standards to latest, no change * bump debhelper compat to 10 for multi-arch and auto dbg, no change * no reason why this is in extra, it should be optional like everything else * remove -dbg package, now build automatically * simplify install target * distribute TUNING.md as part of the documentation * WIP: prepare 1.2.1 changelog * setup gbp config for native package * remove unused substitution variables in control file * depend on lsb-base since we still use an init.d file * use standard BSD-3-clause short name to please lintian * remove boilerplate comment in copyright file * follow FHS for manpages location * remove unnused sbin dir * switch maintainership away from the original packager to upstream [ Tw ] * support adding files by file list [ Doug ] * typo/formatting * support "-b -" to read file list from stdin * support -0 in -b "batch mode" * document -b and -0 flags in manual [ Doug Hoyte ] * Use standard path for manpages (thanks anarcat) * -F option to avoid recursing into separate filesystems (requested by mailinglists35) [ doug ] * releasing 1.3.0 -- doug Thu, 16 Mar 2017 11:30:27 -0400 vmtouch (0.7.3) unstable; urgency=low * Initial release -- Alexey Ivanov Tue, 03 Apr 2012 17:25:33 +0400 vmtouch-1.3.0/debian/compat000066400000000000000000000000031306253002300156130ustar00rootroot0000000000000010 vmtouch-1.3.0/debian/control000066400000000000000000000011431306253002300160160ustar00rootroot00000000000000Source: vmtouch Section: admin Priority: optional Maintainer: Doug Hoyte Uploaders: Antoine Beaupré Build-Depends: debhelper (>= 10~) Standards-Version: 3.9.8 Homepage: https://hoytech.com/vmtouch/ Vcs-Git: git://github.com/hoytech/vmtouch.git Vcs-Browser: https://github.com/hoytech/vmtouch Package: vmtouch Architecture: any Depends: ${misc:Depends}, ${shlibs:Depends} , lsb-base (>= 3.0-6) Description: Portable file system cache diagnostics and control vmtouch is a tool for learning about and controlling the file system cache of unix and unix-like systems. vmtouch-1.3.0/debian/copyright000066400000000000000000000046251306253002300163560ustar00rootroot00000000000000Format: http://dep.debian.net/deps/dep5 Upstream-Name: vmtouch Source: https://github.com/hoytech/vmtouch Files: * Copyright: 2012 Doug Hoyte License: BSD-3-clause Copyright (c) 2009-2016 Doug Hoyte and contributors. All rights reserved. . Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. . THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Files: debian/* Copyright: 2012 Alexey Ivanov License: GPL-2+ This package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. . This package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with this program. If not, see . On Debian systems, the complete text of the GNU General Public License version 2 can be found in "/usr/share/common-licenses/GPL-2". vmtouch-1.3.0/debian/gbp.conf000066400000000000000000000000441306253002300160310ustar00rootroot00000000000000[DEFAULT] debian-tag = v%(version)s vmtouch-1.3.0/debian/rules000077500000000000000000000003011306253002300154660ustar00rootroot00000000000000#!/usr/bin/make -f # -*- makefile -*- # Uncomment this to turn on verbose mode. #export DH_VERBOSE=1 %: dh $@ override_dh_auto_install: $(MAKE) PREFIX=$(CURDIR)/debian/vmtouch/usr install vmtouch-1.3.0/debian/source/000077500000000000000000000000001306253002300157145ustar00rootroot00000000000000vmtouch-1.3.0/debian/source/format000066400000000000000000000000151306253002300171230ustar00rootroot000000000000003.0 (native) vmtouch-1.3.0/debian/vmtouch.default000066400000000000000000000005071306253002300174510ustar00rootroot00000000000000# Change to yes to enable running vmtouch as a daemon ENABLE_VMTOUCH=no # User and group to run as VMTOUCH_USER_GROUP=nobody:nogroup # Whitespace separated list of files and directories for vmtouch to operate on VMTOUCH_FILES="/nonexistent" # Options to pass to vmtouch itself. See vmtouch(8). VMTOUCH_OPTIONS="-d -l -m 1G" vmtouch-1.3.0/debian/vmtouch.docs000066400000000000000000000000311306253002300167450ustar00rootroot00000000000000README.md TODO TUNING.md vmtouch-1.3.0/debian/vmtouch.init000066400000000000000000000025761306253002300170000ustar00rootroot00000000000000#! /bin/sh ### BEGIN INIT INFO # Provides: vmtouch # Required-Start: $local_fs $remote_fs # Required-Stop: $local_fs $remote_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start vmtouch daemon # Description: Start vmtouch, the Virtual Memory Toucher ### END INIT INFO PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/sbin/vmtouch NAME=vmtouch DESC=vmtouch test -x $DAEMON || exit 0 # Override these default values in /etc/default/vmtouch ENABLE_VMTOUCH=no VMTOUCH_USER_GROUP=nobody:nogroup VMTOUCH_FILES="/nonexistent" VMTOUCH_OPTIONS="-d -l -m 1G" [ -r /etc/default/$NAME ] && . /etc/default/$NAME . /lib/lsb/init-functions do_start() { if [ $ENABLE_VMTOUCH = yes ]; then start-stop-daemon --start --quiet --chuid $VMTOUCH_USER_GROUP --exec $DAEMON -- $VMTOUCH_OPTIONS $VMTOUCH_FILES echo "$NAME." else echo "$NAME disabled in /etc/default/vmtouch" fi } do_stop() { start-stop-daemon --stop --quiet --exec $DAEMON } case "$1" in start) echo -n "Starting: " do_start ;; stop) echo -n "Stopping: " do_stop echo "$NAME." ;; restart|force-reload) echo -n "Restarting: " do_stop sleep 1 do_start ;; status) status_of_proc $DAEMON $NAME ;; *) N=/etc/init.d/$NAME echo "Usage: $N {start|stop|restart|force-reload|status}" >&2 exit 1 ;; esac exit 0 vmtouch-1.3.0/debian/vmtouch.manpages000066400000000000000000000000121306253002300176070ustar00rootroot00000000000000vmtouch.8 vmtouch-1.3.0/scripts/000077500000000000000000000000001306253002300146615ustar00rootroot00000000000000vmtouch-1.3.0/scripts/render-man-page.sh000077500000000000000000000000671306253002300201650ustar00rootroot00000000000000#!/bin/sh pod2man --section 8 vmtouch.pod > vmtouch.8 vmtouch-1.3.0/scripts/watch-vmtouch.pl000077500000000000000000000006441306253002300200160ustar00rootroot00000000000000#!/usr/bin/env perl use strict; my $path_to_vmtouch = ''; # leave empty if it is in path. otherwise it must end in / my $frequency = 0.25; # seconds between refreshes if (!@ARGV) { print "Usage: $0 \n"; exit -1; } while(1) { my $out = ''; foreach my $v (@ARGV) { $out .= `${path_to_vmtouch}vmtouch -v $v`; } system("clear"); print $out; select(undef, undef, undef, $frequency); } vmtouch-1.3.0/vmtouch.c000066400000000000000000000606711306253002300150350ustar00rootroot00000000000000/*********************************************************************** vmtouch - the Virtual Memory Toucher Portable file system cache diagnostics and control by Doug Hoyte (doug@hcsw.org) Compilation: gcc -Wall -O3 -o vmtouch vmtouch.c ************************************************************************ Copyright (c) 2009-2016 Doug Hoyte and contributors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ***********************************************************************/ #define VMTOUCH_VERSION "1.3.0" #define RESIDENCY_CHART_WIDTH 60 #define CHART_UPDATE_INTERVAL 0.1 #define MAX_CRAWL_DEPTH 1024 #define MAX_NUMBER_OF_IGNORES 1024 #define MAX_NUMBER_OF_FILENAME_FILTERS 1024 #define MAX_FILENAME_LENGTH 1024 #if defined(__linux__) || (defined(__hpux) && !defined(__LP64__)) // Make sure off_t is 64 bits on linux and when creating 32bit programs // on HP-UX. #define _FILE_OFFSET_BITS 64 #endif #ifdef __linux__ // Required for posix_fadvise() on some linux systems #define _XOPEN_SOURCE 600 // Required for mincore() on some linux systems #define _DEFAULT_SOURCE #define _BSD_SOURCE // for O_NOATIME #define _GNU_SOURCE #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #if defined(__linux__) // Used to find size of block devices #include #include #endif /* * To find out if the stat results from a single file correspond to a file we * have already seen, we need to compare both the device and the inode */ struct dev_and_inode { dev_t dev; ino_t ino; }; long pagesize; int64_t total_pages=0; int64_t total_pages_in_core=0; int64_t total_files=0; int64_t total_dirs=0; int64_t offset=0; int64_t max_len=0; unsigned int junk_counter; // just to prevent any compiler optimizations int curr_crawl_depth=0; ino_t crawl_inodes[MAX_CRAWL_DEPTH]; // remember all inodes (for files with inode count > 1) to find duplicates void *seen_inodes = NULL; dev_t orig_device = 0; int orig_device_inited = 0; int o_touch=0; int o_evict=0; int o_quiet=0; int o_verbose=0; int o_lock=0; int o_lockall=0; int o_daemon=0; int o_followsymlinks=0; int o_singlefilesystem=0; int o_ignorehardlinkeduplictes=0; size_t o_max_file_size=SIZE_MAX; int o_wait=0; static char *o_batch = NULL; int o_0_delim = 0; char *ignore_list[MAX_NUMBER_OF_IGNORES]; char *filename_filter_list[MAX_NUMBER_OF_FILENAME_FILTERS]; int number_of_ignores=0; int number_of_filename_filters=0; int exit_pipe[2]; int daemon_pid; void send_exit_signal(char code) { if (daemon_pid == 0 && o_wait) { if (write(exit_pipe[1], &code, 1) < 0) fprintf(stderr, "vmtouch: FATAL: write: %s", strerror(errno)); } } void usage() { printf("\n"); printf("vmtouch v%s - the Virtual Memory Toucher by Doug Hoyte\n", VMTOUCH_VERSION); printf("Portable file system cache diagnostics and control\n\n"); printf("Usage: vmtouch [OPTIONS] ... FILES OR DIRECTORIES ...\n\nOptions:\n"); printf(" -t touch pages into memory\n"); printf(" -e evict pages from memory\n"); printf(" -l lock pages in physical memory with mlock(2)\n"); printf(" -L lock pages in physical memory with mlockall(2)\n"); printf(" -d daemon mode\n"); printf(" -m max file size to touch\n"); printf(" -p use the specified portion instead of the entire file\n"); printf(" -f follow symbolic links\n"); printf(" -F don't crawl different filesystems\n"); printf(" -h also count hardlinked copies\n"); printf(" -i ignores files and directories that match this pattern\n"); printf(" -I only process files that match this pattern\n"); printf(" -b get files or directories from the list file\n"); printf(" -0 in batch mode (-b) separate paths with NUL byte instead of newline\n"); printf(" -w wait until all pages are locked (only useful together with -d)\n"); printf(" -v verbose\n"); printf(" -q quiet\n"); exit(1); } static void fatal(const char *fmt, ...) { va_list ap; char buf[4096]; va_start(ap, fmt); vsnprintf(buf, sizeof(buf), fmt, ap); va_end(ap); fprintf(stderr, "vmtouch: FATAL: %s\n", buf); send_exit_signal(1); exit(1); } static void warning(const char *fmt, ...) { va_list ap; char buf[4096]; va_start(ap, fmt); vsnprintf(buf, sizeof(buf), fmt, ap); va_end(ap); if (!o_quiet) fprintf(stderr, "vmtouch: WARNING: %s\n", buf); } static void reopen_all() { if (freopen("/dev/null", "r", stdin) == NULL || freopen("/dev/null", "w", stdout) == NULL || freopen("/dev/null", "w", stdout) == NULL) fatal("freopen: %s", strerror(errno)); } static int wait_for_child() { int exit_read = 0; char exit_value = 0; int wait_status; while (1) { struct timeval tv; fd_set rfds; FD_ZERO(&rfds); tv.tv_sec = 1; tv.tv_usec = 0; FD_SET(exit_pipe[0], &rfds); if (select(exit_pipe[0] + 1, &rfds, NULL, NULL, &tv) < 0) fatal("select: %s", strerror(errno)); if (waitpid(daemon_pid, &wait_status, WNOHANG) > 0) fatal("daemon shut down unexpectedly"); if (FD_ISSET(exit_pipe[0], &rfds)) break; } exit_read = read(exit_pipe[0], &exit_value, 1); if (exit_read < 0) fatal("read: %s", strerror(errno)); return exit_value; } void go_daemon() { daemon_pid = fork(); if (daemon_pid == -1) fatal("fork: %s", strerror(errno)); if (daemon_pid) { if (o_wait) exit(wait_for_child()); exit(0); } if (setsid() == -1) fatal("setsid: %s", strerror(errno)); if (!o_wait) reopen_all(); } char *pretty_print_size(int64_t inp) { static char output[100]; if (inp<1024) { snprintf(output, sizeof(output), "%" PRId64, inp); return output; } inp /= 1024; if (inp<1024) { snprintf(output, sizeof(output), "%" PRId64 "K", inp); return output; } inp /= 1024; if (inp<1024) { snprintf(output, sizeof(output), "%" PRId64 "M", inp); return output; } inp /= 1024; snprintf(output, sizeof(output), "%" PRId64 "G", inp); return output; } /* * Convert ASCII string to int64_t number * Note: The inp parameter can't be a character constant * because it will be overwritten. */ int64_t parse_size(char *inp) { char *tp; int len=strlen(inp); char *errstr = "bad size. examples: 4096, 4k, 100M, 1.5G"; char mult_char; int mult=1; double val; if (len < 1) fatal(errstr); mult_char = tolower(inp[len-1]); if (isalpha(mult_char)) { switch(mult_char) { case 'k': mult = 1024; break; case 'm': mult = 1024*1024; break; case 'g': mult = 1024*1024*1024; break; default: fatal("unknown size multiplier: %c", mult_char); } inp[len-1] = '\0'; } val = strtod(inp, &tp); if (val < 0 || val == HUGE_VAL || *tp != '\0') fatal(errstr); val *= mult; if (val > INT64_MAX) fatal(errstr); return (int64_t) val; } int64_t bytes2pages(int64_t bytes) { return (bytes+pagesize-1) / pagesize; } void parse_range(char *inp) { char *token; int64_t upper_range=0; int64_t lower_range=0; token = strsep(&inp,"-"); if (inp == NULL) upper_range = parse_size(token); // single value provided else { if (*token != '\0') lower_range = parse_size(token); // value before hyphen token = strsep(&inp,"-"); if (*token != '\0') upper_range = parse_size(token); // value after hyphen if ((token = strsep(&inp,"-")) != NULL) fatal("malformed range: multiple hyphens"); } // offset must be multiple of pagesize offset = (lower_range / pagesize) * pagesize; if (upper_range) { if (upper_range <= offset) fatal("range limits out of order"); max_len = upper_range - offset; } } void parse_ignore_item(char *inp) { if (inp == NULL) { return; } if (strlen(inp) > MAX_FILENAME_LENGTH) { fatal("too long pattern provided to -i: %s", inp); return; } if (number_of_ignores >= MAX_NUMBER_OF_IGNORES) { fatal("too many patterns passed to -i. Max is %d", MAX_NUMBER_OF_IGNORES); return; } ignore_list[number_of_ignores] = strdup(inp); number_of_ignores++; } void parse_filename_filter_item(char *inp) { if (inp == NULL) { return; } if (strlen(inp) > MAX_FILENAME_LENGTH) { fatal("too long pattern provided to -I: %s", inp); return; } if (number_of_filename_filters >= MAX_NUMBER_OF_FILENAME_FILTERS) { fatal("too many patterns passed to -I. Max is %d", MAX_NUMBER_OF_FILENAME_FILTERS); return; } filename_filter_list[number_of_filename_filters] = strdup(inp); number_of_filename_filters++; } int aligned_p(void *p) { return 0 == ((long)p & (pagesize-1)); } int is_mincore_page_resident(char p) { return p & 0x1; } void increment_nofile_rlimit() { struct rlimit r; if (getrlimit(RLIMIT_NOFILE, &r)) fatal("increment_nofile_rlimit: getrlimit (%s)", strerror(errno)); r.rlim_cur = r.rlim_max + 1; r.rlim_max = r.rlim_max + 1; if (setrlimit(RLIMIT_NOFILE, &r)) { if (errno == EPERM) { if (getuid() == 0 || geteuid() == 0) fatal("system open file limit reached"); fatal("open file limit reached and unable to increase limit. retry as root"); } fatal("increment_nofile_rlimit: setrlimit (%s)", strerror(errno)); } } double gettimeofday_as_double() { struct timeval tv; gettimeofday(&tv, NULL); return tv.tv_sec + (tv.tv_usec/1000000.0); } void print_page_residency_chart(FILE *out, char *mincore_array, int64_t pages_in_file) { int64_t pages_in_core=0; int64_t pages_per_char; int64_t i,j=0,curr=0; if (pages_in_file <= RESIDENCY_CHART_WIDTH) pages_per_char = 1; else pages_per_char = (pages_in_file / RESIDENCY_CHART_WIDTH) + 1; fprintf(out, "\r["); for (i=0; i o_max_file_size) { warning("file %s too large, skipping", path); goto bail; } if (max_len > 0 && (offset + max_len) < len_of_file) { len_of_range = max_len; } else if (offset >= len_of_file) { warning("file %s smaller than offset, skipping", path); goto bail; } else { len_of_range = len_of_file - offset; } mem = mmap(NULL, len_of_range, PROT_READ, MAP_SHARED, fd, offset); if (mem == MAP_FAILED) { warning("unable to mmap file %s (%s), skipping", path, strerror(errno)); goto bail; } if (!aligned_p(mem)) fatal("mmap(%s) wasn't page aligned", path); pages_in_range = bytes2pages(len_of_range); total_pages += pages_in_range; if (o_evict) { if (o_verbose) printf("Evicting %s\n", path); #if defined(__linux__) || defined(__hpux) if (posix_fadvise(fd, offset, len_of_range, POSIX_FADV_DONTNEED)) warning("unable to posix_fadvise file %s (%s)", path, strerror(errno)); #elif defined(__FreeBSD__) || defined(__sun__) || defined(__APPLE__) if (msync(mem, len_of_range, MS_INVALIDATE)) warning("unable to msync invalidate file %s (%s)", path, strerror(errno)); #else fatal("cache eviction not (yet?) supported on this platform"); #endif } else { double last_chart_print_time=0.0, temp_time; char *mincore_array = malloc(pages_in_range); if (mincore_array == NULL) fatal("Failed to allocate memory for mincore array (%s)", strerror(errno)); // 3rd arg to mincore is char* on BSD and unsigned char* on linux if (mincore(mem, len_of_range, (void*)mincore_array)) fatal("mincore %s (%s)", path, strerror(errno)); for (i=0; i (last_chart_print_time+CHART_UPDATE_INTERVAL)) { last_chart_print_time = temp_time; print_page_residency_chart(stdout, mincore_array, pages_in_range); } } } } if (o_verbose) { print_page_residency_chart(stdout, mincore_array, pages_in_range); printf("\n"); } free(mincore_array); } if (o_lock) { if (mlock(mem, len_of_range)) fatal("mlock: %s (%s)", path, strerror(errno)); } bail: if (!o_lock && !o_lockall && mem) { if (munmap(mem, len_of_range)) warning("unable to munmap file %s (%s)", path, strerror(errno)); } if (fd != -1) { close(fd); } } // compare device and inode information int compare_func(const void *p1, const void *p2) { const struct dev_and_inode *kp1 = p1, *kp2 = p2; int cmp1; cmp1 = (kp1->ino > kp2->ino) - (kp1->ino < kp2->ino); if (cmp1 != 0) return cmp1; return (kp1->dev > kp2->dev) - (kp1->dev < kp2->dev); } // add device and inode information to the tree of known inodes static inline void add_object (struct stat *st) { struct dev_and_inode *newp = malloc (sizeof (struct dev_and_inode)); if (newp == NULL) { fatal("malloc: out of memory"); } newp->dev = st->st_dev; newp->ino = st->st_ino; if (tsearch(newp, &seen_inodes, compare_func) == NULL) { fatal("tsearch: out of memory"); } } int is_ignored(const char* path) { char *path_copy; int match; if (!number_of_ignores) return 0; path_copy = strdup(path); match = 0; char *filename = basename(path_copy); for (int i = 0; i < number_of_ignores; i++) { if (fnmatch(ignore_list[i], filename, 0) == 0) { match = 1; break; } } free(path_copy); return match; } int is_filename_filtered(const char* path) { char *path_copy; int match; if (!number_of_filename_filters) return 1; path_copy = strdup(path); match = 0; char *filename = basename(path_copy); for (int i = 0; i < number_of_filename_filters; i++) { if (fnmatch(filename_filter_list[i], filename, 0) == 0) { match = 1; break; } } free(path_copy); return match; } // return true only if the device and inode information has not been added before static inline int find_object(struct stat *st) { struct dev_and_inode obj; void *res; obj.dev = st->st_dev; obj.ino = st->st_ino; res = (void *) tfind(&obj, &seen_inodes, compare_func); return res != (void *) NULL; } void vmtouch_crawl(char *path) { struct stat sb; DIR *dirp; struct dirent *de; char npath[PATH_MAX]; int res; int tp_path_len = strlen(path); int i; if (path[tp_path_len-1] == '/' && tp_path_len > 1) path[tp_path_len-1] = '\0'; // prevent ugly double slashes when printing path names if (is_ignored(path)) { return; } res = o_followsymlinks ? stat(path, &sb) : lstat(path, &sb); if (res) { warning("unable to stat %s (%s)", path, strerror(errno)); return; } else { if (S_ISLNK(sb.st_mode)) { warning("not following symbolic link %s", path); return; } if (o_singlefilesystem) { if (!orig_device_inited) { orig_device = sb.st_dev; orig_device_inited = 1; } else { if (sb.st_dev != orig_device) { warning("not recursing into separate filesystem %s", path); return; } } } if (!o_ignorehardlinkeduplictes && sb.st_nlink > 1) { /* * For files with more than one link to it, ignore it if we already know * inode. Without this check files copied as hardlinks (cp -al) are * counted twice (which may lead to a cache usage of more than 100% of * RAM). */ if (find_object(&sb)) { // we already saw the device and inode referenced by this file return; } else { add_object(&sb); } } if (S_ISDIR(sb.st_mode)) { for (i=0; id_name, ".") == 0 || strcmp(de->d_name, "..") == 0) continue; if (snprintf(npath, sizeof(npath), "%s/%s", path, de->d_name) >= sizeof(npath)) { warning("path too long %s", path); goto bail; } curr_crawl_depth++; vmtouch_crawl(npath); curr_crawl_depth--; } bail: if (closedir(dirp)) { warning("unable to closedir %s (%s)", path, strerror(errno)); return; } } else if (S_ISLNK(sb.st_mode)) { warning("not following symbolic link %s", path); return; } else if (S_ISREG(sb.st_mode) || S_ISBLK(sb.st_mode)) { if (is_filename_filtered(path)) { total_files++; vmtouch_file(path); } } else { warning("skipping non-regular file: %s", path); } } } static void vmtouch_batch_crawl(const char *path) { FILE *f; char *line = NULL; size_t len = 0; ssize_t read; int delim = o_0_delim ? '\0' : '\n'; if (!strcmp(path, "-")) { f = stdin; } else { f = fopen(path, "r"); if (!f) { warning("unable to open %s (%s), skipping", path, strerror(errno)); return; } } while ((read = getdelim(&line, &len, delim, f)) != -1) { // strip the newline character line[read-1] = '\0'; vmtouch_crawl(line); } free(line); fclose(f); } int main(int argc, char **argv) { int ch, i; char *prog = argv[0]; struct timeval start_time; struct timeval end_time; if (pipe(exit_pipe)) fatal("pipe: %s", strerror(errno)); pagesize = sysconf(_SC_PAGESIZE); while((ch = getopt(argc, argv,"tevqlLdfFh0i:I:p:b:m:w")) != -1) { switch(ch) { case '?': usage(); break; case 't': o_touch = 1; break; case 'e': o_evict = 1; break; case 'q': o_quiet = 1; break; case 'v': o_verbose++; break; case 'l': o_lock = 1; o_touch = 1; break; case 'L': o_lockall = 1; o_touch = 1; break; case 'd': o_daemon = 1; break; case 'f': o_followsymlinks = 1; break; case 'F': o_singlefilesystem = 1; break; case 'h': o_ignorehardlinkeduplictes = 1; break; case 'p': parse_range(optarg); break; case 'i': parse_ignore_item(optarg); break; case 'I': parse_filename_filter_item(optarg); break; case 'm': { int64_t val = parse_size(optarg); o_max_file_size = (size_t) val; if (val != (int64_t) o_max_file_size) fatal("value for -m too big to fit in a size_t"); break; } case 'w': o_wait = 1; break; case 'b': o_batch = optarg; break; case '0': o_0_delim = 1; break; } } argc -= optind; argv += optind; if (o_touch) { if (o_evict) fatal("invalid option combination: -t and -e"); } if (o_evict) { if (o_lock) fatal("invalid option combination: -e and -l"); } if (o_lock && o_lockall) fatal("invalid option combination: -l and -L"); if (o_daemon) { if (!(o_lock || o_lockall)) fatal("daemon mode must be combined with -l or -L"); if (!o_wait) { o_quiet = 1; o_verbose = 0; } } if (o_wait && !o_daemon) fatal("wait mode must be combined with -d"); if (o_quiet && o_verbose) fatal("invalid option combination: -q and -v"); if (!argc && !o_batch) { printf("%s: no files or directories specified\n", prog); usage(); } // Must be done now because mlock() not inherited across fork() if (o_daemon) go_daemon(); gettimeofday(&start_time, NULL); if (o_batch) { vmtouch_batch_crawl(o_batch); } for (i=0; i. The mappings are opened read-only. It recursively crawls any directories and does the same to all files it finds within them. With no options, vmtouch will not read from (touch) any memory pages. It will only use C to determine how many pages of each file are actually resident in memory. Before exiting, it will print a summary of the total pages encountered and how many were resident. =over =item -t Touch virtual memory pages. Reads a byte from each page of the file. If the page is not resident in memory, a page fault will be generated and the page will be read from disk into the file system's memory cache. Note: Although each page is guaranteed to have been brought into memory, the page might be evicted from memory by the time the vmtouch command completes. =item -e Evict the mapped pages from the file system cache. They will need to be read in from disk the next time they are accessed. This is the inverse of C<-t>. Note: Even if the eviction is successful, pages may be paged back into memory by the time the vmtouch command completes. Note: This option is not portable to all systems. See L below. =item -l Lock pages into physical memory. This option works the same as C<-t> except it calls C on all the memory mappings and doesn't close the descriptors when finished. At the end of the crawl, if successful, vmtouch will block indefinitely. The files will be locked in physical memory until the vmtouch process is killed. Note: While the vmtouch process is holding memory locks, any processes that access the locked pages will not cause non-resident page faults or address-translation faults although they may still cause TLB misses. Note: Because vmtouch holds file descriptors open it may reach the C process file descriptor limit. In this case it will try to increase the descriptor limit which will only work if the process is run with root privileges. Similarly, root privileges are required to exceed the C limit. Even with root privileges, C<-l> is limited by both the system file descriptor limit and the system limit on "wired memory". =item -L This option is the same as C<-l> except that it uses C at the end of the crawl rather than individually Cing each file. Because of this, other unrelated pages belonging to the vmtouch process will also be locked in memory. =item -d Daemon mode. After performing the crawl, disassociate from the terminal and run in the background as a daemon. This option can only be used with the C<-l> or C<-L> locking modes. =item -m Maximum file size to map into virtual memory. Files that are larger than this will be skipped. Examples: 4096, 4k, 100M, 1.5G. The default is 500M. =item -p or Page mode. Maps the portion of the file specified by a range instead of the entire file. Size format same as for C<-m>. Omitted range start (end) value means start (end) of file. Single value is equivalent to -, i.e. map the first bytes. Examples: 4k-50k, 100M-2G, -5M, -. =item -f Follow symbolic links. With this option, vmtouch will descend into symbolic links that point to directories and will touch regular files pointed to by symbolic links. Symbolic link loops are detected and issue warnings. =item -i Can be specified multiple times. Ignores files and directories that match any of the provided patterns. The pattern may include wildcards (remember to escape them from your shell). This option stops the crawl, so can be used to ignore directories and all their contents. Example: vmtouch -i .git -i '*.bak' . =item -I Can be specified multiple times. Only processes filenames matching one or more of the provided patterns. The pattern may include wildcards (remember to escape them from your shell). Example: vmtouch -I '*.c' -I '*.h' . =item -b The list of files/directories to crawl is read from the specified list file, which by default should be a newline-separated list, for example the output from the find command. If the list file is "-" then this list is read from standard input. Example: find /usr/lib -type f | vmtouch -b - =item -0 If -b ("batch mode") is in effect, assume the list file is delimited with NUL bytes instead of newlines, for example the output from find -print0. This is useful in case your filenames contain newline characters themselves. =item -v Verbose mode. While crawling, print out every file being processed along with its total number of pages and the number of its pages that are currently resident in memory to standard output. =item -q Quiet mode. Suppress the end of crawl summary and all warnings that are normally printed to standard error. On success print nothing. Fatal errors print a single error message line to standard error. =item -h Normally, if multiple files both point to the same inode then vmtouch will ignore all but the first it finds so as to avoid double-counting their pages. This option overrides this behaviour and double-counts anyways. =back =head1 PORTABILITY The page residency summaries depend on C which first appeared in 4.4BSD but is not present on all unix systems. The C<-l> and C<-L> locking options depends on C or C, both of which are specified by POSIX.1b-1993, Real-Time Extensions. The C<-e> page eviction option is the least portable. On Linux it uses C with C advice to inform the kernel that the file should be evicted from the file system cache. C is specified by POSIX.1-2003 TC1. On FreeBSD, the pages are invalidated with C's C flag. C is specified by POSIX.1b-1993, Real-Time Extensions, although this call is not required to remove pages from the file system cache. Some systems like OpenBSD 4.3 don't have C, don't evict the pages on an C/C, and don't evict the pages with C/C so C<-e> isn't supported on those systems yet. Using C<-e> on systems that don't yet support it is a fatal error. =head1 SUPPORTED SYSTEMS All vmtouch features have been confirmed to work on the following systems: =over =item Linux 2.6+ =item FreeBSD 4.X =item FreeBSD 7.X =item Solaris 10 =item OS X 10.x =item HP-UX 11.31+patches (see below) =back Systems that support everything except eviction: =over =item OpenBSD 4.3 =back CPUs that have been tested: =over =item x86 =item amd64 (x86-64) =item SPARC =item ARMv7 =back We would like to support as many systems as possible so please send us any success reports, failure reports or patches. Thanks! =head1 SYSTEM NOTES Shane Seymour did the HP-UX port and says that either 32-bit or 64-bit binaries can be compiled (just use C<+DD64> for 64-bit). However, C was added to HP-UX 11.31 via patches and at least the following patches need to be installed: PHKL_38651, PHKL_38708, PHKL_38686, PHKL_38688, and PHCO_38658 (or patches that supersede those ones). =head1 SEE ALSO Not all the following manual pages may exist in every unix dialect to which vmtouch has been ported. vmstat(8), touch(1), mmap(2), mincore(2), mlock(2), mlockall(2), msync(2), madvise(2), posix_fadvise(2) =head1 AUTHOR Written by Doug Hoyte =cut