agedu-20211129.8cd63c5/ 0000755 0001750 0001750 00000000000 14151034324 013151 5 ustar simon simon agedu-20211129.8cd63c5/agedu.1 0000644 0001750 0001750 00000111727 14151034324 014331 0 ustar simon simon .\" agedu version 20211129.8cd63c5
.ie \n(.g .ds Aq \(aq
.el .ds Aq '
.TH "agedu" "1" "2008\(hy11\(hy02" "Simon\ Tatham" "Simon\ Tatham"
.SH "NAME"
.PP
\fBagedu\fP - correlate disk usage with last-access times to identify large and disused data
.SH "SYNOPSIS"
.PP
.nf
\fBagedu\fP\ [\ \fIoptions\fP\ ]\ \fIaction\fP\ [\fIaction\fP...]
.fi
.SH "DESCRIPTION"
.PP
\fBagedu\fP scans a directory tree and produces reports about how much disk space is used in each directory and subdirectory, and also how that usage of disk space corresponds to files with last-access times a long time ago.
.PP
In other words, \fBagedu\fP is a tool you might use to help you free up disk space. It lets you see which directories are taking up the most space, as \fBdu\fP does; but unlike \fBdu\fP, it also distinguishes between large collections of data which are still in use and ones which have not been accessed in months or years - for instance, large archives downloaded, unpacked, used once, and never cleaned up. Where \fBdu\fP helps you find what\*(Aqs using your disk space, \fBagedu\fP helps you find what\*(Aqs \fIwasting\fP your disk space.
.PP
\fBagedu\fP has several operating modes. In one mode, it scans your disk and builds an index file containing a data structure which allows it to efficiently retrieve any information it might need. Typically, you would use it in this mode first, and then run it in one of a number of `query' modes to display a report of the disk space usage of a particular directory and its subdirectories. Those reports can be produced as plain text (much like \fBdu\fP) or as HTML. \fBagedu\fP can even run as a miniature web server, presenting each directory\*(Aqs HTML report with hyperlinks to let you navigate around the file system to similar reports for other directories.
.PP
So you would typically start using \fBagedu\fP by telling it to do a scan of a directory tree and build an index. This is done with a command such as
.PP
.nf
$\ \fBagedu\ \-s\ /home/fred\fP
.fi
.PP
which will build a large data file called \fBagedu.dat\fP in your current directory. (If that current directory is \fIinside\fP \fB/home/fred\fP, don\*(Aqt worry - \fBagedu\fP is smart enough to discount its own index file.)
.PP
Having built the index, you would now query it for reports of disk space usage. If you have a graphical web browser, the simplest and nicest way to query the index is by running \fBagedu\fP in web server mode:
.PP
.nf
$\ \fBagedu\ \-w\fP
.fi
.PP
which will print (among other messages) a URL on its standard output along the lines of
.PP
.nf
URL:\ http://127.0.0.1:48638/
.fi
.PP
(That URL will always begin with `\fB127.\fP', meaning that it\*(Aqs in the \fBlocalhost\fP address space. So only processes running on the same computer can even try to connect to that web server, and also there is access control to prevent other users from seeing it - see below for more detail.)
.PP
Now paste that URL into your web browser, and you will be shown a graphical representation of the disk usage in \fB/home/fred\fP and its immediate subdirectories, with varying colours used to show the difference between disused and recently-accessed data. Click on any subdirectory to descend into it and see a report for its subdirectories in turn; click on parts of the pathname at the top of any page to return to higher-level directories. When you\*(Aqve finished browsing, you can just press Ctrl-D to send an end-of-file indication to \fBagedu\fP, and it will shut down.
.PP
After that, you probably want to delete the data file \fBagedu.dat\fP, since it\*(Aqs pretty large. In fact, the command \fBagedu -R\fP will do this for you; and you can chain \fBagedu\fP commands on the same command line, so that instead of the above you could have done
.PP
.nf
$\ \fBagedu\ \-s\ /home/fred\ \-w\ \-R\fP
.fi
.PP
for a single self-contained run of \fBagedu\fP which builds its index, serves web pages from it, and cleans it up when finished.
.PP
In some situations, you might want to scan the directory structure of one computer, but run \fBagedu\fP\*(Aqs user interface on another. In that case, you can do your scan using the \fBagedu -S\fP option in place of \fBagedu -s\fP, which will make \fBagedu\fP not bother building an index file but instead just write out its scan results in plain text on standard output; then you can funnel that output to the other machine using SSH (or whatever other technique you prefer), and there, run \fBagedu -L\fP to load in the textual dump and turn it into an index file. For example, you might run a command like this (plus any \fBssh\fP options you need) on the machine you want to scan:
.PP
.nf
$\ \fBagedu\ \-S\ /home/fred\ |\ ssh\ indexing\-machine\ agedu\ \-L\fP
.fi
.PP
or, equivalently, run something like this on the other machine:
.PP
.nf
$\ \fBssh\ machine\-to\-scan\ agedu\ \-S\ /home/fred\ |\ agedu\ \-L\fP
.fi
.PP
Either way, the \fBagedu -L\fP command will create an \fBagedu.dat\fP index file, which you can then use with \fBagedu -w\fP just as above.
.PP
(Another way to do this might be to build the index file on the first machine as normal, and then just copy it to the other machine once it's complete. However, for efficiency, the index file is formatted differently depending on the CPU architecture that \fBagedu\fP is compiled for. So if that doesn\*(Aqt match between the two machines - e.g. if one is a 32-bit machine and one 64-bit - then \fBagedu.dat\fP files written on one machine will not work on the other. The technique described above using \fB-S\fP and \fB-L\fP should work between any two machines.)
.PP
If you don't have a graphical web browser, you can do text-based queries instead of using \fBagedu\fP\*(Aqs web interface. Having scanned \fB/home/fred\fP in any of the ways suggested above, you might run
.PP
.nf
$\ \fBagedu\ \-t\ /home/fred\fP
.fi
.PP
which again gives a summary of the disk usage in \fB/home/fred\fP and its immediate subdirectories; but this time \fBagedu\fP will print it on standard output, in much the same format as \fBdu\fP. If you then want to find out how much \fIold\fP data is there, you can add the \fB-a\fP option to show only files last accessed a certain length of time ago. For example, to show only files which haven\*(Aqt been looked at in six months or more:
.PP
.nf
$\ \fBagedu\ \-t\ /home/fred\ \-a\ 6m\fP
.fi
.PP
That's the essence of what \fBagedu\fP does. It has other modes of operation for more complex situations, and the usual array of configurable options. The following sections contain a complete reference for all its functionality.
.SH "OPERATING MODES"
.PP
This section describes the operating modes supported by \fBagedu\fP. Each of these is in the form of a command-line option, sometimes with an argument. Multiple operating-mode options may appear on the command line, in which case \fBagedu\fP will perform the specified actions one after another. For instance, as shown in the previous section, you might want to perform a disk scan and immediately launch a web server giving reports from that scan.
.IP "\fB-s\fP \fIdirectory\fP or \fB--scan\fP \fIdirectory\fP"
In this mode, \fBagedu\fP scans the file system starting at the specified directory, and indexes the results of the scan into a large data file which other operating modes can query.
.RS
.PP
By default, the scan is restricted to a single file system (since the expected use of \fBagedu\fP is that you would probably use it because a particular disk partition was running low on space). You can remove that restriction using the \fB--cross-fs\fP option; other configuration options allow you to include or exclude files or entire subdirectories from the scan. See the next section for full details of the configurable options.
.PP
The index file is created with restrictive permissions, in case the file system you are scanning contains confidential information in its structure.
.PP
Index files are dependent on the characteristics of the CPU architecture you created them on. You should not expect to be able to move an index file between different types of computer and have it continue to work. If you need to transfer the results of a disk scan to a different kind of computer, see the \fB-D\fP and \fB-L\fP options below.
.RE
.IP "\fB-w\fP or \fB--web\fP"
In this mode, \fBagedu\fP expects to find an index file already written. It allocates a network port, and starts up a web server on that port which serves reports generated from the index file. By default it invents its own URL and prints it out.
.RS
.PP
The web server runs until \fBagedu\fP receives an end-of-file event on its standard input. (The expected usage is that you run it from the command line, immediately browse web pages until you\*(Aqre satisfied, and then press Ctrl-D.) To disable the EOF behaviour, use the \fB--no-eof\fP option.
.PP
In case the index file contains any confidential information about your file system, the web server protects the pages it serves from access by other people. On Linux, this is done transparently by means of using \fB/proc/net/tcp\fP to check the owner of each incoming connection; failing that, the web server will require a password to view the reports, and \fBagedu\fP will print the password it invented on standard output along with the URL.
.PP
Configurable options for this mode let you specify your own address and port number to listen on, and also specify your own choice of authentication method (including turning authentication off completely) and a username and password of your choice.
.RE
.IP "\fB-t\fP \fIdirectory\fP or \fB--text\fP \fIdirectory\fP"
In this mode, \fBagedu\fP generates a textual report on standard output, listing the disk usage in the specified directory and all its subdirectories down to a given depth. By default that depth is 1, so that you see a report for \fIdirectory\fP itself and all of its immediate subdirectories. You can configure a different depth (or no depth limit) using \fB-d\fP, described in the next section.
.RS
.PP
Used on its own, \fB-t\fP merely lists the \fItotal\fP disk usage in each subdirectory; \fBagedu\fP\*(Aqs additional ability to distinguish unused from recently-used data is not activated. To activate it, use the \fB-a\fP option to specify a minimum age.
.PP
The directory structure stored in \fBagedu\fP\*(Aqs index file is treated as a set of literal strings. This means that you cannot refer to directories by synonyms. So if you ran \fBagedu -s .\fP, then all the path names you later pass to the \fB-t\fP option must be either `\fB.\fP' or begin with `\fB./\fP'. Similarly, symbolic links within the directory you scanned will not be followed; you must refer to each directory by its canonical, symlink-free pathname.
.RE
.IP "\fB-R\fP or \fB--remove\fP"
In this mode, \fBagedu\fP deletes its index file. Running just \fBagedu -R\fP on its own is therefore equivalent to typing \fBrm agedu.dat\fP. However, you can also put \fB-R\fP on the end of a command line to indicate that \fBagedu\fP should delete its index file after it finishes performing other operations.
.IP "\fB-S\fP \fIdirectory\fP or \fB--scan-dump\fP \fIdirectory\fP"
In this mode, \fBagedu\fP will scan a directory tree and convert the results straight into a textual dump on standard output, without generating an index file at all. The dump data is intended for \fBagedu -L\fP to read.
.IP "\fB-L\fP or \fB--load\fP"
In this mode, \fBagedu\fP expects to read a dump produced by the \fB-S\fP option from its standard input. It constructs an index file from that dump, exactly as it would have if it had read the same data from a disk scan in \fB-s\fP mode.
.IP "\fB-D\fP or \fB--dump\fP"
In this mode, \fBagedu\fP reads an existing index file and produces a dump of its contents on standard output, in the same format used by \fB-S\fP and \fB-L\fP. This option could be used to convert an existing index file into a format acceptable to a different kind of computer, by dumping it using \fB-D\fP and then loading the dump back in on the other machine using \fB-L\fP.
.RS
.PP
(The output of \fBagedu -D\fP on an existing index file will not be exactly \fIidentical\fP to what \fBagedu -S\fP would have originally produced, due to a difference in treatment of last-access times on directories. However, it should be effectively equivalent for most purposes. See the documentation of the \fB--dir-atime\fP option in the next section for further detail.)
.RE
.IP "\fB-H\fP \fIdirectory\fP or \fB--html\fP \fIdirectory\fP"
In this mode, \fBagedu\fP will generate an HTML report of the disk usage in the specified directory and its immediate subdirectories, in the same form that it serves from its web server in \fB-w\fP mode.
.RS
.PP
By default, a single HTML report will be generated and simply written to standard output, with no hyperlinks pointing to other similar pages. If you also specify the \fB-d\fP option (see below), \fBagedu\fP will instead write out a collection of HTML files with hyperlinks between them, and call the top-level file \fBindex.html\fP.
.RE
.IP "\fB--cgi\fP"
In this mode, \fBagedu\fP will run as the bulk of a CGI script which provides the same set of web pages as the built-in web server would. It will read the usual CGI environment variables, and write CGI-style data to its standard output.
.RS
.PP
The actual CGI program itself should be a tiny wrapper around \fBagedu\fP which passes it the \fB--cgi\fP option, and also (probably) \fB-f\fP to locate the index file. \fBagedu\fP will do everything else. For example, your script might read
.PP
.nf
#!/bin/sh
\fI/some/path/to/\fPagedu\ \-\-cgi\ \-f\ \fI/some/other/path/to/\fPagedu.dat
.fi
.PP
(Note that \fBagedu\fP will produce the \fIentire\fP CGI output, including status code, HTTP headers and the full HTML document. If you try to surround the call to \fBagedu --cgi\fP with code that adds your own HTML header and footer, you won\*(Aqt get the results you want, and \fBagedu\fP\*(Aqs HTTP-level features such as auto-redirecting to canonical versions of URIs will stop working.)
.PP
No access control is performed in this mode: restricting access to CGI scripts is assumed to be the job of the web server.
.RE
.IP "\fB--presort\fP and \fB--postsort\fP"
In these two modes, \fBagedu\fP will expect to read a textual data dump from its standard input of the form produced by \fB-S\fP (and \fB-D\fP). It will transform the data into a different version of its text dump format, and write the transformed version on standard output.
.RS
.PP
The ordinary dump file format is reasonably readable, but loading it into an index file using \fBagedu -L\fP requires it to be sorted in a specific order, which is complicated to describe and difficult to implement using ordinary Unix sorting tools. So if you want to construct your own data dump from a source of your own that \fBagedu\fP itself doesn\*(Aqt know how to scan, you will need to make sure it\*(Aqs sorted in the right order.
.PP
To help with this, \fBagedu\fP provides a secondary dump format which is `sortable', in the sense that ordinary \fBsort\fP(\fI1\fP) without arguments will arrange it into the right order. However, the sortable format is much more unreadable and also twice the size, so you wouldn\*(Aqt want to write it directly!
.PP
So the recommended procedure is to generate dump data in the ordinary format; then pipe it through \fBagedu --presort\fP to turn it into the sortable format; then sort it; \fIthen\fP pipe it into \fBagedu -L\fP (which can accept either the normal or the sortable format as input). For example:
.PP
.nf
\fIgenerate_custom_data.sh\fP\ |\ agedu\ \-\-presort\ |\ sort\ |\ agedu\ \-L
.fi
.PP
If you need to transform the sorted dump file back into the ordinary format, \fBagedu --postsort\fP can do that. But since \fBagedu -L\fP can accept either format as input, you may not need to.
.RE
.IP "\fB-h\fP or \fB--help\fP"
Causes \fBagedu\fP to print some help text and terminate immediately.
.IP "\fB-V\fP or \fB--version\fP"
Causes \fBagedu\fP to print its version number and terminate immediately.
.SH "OPTIONS"
.PP
This section describes the various configuration options that affect \fBagedu\fP\*(Aqs operation in one mode or another.
.PP
The following option affects nearly all modes (except \fB-S\fP):
.IP "\fB-f\fP \fIfilename\fP or \fB--file\fP \fIfilename\fP"
Specifies the location of the index file which \fBagedu\fP creates, reads or removes depending on its operating mode. By default, this is simply `\fBagedu.dat\fP', in whatever is the current working directory when you run \fBagedu\fP.
.PP
The following options affect the disk-scanning modes, \fB-s\fP and \fB-S\fP:
.IP "\fB--cross-fs\fP and \fB--no-cross-fs\fP"
These configure whether or not the disk scan is permitted to cross between different file systems. The default is not to: \fBagedu\fP will normally skip over subdirectories on which a different file system is mounted. This makes it convenient when you want to free up space on a particular file system which is running low. However, in other circumstances you might wish to see general information about the use of space no matter which file system it\*(Aqs on (for instance, if your real concern is your backup media running out of space, and if your backups do not treat different file systems specially); in that situation, use \fB--cross-fs\fP.
.RS
.PP
(Note that this default is the opposite way round from the corresponding option in \fBdu\fP.)
.RE
.IP "\fB--prune\fP \fIwildcard\fP and \fB--prune-path\fP \fIwildcard\fP"
These cause particular files or directories to be omitted entirely from the scan. If \fBagedu\fP\*(Aqs scan encounters a file or directory whose name matches the wildcard provided to the \fB--prune\fP option, it will not include that file in its index, and also if it\*(Aqs a directory it will skip over it and not scan its contents.
.RS
.PP
Note that in most Unix shells, wildcards will probably need to be escaped on the command line, to prevent the shell from expanding the wildcard before \fBagedu\fP sees it.
.PP
\fB--prune-path\fP is similar to \fB--prune\fP, except that the wildcard is matched against the entire pathname instead of just the filename at the end of it. So whereas \fB--prune *a*b*\fP will match any file whose actual name contains an \fBa\fP somewhere before a \fBb\fP, \fB--prune-path *a*b*\fP will also match a file whose name contains \fBb\fP and which is inside a directory containing an \fBa\fP, or any file inside a directory of that form, and so on.
.RE
.IP "\fB--exclude\fP \fIwildcard\fP and \fB--exclude-path\fP \fIwildcard\fP"
These cause particular files or directories to be omitted from the index, but not from the scan. If \fBagedu\fP\*(Aqs scan encounters a file or directory whose name matches the wildcard provided to the \fB--exclude\fP option, it will not include that file in its index - but unlike \fB--prune\fP, if the file in question is a directory it will still scan its contents and index them if they are not ruled out themselves by \fB--exclude\fP options.
.RS
.PP
As above, \fB--exclude-path\fP is similar to \fB--exclude\fP, except that the wildcard is matched against the entire pathname.
.RE
.IP "\fB--include\fP \fIwildcard\fP and \fB--include-path\fP \fIwildcard\fP"
These cause particular files or directories to be re-included in the index and the scan, if they had previously been ruled out by one of the above exclude or prune options. You can interleave include, exclude and prune options as you wish on the command line, and if more than one of them applies to a file then the last one takes priority.
.RS
.PP
For example, if you wanted to see only the disk space taken up by MP3 files, you might run
.PP
.nf
$\ \fBagedu\ \-s\ .\ \-\-exclude\ \*(Aq*\*(Aq\ \-\-include\ \*(Aq*.mp3\*(Aq\fP
.fi
.PP
which will cause everything to be omitted from the scan, but then the MP3 files to be put back in. If you then wanted only a subset of those MP3s, you could then exclude some of them again by adding, say, `\fB--exclude-path \*(Aq./queen/*\*(Aq\fP' (or, more efficiently, `\fB--prune ./queen\fP') on the end of that command.
.PP
As with the previous two options, \fB--include-path\fP is similar to \fB--include\fP except that the wildcard is matched against the entire pathname.
.RE
.IP "\fB--progress\fP, \fB--no-progress\fP and \fB--tty-progress\fP"
When \fBagedu\fP is scanning a directory tree, it will typically print a one-line progress report every second showing where it has reached in the scan, so you can have some idea of how much longer it will take. (Of course, it can\*(Aqt predict \fIexactly\fP how long it will take, since it doesn\*(Aqt know which of the directories it hasn\*(Aqt scanned yet will turn out to be huge.)
.RS
.PP
By default, those progress reports are displayed on \fBagedu\fP\*(Aqs standard error channel, if that channel points to a terminal device. If you need to manually enable or disable them, you can use the above three options to do so: \fB--progress\fP unconditionally enables the progress reports, \fB--no-progress\fP unconditionally disables them, and \fB--tty-progress\fP reverts to the default behaviour which is conditional on standard error being a terminal.
.RE
.IP "\fB--dir-atime\fP and \fB--no-dir-atime\fP"
In normal operation, \fBagedu\fP ignores the atimes (last access times) on the \fIdirectories\fP it scans: it only pays attention to the atimes of the \fIfiles\fP inside those directories. This is because directory atimes tend to be reset by a lot of system administrative tasks, such as \fBcron\fP jobs which scan the file system for one reason or another - or even other invocations of \fBagedu\fP itself, though it tries to avoid modifying any atimes if possible. So the literal atimes on directories are typically not representative of how long ago the data in question was last accessed with real intent to use that data in particular.
.RS
.PP
Instead, \fBagedu\fP makes up a fake atime for every directory it scans, which is equal to the newest atime of any file in or below that directory (or the directory\*(Aqs last \fImodification\fP time, whichever is newest). This is based on the assumption that all \fIimportant\fP accesses to directories are actually accesses to the files inside those directories, so that when any file is accessed all the directories on the path leading to it should be considered to have been accessed as well.
.PP
In unusual cases it is possible that a directory itself might embody important data which is accessed by reading the directory. In that situation, \fBagedu\fP\*(Aqs atime-faking policy will misreport the directory as disused. In the unlikely event that such directories form a significant part of your disk space usage, you might want to turn off the faking. The \fB--dir-atime\fP option does this: it causes the disk scan to read the original atimes of the directories it scans.
.PP
The faking of atimes on directories also requires a processing pass over the index file after the main disk scan is complete. \fB--dir-atime\fP also turns this pass off. Hence, this option affects the \fB-L\fP option as well as \fB-s\fP and \fB-S\fP.
.PP
(The previous section mentioned that there might be subtle differences between the output of \fBagedu -s /path -D\fP and \fBagedu -S /path\fP. This is why. Doing a scan with \fB-s\fP and then dumping it with \fB-D\fP will dump the fully faked atimes on the directories, whereas doing a scan-to-dump with \fB-S\fP will dump only \fIpartially\fP faked atimes - specifically, each directory\*(Aqs last modification time - since the subsequent processing pass will not have had a chance to take place. However, loading either of the resulting dump files with \fB-L\fP will perform the atime-faking processing pass, leading to the same data in the index file in each case. In normal usage it should be safe to ignore all of this complexity.)
.RE
.IP "\fB--mtime\fP"
This option causes \fBagedu\fP to index files by their last modification time instead of their last access time. You might want to use this if your last access times were completely useless for some reason: for example, if you had recently searched every file on your system, the system would have lost all the information about what files you hadn\*(Aqt recently accessed before then. Using this option is liable to be less effective at finding genuinely wasted space than the normal mode (that is, it will be more likely to flag things as disused when they\*(Aqre not, so you will have more candidates to go through by hand looking for data you don\*(Aqt need), but may be better than nothing if your last-access times are unhelpful.
.RS
.PP
Another use for this mode might be to find \fIrecently created\fP large data. If your disk has been gradually filling up for years, the default mode of \fBagedu\fP will let you find unused data to delete; but if you know your disk had plenty of space recently and now it\*(Aqs suddenly full, and you suspect that some rogue program has left a large core dump or output file, then \fBagedu --mtime\fP might be a convenient way to locate the culprit.
.RE
.IP "\fB--logicalsize\fP"
This option causes \fBagedu\fP to consider the size of each file to be its `logical' size, rather than the amount of space it consumes on disk. (That is, it will use \fBst_size\fP instead of \fBst_blocks\fP in the data returned from \fBstat\fP(\fI2\fP).) This option makes \fBagedu\fP less accurate at reporting how much of your disk is used, but it might be useful in specialist cases, such as working around a file system that is misreporting physical sizes.
.RS
.PP
For most files, the physical size of a file will be larger than the logical size, reflecting the fact that filesystem layouts generally allocate a whole number of blocks of the disk to each file, so some space is wasted at the end of the last block. So counting only the logical file size will typically cause under-reporting of the disk usage (perhaps \fIlarge\fP under-reporting in the case of a very large number of very small files).
.PP
On the other hand, sometimes a file with a very large logical size can have `holes' where no data is actually stored, in which case using the logical size of the file will \fIover\fP-report its disk usage. So the use of logical sizes can give wrong answers in both directions.
.RE
.PP
The following option affects all the modes that generate reports: the web server mode \fB-w\fP, the stand-alone HTML generation mode \fB-H\fP and the text report mode \fB-t\fP.
.IP "\fB--files\fP"
This option causes \fBagedu\fP\*(Aqs reports to list the individual files in each directory, instead of just giving a combined report for everything that\*(Aqs not in a subdirectory.
.PP
The following option affects the text report mode \fB-t\fP.
.IP "\fB-a\fP \fIage\fP or \fB--age\fP \fIage\fP"
This option tells \fBagedu\fP to report only files of at least the specified age. An age is specified as a number, followed by one of `\fBy\fP' (years), `\fBm\fP' (months), `\fBw\fP' (weeks) or `\fBd\fP' (days). (This syntax is also used by the \fB-r\fP option.) For example, \fB-a 6m\fP will produce a text report which includes only files at least six months old.
.PP
The following options affect the stand-alone HTML generation mode \fB-H\fP and the text report mode \fB-t\fP.
.IP "\fB-d\fP \fIdepth\fP or \fB--depth\fP \fIdepth\fP"
This option controls the maximum depth to which \fBagedu\fP recurses when generating a text or HTML report.
.RS
.PP
In text mode, the default is 1, meaning that the report will include the directory given on the command line and all of its immediate subdirectories. A depth of two includes another level below that, and so on; a depth of zero means \fIonly\fP the directory on the command line.
.PP
In HTML mode, specifying this option switches \fBagedu\fP from writing out a single HTML file to writing out multiple files which link to each other. A depth of 1 means \fBagedu\fP will write out an HTML file for the given directory and also one for each of its immediate subdirectories.
.PP
If you want \fBagedu\fP to recurse as deeply as possible, give the special word `\fBmax\fP' as an argument to \fB-d\fP.
.RE
.IP "\fB-o\fP \fIfilename\fP or \fB--output\fP \fIfilename\fP"
This option is used to specify an output file for \fBagedu\fP to write its report to. In text mode or single-file HTML mode, the argument is treated as the name of a file. In multiple-file HTML mode, the argument is treated as the name of a directory: the directory will be created if it does not already exist, and the output HTML files will be created inside it.
.PP
The following option affects only the stand-alone HTML generation mode \fB-H\fP, and even then, only in recursive mode (with \fB-d\fP):
.IP "\fB--numeric\fP"
This option tells \fBagedu\fP to name most of its output HTML files numerically. The root of the whole output file collection will still be called \fBindex.html\fP, but all the rest will have names like \fB73.html\fP or \fB12525.html\fP. (The numbers are essentially arbitrary; in fact, they\*(Aqre indices of nodes in the data structure used by \fBagedu\fP\*(Aqs index file.)
.RS
.PP
This system of file naming is less intuitive than the default of naming files after the sub-pathname they index. It's also less stable: the same pathname will not necessarily be represented by the same filename if \fBagedu -H\fP is re-run after another scan of the same directory tree. However, it does have the virtue that it keeps the filenames \fIshort\fP, so that even if your directory tree is very deep, the output HTML files won\*(Aqt exceed any OS limit on filename length.
.RE
.PP
The following options affect the web server mode \fB-w\fP, and in some cases also the stand-alone HTML generation mode \fB-H\fP:
.IP "\fB-r\fP \fIage range\fP or \fB--age-range\fP \fIage range\fP"
The HTML reports produced by \fBagedu\fP use a range of colours to indicate how long ago data was last accessed, running from red (representing the most disused data) to green (representing the newest). By default, the lengths of time represented by the two ends of that spectrum are chosen by examining the data file to see what range of ages appears in it. However, you might want to set your own limits, and you can do this using \fB-r\fP.
.RS
.PP
The argument to \fB-r\fP consists of a single age, or two ages separated by a minus sign. An age is a number, followed by one of `\fBy\fP' (years), `\fBm\fP' (months), `\fBw\fP' (weeks) or `\fBd\fP' (days). (This syntax is also used by the \fB-a\fP option.) The first age in the range represents the oldest data, and will be coloured red in the HTML; the second age represents the newest, coloured green. If the second age is not specified, it will default to zero (so that green means data which has been accessed \fIjust now\fP).
.PP
For example, \fB-r 2y\fP will mark data in red if it has been unused for two years or more, and green if it has been accessed just now. \fB-r 2y-3m\fP will similarly mark data red if it has been unused for two years or more, but will mark it green if it has been accessed three months ago or later.
.RE
.IP "\fB--address\fP \fIaddr\fP[\fB:\fP\fIport\fP]"
Specifies the network address and port number on which \fBagedu\fP should listen when running its web server. If you want \fBagedu\fP to listen for connections coming in from any source, specify the address as the special value \fBANY\fP. If the port number is omitted, an arbitrary unused port will be chosen for you and displayed.
.RS
.PP
If you specify this option, \fBagedu\fP will not print its URL on standard output (since you are expected to know what address you told it to listen to).
.RE
.IP "\fB--auth\fP \fIauth-type\fP"
Specifies how \fBagedu\fP should control access to the web pages it serves. The options are as follows:
.RS
.IP "\fBmagic\fP"
This option only works on Linux, and only when the incoming connection is from the same machine that \fBagedu\fP is running on. On Linux, the special file \fB/proc/net/tcp\fP contains a list of network connections currently known to the operating system kernel, including which user id created them. So \fBagedu\fP will look up each incoming connection in that file, and allow access if it comes from the same user id under which \fBagedu\fP itself is running. Therefore, in \fBagedu\fP\*(Aqs normal web server mode, you can safely run it on a multi-user machine and no other user will be able to read data out of your index file.
.IP "\fBbasic\fP"
In this mode, \fBagedu\fP will use HTTP Basic authentication: the user will have to provide a username and password via their browser. \fBagedu\fP will normally make up a username and password for the purpose, but you can specify your own; see below.
.IP "\fBnone\fP"
In this mode, the web server is unauthenticated: anyone connecting to it has full access to the reports generated by \fBagedu\fP. Do not do this unless there is nothing confidential at all in your index file, or unless you are certain that nobody but you can run processes on your computer.
.IP "\fBdefault\fP"
This is the default mode if you do not specify one of the above. In this mode, \fBagedu\fP will attempt to use Linux magic authentication, but if it detects at startup time that \fB/proc/net/tcp\fP is absent or non-functional then it will fall back to using HTTP Basic authentication and invent a user name and password.
.RE
.IP "\fB--auth-file\fP \fIfilename\fP or \fB--auth-fd\fP \fIfd\fP"
When \fBagedu\fP is using HTTP Basic authentication, these options allow you to specify your own user name and password. If you specify \fB--auth-file\fP, these will be read from the specified file; if you specify \fB--auth-fd\fP they will instead be read from a given file descriptor which you should have arranged to pass to \fBagedu\fP. In either case, the authentication details should consist of the username, followed by a colon, followed by the password, followed \fIimmediately\fP by end of file (no trailing newline, or else it will be considered part of the password).
.IP "\fB--title\fP \fItitle\fP"
Specify the string that appears at the start of the \fB\fP section of the output HTML pages. The default is `\fBagedu\fP'. This title is followed by a colon and then the path you\*(Aqre viewing within the index file. You might use this option if you were serving \fBagedu\fP reports for several different servers and wanted to make it clearer which one a user was looking at.
.IP "\fB--launch\fP \fIshell-command\fP"
Specify a command to be run with the base URL of the web user interface, once the web server has started up. The command will be interpreted by \fB/bin/sh\fP, and the base URL will be appended to it as an extra argument word.
.RS
.PP
A typical use for this would be `\fB--launch=browse\fP', which uses the XDG `\fBbrowse\fP' command to automatically open the \fBagedu\fP web interface in your default browser. However, other uses are possible: for example, you could provide a command which communicates the URL to some other software that will use it for something.
.RE
.IP "\fB--no-eof\fP"
Stop \fBagedu\fP in web server mode from looking for end-of-file on standard input and treating it as a signal to terminate.
.SH "LIMITATIONS"
.PP
The data file is pretty large. The core of \fBagedu\fP is the tree-based data structure it uses in its index in order to efficiently perform the queries it needs; this data structure requires \fBO(N log N)\fP storage. This is larger than you might expect; a scan of my own home directory, containing half a million files and directories and about 20Gb of data, produced an index file over 60Mb in size. Furthermore, since the data file must be memory-mapped during most processing, it can never grow larger than available address space, so a \fIreally\fP big filesystem may need to be indexed on a 64-bit computer. (This is one reason for the existence of the \fB-D\fP and \fB-L\fP options: you can do the scanning on the machine with access to the filesystem, and the indexing on a machine big enough to handle it.)
.PP
The data structure also does not usefully permit access control within the data file, so it would be difficult - even given the willingness to do additional coding - to run a system-wide \fBagedu\fP scan on a \fBcron\fP job and serve the right subset of reports to each user.
.PP
In certain circumstances, \fBagedu\fP can report false positives (reporting files as disused which are in fact in use) as well as the more benign false negatives (reporting files as in use which are not). This arises when a file is, semantically speaking, `read' without actually being physically \fIread\fP. Typically this occurs when a program checks whether the file\*(Aqs mtime has changed and only bothers re-reading it if it has; programs which do this include \fBrsync\fP(\fI1\fP) and \fBmake\fP(\fI1\fP). Such programs will fail to update the atime of unmodified files despite depending on their continued existence; a directory full of such files will be reported as disused by \fBagedu\fP even in situations where deleting them will cause trouble.
.PP
Finally, of course, \fBagedu\fP\*(Aqs normal usage mode depends critically on the OS providing last-access times which are at least approximately right. So a file system mounted with Linux\*(Aqs `\fBnoatime\fP' option, or the equivalent on any other OS, will not give useful results! (However, the Linux mount option `\fBrelatime\fP', which distributions now tend to use by default, should be fine for all but specialist purposes: it reduces the accuracy of last-access times so that they might be wrong by up to 24 hours, but if you\*(Aqre looking for files that have been unused for months or years, that\*(Aqs not a problem.)
.SH "LICENCE"
.PP
\fBagedu\fP is free software, distributed under the MIT licence. Type \fBagedu --licence\fP to see the full licence text.
agedu-20211129.8cd63c5/winscan.c 0000644 0001750 0001750 00000014753 14151034324 014771 0 ustar simon simon #include <windows.h>
#include <stdio.h>
#include <stdbool.h>
#include <string.h>
#define lenof(x) (sizeof((x))/sizeof(*(x)))
#define snew(type) \
( (type *) malloc (sizeof (type)) )
#define snewn(number, type) \
( (type *) malloc ((number) * sizeof (type)) )
#define sresize(array, number, type) \
( (void)sizeof((array)-(type *)0), \
(type *) realloc ((array), (number) * sizeof (type)) )
#define sfree free
char *dupstr(const char *s) {
char *r = malloc(1+strlen(s));
strcpy(r,s);
return r;
}
typedef struct {
HANDLE hdl;
WIN32_FIND_DATA fdata;
bool got_one, eod;
} dirhandle;
int open_dir(char *path, dirhandle *dh)
{
strcat(path, "\\*");
dh->hdl = FindFirstFile(path, &dh->fdata);
if (dh->hdl == INVALID_HANDLE_VALUE) {
int err = GetLastError();
if (err == ERROR_FILE_NOT_FOUND) {
dh->eod = true;
dh->got_one = false;
return 0;
} else {
return -err;
}
} else {
dh->eod = false;
dh->got_one = true;
return 0;
}
}
const char *read_dir(dirhandle *dh)
{
if (!dh->got_one) {
if (dh->eod)
return NULL;
if (FindNextFile(dh->hdl, &dh->fdata)) {
dh->got_one = true;
} else {
dh->eod = true;
return NULL;
}
}
dh->got_one = false;
return dh->fdata.cFileName;
}
void close_dir(dirhandle *dh)
{
CloseHandle(dh->hdl);
}
static int str_cmp(const void *av, const void *bv)
{
return strcmp(*(const char **)av, *(const char **)bv);
}
typedef int (*gotdata_fn_t)(void *ctx, const char *pathname,
WIN32_FIND_DATA *dat);
static void format_error(char *buf, size_t size, int err)
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, err, 0, buf, size, NULL);
buf[size-1] = '\0';
if (buf[strlen(buf)-1] == '\n')
buf[strlen(buf)-1] = '\0';
}
static void du_recurse(char **path, size_t pathlen, size_t *pathsize,
gotdata_fn_t gotdata, void *gotdata_ctx)
{
const char *name;
dirhandle d;
char **names;
int error;
size_t i, nnames, namesize;
WIN32_FIND_DATA dat;
HANDLE h;
if (*pathsize <= pathlen + 10) {
*pathsize = pathlen * 3 / 2 + 256;
*path = sresize(*path, *pathsize, char);
}
h = FindFirstFile(*path, &dat);
if (h != INVALID_HANDLE_VALUE) {
/* The normal case: we retrieved information about the file in
* 'dat', and we also got a handle we can pass to
* FindNextFile. We don't need the latter, so close it
* immediately. */
CloseHandle(h);
} else if (pathlen > 0 && (*path)[pathlen-1] == '\\') {
/* Special case: we're scanning a subdirectory such as c:\ .
* In that case, take it to have zero size, and resort to
* GetFileTime to find its times. */
dat.nFileSizeHigh = dat.nFileSizeLow = 0;
dat.ftLastWriteTime.dwHighDateTime = 0x19DB1DE;
dat.ftLastWriteTime.dwLowDateTime = 0xD53E8000;
dat.ftLastAccessTime.dwHighDateTime = 0x19DB1DE;
dat.ftLastAccessTime.dwLowDateTime = 0xD53E8000;
h = CreateFile(*path, GENERIC_READ, FILE_SHARE_WRITE, NULL,
OPEN_EXISTING, 0, NULL);
if (h != INVALID_HANDLE_VALUE) {
GetFileTime(h, &dat.ftCreationTime, &dat.ftLastAccessTime,
&dat.ftLastWriteTime);
CloseHandle(h);
}
} else {
/* Total failure: FindFirstFile returned INVALID_HANDLE_VALUE
* and it _wasn't_ because the file is a special directory. */
char buf[4096];
format_error(buf, lenof(buf), GetLastError());
fprintf(stderr, "Unable to scan '%s': %s\n", *path, buf);
return;
}
if (!gotdata(gotdata_ctx, *path, &dat))
return;
if (!(GetFileAttributes(*path) & FILE_ATTRIBUTE_DIRECTORY))
return;
names = NULL;
nnames = namesize = 0;
if ((error = open_dir(*path, &d)) < 0) {
char buf[4096];
format_error(buf, lenof(buf), -error);
fprintf(stderr, "Unable to open directory '%s': %s\n", *path, buf);
return;
}
while ((name = read_dir(&d)) != NULL) {
if (name[0] == '.' && (!name[1] || (name[1] == '.' && !name[2]))) {
/* do nothing - we skip "." and ".." */
} else {
if (nnames >= namesize) {
namesize = nnames * 3 / 2 + 64;
names = sresize(names, namesize, char *);
}
names[nnames++] = dupstr(name);
}
}
close_dir(&d);
if (nnames == 0)
return;
qsort(names, nnames, sizeof(*names), str_cmp);
for (i = 0; i < nnames; i++) {
size_t newpathlen = pathlen + 1 + strlen(names[i]);
if (*pathsize <= newpathlen) {
*pathsize = newpathlen * 3 / 2 + 256;
*path = sresize(*path, *pathsize, char);
}
/*
* Avoid duplicating a slash if we got a trailing one to
* begin with (i.e. if we're starting the scan in '\\' itself).
*/
if (pathlen > 0 && (*path)[pathlen-1] == '\\') {
strcpy(*path + pathlen, names[i]);
newpathlen--;
} else {
sprintf(*path + pathlen, "\\%s", names[i]);
}
du_recurse(path, newpathlen, pathsize, gotdata, gotdata_ctx);
sfree(names[i]);
}
sfree(names);
}
void du(const char *inpath, gotdata_fn_t gotdata, void *gotdata_ctx)
{
char *path;
size_t pathlen, pathsize;
pathlen = strlen(inpath);
pathsize = pathlen + 256;
path = snewn(pathsize, char);
strcpy(path, inpath);
du_recurse(&path, pathlen, &pathsize, gotdata, gotdata_ctx);
}
int gd(void *ctx, const char *pathname, WIN32_FIND_DATA *dat)
{
unsigned long long size, t;
FILETIME ft;
const char *p;
size = dat->nFileSizeHigh;
size = (size << 32) | dat->nFileSizeLow;
printf("%llu ", size);
if (dat->dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
ft = dat->ftLastWriteTime;
else
ft = dat->ftLastAccessTime;
t = ft.dwHighDateTime;
t = (t << 32) | ft.dwLowDateTime;
t /= 10000000;
/*
* Correction factor: number of seconds between Windows's file
* time epoch of 1 Jan 1601 and Unix's time_t epoch of 1 Jan 1970.
*
* That's 369 years, of which 92 were divisible by 4, but
* three of those were century points.
*/
t -= (369 * 365 + 92 - 3) * (unsigned long long)86400;
printf("%llu ", t);
for (p = pathname; *p; p++) {
if (*p >= ' ' && *p < 127 && *p != '%')
putchar(*p);
else
printf("%%%02x", (unsigned char)*p);
}
putchar('\n');
return 1;
}
int main(int argc, char **argv)
{
char *dir;
int dirlen, dirsize;
if (argc > 1)
SetCurrentDirectory(argv[1]);
dirsize = 512;
dir = snewn(dirsize, char);
while ((dirlen = GetCurrentDirectory(dirsize, dir)) >= dirsize) {
dirsize = dirlen + 256;
dir = sresize(dir, dirsize, char);
}
printf("agedu dump file. pathsep=5c\n");
du(dir, gd, NULL);
}
agedu-20211129.8cd63c5/version.h 0000644 0001750 0001750 00000000453 14151034324 015011 0 ustar simon simon /*
* Header file that defines AGEDU_VERSION to a string that goes in
* 'agedu --version', in the distribution tarball.
*
* This empty copy of it, in source control, exists so that #including
* it won't fail when we're not building from the tarball.
*/
#define AGEDU_VERSION "20211129.8cd63c5"
agedu-20211129.8cd63c5/trie.h 0000644 0001750 0001750 00000011736 14151034324 014275 0 ustar simon simon /*
* trie.h: functions to build and search a trie-like structure
* mapping pathnames to file records.
*/
#include <sys/types.h>
/*
* An entry in the trie file describing an actual file.
*/
struct trie_file {
unsigned long long size;
unsigned long long atime;
};
/* ----------------------------------------------------------------------
* Functions which can be passed a list of pathnames and file data
* in strict order, and will build up a trie and write it out to a
* file.
*/
typedef struct triebuild triebuild;
/*
* Create a new trie-building context given a writable file
* descriptor, which should point to the start of an empty file.
*/
triebuild *triebuild_new(int fd);
/*
* Add a trie_file record to the trie. The pathnames should appear
* in trie collation order (i.e. strict ASCII sorting order except
* that / is moved so that it sorts before any other non-NUL
* character), on penalty of assertion failure.
*/
void triebuild_add(triebuild *tb, const char *pathname,
const struct trie_file *file);
/*
* Put the finishing touches to the contents of the output file
* once all the trie_file records are present. Returns the total
* number of elements in the trie.
*/
int triebuild_finish(triebuild *tb);
/*
* Free the context. (Does not close the file, but may leave the
* file pointer in an arbitrary place.)
*/
void triebuild_free(triebuild *tb);
/* ----------------------------------------------------------------------
* Anomalous function which modifies a trie after it's memory-mapped.
*/
/*
* Invent new fake atimes for each directory in the trie, by
* taking the maximum (latest) of the directory's previously
* stored atime and the atimes of everything below it.
*/
void trie_fake_dir_atimes(void *t);
/* ----------------------------------------------------------------------
* Functions to query a trie given a pointer to the start of the
* memory-mapped file.
*/
/*
* Check the magic numbers at the start of the file. This should also
* verify that the file was built on a platform whose structure layout
* matches that of the agedu reading it. Returns nonzero on successful
* match, zero on mismatch.
*/
int trie_check_magic(const void *t);
/*
* Return the path separator character in use in the trie.
*/
char trie_pathsep(const void *t);
/*
* Return the length of the longest pathname stored in the trie,
* including its trailing NUL.
*/
size_t trie_maxpathlen(const void *t);
/*
* Determine the total number of entries in the trie which sort
* strictly before the given pathname (irrespective of whether the
* pathname actually exists in the trie).
*/
unsigned long trie_before(const void *t, const char *pathname);
/*
* Return the pathname for the nth entry in the trie, written into
* the supplied buffer (which must be large enough, i.e. at least
* trie_maxpathlen(t) bytes).
*/
void trie_getpath(const void *t, unsigned long n, char *buf);
/*
* Return the trie_file * for the nth entry in the trie.
*/
const struct trie_file *trie_getfile(const void *t, unsigned long n);
/*
* Return the total number of entries in the whole trie.
*/
unsigned long trie_count(const void *t);
/*
* Enumerate all the trie_file records in the trie, in sequence,
* and return pointers to their trie_file structures. Returns NULL
* at end of list, naturally.
*
* Optionally returns the pathnames as well: if "buf" is non-NULL
* then it is expected to point to a buffer large enough to store
* all the pathnames in the trie (i.e. at least trie_maxpathlen(t)
* bytes). This buffer is also expected to remain unchanged
* between calls to triewalk_next(), or else the returned
* pathnames will be corrupted.
*/
typedef struct triewalk triewalk;
triewalk *triewalk_new(const void *t);
const struct trie_file *triewalk_next(triewalk *tw, char *buf);
void triewalk_rebase(triewalk *tw, const void *t);
void triewalk_free(triewalk *tw);
/* ----------------------------------------------------------------------
* The trie file also contains an index managed by index.h, so we
* must be able to ask about that too.
*/
void trie_set_index_offset(void *t, off_t ptr);
off_t trie_get_index_offset(const void *t);
/* ----------------------------------------------------------------------
* Utility functions not directly involved with the trie.
*/
/*
* Given a pathname in a buffer, adjust the pathname in place so
* that it points at a string which, when passed to trie_before,
* will reliably return the index of the thing that comes after
* that pathname and all its descendants. Usually this is done by
* suffixing ^A (since foo^A is guaranteeably the first thing that
* sorts after foo and foo/stuff); however, if the pathname
* actually ends in a path separator (e.g. if it's just "/"), that
* must be stripped off first.
*/
void make_successor(char *pathbuf);
/*
* String comparison function consistent with the trie's sort order.
* Requires the current path separator as a parameter.
*/
int triecmp(const char *a, const char *b, int *offset, unsigned char pathsep);
agedu-20211129.8cd63c5/trie.c 0000644 0001750 0001750 00000053616 14151034324 014273 0 ustar simon simon /*
* trie.c: implementation of trie.h.
*/
#include "agedu.h"
#include "alloc.h"
#include "trie.h"
#define alignof(typ) ( offsetof(struct { char c; typ t; }, t) )
/*
* Compare functions for pathnames. Returns the relative order of
* the names, like strcmp; also passes back the offset of the
* first differing character if desired.
*/
static int trieccmp(unsigned char a, unsigned char b, unsigned char pathsep)
{
int ia = (a == '\0' ? '\0' : a == pathsep ? '\1' : a+1);
int ib = (b == '\0' ? '\0' : b == pathsep ? '\1' : b+1);
return ia - ib;
}
static int triencmp(const char *a, size_t alen, const char *b, size_t blen,
int *offset, unsigned char pathsep)
{
int off = 0;
while (off < alen && off < blen && a[off] == b[off])
off++;
if (offset)
*offset = off;
if (off == alen || off == blen) return (off == blen) - (off == alen);
return trieccmp(a[off], b[off], pathsep);
}
int triecmp(const char *a, const char *b, int *offset, unsigned char pathsep)
{
return triencmp(a, strlen(a), b, strlen(b), offset, pathsep);
}
/* ----------------------------------------------------------------------
* Trie node structures.
*
* The trie format stored in the file consists of three distinct
* node types, each with a distinguishing type field at the start.
*
* TRIE_LEAF is a leaf node; it contains an actual trie_file
* structure, and indicates that when you're searching down the
* trie with a string, you should now expect to encounter
* end-of-string.
*
* TRIE_SWITCH indicates that the set of strings in the trie
* include strings with more than one distinct character after the
* prefix leading up to this point. Hence, it stores multiple
* subnode pointers and a different single character for each one.
*
* TRIE_STRING indicates that at this point everything in the trie
* has the same next few characters; it stores a single mandatory
* string fragment and exactly one subnode pointer.
*/
enum {
TRIE_LEAF = 0x7fffe000,
TRIE_SWITCH,
TRIE_STRING
};
struct trie_common {
int type;
};
struct trie_switchentry {
off_t subnode;
int subcount;
};
struct trie_leaf {
struct trie_common c;
struct trie_file file;
};
struct trie_switch {
struct trie_common c;
/*
* sw[0] to sw[len-1] give the subnode pointers and element
* counts. At &sw[len] is stored len single bytes which are
* the characters corresponding to each subnode.
*/
int len;
struct trie_switchentry sw[];
};
struct trie_string {
struct trie_common c;
int stringlen;
off_t subnode;
char string[];
};
static const char magic_ident_string[16] = "agedu index file";
struct trie_magic {
/*
* 'Magic numbers' to go at the start of an agedu index file.
* These are checked (using trie_check_magic) by every agedu mode
* which reads a pre-existing index.
*
* As well as identifying an agedu file from any other kind of
* file, this magic-number structure is also intended to detect
* agedu files which were written on the wrong platform and hence
* whose structure layouts are incompatible. To make that as
* reliable as possible, I design the structure of magic numbers
* as follows: it contains one of each integer type I might use,
* each containing a different magic number, and each followed by
* a char to indicate where it ends in the file. One byte is set
* to the length of the magic-number structure itself, which means
* that no two structures of different lengths can possibly
* compare equal even if by some chance they match up to the
* length of the shorter one. Finally, the whole magic number
* structure is memset to another random byte value before
* initialising any of these fields, so that padding in between
* can be readily identified.
*/
char ident[16]; /* human-readable string */
unsigned char magic_len;
unsigned long long longlong_magic;
unsigned char postlonglong_char_magic;
off_t offset_magic;
unsigned char postoffset_char_magic;
size_t size_magic;
unsigned char postsize_char_magic;
void *null_pointer;
unsigned char postptr_char_magic;
unsigned long long_magic;
unsigned char postlong_char_magic;
unsigned short short_magic;
unsigned char postshort_char_magic;
};
struct trie_header {
struct trie_magic magic;
off_t root, indexroot;
int count;
size_t maxpathlen;
int pathsep;
};
/* Union only used for computing alignment */
union trie_node {
struct trie_leaf leaf;
struct { /* fake trie_switch with indeterminate array length filled in */
struct trie_common c;
int len;
struct trie_switchentry sw[1];
} sw;
struct { /* fake trie_string with indeterminate array length filled in */
struct trie_common c;
int stringlen;
off_t subnode;
char string[1];
} str;
};
#define TRIE_ALIGN alignof(union trie_node)
static void setup_magic(struct trie_magic *th)
{
/*
* Magic values are chosen so that every byte value used is
* distinct (so that we can't fail to spot endianness issues), and
* we cast 64 bits of data into each integer type just in case
* the platform makes it longer than we expect it to be.
*/
memset(th, 0xCDU, sizeof(*th));
th->magic_len = sizeof(*th);
memcpy(th->ident, magic_ident_string, 16);
th->longlong_magic = 0x5583F34D5D84F73CULL;
th->postlonglong_char_magic = 0xDDU;
th->offset_magic = (off_t)0xB39BF9AD56D48E0BULL;
th->postoffset_char_magic = 0x95U;
th->size_magic = (size_t)0x6EC752B0EEAEBAC1ULL;
th->postsize_char_magic = 0x77U;
th->null_pointer = NULL;
th->postptr_char_magic = 0x71U;
th->long_magic = (unsigned long)0xA81A5E1F44334716ULL;
th->postlong_char_magic = 0x99U;
th->short_magic = (unsigned short)0x0C8BD7984B68D9FCULL;
th->postshort_char_magic = 0x35U;
}
/* ----------------------------------------------------------------------
* Trie-building functions.
*/
struct tbswitch {
int len;
char c[256];
off_t off[256];
int count[256];
};
struct triebuild {
int fd;
off_t offset;
char *lastpath;
int lastlen, lastsize;
off_t lastoff;
struct tbswitch *switches;
int switchsize;
size_t maxpathlen;
};
static void tb_seek(triebuild *tb, off_t off)
{
tb->offset = off;
if (lseek(tb->fd, off, SEEK_SET) < 0) {
fprintf(stderr, PNAME ": lseek: %s\n", strerror(errno));
exit(1);
}
}
static void tb_write(triebuild *tb, const void *buf, size_t len)
{
tb->offset += len;
while (len > 0) {
int ret = write(tb->fd, buf, len);
if (ret < 0) {
fprintf(stderr, PNAME ": write: %s\n", strerror(errno));
exit(1);
}
len -= ret;
buf = (const void *)((const char *)buf + ret);
}
}
static char trie_align_zeroes[TRIE_ALIGN];
static void tb_align(triebuild *tb)
{
int off = (TRIE_ALIGN - ((tb)->offset % TRIE_ALIGN)) % TRIE_ALIGN;
tb_write(tb, trie_align_zeroes, off);
}
triebuild *triebuild_new(int fd)
{
triebuild *tb = snew(triebuild);
struct trie_header th;
tb->fd = fd;
tb->lastpath = NULL;
tb->lastlen = tb->lastsize = 0;
tb->lastoff = 0;
tb->switches = NULL;
tb->switchsize = 0;
tb->maxpathlen = 0;
setup_magic(&th.magic);
th.root = th.count = 0;
th.indexroot = 0;
th.maxpathlen = 0;
th.pathsep = (unsigned char)pathsep;
tb_seek(tb, 0);
tb_write(tb, &th, sizeof(th));
return tb;
}
static off_t triebuild_unwind(triebuild *tb, int targetdepth, int *outcount)
{
off_t offset;
int count, depth;
if (tb->lastoff == 0) {
*outcount = 0;
return 0;
}
offset = tb->lastoff;
count = 1;
depth = tb->lastlen + 1;
assert(depth >= targetdepth);
while (depth > targetdepth) {
int odepth = depth;
while (depth > targetdepth &&
(depth-1 >= tb->switchsize || !tb->switches ||
tb->switches[depth-1].len == 0))
depth--;
if (odepth > depth) {
/*
* Write out a string node.
*/
size_t nodesize = sizeof(struct trie_string) + odepth - depth;
struct trie_string *st = (struct trie_string *)smalloc(nodesize);
st->c.type = TRIE_STRING;
st->stringlen = odepth - depth;
st->subnode = offset;
memcpy(st->string, tb->lastpath + depth, odepth - depth);
tb_align(tb);
offset = tb->offset;
tb_write(tb, st, nodesize);
sfree(st);
}
assert(depth >= targetdepth);
if (depth <= targetdepth)
break;
/*
* Now we expect to be sitting just below a switch node.
* Add our final entry to it and write it out.
*/
depth--;
{
struct trie_switch *sw;
char *chars;
size_t nodesize;
int swlen = tb->switches[depth].len;
int i;
assert(swlen > 0);
tb->switches[depth].c[swlen] = tb->lastpath[depth];
tb->switches[depth].off[swlen] = offset;
tb->switches[depth].count[swlen] = count;
swlen++;
nodesize = sizeof(struct trie_switch) +
swlen * sizeof(struct trie_switchentry) + swlen;
sw = (struct trie_switch *)smalloc(nodesize);
chars = (char *)&sw->sw[swlen];
sw->c.type = TRIE_SWITCH;
sw->len = swlen;
count = 0;
for (i = 0; i < swlen; i++) {
sw->sw[i].subnode = tb->switches[depth].off[i];
sw->sw[i].subcount = tb->switches[depth].count[i];
chars[i] = tb->switches[depth].c[i];
count += tb->switches[depth].count[i];
}
tb_align(tb);
offset = tb->offset;
tb_write(tb, sw, nodesize);
sfree(sw);
tb->switches[depth].len = 0; /* clear this node */
}
}
*outcount = count;
return offset;
}
void triebuild_add(triebuild *tb, const char *pathname,
const struct trie_file *file)
{
int pathlen = strlen(pathname);
int depth;
if (tb->maxpathlen < pathlen+1)
tb->maxpathlen = pathlen+1;
if (tb->lastpath) {
off_t offset;
int count;
/*
* Find the first differing character between this pathname
* and the previous one.
*/
int ret = triecmp(tb->lastpath, pathname, &depth, pathsep);
assert(ret < 0);
/*
* Finalise all nodes above this depth.
*/
offset = triebuild_unwind(tb, depth+1, &count);
/*
* Add the final node we just acquired to the switch node
* at our chosen depth, creating it if it isn't already
* there.
*/
if (tb->switchsize <= depth) {
int oldsize = tb->switchsize;
tb->switchsize = depth * 3 / 2 + 64;
tb->switches = sresize(tb->switches, tb->switchsize,
struct tbswitch);
while (oldsize < tb->switchsize)
tb->switches[oldsize++].len = 0;
}
tb->switches[depth].c[tb->switches[depth].len] = tb->lastpath[depth];
tb->switches[depth].off[tb->switches[depth].len] = offset;
tb->switches[depth].count[tb->switches[depth].len] = count;
tb->switches[depth].len++;
}
/*
* Write out a leaf node for the new file, and remember its
* file offset.
*/
{
struct trie_leaf leaf;
leaf.c.type = TRIE_LEAF;
leaf.file = *file; /* structure copy */
tb_align(tb);
tb->lastoff = tb->offset;
tb_write(tb, &leaf, sizeof(leaf));
}
/*
* Store this pathname for comparison with the next one.
*/
if (tb->lastsize < pathlen+1) {
tb->lastsize = pathlen * 3 / 2 + 64;
tb->lastpath = sresize(tb->lastpath, tb->lastsize, char);
}
strcpy(tb->lastpath, pathname);
tb->lastlen = pathlen;
}
int triebuild_finish(triebuild *tb)
{
struct trie_header th;
setup_magic(&th.magic);
th.root = triebuild_unwind(tb, 0, &th.count);
th.indexroot = 0;
th.maxpathlen = tb->maxpathlen;
th.pathsep = (unsigned char)pathsep;
tb_seek(tb, 0);
tb_write(tb, &th, sizeof(th));
return th.count;
}
void triebuild_free(triebuild *tb)
{
sfree(tb->switches);
sfree(tb->lastpath);
sfree(tb);
}
/* ----------------------------------------------------------------------
* Memory-mapped trie modification.
*/
#define MNODE(t, off, type) \
((struct type *)((char *)(t) + (off)))
static unsigned long long fake_atime_recurse(void *t, struct trie_common *node,
int last_seen_pathsep)
{
while (node->type == TRIE_STRING) {
struct trie_string *st = (struct trie_string *)node;
last_seen_pathsep = (st->string[st->stringlen-1] == pathsep);
node = MNODE(t, st->subnode, trie_common);
}
if (node->type == TRIE_LEAF) {
struct trie_leaf *leaf = (struct trie_leaf *)node;
return leaf->file.atime;
} else if (assert(node->type == TRIE_SWITCH), 1) {
struct trie_switch *sw = (struct trie_switch *)node;
const char *chars = (const char *)&sw->sw[sw->len];
unsigned long long max = 0, subdir, ret;
int i;
int slashindex = -1, bareindex = -1;
/*
* First, process all the children of this node whose
* switch characters are not \0 or pathsep. We do this in
* reverse order so as to maintain best cache locality
* (tracking generally backwards through the file), though
* it doesn't matter semantically.
*
* For each of these children, we're just recursing into
* it to do any fixups required below it, and amalgamating
* the max atimes we get back.
*/
for (i = sw->len; i-- > 0 ;) {
if (chars[i] == '\0') {
bareindex = i;
} else if (chars[i] == pathsep) {
slashindex = i;
} else {
ret = fake_atime_recurse(t, MNODE(t, sw->sw[i].subnode,
trie_common), 0);
if (max < ret)
max = ret;
}
}
/*
* Now we have at most two child nodes left to deal with:
* one with a slash (or general pathsep) and one with \0.
*
* If there's a slash node and a bare node, then the slash
* node contains precisely everything inside the directory
* described by the bare node; so we must retrieve the max
* atime for the slash node and use it to fix up the bare
* node.
*
* If there's only a bare node but the pathname leading up
* to this point ends in a slash, then _all_ of the child
* nodes of this node contain stuff inside the directory
* described by the bare node; so we use the whole of the
* maximum value we've computed so far to update the bare
* node.
*/
if (slashindex >= 0) {
ret = fake_atime_recurse(t, MNODE(t, sw->sw[slashindex].subnode,
trie_common), 1);
if (max < ret)
max = ret;
subdir = ret;
} else if (last_seen_pathsep) {
subdir = max;
} else {
/* Don't update the bare subnode at all. */
subdir = 0;
}
if (bareindex >= 0) {
struct trie_leaf *leaf;
leaf = MNODE(t, sw->sw[bareindex].subnode, trie_leaf);
if (leaf && leaf->c.type == TRIE_LEAF) {
if (leaf->file.atime < subdir)
leaf->file.atime = subdir;
ret = leaf->file.atime;
} else {
/* Shouldn't really happen, but be cautious anyway */
ret = fake_atime_recurse(t, &leaf->c, 0);
}
if (max < ret)
max = ret;
}
return max;
}
return 0; /* placate lint */
}
void trie_fake_dir_atimes(void *t)
{
struct trie_header *hdr = MNODE(t, 0, trie_header);
struct trie_common *node = MNODE(t, hdr->root, trie_common);
fake_atime_recurse(t, node, 1);
}
/* ----------------------------------------------------------------------
* Querying functions.
*/
#define NODE(t, off, type) \
((const struct type *)((const char *)(t) + (off)))
int trie_check_magic(const void *t)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
struct trie_magic magic;
setup_magic(&magic);
return !memcmp(&magic, &hdr->magic, sizeof(magic));
}
size_t trie_maxpathlen(const void *t)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
return hdr->maxpathlen;
}
unsigned long trie_before(const void *t, const char *pathname)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
int ret = 0, lastcount = hdr->count;
int len = 1 + strlen(pathname), depth = 0;
off_t off = hdr->root;
while (1) {
const struct trie_common *node = NODE(t, off, trie_common);
if (node->type == TRIE_LEAF) {
if (depth < len)
ret += lastcount; /* _shouldn't_ happen, but in principle */
return ret;
} else if (node->type == TRIE_STRING) {
const struct trie_string *st = NODE(t, off, trie_string);
int offset;
int cmp = triencmp(st->string, st->stringlen,
pathname + depth, len-depth, &offset, pathsep);
if (offset < st->stringlen) {
if (cmp < 0)
ret += lastcount;
return ret;
}
depth += st->stringlen;
off = st->subnode;
} else if (node->type == TRIE_SWITCH) {
const struct trie_switch *sw = NODE(t, off, trie_switch);
const char *chars = (const char *)&sw->sw[sw->len];
int i;
for (i = 0; i < sw->len; i++) {
int c = chars[i];
int cmp = trieccmp(pathname[depth], c, pathsep);
if (cmp > 0)
ret += sw->sw[i].subcount;
else if (cmp < 0)
return ret;
else {
off = sw->sw[i].subnode;
lastcount = sw->sw[i].subcount;
depth++;
break;
}
}
if (i == sw->len)
return ret;
}
}
}
void trie_getpath(const void *t, unsigned long n, char *buf)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
int depth = 0;
off_t off = hdr->root;
while (1) {
const struct trie_common *node = NODE(t, off, trie_common);
if (node->type == TRIE_LEAF) {
assert(depth > 0 && buf[depth-1] == '\0');
return;
} else if (node->type == TRIE_STRING) {
const struct trie_string *st = NODE(t, off, trie_string);
memcpy(buf + depth, st->string, st->stringlen);
depth += st->stringlen;
off = st->subnode;
} else if (node->type == TRIE_SWITCH) {
const struct trie_switch *sw = NODE(t, off, trie_switch);
const char *chars = (const char *)&sw->sw[sw->len];
int i;
for (i = 0; i < sw->len; i++) {
if (n < sw->sw[i].subcount) {
buf[depth++] = chars[i];
off = sw->sw[i].subnode;
break;
} else
n -= sw->sw[i].subcount;
}
assert(i < sw->len);
}
}
}
const struct trie_file *trie_getfile(const void *t, unsigned long n)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
off_t off = hdr->root;
while (1) {
const struct trie_common *node = NODE(t, off, trie_common);
if (node->type == TRIE_LEAF) {
const struct trie_leaf *leaf = NODE(t, off, trie_leaf);
return &leaf->file;
} else if (node->type == TRIE_STRING) {
const struct trie_string *st = NODE(t, off, trie_string);
off = st->subnode;
} else if (node->type == TRIE_SWITCH) {
const struct trie_switch *sw = NODE(t, off, trie_switch);
int i;
for (i = 0; i < sw->len; i++) {
if (n < sw->sw[i].subcount) {
off = sw->sw[i].subnode;
break;
} else
n -= sw->sw[i].subcount;
}
assert(i < sw->len);
}
}
}
unsigned long trie_count(const void *t)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
return hdr->count;
}
char trie_pathsep(const void *t)
{
const struct trie_header *hdr = NODE(t, 0, trie_header);
return (char)hdr->pathsep;
}
struct triewalk_switch {
const struct trie_switch *sw;
int pos, depth, count;
};
struct triewalk {
const void *t;
struct triewalk_switch *switches;
int nswitches, switchsize;
int count;
};
triewalk *triewalk_new(const void *vt)
{
triewalk *tw = snew(triewalk);
tw->t = (const char *)vt;
tw->switches = NULL;
tw->switchsize = 0;
tw->nswitches = -1;
tw->count = 0;
return tw;
}
void triewalk_rebase(triewalk *tw, const void *t)
{
ptrdiff_t diff = ((const unsigned char *)t - (const unsigned char *)(tw->t));
int i;
tw->t = t;
for (i = 0; i < tw->nswitches; i++)
tw->switches[i].sw = (const struct trie_switch *)
((const unsigned char *)(tw->switches[i].sw) + diff);
}
const struct trie_file *triewalk_next(triewalk *tw, char *buf)
{
off_t off;
int depth;
if (tw->nswitches < 0) {
const struct trie_header *hdr = NODE(tw->t, 0, trie_header);
off = hdr->root;
depth = 0;
tw->nswitches = 0;
} else {
while (1) {
int swpos;
const struct trie_switch *sw;
const char *chars;
if (tw->nswitches == 0) {
assert(tw->count == NODE(tw->t, 0, trie_header)->count);
return NULL; /* run out of trie */
}
swpos = tw->switches[tw->nswitches-1].pos;
sw = tw->switches[tw->nswitches-1].sw;
chars = (const char *)&sw->sw[sw->len];
if (swpos < sw->len) {
depth = tw->switches[tw->nswitches-1].depth;
off = sw->sw[swpos].subnode;
if (buf)
buf[depth++] = chars[swpos];
assert(tw->count == tw->switches[tw->nswitches-1].count);
tw->switches[tw->nswitches-1].count += sw->sw[swpos].subcount;
tw->switches[tw->nswitches-1].pos++;
break;
}
tw->nswitches--;
}
}
while (1) {
const struct trie_common *node = NODE(tw->t, off, trie_common);
if (node->type == TRIE_LEAF) {
const struct trie_leaf *lf = NODE(tw->t, off, trie_leaf);
if (buf)
assert(depth > 0 && buf[depth-1] == '\0');
tw->count++;
return &lf->file;
} else if (node->type == TRIE_STRING) {
const struct trie_string *st = NODE(tw->t, off, trie_string);
if (buf)
memcpy(buf + depth, st->string, st->stringlen);
depth += st->stringlen;
off = st->subnode;
} else if (node->type == TRIE_SWITCH) {
const struct trie_switch *sw = NODE(tw->t, off, trie_switch);
const char *chars = (const char *)&sw->sw[sw->len];
if (tw->nswitches >= tw->switchsize) {
tw->switchsize = tw->nswitches * 3 / 2 + 32;
tw->switches = sresize(tw->switches, tw->switchsize,
struct triewalk_switch);
}
tw->switches[tw->nswitches].sw = sw;
tw->switches[tw->nswitches].pos = 1;
tw->switches[tw->nswitches].depth = depth;
tw->switches[tw->nswitches].count = tw->count + sw->sw[0].subcount;
off = sw->sw[0].subnode;
if (buf)
buf[depth++] = chars[0];
tw->nswitches++;
}
}
}
void triewalk_free(triewalk *tw)
{
sfree(tw->switches);
sfree(tw);
}
void trie_set_index_offset(void *t, off_t ptr)
{
((struct trie_header *)t)->indexroot = ptr;
}
off_t trie_get_index_offset(const void *t)
{
return ((const struct trie_header *)t)->indexroot;
}
void make_successor(char *pathbuf)
{
int len = strlen(pathbuf);
if (len > 0 && pathbuf[len-1] == pathsep)
len--;
pathbuf[len] = '\001';
pathbuf[len+1] = '\0';
}
agedu-20211129.8cd63c5/tree.but 0000644 0001750 0001750 00000031301 14151034324 014622 0 ustar simon simon \cfg{html-chapter-numeric}{true}
\cfg{html-chapter-suffix}{. }
\cfg{chapter}{Section}
\define{dash} \u2013{-}
\title A tree structure for log-time quadrant counting
This article describes a general form of data structure which
permits the storing of two-dimensional data in such a way that a
log-time lookup can reliably return the total \e{amount} of data in
an arbitrary quadrant (i.e. a region both upward and leftward of a
user-specified point).
The program
\W{https://www.chiark.greenend.org.uk/~sgtatham/agedu/}\cw{agedu},
which uses a data structure of this type for its on-disk index
files, is used as a case study.
\C{problem} The problem
The basic problem can be stated as follows: you have a collection of
\cw{N} points, each specified as an \cw{(x,y)} coordinate pair, and
you want to be able to efficiently answer questions of the general
form \q{How many points have both \cw{x\_<\_x0} and \cw{y\_<\_y0}?},
for arbitrary \cw{(x0,y0)}.
A slightly enhanced form of the problem, which is no more difficult
to solve, allows each point to have a \e{weight} as well as
coordinates; now the question to be answered is \q{What is the
\e{total weight} of the points with both \cw{x\_<\_x0} and \cw{y\_<\_y0}?}. The
previous version of the question, of course, is a special case of
this in which all weights are set to 1.
The data structure presented in this article answers any question of
this type in time \cw{O(log N)}.
\C{onedim} The one-dimensional case
To begin with, we address the one-dimensional analogue of the
problem. If we have a set of points which each have an
\cw{x}-coordinate and a weight, how do we represent them so as to be
able to find the total weight of all points with \cw{x\_<\_x0}?
An obvious solution is to sort the points into order of their
\cw{x}-coordinate, and alongside each point to list the total weight of
everything up to and including that point. Then, given any \cw{x0}, a
log-time binary search will find the last point in the list with
\cw{x\_<\_x0}, and we can immediately read off the cumulative weight
figure.
Now let's be more ambitious. What if the set of points were
constantly changing \dash new points arriving, old points going away
\dash and we wanted a data structure which would cope efficiently
with dynamic updates while maintaining the ability to answer these
count queries?
An efficient solution begins by storing the points in a balanced
tree, sorted by their \cw{x}-coordinates. Any of the usual types \dash
AVL tree, red-black tree, the various forms of B-tree \dash will
suffice. We then annotate every node of the tree with an extra field
giving the total weight of the points in and below that node.
These annotations can be maintained through updates to the tree,
without sacrificing the \cw{O(log N)} time bound on insertions or
deletions. So we can start with an empty tree and insert a complete
set of points, or we can insert and delete points in arbitrary
order, and the tree annotations will remain valid at all times.
A balanced tree sorted by \cw{x} can easily be searched to find the
last point with \cw{x\_<\_x0}, for any \cw{x0}. If the tree has total-weight
annotations as described above, we can arrange that this search
calculates the total weight of all points with \cw{x\_<\_x0}: every time
we examine a tree node and decide which subtree of it to descend to,
we look at the top node of each subtree to the left of that one, and
add their weight annotations to a running total. When we reach a
leaf node and find our chosen subtree is \cw{NULL}, the running
total is the required answer.
So this data structure allows us to maintain a changing set of
points in such a way that we can do an efficient one-dimensional
count query at any time.
\C{incremental} An incremental approach
Now we'll use the above one-dimensional structure to answer a
restricted form of the original 2-D problem. Suppose we have some
large set of \cw{(x0,y0)} pairs for which we want to answer
counted-quadrant queries, and suppose (for the moment) that we have
\e{all of the queries presented in advance}. Is there any way we can
do that efficiently?
There is. We sort our points by their \cw{y}-coordinate, and then go
through them in that order adding them one by one to a balanced tree
sorted by \cw{x} as described above.
So for any query coordinates \cw{(x0,y0)}, there must be some moment
during that process at which we have added to our tree precisely
those points with \cw{y\_<\_y0}. At that moment, we could search the tree
to find the total weight of everything it contained with \cw{x\_<\_x0},
and that would be the answer to our two-dimensional query.
Hence, if we also sorted our queries into order by \cw{y0}, we could
progress through the list of queries in parallel with the list of
points (much like merging two sorted lists), answering each query at
the moment when the tree contained just the right set of points to
make it easy.
\C{cow} Copy-on-write
In real life, of course, we typically don't receive all our queries
in a big batch up front. We want to construct a data structure
capable of answering \e{any} query efficiently, and then be able to
deal with queries as they arise.
A data structure capable of this, it turns out, is only a small step
away from the one described in the previous section. The small step
is \e{copy-on-write}.
As described in the previous section, we go through our list of
points in order of their \cw{y}-coordinate, and we add each one in turn
to a balanced tree sorted by \cw{x} with total-weight annotations. The
catch is that, this time, we never \e{modify} any node in the tree:
whenever the process of inserting an element into a tree wants to
modify a node, we instead make a \e{copy} of that node containing
the modified data. The parent of that node must now be modified
(even if it would previously not have needed modification) so that
it points at the copied node instead of the original \dash so we do
the same thing there, and copy that one too, and so on up to the
root.
So after we have done a single insertion by this method, we end up
with a new tree root which describes the new tree \dash but the old
tree root, describing the tree before the insertion, is still valid,
because every node reachable from the old root is unchanged.
Therefore, if we start with an empty tree and repeatedly do this, we
will end up with a distinct tree root for each point in the process,
and \e{all of them will be valid at once}. It's as if we had done
the incremental tree construction in the previous section, but could
rewind time to any point in the process.
So now all we have to do is make a list of those tree roots, with
their associated \cw{y}-coordinates. Any way of doing this will do
\dash another balanced tree, or a simple sorted list to be
binary-searched, or something more exotic, whatever is convenient.
Then we can answer an arbitrary quadrant query using only a pair of
log-time searches: given a query coordinate pair \cw{(x0,y0)}, we
look through our list of tree roots to find the one describing
precisely the set of points with \cw{y\_<\_y0}, and then do a
one-dimensional count query into that tree for the total weight of
points with \cw{x\_<\_x0}. Done!
The nice thing about all of this is that \e{nearly all} the nodes in
each tree are shared with the next one. Consider: since the
operation of adding an element to a balanced tree takes \cw{O(log N)}
time, it must modify at most \cw{O(log N)} nodes. Each of these
node-copying insertion processes must copy all of those nodes, but
need not copy any others \dash so it creates at most \cw{O(log N)} new
nodes. Hence, the total storage used by the combined set of trees is
\cw{O(N log N)}, much smaller than the \cw{O(N^2 log N)} you'd expect if
the trees were all separate or even mostly separate.
\C{limitations} Limitations
The one-dimensional data structure described at the start of this
article is dynamically updatable: if the set of points to be
searched changes, the structure can be modified efficiently without
losing its searching properties. The two-dimensional structure we
ended up with is not: if a single point changes its coordinates or
weight, or appears, or disappears, then the whole structure must be
rebuilt.
Since the technique I used to add an extra dimension is critically
dependent on the dynamic updatability of the one-dimensional base
structure, but the resulting structure is not dynamically updatable
in turn, it follows that this technique cannot be applied twice: no
analogous transformation will construct a \e{three}-dimensional
structure capable of counting the total weight of an octant
\cw{\{x\_<\_x0, y\_<\_y0, z\_<\_z0\}}. I know of no efficient way
to do that.
The structure as described above uses \cw{O(N log N)} storage. Many
algorithms using \cw{O(N log N)} time are considered efficient (e.g.
sorting), but space is generally more expensive than time, and \cw{O(N
log N)} space is larger than you think!
\C{application} An application: \cw{agedu}
The application for which I developed this data structure is
\W{https://www.chiark.greenend.org.uk/~sgtatham/agedu/}\cw{agedu}, a
program which scans a file system and indexes pathnames against
last-access times (\q{atimes}, in Unix terminology) so as to be able
to point out directories which take up a lot of disk space but whose
contents have not been accessed in years (making them strong
candidates for tidying up or archiving to save space).
So the fundamental searching primitive we want for \cw{agedu} is
\q{What is the total size of the files contained within some
directory path \cw{Ptop} which have atime at most \cw{T0}?}
We begin by sorting the files into order by their full pathname.
This brings every subdirectory, at every level, together into a
contiguous sequence of files. So now our query primitive becomes
\q{What is the total size of files whose pathname falls between
\cw{P0} and \cw{P1}, and whose atime is at most \cw{T0}?}
Clearly, we can simplify this to the same type of quadrant query as
discussed above, by splitting this into two subqueries: the total
size of files with \cw{P0\_<=\_P\_<\_P1} and \cw{T\_<\_T0} is clearly the
total size of files with \cw{P\_<\_P1, T\_<\_T0} minus the total size
of files with \cw{P\_<\_P0, T\_<\_T0}. Each of those subqueries is of
precisely the type we have just derived a data structure to answer.
So we want to sort our files by two \q{coordinates}: one is atime,
and the other is pathname (sorted in ASCII collation order). So
which should be \cw{x} and which \cw{y}, in the above notation?
Well, either way round would work in principle, but two criteria
inform the decision. Firstly, \cw{agedu} typically wants to do many
queries on the same pathname for different atimes, so as to build up
a detailed profile of size against atime for a given subdirectory.
So it makes sense to have the first-level lookup (on \cw{y}, to find a
tree root) be done on pathname, and the secondary lookup (on \cw{x},
within that tree) be done on atime; then we can cache the tree root
found in the first lookup, and use it many times without wasting
time repeating the pathname search.
Another important point for \cw{agedu} is that not all tree roots
are actually used: the only pathnames ever submitted to a quadrant
search are those at the start or the end of a particular
subdirectory. This allows us to save a lot of disk space by limiting
the copy-on-write behaviour: instead of \e{never} modifying an
existing tree node, we now rule that we may modify a tree node \e{if
it has already been modified once since the last tree root we
saved}. In real-world cases, this cuts down the total space usage by
about a factor of five, so it's well worthwhile \dash and we
wouldn't be able to do it if we'd used atime as the \cw{y}-coordinate
instead of pathname.
Since the \q{\cw{y}-coordinates} in \cw{agedu} are strings, the
top-level lookup to find a tree root is most efficiently done using
neither a balanced tree nor a sorted list, but a trie: tries allow
lookup of a string in time proportional to the length of the string,
whereas either of the other approaches would require \cw{O(log N)}
compare operations \e{each} of which could take time proportional to
the length of the string.
Finally, the two-dimensions limitation on the above data structure
unfortunately imposes limitations on what \cw{agedu} can do. One
would like to run a single \cw{agedu} as a system-wide job on a file
server (perhaps nightly or weekly), and present the results to all
users in such a way that each user's view of the data was filtered
to only what their access permissions permitted them to see. Sadly,
to do this would require a third dimension in the data structure
(representing ownership or access control, in some fashion), and
hence cannot be done efficiently. \cw{agedu} is therefore instead
most sensibly used on demand by an individual user, so that it
generates a custom data set for that user every time.
agedu-20211129.8cd63c5/licence.c 0000644 0001750 0001750 00000002472 14151034324 014724 0 ustar simon simon #include "agedu.h"
const char *const licence[] = {
PNAME " is copyright 2008 Simon Tatham. All rights reserved.\n",
"\n",
"Permission is hereby granted, free of charge, to any person\n",
"obtaining a copy of this software and associated documentation files\n",
"(the \"Software\"), to deal in the Software without restriction,\n",
"including without limitation the rights to use, copy, modify, merge,\n",
"publish, distribute, sublicense, and/or sell copies of the Software,\n",
"and to permit persons to whom the Software is furnished to do so,\n",
"subject to the following conditions:\n",
"\n",
"The above copyright notice and this permission notice shall be\n",
"included in all copies or substantial portions of the Software.\n",
"\n",
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\n",
"EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\n",
"MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\n",
"NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS\n",
"BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN\n",
"ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN\n",
"CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
"SOFTWARE.\n",
0
};
agedu-20211129.8cd63c5/index.h 0000644 0001750 0001750 00000004737 14151034324 014444 0 ustar simon simon /*
* index.h: Manage indexes for agedu.
*/
#include <sys/types.h>
/*
* Given the size of an existing data file and the number of
* entries required in the index, return the file size required to
* store the fixed initial fragment of the index before the
* indexbuild functions are invoked.
*/
off_t index_initial_size(off_t currentsize, int nodecount);
/*
* Build an index, given the address of a memory-mapped data file
* and the starting offset within it.
*
* trie_file structures passed to tf must of course be within the
* bounds of the memory-mapped file.
*
* The "delta" parameter to indexbuild_new is filled in with the
* maximum size by which the index can ever grow in a single
* indexbuild_add call. This can be used, together with
* indexbuild_realsize, to detect when the index is about to get
* too big for its file and the file needs resizing.
*
* indexbuild_add adds a trie_file structure to the index.
* indexbuild_tag, called after that, causes the current state of
* the index to be preserved so that it can be queried later. It's
* idempotent, and will safely do nothing if called before
* indexbuild_add.
*
* indexbuild_realsize returns the total amount of data _actually_
* written into the file, to allow it to be truncated afterwards,
* and to allow the caller of the indexbuild functions to know
* when the file is about to need to grow bigger during index
* building.
*
* indexbuild_rebase is used when the file has been
* re-memory-mapped at a different address (because it needs to
* grow).
*/
typedef struct indexbuild indexbuild;
indexbuild *indexbuild_new(void *t, off_t startoff, int nodecount,
size_t *delta);
void indexbuild_add(indexbuild *ib, const struct trie_file *tf);
void indexbuild_tag(indexbuild *ib);
void indexbuild_rebase(indexbuild *ib, void *t);
off_t indexbuild_realsize(indexbuild *ib);
void indexbuild_free(indexbuild *ib);
/*
* Check to see if a name index has an index tree root available
* (i.e. represents a directory rather than a file).
*/
int index_has_root(const void *t, int n);
/*
* Query an index to find the total size of records with name
* index strictly less than n, with atime less than at.
*/
unsigned long long index_query(const void *t, int n, unsigned long long at);
/*
* Retrieve an order statistic from the index: given a fraction f,
* return an atime such that (at most) the requested fraction of
* the total data is older than it.
*/
unsigned long long index_order_stat(const void *t, double f);
agedu-20211129.8cd63c5/index.c 0000644 0001750 0001750 00000023756 14151034324 014441 0 ustar simon simon /*
* index.c: Implementation of index.h.
*/
#include "agedu.h"
#include "trie.h"
#include "index.h"
#include "alloc.h"
#define alignof(typ) ( offsetof(struct { char c; typ t; }, t) )
#define PADDING(x, mod) ( ((mod) - ((x) % (mod))) % (mod) )
struct avlnode {
off_t children[2], element;
int maxdepth; /* maximum depth of this subtree */
unsigned long long totalsize;
};
/*
* Determine the maximum depth of an AVL tree containing a certain
* number of nodes.
*/
static int index_maxdepth(int nodecount)
{
int b, c, maxdepth;
/*
* Model the tree growing at maximum imbalance. We do this by
* determining the number of nodes in the most unbalanced
* (i.e. smallest) tree of any given depth, and stopping when
* that's larger than nodecount.
*/
maxdepth = 1;
b = 0;
c = 1;
while (b <= nodecount) {
int tmp;
tmp = 1 + b + c;
b = c;
c = tmp;
maxdepth++;
}
return maxdepth;
}
off_t index_initial_size(off_t currentsize, int nodecount)
{
currentsize += PADDING(currentsize, alignof(off_t));
currentsize += nodecount * sizeof(off_t);
currentsize += PADDING(currentsize, alignof(struct avlnode));
return currentsize;
}
/* ----------------------------------------------------------------------
* Functions to build the index.
*/
struct indexbuild {
void *t;
int n, nnodes;
struct avlnode *nodes;
off_t *roots;
struct avlnode *currroot;
struct avlnode *firstmutable;
};
#define ELEMENT(t,offset) \
((struct trie_file *) ((offset) ? ((char *)(t) + (offset)) : NULL))
#define NODE(t,offset) \
((struct avlnode *) ((offset) ? ((char *)(t) + (offset)) : NULL))
#define OFFSET(t,node) \
((node) ? (off_t)((const char *)node - (const char *)t) : (off_t)0)
#define MAXDEPTH(node) ((node) ? (node)->maxdepth : 0)
indexbuild *indexbuild_new(void *t, off_t startoff, int nodecount,
size_t *delta)
{
indexbuild *ib = snew(indexbuild);
ib->t = t;
startoff += PADDING(startoff, alignof(off_t));
ib->roots = (off_t *)((char *)t + startoff);
trie_set_index_offset(t, startoff);
startoff += nodecount * sizeof(off_t);
startoff += PADDING(startoff, alignof(struct avlnode));
ib->nodes = (struct avlnode *)((char *)t + startoff);
ib->nnodes = ib->n = 0;
ib->currroot = NULL;
ib->firstmutable = ib->nodes + ib->nnodes;
if (delta)
*delta = sizeof(struct avlnode) * (1 + index_maxdepth(nodecount));
return ib;
}
/*
* Return a mutable node, which is n or a copy of n if n is
* non-NULL.
*/
static struct avlnode *avl_makemutable(indexbuild *ib, struct avlnode *n)
{
struct avlnode *newnode;
if (n && n >= ib->firstmutable)
return n; /* already mutable */
newnode = ib->nodes + ib->nnodes++;
if (n)
*newnode = *n; /* structure copy */
return newnode;
}
/*
* Fix the annotations in a tree node.
*/
static void avl_fix(indexbuild *ib, struct avlnode *n)
{
/*
* Make sure the max depth field is right.
*/
n->maxdepth = 1 + max(MAXDEPTH(NODE(ib->t, n->children[0])),
MAXDEPTH(NODE(ib->t, n->children[1])));
n->totalsize =
(ELEMENT(ib->t, n->element)->size +
(n->children[0] ? NODE(ib->t, n->children[0])->totalsize : 0) +
(n->children[1] ? NODE(ib->t, n->children[1])->totalsize : 0));
}
static struct avlnode *avl_insert(indexbuild *ib, struct avlnode *n,
off_t node)
{
struct trie_file *newfile;
struct trie_file *oldfile;
int subtree;
struct avlnode *nn;
/*
* Recursion bottoming out: if the subtree we're inserting
* into is null, just construct and return a fresh node.
*/
if (!n) {
n = avl_makemutable(ib, NULL);
n->children[0] = n->children[1] = 0;
n->element = node;
avl_fix(ib, n);
return n;
}
/*
* Otherwise, we have to insert into an existing tree.
*/
/*
* Determine which subtree to insert this node into. Ties
* aren't important, so we just break them any old way.
*/
newfile = (struct trie_file *)((char *)ib->t + node);
oldfile = (struct trie_file *)((char *)ib->t + n->element);
if (newfile->atime > oldfile->atime)
subtree = 1;
else
subtree = 0;
/*
* Construct a copy of the node we're looking at.
*/
n = avl_makemutable(ib, n);
/*
* Recursively insert into the next subtree down.
*/
nn = avl_insert(ib, NODE(ib->t, n->children[subtree]), node);
n->children[subtree] = OFFSET(ib->t, nn);
/*
* Rebalance if necessary, to ensure that our node's children
* differ in maximum depth by at most one. Of course, the
* subtree we've just modified will be the deeper one if so.
*/
if (MAXDEPTH(NODE(ib->t, n->children[subtree])) >
MAXDEPTH(NODE(ib->t, n->children[1-subtree])) + 1) {
struct avlnode *p, *q;
/*
* There are two possible cases, one of which requires a
* single tree rotation and the other requires two. It all
* depends on which subtree of the next node down (here p)
* is the taller. (It turns out that they can't both be
* the same height: any tree which has just increased in
* depth must have one subtree strictly taller than the
* other.)
*/
p = NODE(ib->t, n->children[subtree]);
assert(p >= ib->firstmutable);
if (MAXDEPTH(NODE(ib->t, p->children[subtree])) >=
MAXDEPTH(NODE(ib->t, p->children[1-subtree]))) {
/*
* n p
* / \ / \
* [k] p -> n [k+1]
* / \ / \
* [k] [k+1] [k] [k]
*/
n->children[subtree] = p->children[1-subtree];
p->children[1-subtree] = OFFSET(ib->t, n);
avl_fix(ib, n);
n = p;
} else {
q = NODE(ib->t, p->children[1-subtree]);
assert(q >= ib->firstmutable);
p->children[1-subtree] = OFFSET(ib->t, q);
/*
* n n q
* / \ / \ / \
* [k] p == [k] p -> n p
* / \ / \ / \ / \
* [k+1] [k] q [k] [k] \ / [k]
* / \ [k-1,k] [k-1,k]
* [k-1,k] [k-1,k]
*/
n->children[subtree] = q->children[1-subtree];
p->children[1-subtree] = q->children[subtree];
q->children[1-subtree] = OFFSET(ib->t, n);
q->children[subtree] = OFFSET(ib->t, p);
avl_fix(ib, n);
avl_fix(ib, p);
n = q;
}
}
/*
* Fix up our maximum depth field.
*/
avl_fix(ib, n);
/*
* Done.
*/
return n;
}
void indexbuild_add(indexbuild *ib, const struct trie_file *tf)
{
off_t node = OFFSET(ib->t, tf);
ib->currroot = avl_insert(ib, ib->currroot, node);
ib->roots[ib->n++] = 0;
}
void indexbuild_tag(indexbuild *ib)
{
if (ib->n > 0)
ib->roots[ib->n - 1] = OFFSET(ib->t, ib->currroot);
ib->firstmutable = ib->nodes + ib->nnodes;
}
void indexbuild_rebase(indexbuild *ib, void *t)
{
ptrdiff_t diff = (unsigned char *)t - (unsigned char *)(ib->t);
ib->t = t;
ib->nodes = (struct avlnode *)((unsigned char *)ib->nodes + diff);
ib->roots = (off_t *)((unsigned char *)ib->roots + diff);
if (ib->currroot)
ib->currroot = (struct avlnode *)
((unsigned char *)ib->currroot + diff);
ib->firstmutable = (struct avlnode *)((unsigned char *)ib->firstmutable + diff);
}
off_t indexbuild_realsize(indexbuild *ib)
{
return OFFSET(ib->t, (ib->nodes + ib->nnodes));
}
void indexbuild_free(indexbuild *ib)
{
assert(ib->n == trie_count(ib->t));
sfree(ib);
}
int index_has_root(const void *t, int n)
{
const off_t *roots;
roots = (const off_t *)((const char *)t + trie_get_index_offset(t));
if (n == 0)
return 1;
if (n < 0 || n >= trie_count(t) || !roots[n-1])
return 0;
return 1;
}
unsigned long long index_query(const void *t, int n, unsigned long long at)
{
const off_t *roots;
const struct avlnode *node;
unsigned long count;
unsigned long long ret;
roots = (const off_t *)((const char *)t + trie_get_index_offset(t));
if (n < 1)
return 0;
count = trie_count(t);
if (n > count)
n = count;
assert(roots[n-1]);
node = NODE(t, roots[n-1]);
ret = 0;
while (node) {
const struct trie_file *tf = ELEMENT(t, node->element);
const struct avlnode *left = NODE(t, node->children[0]);
const struct avlnode *right = NODE(t, node->children[1]);
if (at <= tf->atime) {
node = left;
} else {
if (left)
ret += left->totalsize;
ret += tf->size;
node = right;
}
}
return ret;
}
unsigned long long index_order_stat(const void *t, double f)
{
const off_t *roots;
const struct avlnode *node;
unsigned long count;
unsigned long long size;
roots = (const off_t *)((const char *)t + trie_get_index_offset(t));
count = trie_count(t);
assert(roots[count-1]);
node = NODE(t, roots[count-1]);
size = node->totalsize * f;
if (size > node->totalsize) {
/*
* This can happen in principle if node->totalsize is so
* enormous (bigger than 2^53 bytes) that it can't be
* represented faithfully in a 'double'. Then it might be
* rounded _up_ by the conversion to double, and if
* multiplication by f doesn't make it smaller again, end up
* just bigger than node->totalsize by the time we convert
* back to an integer.
*
* It takes a huge filesystem to make that happen in reality -
* but a corrupt index file could also trip the same case, so
* we should at least handle it gracefully.
*/
size = node->totalsize;
}
while (1) {
const struct trie_file *tf = ELEMENT(t, node->element);
const struct avlnode *left = NODE(t, node->children[0]);
const struct avlnode *right = NODE(t, node->children[1]);
if (left && size < left->totalsize) {
node = left;
} else if (!right ||
size < (left ? left->totalsize : 0) + tf->size) {
return tf->atime;
} else {
if (left)
size -= left->totalsize;
size -= tf->size;
node = right;
}
}
}
agedu-20211129.8cd63c5/httpd.h 0000644 0001750 0001750 00000000677 14151034324 014457 0 ustar simon simon /*
* httpd.h: a minimal custom HTTP server designed to serve the
* pages generated by html.h.
*/
#define HTTPD_AUTH_MAGIC 1
#define HTTPD_AUTH_BASIC 2
#define HTTPD_AUTH_NONE 4
struct httpd_config {
const char *address, *port;
bool closeoneof;
const char *basicauthdata;
const char *url_launch_command;
};
void run_httpd(const void *t, int authmask, const struct httpd_config *dcfg,
const struct html_config *pcfg);
agedu-20211129.8cd63c5/httpd.c 0000644 0001750 0001750 00000074577 14151034324 014464 0 ustar simon simon /*
* httpd.c: implementation of httpd.h.
*/
#include "agedu.h"
#include "alloc.h"
#include "html.h"
#include "httpd.h"
/* --- Logic driving what the web server's responses are. --- */
enum { /* connctx states */
READING_REQ_LINE,
READING_HEADERS,
DONE
};
struct connctx {
const void *t;
char *data;
int datalen, datasize;
char *method, *url, *headers, *auth;
int state;
};
/*
* Called when a new connection arrives on a listening socket.
* Returns a connctx for the new connection.
*/
struct connctx *new_connection(const void *t)
{
struct connctx *cctx = snew(struct connctx);
cctx->t = t;
cctx->data = NULL;
cctx->datalen = cctx->datasize = 0;
cctx->state = READING_REQ_LINE;
cctx->method = cctx->url = cctx->headers = cctx->auth = NULL;
return cctx;
}
void free_connection(struct connctx *cctx)
{
sfree(cctx->data);
sfree(cctx);
}
static char *http_error(char *code, char *errmsg, char *extraheader,
char *errtext, ...)
{
return dupfmt("HTTP/1.1 %s %s\r\n"
"Date: %D\r\n"
"Server: " PNAME "\r\n"
"Connection: close\r\n"
"%s"
"Content-Type: text/html; charset=US-ASCII\r\n"
"\r\n"
"<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\r\n"
"<HTML><HEAD>\r\n"
"<TITLE>%s %s\r\n"
"\r\n"
"%s %s
\r\n"
"%s
\r\n"
"