./ 0000755 0000041 0000041 00000000000 12756720745 011262 5 ustar www-data www-data ./doc/ 0000755 0000041 0000041 00000000000 12756720745 012027 5 ustar www-data www-data ./doc/main_page.dox.in 0000644 0000041 0000041 00000013246 12756720745 015076 0 ustar www-data www-data /*!
\mainpage
\section overview Overview
This API provides a persistent cache of key-value pairs. core::PersistentStringCache stores key,
value, and metadata as type `std::string`. A simple type adapter template, core::PersistentCache,
is provided to permit the use of types other than `string`.
The cache is intended for caching arbitrary (possibly large) amounts of data,
such as might be needed by a web browser cache. The cache scales to large
numbers (millions) of entries and is [very fast](@ref performance).
The implementation is based on [leveldb](http://leveldb.org) and typically
provides throughput many times larger than the I/O bandwidth to disk.
The cache is robust in the face of crashes and power loss. After a
re-start, it is guaranteed to be in a consistent state with correct
data. (Some number of updates that were made just prior to a power loss
or kernel crash can be lost; however, if just the calling process
crashes, no updates are lost.)
A cache has a maximum size (which can be changed at any time). Once
the cache reaches its maximum size, when adding an entry,
the cache automatically discards enough entries to make room for the new entry.
Keys can be (possibly binary) strings of size > 0. Values can be
(possibly binary) strings including the empty string. Using the
core::PersistentCache template, keys, values, and [metadata](@ref metadata)
can be arbitrary user-defined types. The template requires the
the application to provide encode and decode functions that
convert each user-defined type to/from `string` and calls
these functions automatically. This relieves the application
from explicitly having to serialize or deserialize user-defined
types when manipulating the cache.
Entries maintain an access time, which is used to keep them in
least-recently-used (LRU) order. In addition, entries can have
an optional expiry time. (If no expiry time is specified, infinite
expiry time is assumed.)
\note The cache is thread-safe; you can call member functions from
different threads without any synchronization. Thread-safety is
provided for convenience, not performance. Calling concurrently
into the cache from multiple threads will not yield improved performance.
\subsection policy Discard policy
The cache provides two different discard policies, `lru_ttl`
and `lru_only`.
For `lru_ttl`, the discard policy of the cache is to first delete all
entries that have expired. If this does not free sufficient space
to make room for a new entry, the cache then deletes entries in oldest
to newest (LRU) order until sufficient space is available. This
deletion in LRU order may delete entries that have an
expiry time, but have not expired yet, as well as entries with
infinite expiry time.
For `lru_only`, entries do not maintain an expiry time and
are therefore discarded strictly in LRU order.
Access and expiry times are recorded with millisecond granularity.
To indicate infinite expiry time, use the defaulted parameter value or
`chrono::system_clock::time_point()`.
\subsection metadata Metadata
Besides storing key-value pairs, the cache allows you to add arbitrary
extra data to each entry. This is useful, for example, to maintain
metadata (such as HTTP header details) for the entries in the cache.
\warning It is not possible to distinguish between "no metadata was added"
and "empty metadata was added". Do not use the metadata in such
a way that you rely the difference between "metadata not there" and
"metadata is the empty string".
\subsection errors Error reporting
Methods throw `std::runtime_error` if the underlying database
(leveldb) reports an error. If leveldb detects database corruption, the code
throws `std::system_error` with with a 666 error code. To recover
from this error, remove all files in the cache directory.
Other errors are indicated by throwing `std::logic_error` or
`std::invalid_argument` as appropriate.
\subsection performance Performance
Some rough performance figures, taken on an Intel Ivy Bridge i7-3770K 3.5 GHz
with 16 GB RAM, appear below. Records are filled with random data to make them non-compressible.
After filling the cache, the code performs cache lookups using random
keys, with an 80% hit probability. On a miss, it inserts a new
random record. This measures the typical steady-state behavior:
whenever a cache miss happens, the caller fetches the data and
inserts a new record into the cache.
Setting | Value
---------- | ------
Cache size | 100 MB
# Records | ~5100
Record size | 20 kB, normal distribution, stddev = 7000
Running the test With a 7200 rpm spinning disk produces:
Parameter | Value
-------------|------------
Reads | 30.9 MB/sec
Writes | 7.0 MB/sec
Records/sec | 1995
Running the test With an Intel 256 GB SSD produces:
Parameter | Value
-------------|------------
Reads | 80.4 MB/sec
Writes | 15.7 MB/sec
Records/sec | 4932
\note When benchmarking, make sure to compile in release mode. In debug
mode, a number of expensive assertions are turned on.
\note Also be aware that leveldb uses Snappy compression beneath the
covers. This means that, if test data is simply filled with
a fixed byte pattern, you will measure artificially high performance.
\subsection linking Compiling and linking
The API is provided as a static library, `lib@LIBNAME@.a`. (Code size on a 64-bit processor is less than 100 kB.)
You can compile and link with the library as follows:
g++ --std=c++11 -o myprog myprog.cpp `pkg-config --cflags --libs libpersistent-cache-cpp` -lleveldb
\subsection examples Examples
See core::PersistentStringCache and core::PersistentCache for simple usage examples.
Additional code examples can be found in
[@LIBNAME@/examples](file:///@CMAKE_INSTALL_PREFIX@/share/doc/@LIBNAME@/examples).
*/
./doc/CMakeLists.txt 0000644 0000041 0000041 00000000065 12756720745 014570 0 ustar www-data www-data configure_file(main_page.dox.in main_page.dox @ONLY)
./valgrind-suppress 0000644 0000041 0000041 00000000000 12756720745 014663 0 ustar www-data www-data ./tests/ 0000755 0000041 0000041 00000000000 12756720745 012424 5 ustar www-data www-data ./tests/headers/ 0000755 0000041 0000041 00000000000 12756720745 014037 5 ustar www-data www-data ./tests/headers/check_public_headers.py 0000755 0000041 0000041 00000006222 12756720745 020524 0 ustar www-data www-data #! /usr/bin/env python3
#
# Copyright (C) 2013 Canonical Ltd
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License version 3 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program. If not, see .
#
# Authored by: Michi Henning
#
#
# Little helper program to test that public header files don't include internal header files.
#
# Usage: check_public_headers.py directory
#
# The directory specifies the location of the header files. All files in that directory ending in .h (but not
# in subdirectories) are tested.
#
import argparse
import os
import sys
import re
#
# Write the supplied message to stderr, preceded by the program name.
#
def error(msg):
print(os.path.basename(sys.argv[0]) + ": " + msg, file=sys.stderr)
#
# Write the supplied message to stdout, preceded by the program name.
#
def message(msg):
print(os.path.basename(sys.argv[0]) + ": " + msg)
#
# For each of the supplied headers, check whether that header includes something in an internal directory.
# Return the count of headers that do this.
#
def test_files(hdr_dir, hdrs):
num_errs = 0
for hdr in hdrs:
try:
hdr_name = os.path.join(hdr_dir, hdr)
file = open(hdr_name, 'r', encoding = 'utf=8')
except OSError as e:
error("cannot open \"" + hdr_name + "\": " + e.strerror)
sys.exit(1)
include_pat = re.compile(r'#[ \t]*include[ \t]+[<"](.*?)[>"]')
lines = file.readlines()
line_num = 0
for l in lines:
line_num += 1
include_mo = include_pat.match(l)
if include_mo:
hdr_path = include_mo.group(1)
if 'internal/' in hdr_path:
num_errs += 1
# Yes, write to stdout because this is expected output
message(hdr_name + " includes an internal header at line " + str(line_num) + ": " + hdr_path)
return num_errs
def run():
#
# Parse arguments.
#
parser = argparse.ArgumentParser(description = 'Test that no public header includes an internal header.')
parser.add_argument('dir', nargs = 1, help = 'The directory to look for header files ending in ".h"')
args = parser.parse_args()
#
# Find all the .h files in specified directory and look for #include directives that mention "internal/".
#
hdr_dir = args.dir[0]
try:
files = os.listdir(hdr_dir)
except OSError as e:
error("cannot open \"" + hdr_dir + "\": " + e.strerror)
sys.exit(1)
hdrs = [hdr for hdr in files if hdr.endswith('.h')]
if test_files(hdr_dir, hdrs) != 0:
sys.exit(1) # Errors were reported earlier
if __name__ == '__main__':
run()
./tests/headers/compile_headers.py 0000755 0000041 0000041 00000014747 12756720745 017554 0 ustar www-data www-data #! /usr/bin/env python3
# Copyright (C) 2013 Canonical Ltd.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 3 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see .
#
# Authored by: Michi Henning
#
# Little helper program to test that header files are stand-alone compilable (and therefore don't depend on
# other headers being included first).
#
# Usage: compile_headers.py directory compiler [compiler_flags]
#
# The directory specifies the location of the header files. All files in that directory ending in .h (but not
# in subdirectories) are tested.
#
# The compiler argument specifies the compiler to use (such as "gcc"), and the compiler_flags argument (which
# must be a single string argument, not a bunch of separate strings) specifies any additional flags, such
# as "-I -g". The flags need not include "-c".
#
# For each header file in the specified directory, the script create a corresponding .cpp that includes the
# header file. The .cpp file is created in the current directory (which isn't necessarily the same one as
# the directory the header files are in). The script runs the compiler on the generated .cpp file and, if the
# compiler returns non-zero exit status, it prints a message (on stdout) reporting the failure.
#
# The script does not stop if a file fails to compile. If all source files compile successfully, no output (other
# than the output from the compiler) is written, and the exit status is zero. If one or more files do not compile,
# or there are any other errors, such as not being able to open a file, exit status is non-zero.
#
# Messages about files that fail to compile are written to stdout. Message about other problems, such as non-existent
# files and the like, are written to stderr.
#
# The compiler's output goes to whatever stream the compiler writes to and is left alone.
#
import argparse
import os
import re
import shlex
import subprocess
import sys
import tempfile
import concurrent.futures, multiprocessing
# Additional #defines that should be set before including a header.
# This is intended for things such as enabling additional features
# that are conditionally compiled in the source.
extra_defines =[]
#
# Write the supplied message to stderr, preceded by the program name.
#
def error(msg):
print(os.path.basename(sys.argv[0]) + ": " + msg, file=sys.stderr)
#
# Write the supplied message to stdout, preceded by the program name.
#
def message(msg):
print(os.path.basename(sys.argv[0]) + ": " + msg)
#
# Create a source file in the current directory that includes the specified header, compile it,
# and check exit status from the compiler. Throw if the compile command itself fails,
# return False if the compile command worked but reported errors, True if the compile succeeded.
#
def run_compiler(hdr, compiler, copts, verbose, hdr_dir):
try:
src = tempfile.NamedTemporaryFile(suffix='.cpp', dir='.')
# Add any extra defines at the beginning of the temporary file.
for flag in extra_defines:
src.write(bytes("#define " + flag + "" + "\n", 'UTF-8'))
src.write(bytes("#include <" + hdr + ">" + "\n", 'UTF-8'))
src.flush() # Need this to make the file visible
src_name = os.path.join('.', src.name)
if verbose:
print(compiler + " -c " + src_name + " " + copts)
status = subprocess.call([compiler] + shlex.split(copts) + ["-c", src_name])
if status != 0:
message("cannot compile \"" + hdr + "\"") # Yes, write to stdout because this is expected output
obj = os.path.splitext(src_name)[0] + ".o"
try:
os.unlink(obj)
except:
pass
gcov = os.path.splitext(src_name)[0] + ".gcno"
try:
os.unlink(gcov)
except:
pass
return status == 0
except OSError as e:
error(e.strerror)
raise
#
# For each of the supplied headers, create a source file in the current directory that includes the header
# and then try to compile the header. Returns normally if all files could be compiled successfully and
# throws, otherwise.
#
def test_files(hdrs, compiler, copts, verbose, hdr_dir):
num_errs = 0
executor = concurrent.futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count())
futures = [executor.submit(run_compiler, h, compiler, copts, verbose, hdr_dir) for h in hdrs]
for f in futures:
try:
if not f.result():
num_errs += 1
except OSError:
num_errs += 1
pass # Error reported already
if num_errs != 0:
msg = str(num_errs) + " file"
if num_errs != 1:
msg += "s"
msg += " failed to compile"
message(msg) # Yes, write to stdout because this is expected output
sys.exit(1)
def run():
#
# Parse arguments.
#
parser = argparse.ArgumentParser(description = 'Test that all headers in the passed directory compile stand-alone.')
parser.add_argument('-v', '--verbose', action='store_true', help = 'Trace invocations of the compiler')
parser.add_argument('dir', nargs = 1, help = 'The directory to look for header files ending in ".h"')
parser.add_argument('compiler', nargs = 1, help = 'The compiler executable, such as "gcc"')
parser.add_argument('copts', nargs = '?', default="",
help = 'The compiler options (excluding -c), such as "-g -Wall -I." as a single string.')
args = parser.parse_args()
#
# Find all the .h files in specified directory and do the compilation for each one.
#
hdr_dir = args.dir[0]
try:
files = os.listdir(hdr_dir)
except OSError as e:
msg = "cannot open \"" + hdr_dir + "\": " + e.strerror
error(msg)
sys.exit(1)
hdrs = [hdr for hdr in files if hdr.endswith('.h')]
try:
test_files(hdrs, args.compiler[0], args.copts, args.verbose, hdr_dir)
except OSError:
sys.exit(1) # Errors were written earlier
if __name__ == '__main__':
run()
./tests/headers/CMakeLists.txt 0000644 0000041 0000041 00000002567 12756720745 016611 0 ustar www-data www-data #
# Test that all header files compile stand-alone and that no public header includes an internal one.
#
set(root_inc_dir ${CMAKE_SOURCE_DIR}/include)
set(subdirs
core
)
foreach(dir ${subdirs})
string(REPLACE "/" "-" location ${dir})
set(public_inc_dir ${root_inc_dir}/${dir})
set(internal_inc_dir ${public_inc_dir}/internal)
if (${slowtests})
# Test that each public header compiles stand-alone.
add_test(stand-alone-${location}-headers
${CMAKE_CURRENT_SOURCE_DIR}/compile_headers.py
${public_inc_dir} ${CMAKE_CXX_COMPILER} "${CMAKE_CXX_COMPILER_ARG1} -fPIC -fsyntax-only -I${root_inc_dir} -I${public_inc_dir} ${other_inc_dirs} ${CMAKE_CXX_FLAGS} ${extra_inc_dirs}")
# Test that each internal header compiles stand-alone.
if (IS_DIRECTORY ${internal_inc_dir})
add_test(stand-alone-${location}-internal-headers
${CMAKE_CURRENT_SOURCE_DIR}/compile_headers.py
${internal_inc_dir} ${CMAKE_CXX_COMPILER} "${CMAKE_CXX_COMPILER_ARG1} -fPIC -fsyntax-only -I${root_inc_dir} -I${internal_inc_dir} ${other_inc_dirs} ${CMAKE_CXX_FLAGS} ${extra_includes}")
endif()
endif()
# Test that no public header includes an internal header
add_test(clean-public-${location}-headers ${CMAKE_CURRENT_SOURCE_DIR}/check_public_headers.py ${public_inc_dir})
endforeach()
./tests/core/ 0000755 0000041 0000041 00000000000 12756720745 013354 5 ustar www-data www-data ./tests/core/internal/ 0000755 0000041 0000041 00000000000 12756720745 015170 5 ustar www-data www-data ./tests/core/internal/persistent_string_cache_impl/ 0000755 0000041 0000041 00000000000 12756720745 023122 5 ustar www-data www-data ./tests/core/internal/persistent_string_cache_impl/persistent_string_cache_impl_test.cpp 0000644 0000041 0000041 00000203115 12756720745 032621 0 ustar www-data www-data /*
* Copyright (C) 2015 Canonical Ltd.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License version 3 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with this program. If not, see .
*
* Authored by: Michi Henning
*/
#include
#include
#include
#include
#include