pax_global_header00006660000000000000000000000064124716346420014523gustar00rootroot0000000000000052 comment=eff928fdf6da8ed256d45e5b2c7d2ea8261d1042 python-darts.lib.utils.lru-0.5~git20140220/000077500000000000000000000000001247163464200202455ustar00rootroot00000000000000python-darts.lib.utils.lru-0.5~git20140220/.gitignore000066400000000000000000000003501247163464200222330ustar00rootroot00000000000000*.py[co] # Packages *.egg *.egg-info dist build eggs parts bin var sdist develop-eggs .installed.cfg # Installer logs pip-log.txt # Unit test / coverage reports .coverage .tox #Translations *.mo #Mr Developer .mr.developer.cfg python-darts.lib.utils.lru-0.5~git20140220/README.txt000066400000000000000000000235661247163464200217570ustar00rootroot00000000000000 LRU Dictionaries ================= >>> from darts.lib.utils.lru import LRUDict An `LRUDict` is basically a simple dictionary, which has a defined maximum capacity, that may be supplied at construction time, or modified at run-time via the `capacity` property:: >>> cache = LRUDict(1) >>> cache.capacity 1 The minimum capacity value is 1, and LRU dicts will complain, if someone attempts to use a value smaller than that:: >>> cache.capacity = -1 #doctest: +ELLIPSIS Traceback (most recent call last): ... ValueError: -1 is not a valid capacity >>> LRUDict(-1) #doctest: +ELLIPSIS Traceback (most recent call last): ... ValueError: -1 is not a valid capacity LRU dictionaries can never contain more elements than their capacity value indicates, so:: >>> cache[1] = "First" >>> cache[2] = "Second" >>> len(cache) 1 In order to ensure this behaviour, the dictionary will evict entries if it needs to make room for new ones. So:: >>> 1 in cache False >>> 2 in cache True The capacity can be adjusted at run-time. Growing the capacity does not affect the number of elements present in an LRU dictionary:: >>> cache.capacity = 3 >>> len(cache) 1 >>> cache[1] = "First" >>> cache[3] = "Third" >>> len(cache) 3 but shrinking does:: >>> cache.capacity = 2 >>> len(cache) 2 >>> sorted(list(cache.iterkeys())) [1, 3] Note, that the entry with key `2` was evicted, because it was the oldest entry at the time of the modification of `capacity`. The new oldest entry is the one with key `1`, which can be seen, when we try to add another entry to the dict:: >>> cache[4] = "Fourth" >>> sorted(list(cache.iterkeys())) [3, 4] The following operations affect an entry's priority:: - `get` - `__getitem__` - `__setitem__` - `__contains__` Calling any of these operations on an existing key will boost the key's priority, making it more unlikely to get evicted, when the dictionary needs to make room for new entries. There is a special `peek` operation, which returns the current value associated to a key without boosting the priority of the entry:: >>> cache.peek(3) 'Third' >>> cache[5] = "Fifth" >>> sorted(list(cache.iterkeys())) [4, 5] As you can see, even though we accessed the entry with key `3` as the last one, the entry is now gone, because it did not get a priority boost from the call to `peek`. The class `LRUDict` supports a subset of the standard Python `dict` interface. In particular, we can iterate over the key, values, and items of an LRU dict:: >>> sorted([k for k in cache.iterkeys()]) [4, 5] >>> sorted([v for v in cache.itervalues()]) ['Fifth', 'Fourth'] >>> sorted([p for p in cache.iteritems()]) [(4, 'Fourth'), (5, 'Fifth')] >>> sorted(list(cache)) [4, 5] Note, that there is no guaranteed order; in particular, the elements are not generated in priority order or somesuch. Similar to regular `dict`s, an LRU dict's `__iter__` is actually any alias for `iterkeys`. Furthermore, we can remove all elements from the dict: >>> cache.clear() >>> sorted(list(cache.iterkeys())) [] Thread-safety -------------- Instances of class `LRUDict` are not thread safe. Worse: even concurrent read-only access is not thread-safe and has to be synchronized by the client application. There is, however, the class `SynchronizedLRUDict`, which exposes the same interface as plain `LRUDict`, but fully thread-safe. The following session contains exactly the steps, we already tried with a plain `LRUDict`, but now using the synchronized version:: >>> from darts.lib.utils.lru import SynchronizedLRUDict >>> cache = SynchronizedLRUDict(1) >>> cache.capacity 1 >>> cache.capacity = -1 #doctest: +ELLIPSIS Traceback (most recent call last): ... ValueError: -1 is not a valid capacity >>> LRUDict(-1) #doctest: +ELLIPSIS Traceback (most recent call last): ... ValueError: -1 is not a valid capacity >>> cache[1] = "First" >>> cache[2] = "Second" >>> len(cache) 1 >>> 1 in cache False >>> 2 in cache True >>> cache.capacity = 3 >>> len(cache) 1 >>> cache[1] = "First" >>> cache[3] = "Third" >>> len(cache) 3 >>> cache.capacity = 2 >>> len(cache) 2 >>> sorted(list(cache.iterkeys())) [1, 3] >>> cache[4] = "Fourth" >>> sorted(list(cache.iterkeys())) [3, 4] >>> cache.peek(3) 'Third' >>> cache[5] = "Fifth" >>> sorted(list(cache.iterkeys())) [4, 5] >>> sorted([k for k in cache.iterkeys()]) [4, 5] >>> sorted([v for v in cache.itervalues()]) ['Fifth', 'Fourth'] >>> sorted([p for p in cache.iteritems()]) [(4, 'Fourth'), (5, 'Fifth')] >>> sorted(list(cache)) [4, 5] >>> cache.clear() >>> sorted(list(cache.iterkeys())) [] Auto-loading Caches ==================== Having some kind of dictionary which is capable of cleaning itself up is nice, but in order to implement caching, there is still something missing: the mechanism, which actually loads something into our dict. This part of the story is implemented by the `AutoLRUCache`:: >>> from darts.lib.utils.lru import AutoLRUCache Let's first define a load function:: >>> def load_resource(key): ... if key < 10: ... print "Loading %r" % (key,) ... return "R(%s)" % (key,) and a cache:: >>> cache = AutoLRUCache(load_resource, capacity=3) >>> cache.load(1) Loading 1 'R(1)' >>> cache.load(1) 'R(1)' As you can see, the first time, an actual element is loaded, the load function provided to the constructor is called, in order to provide the actual resource value. On subsequent calls to `load`, the cached value is returned. Internally, the `AutoLRUCache` class uses an `LRUDict` to cache values, so:: >>> cache.load(2) Loading 2 'R(2)' >>> cache.load(3) Loading 3 'R(3)' >>> cache.load(4) Loading 4 'R(4)' >>> cache.load(1) Loading 1 'R(1)' Note the "Loading 1" line in the last example. The cache has been initialized with a capacity of 3, so the value of key `1` had to be evicted when the one for key `4` was loaded. When we tried to obtain `1` again, the cache had to properly reload it, calling the loader function. If there is actually no resource for a given key value, the loader function must return `None`. It follows, that `None` is never a valid resource value to be associated with some key in an `AutoLRUCache`. >>> cache.load(11, 'Oops') 'Oops' Thread-safety -------------- Instances of class `AutoLRUCache` are fully thread safe. Be warned, though, that the loader function is called outside of any synchronization scope the class may internally use, and has to provide its own synchronization if required. The cache class actually tries to minimize the number of invocations of the loader by making sure, that no two concurrent threads will try to load the same key value (though any number of concurrent threads might be busy loading the resources associated with different keys). Caching and stale entries ========================== There is another `AutoLRUCache`-like class provided by the LRU module, which gives more control over timing out of entries than `AutoLRUCache` does. >>> from darts.lib.utils.lru import DecayingLRUCache >>> current_time = 0 >>> def tick(): ... global current_time ... current_time += 1 Here, we defined a simple "clock". We could have used the system clock, but roling our own here gives us more control over the notion of "time". Now, let's define a simple cache entry: >>> from collections import namedtuple >>> Entry = namedtuple("Entry", "timestamp payload") >>> def make_entry(payload): ... return Entry(current_time, payload) and a loader function >>> def load(full_key): ... print "Loading", full_key ... return make_entry(u"Entry for %r" % (full_key,)) For the following parts, we consider an entry to be "too old", if it has been created more then two "ticks" ago: >>> def is_still_current(entry): ... return current_time - entry.timestamp <= 2 Finally, we create another cache thingy >>> cache = DecayingLRUCache(load, tester=is_still_current, capacity=3) The `DecayingLRUCache` shows much of the same behaviour of the `AutoLRUCache`, namely: >>> cache.load(1) Loading 1 Entry(timestamp=0, payload=u'Entry for 1') >>> cache.load(2) Loading 2 Entry(timestamp=0, payload=u'Entry for 2') >>> cache.load(3) Loading 3 Entry(timestamp=0, payload=u'Entry for 3') >>> cache.load(4) Loading 4 Entry(timestamp=0, payload=u'Entry for 4') >>> cache.load(1) Loading 1 Entry(timestamp=0, payload=u'Entry for 1') The entry with key `1` had to be reloaded, since the cache has a capacity of 3, and the old entry for `1` was evicted when the entry for `4` was loaded and we needed to make room. >>> cache.load(3) Entry(timestamp=0, payload=u'Entry for 3') Now, let's advance time >>> tick() >>> cache.load(3) Entry(timestamp=0, payload=u'Entry for 3') The entry is still available. >>> tick() >>> cache.load(3) Entry(timestamp=0, payload=u'Entry for 3') >>> tick() >>> cache.load(3) Loading 3 Entry(timestamp=3, payload=u'Entry for 3') Note, that eviction is still based on LRU, not on the age test. Change Log ========== Version 0.5 ------------ Added a "from __future__ import with_statement" for Python 2.5 compatibility. Note, that supporting py2.5 is not a real goal, and I did not test the code using that version. Version 0.4 ------------ Added class `DecayingLRUCache` Version 0.3 ------------ Added class `SynchronizedLRUDict` as thread-safe counterpart for `LRUDict`. python-darts.lib.utils.lru-0.5~git20140220/darts/000077500000000000000000000000001247163464200213625ustar00rootroot00000000000000python-darts.lib.utils.lru-0.5~git20140220/darts/__init__.py000066400000000000000000000000711247163464200234710ustar00rootroot00000000000000__import__('pkg_resources').declare_namespace(__name__) python-darts.lib.utils.lru-0.5~git20140220/darts/lib/000077500000000000000000000000001247163464200221305ustar00rootroot00000000000000python-darts.lib.utils.lru-0.5~git20140220/darts/lib/__init__.py000066400000000000000000000000711247163464200242370ustar00rootroot00000000000000__import__('pkg_resources').declare_namespace(__name__) python-darts.lib.utils.lru-0.5~git20140220/darts/lib/utils/000077500000000000000000000000001247163464200232705ustar00rootroot00000000000000python-darts.lib.utils.lru-0.5~git20140220/darts/lib/utils/__init__.py000066400000000000000000000000711247163464200253770ustar00rootroot00000000000000__import__('pkg_resources').declare_namespace(__name__) python-darts.lib.utils.lru-0.5~git20140220/darts/lib/utils/lru.py000066400000000000000000001137711247163464200244560ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Deterministic Arts Utilities # Copyright (c) 2011 Dirk Esser # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN # THE SOFTWARE. """Trivial LRU-Dictionary implementation """ from __future__ import with_statement import sys from threading import RLock, Lock, Condition, Thread __all__ = ( 'LRUDict', 'AutoLRUCache', 'CacheLoadError', 'CacheAbandonedError', 'CachingError', 'SynchronizedLRUDict', 'DecayingLRUCache', ) def log(): from logging import getLogger return getLogger(__name__) log = log() class CachingError(Exception): """Base class for caching-related errors """ class CacheLoadError(CachingError): """Exception raised, when loading a key fails This exception is raised by an `AutoLRUCache` instance, if loading the value associated with some key fails. The `key` property of the exception instance provides the key being looked up and the `exc_info` property contains a tuple, which provides information about the actual exception caught while loading. """ def __init__(self, key=None, exc_info=None): super(CacheLoadError, self).__init__() self.key = key self.exc_info = exc_info class CacheAbandonedError(CachingError): """A loading operation has been discarded This exception is raised to indicate, that the result of a load operation had to be discarded, because the cache has been reset. """ def __init__(self, key=None, value=None, exc_info=None): super(CacheAbandonedError, self).__init__() self.key = key self.value = value self.exc_info = exc_info class LRUItem(object): """Element of an LRUDict Instances of this class are used internally to hold the entries of `LRUDict` instances. The entries form a doubly-linked list, and are also interned in a dictionary for fast look-up. """ __slots__ = ( '_previous', '_next', '_key', '_value', ) def __init__(self, key, value): super(LRUItem, self).__init__() self._key = key self._value = value self._previous = None self._next = None def __str__(self): return "" % (self._key, self._value) def __repr__(self): return "LRUItem(%r, %r)" % (self._key, self._value) # Internal tag value used to mark optional arguments, for # which no value has been supplied by the caller. missing = object() class LRUDict(object): """Dictionary with LRU behaviour This class implements a dictionary, whose size is limited to a pre-defined capacity. Adding new elements to a full LRU dictionary will cause \"old\" elements to be evicted from the dictionary in order to make room for new elements. Instances of this class are not thread-safe; if an instance is to be shared across multiple concurrently running threads, the client application is responsible for providing proper synchronization. In particular, be aware, that even read-only operations on instances of this class might internally modify data structures, and thus, concurrent read-only access has to be synchronized as well. """ __slots__ = ( '__weakref__', '_LRUDict__index', '_LRUDict__first', '_LRUDict__last', '_LRUDict__capacity', ) def __init__(self, capacity=1024): super(LRUDict, self).__init__() self.__index = dict() self.__first = None self.__last = None self.__capacity = self.__check_capacity(capacity) def __set_capacity(self, value): value = self.__check_capacity(value) if value > self.__capacity: self.__capacity = value else: if value < self.__capacity: self.__capacity = value self.__ensure_room(0) def capacity(): def getter(self): return self.__capacity def setter(self, value): self.__set_capacity(value) return property(getter, setter, doc="""Capacity This property holds an integer number, which defines the capacity of this dictionary (i.e., the maximum number of elements it may hold at any time). Setting this property to a new value may cause elements to be evicted from the dictionary.""") capacity = capacity() def __check_capacity(self, value): if value < 1: raise ValueError("%r is not a valid capacity" % (value,)) return value def clear(self): """Removes all entries """ first = self.__first self.__index.clear() self.__first = self.__last = None # XXX: This code should be reconsidered. We clean up the # doubly-linked list primarily for the sake of CPython and # its reference-counting based memory management. However, # maybe we should just forget about this and rely on the # GC to clean up the cycles here. There should not be any # __del__ methods involved here anyway (at least not in the # objects we are responsible for, i.e., the LRUItems) while first is not None: next = first._next first._key = first._value = None first._previous = first._next = None first = next def __len__(self): """Answers the current number of elements in this dictionary """ return len(self.__index) def __contains__(self, key): """Tests, whether a key is contained This method returns true, if there is an entry with the given `key` in this dictionary. If so, the entry's priority will be boosted by the call, making it more unlikely, that the entry will be evicted when the dictionary needs to make room for new entries. """ item = self.__index.get(key) if item is None: return False else: self.__make_first(item) return True def __iter__(self): """Iterator for all keys of this dictionary See `iterkeys`. """ return self.__index.iterkeys() def iterkeys(self): """Iterator for all keys of this dictionary This method returns a generator, which yields all keys currently present in this dictionary. Note, that iterating over the elements of an LRU dictionary does not change the priorities of individual entries. Also note, that the order, in which entries are generated, is undefined. In particular, the order does usually *not* reflect the LRU priority in any way. """ return self.__index.iterkeys() def itervalues(self): """Iterator for all values of this dictionary This method returns a generator, which yields all values currently present in this dictionary. Note, that iterating over the elements of an LRU dictionary does not change the priorities of individual entries. Also note, that the order, in which entries are generated, is undefined. In particular, the order does usually *not* reflect the LRU priority in any way. """ for item in self.__index.itervalues(): yield item._value def iteritems(self): """Iterator for all entries of this dictionary This method returns a generator, which yields all keys and values currently present in this dictionary as tuples of `key, value`. Note, that iterating over the elements of an LRU dictionary does not change the priorities of individual entries. Also note, that the order, in which entries are generated, is undefined. In particular, the order does usually *not* reflect the LRU priority in any way. """ for key, item in self.__index.iteritems(): yield key, item._value def __delitem__(self, key): """Removes an entry from this dictionary This method removes the entry identified by the given `key` from this dictionary. If no matching entry exists, this method raises a `KeyError` exception. This method does not affect the priorities of entries remaining in the dictionary. """ item = self.__index.pop(key) self.__unlink(item) def pop(self, key, default=missing): """Removes an entry from this dictionary This method removes the entry identified by the given `key` from this dictionary, returning its associated value. If no matching entry exists, this method returns the value supplied as `default` or raises a `KeyError` exception, if no default value was supplied. This method does not affect the priorities of entries remaining in the dictionary. """ item = self.__index.pop(key, None) if item is None: if default is not missing: return default else: raise KeyError(key) else: self.__unlink(item) return item._value def get(self, key, default=None): """Obtains an entry's value This method returns the value associated with the given `key` in this dictionary. If no matching entry exists, the method returns the value of `default` instead. If a matching entry is found, this method will boost that entry's priority, making it more unlikely, that the entry will be evicted from the dictionary the next time, the dict needs to make room for a new entry. """ item = self.__index.get(key) if item is None: return default else: self.__make_first(item) return item._value def peek(self, key, default=None): """Obtains an entry's value This method returns the value associated with the given `key` in this dictionary. If no matching entry exists, the method returns the value of `default` instead. This method differs from `get` in that it does not alter the priority of a matching entry, i.e., the likelyhood of the entry being evicted the next time, the dict needs to make room, is not affected by calls to this method. """ item = self.__index.get(key) if item is None: return default else: return item._value def __getitem__(self, key): """Obtains an entry's value This method returns the value associated with the given `key` in this dictionary. If no matching entry exists, the method raises a `KeyError` instead. If a matching entry is found, this method will boost that entry's priority, making it more unlikely, that the entry will be evicted from the dictionary the next time, the dict needs to make room for a new entry. """ item = self.__index.get(key) if item is None: raise KeyError(key) else: self.__make_first(item) return item._value def __setitem__(self, key, value): """Adds or modifies an entry This method sets the value associated with `key` in this dictionary to `value`. If there is no present entry for `key` in this dictionary, this method may need to evict entries from the dictionary in order to make room for the new entry. This method boosts the priority of the entry associated with `key` making it more unlikely, that the entry will be evicted from the dictionary the next time, the dict needs to make room for a new entry. """ present = self.__index.get(key) if present is not None: present._value = value self.__make_first(present) else: self.__ensure_room(1) item = LRUItem(key, value) item._previous = None item._next = self.__first if self.__first is None: self.__last = item else: self.__first._previous = item self.__index[key] = item self.__first = item def __ensure_room(self, size): """(internal) This method makes sure, that there is room for `size` new elements in this dictionary, potentially evicting old elements. """ index = self.__index capacity = self.__capacity if size > capacity: raise ValueError(size) else: while len(index) + size > capacity: last = self.__last del index[last._key] self.__unlink(last) return self def __unlink(self, item): """(internal) Unlink `item` from the doubly-linked list of LRU items maintained by this dictionary. Makes sure, that the dictionary's `__first` and `__last` pointers are properly updated, if the item happens to be the first/last in the list. """ p, n = item._previous, item._next if p is None: self.__first = n else: p._next = n if n is None: self.__last = p else: n._previous = p item._previous = None item._next = None return item def __make_first(self, item): """(internal) Boosts `item`'s priority by making it the first item of the doubly linked list of all items. Since the eviction process removes elements from the end of the list, the items closer to the head are more unlikely to be removed if we need to make room for new elements. """ p, n = item._previous, item._next if p is None: assert item is self.__first, "no previous entry in %r, but first is %r" % (item, self.__first,) return item else: p._next = n if n is None: self.__last = p else: n._previous = p item._previous = None item._next = self.__first self.__first._previous = item self.__first = item return item class SynchronizedLRUDict(object): """Thread-safe LRU Dictionary This class acts as a thread-safe wrapper around a regular LRU dictionary. The primary interface is that of a standard `LRUDict`, but all methods are internally synchronized. Note, that there is nothing fancy going on this class; all methods simply ensure, that the lock is held around calls to the underlying plain `LRUDict` instance, which does the actual work. """ __slots__ = ( '__weakref__', '_SynchronizedLRUDict__dict', '_SynchronizedLRUDict__lock', ) def __init__(self, capacity=1024, lock=None): """Initialize a new instance This method initializes a new instance, providing an initial capacity of `capacity` elements. If `lock` is provided, it must be a `threading.Lock` or `threading.RLock` instance. The lock will be used to synchronize all access to the underyling `LRUDict`. If no lock is provided, the method will create a new one. """ super(SynchronizedLRUDict, self).__init__() self.__dict = LRUDict(capacity) self.__lock = RLock() if lock is None else lock def capacity(): def getter(self): with self.__lock: return self.__dict.capacity def setter(self, value): with self.__lock: self.__dict.capacity = value return property(getter, setter, doc="""Capacity This property holds an integer number, which defines the capacity of this dictionary (i.e., the maximum number of elements it may hold at any time). Setting this property to a new value may cause elements to be evicted from the dictionary.""") capacity = capacity() def clear(self): """Removes all entries """ with self.__lock: self.__dict.clear() def get(self, key, default=None): """Obtains an entry's value This method returns the value associated with the given `key` in this dictionary. If no matching entry exists, the method returns the value of `default` instead. If a matching entry is found, this method will boost that entry's priority, making it more unlikely, that the entry will be evicted from the dictionary the next time, the dict needs to make room for a new entry. """ with self.__lock: return self.__dict.get(key, None) def peek(self, key, default=None): """Obtains an entry's value This method returns the value associated with the given `key` in this dictionary. If no matching entry exists, the method returns the value of `default` instead. This method differs from `get` in that it does not alter the priority of a matching entry, i.e., the likelyhood of the entry being evicted the next time, the dict needs to make room, is not affected by calls to this method. """ with self.__lock: return self.__dict.peek(key, None) def pop(self, key, default=missing): """Removes an entry from this dictionary This method removes the entry identified by the given `key` from this dictionary, returning its associated value. If no matching entry exists, this method returns the value supplied as `default` or raises a `KeyError` exception, if no default value was supplied. This method does not affect the priorities of entries remaining in the dictionary. """ with self.__lock: return self.__dict.pop(key, default) def __len__(self): """Answers the current number of elements in this dictionary """ with self.__lock: return self.__dict.__len__() def __contains__(self, key): """Tests, whether a key is contained This method returns true, if there is an entry with the given `key` in this dictionary. If so, the entry's priority will be boosted by the call, making it more unlikely, that the entry will be evicted when the dictionary needs to make room for new entries. """ with self.__lock: return self.__dict.__contains__(key) def __iter__(self): """Iterator for all keys of this dictionary See `iterkeys`. """ return self.iterkeys() def iterkeys(self): """Iterator for all keys of this dictionary This method returns a generator, which yields all keys currently present in this dictionary. Note, that iterating over the elements of an LRU dictionary does not change the priorities of individual entries. Also note, that the order, in which entries are generated, is undefined. In particular, the order does usually *not* reflect the LRU priority in any way. Note, that the resulting generator actually traverses a snapshot copy of the key set taken, when the `iterkeys` method was called. """ with self.__lock: return iter(tuple(self.__dict.iterkeys())) def itervalues(self): """Iterator for all values of this dictionary This method returns a generator, which yields all values currently present in this dictionary. Note, that iterating over the elements of an LRU dictionary does not change the priorities of individual entries. Also note, that the order, in which entries are generated, is undefined. In particular, the order does usually *not* reflect the LRU priority in any way. Note, that the resulting generator actually traverses a snapshot copy of the value collection taken, when the `itervalues` method was called. """ with self.__lock: return iter(tuple(self.__dict.itervalues())) def iteritems(self): """Iterator for all entries of this dictionary This method returns a generator, which yields all keys and values currently present in this dictionary as tuples of `key, value`. Note, that iterating over the elements of an LRU dictionary does not change the priorities of individual entries. Also note, that the order, in which entries are generated, is undefined. In particular, the order does usually *not* reflect the LRU priority in any way. Note, that the resulting generator actually traverses a snapshot copy of the items collection taken, when the `iteritems` method was called. """ with self.__lock: return iter(tuple(self.__dict.iteritems())) def __getitem__(self, key): """Obtains an entry's value This method returns the value associated with the given `key` in this dictionary. If no matching entry exists, the method raises a `KeyError` instead. If a matching entry is found, this method will boost that entry's priority, making it more unlikely, that the entry will be evicted from the dictionary the next time, the dict needs to make room for a new entry. """ with self.__lock: return self.__dict.__getitem__(key) def __setitem__(self, key, value): """Adds or modifies an entry This method sets the value associated with `key` in this dictionary to `value`. If there is no present entry for `key` in this dictionary, this method may need to evict entries from the dictionary in order to make room for the new entry. This method boosts the priority of the entry associated with `key` making it more unlikely, that the entry will be evicted from the dictionary the next time, the dict needs to make room for a new entry. """ with self.__lock: self.__dict.__setitem__(key, value) def __delitem__(self, key): """Removes an entry from this dictionary This method removes the entry identified by the given `key` from this dictionary. If no matching entry exists, this method raises a `KeyError` exception. This method does not affect the priorities of entries remaining in the dictionary. """ with self.__lock: self.__dict.__delitem__(key) loading = object() available = object() failed = object() discarded = object() class Placeholder(object): """Placeholder for a value-yet-to-come The `condition` property holds a condition variable, which is synchronized using the creating cache's lock. In order to wait for a load to finish, wait on this condition variable. The state property indicates, what's the current state of the load operation, which this object represent: - `loading` the operation is still in progress. Wait on the placeholder's `condition` for updates - `available` the operation has successfully been performed and the value is available in the `value` property - `failed` the loader raised an exception. The `value` property contains the `exc_info` tuple of the exception. - `discarded` the cache has been reset with the `discard_loads` option. When seen, a `CacheAbandonedError` should be raised """ def __init__(self, lock): super(Placeholder, self).__init__() self._condition = Condition(lock) self._state = loading self._value = None class AutoLRUCache(object): """Auto-loading LRU cache Instances of this class provide simple caches, which support automatic loading of resources, when cache look-up fails. Loading of resources is accomplished by providing a `loader` function to the cache's constructor. The function must have the signature lambda key: ... It may return `None` to indicate, that the resource has not been found. Any other value is taken to be the resource value associated with the given `key`. Note, that `None` is never a valid resource value and will always be taken to mean, that no value is available. If the loader raises an exception, the cache will wrap the exception in a `CacheLoadError` and propagate the information by raising that one. This class carefully tries to ensure, that only a single thread attempts to actually load a given key value (though multiple threads may be busy concurrently loading resources for different key values at the same time). If the value for a key `K` is currently being loaded by a thread `A`, and another thread `B` requests the value of `K` from the same cache, then `B` will block until `A` has loaded the value (or bailed out due to an exception). Thread-safety. Instances of this class are fully thread safe, using an internal locking discipline. Note, though, that the load function is called unsynchronized, and needs to provide its own locking discipline. """ __slots__ = ( '__weakref__', '_AutoLRUCache__lock', '_AutoLRUCache__cache', '_AutoLRUCache__loader', '_AutoLRUCache__loading', ) def __init__(self, loader, capacity=1024): super(AutoLRUCache, self).__init__() self.__lock = Lock() self.__cache = LRUDict(capacity) self.__loader = loader self.__loading = dict() def clear(self, discard_loads=False): """Remove all cached values This method removes all values from this cache. If `discard_loads`, then the method also forces all currently running load operations to fail with a `CacheAbandonedError`. Note, that this method has no way of interrupting load operations, so all pending operations will have to complete before the discard condition can be detected. """ with self.__lock: self.__cache.clear() if discard_loads: conditions = list() keys = tuple(self.__loading.iterkeys()) for k in keys: placeholder = self.__loading.pop(k) if placeholder._state is loading: placeholder._state = discarded conditions.append(placeholder._condition) while conditions: conditions.pop().notifyAll() def load(self, key, default=None): """Load a value This method returns the value associated with `key`, possibly loading it, if it is not already available in the cache. If no matching value can be found, the method returns the value supplied as `default`. """ # TODO: I'd really like to have a timeout parameter # on this method. This would be trivial to implement # for threads, which can piggy back on load operations # performed by other threads (since `Condition.wait` # already supports a timeout parameter). However, it # will be hard to do, when the current thread is the # actual loading one. The only way I can think of right # now is to always move the load operations to a fresh # thread, and make the triggering thread block like a # piggy-backing one. with self.__lock: item = self.__cache.get(key, missing) if item is not missing: if item is None: return default else: return item else: placeholder = self.__loading.get(key) if placeholder is not None: while placeholder._state is loading: placeholder._condition.wait() if placeholder._state is failed: raise CacheLoadError(key, placeholder._value) else: if placeholder._state is available: value = placeholder._value if value is None: return default else: return value else: assert placeholder._state is discarded raise CacheAbandonedError(key=key) assert False, "this line should never be reached" else: placeholder = Placeholder(self.__lock) self.__loading[key] = placeholder # The previous line was the point of no return. # Reaching this point means, that we are the thread # which has to do the actual loading. It also means, # that we must never leave this method without properly # cleaning up behind us. try: value = self.__loader(key) except: with self.__lock: if placeholder._state is loading: # We are still responsible for the placeholder. del self.__loading[key] placeholder._value = sys.exc_info() placeholder._state = failed placeholder._condition.notifyAll() raise CacheLoadError(key, placeholder._value) else: # Do not notify the condition variable, since # that should already have been done by whoever # changed the placeholder's state raise CacheAbandonedError(key=key, exc_info=sys.exc_info()) else: with self.__lock: if placeholder._state is loading: # We are still responsible for the placeholder. del self.__loading[key] placeholder._value = value placeholder._state = available self.__cache[key] = value placeholder._condition.notifyAll() else: # Do not notify the condition variable, since # that should already have been done by whoever # changed the placeholder's state raise CacheAbandonedError(key=key, value=value) if value is None: return default else: return value def identity(value): return value def good(value): return True class DecayingLRUCache(object): """Auto-LRU cache with support for stale entry detection This class implements a variant of `AutoLRUCache`, which also supports the detection of entries, that are too old and need to be reloaded even if they are are present. The client application needs to provide the following parameters: - `loader` A callable, which is applied to a key value in order to load the associated object. This must be a function with the signature `lambda key: ...` - `tester` Is applied to a cached element before that is handed out to the caller; if this function returns false, the cached element is considered "too old" and dropped from the cache. - `key` A callable, which is applied to a key value in order to produce a properly hashable value from it. The default key extraction function is `identity`, i.e., we use the supplied key value unchanged. - `capacity` Maximum number of elements in kept in the LRU cache. The cache starts evicting elements, which have not been recently accessed, if the number of preserved elements reaches this limit. Note, that eviction is *not* in any way controlled by the `tester` function, but by access order only! Note, that unlike `AutoLRUCache`, this class does not consider a `None` result from the `loader` function to be special in any way. It is up to the `tester` function to decide, whether the entry is "good" (i.e., should be reported back to the application) or not (i.e., must be reloaded upon access). """ __slots__ = ( '__weakref__', '_DecayingLRUCache__lock', '_DecayingLRUCache__cache', '_DecayingLRUCache__loader', '_DecayingLRUCache__loading', '_DecayingLRUCache__tester', '_DecayingLRUCache__key', ) def __init__(self, loader, tester=good, key=identity, capacity=1024): """Initialize a new instance """ super(DecayingLRUCache, self).__init__() self.__lock = Lock() self.__cache = LRUDict(capacity) self.__loader = loader self.__loading = dict() self.__tester = tester self.__key = key def clear(self, discard_loads=False): """Remove all cached values This method removes all values from this cache. If `discard_loads`, then the method also forces all currently running load operations to fail with a `CacheAbandonedError`. Note, that this method has no way of interrupting load operations, so all pending operations will have to complete before the discard condition can be detected. """ with self.__lock: self.__cache.clear() if discard_loads: conditions = list() keys = tuple(self.__loading.iterkeys()) for k in keys: placeholder = self.__loading.pop(k) if placeholder._state is loading: placeholder._state = discarded conditions.append(placeholder._condition) while conditions: conditions.pop().notifyAll() def load(self, key): """Load a value Returns the value associated with `key`. If no matching value is currently present in this cache, or if the value present is considered "too old" by the `tester` function, then a value is loaded via the cache's `loader` function. """ loader, tester, keyfn = self.__loader, self.__tester, self.__key kref = keyfn(key) with self.__lock: item = self.__cache.get(kref, missing) if item is not missing: if tester(item): return item else: del self.__cache[kref] placeholder = self.__loading.get(kref) if placeholder is not None: while placeholder._state is loading: placeholder._condition.wait() if placeholder._state is failed: raise CacheLoadError(key, placeholder._value) else: if placeholder._state is available: return placeholder._value else: assert placeholder._state is discarded raise CacheAbandonedError(key=key) assert False, "this line should never be reached" else: placeholder = Placeholder(self.__lock) self.__loading[kref] = placeholder # The previous line was the point of no return. # Reaching this point means, that we are the thread # which has to do the actual loading. It also means, # that we must never leave this method without properly # cleaning up behind us. try: value = loader(key) except: with self.__lock: if placeholder._state is loading: # We are still responsible for the placeholder. del self.__loading[kref] placeholder._value = sys.exc_info() placeholder._state = failed placeholder._condition.notifyAll() raise CacheLoadError(key, placeholder._value) else: # Do not notify the condition variable, since # that should already have been done by whoever # changed the placeholder's state raise CacheAbandonedError(key=key, exc_info=sys.exc_info()) else: with self.__lock: if placeholder._state is loading: # We are still responsible for the placeholder. del self.__loading[kref] placeholder._value = value placeholder._state = available self.__cache[kref] = value placeholder._condition.notifyAll() else: # Do not notify the condition variable, since # that should already have been done by whoever # changed the placeholder's state raise CacheAbandonedError(key=key, value=value) return value python-darts.lib.utils.lru-0.5~git20140220/doc/000077500000000000000000000000001247163464200210125ustar00rootroot00000000000000python-darts.lib.utils.lru-0.5~git20140220/doc/Makefile000066400000000000000000000110541247163464200224530ustar00rootroot00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = /home/dirk/Python/default/bin/sphinx-build PAPER = BUILDDIR = _build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " text to make text files" @echo " man to make manual pages" @echo " changes to make an overview of all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: -rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/dartutilsevent.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/dartutilsevent.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/dartutilsevent" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/dartutilsevent" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." make -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." python-darts.lib.utils.lru-0.5~git20140220/doc/conf.py000066400000000000000000000156031247163464200223160ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # dart.utils.event documentation build configuration file, created by # sphinx-quickstart on Tue Dec 14 19:04:07 2010. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [] # Add any paths that contain templates here, relative to this directory. templates_path = [] # '_templates'] # The suffix of source filenames. source_suffix = '.txt' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'dart.utils.lru' copyright = u'2010, Deterministic Arts' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = '0.2' # The full version, including alpha/beta/rc tags. release = '0.2' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['_build'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = [] #'_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'dartutilslrudoc' # -- Options for LaTeX output -------------------------------------------------- # The paper size ('letter' or 'a4'). #latex_paper_size = 'letter' # The font size ('10pt', '11pt' or '12pt'). #latex_font_size = '10pt' # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'dartutilslru.tex', u'dart.utils.lru Documentation', u'Deterministic Arts', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Additional stuff for the LaTeX preamble. #latex_preamble = '' # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'dartutilslru', u'dart.utils.lru Documentation', [u'Deterministic Arts'], 1) ] python-darts.lib.utils.lru-0.5~git20140220/doc/index.txt000066400000000000000000000272501247163464200226700ustar00rootroot00000000000000 Caching and Least-Recently-Used Eviction ========================================= .. py:module:: darts.lib.utils.lru Reference --------- .. py:class:: LRUDict ([capacity=1024]) Instances of this class are basically dictionaries, which limit the number of entries they can hold. Each dictionary has a `capacity` (initialized by the capacity argument to the constructor), which defines this maximum number. The value can be changed at run-time by modifying the :py:attr:`capacity` property of an instance. If the dictionary has to remove entries in order to make room for new ones, it will remove those, which haven't been accessed for a "long time". Thread-safety: Instances of this class **are not thread-safe**, even if only the read accessors are used concurrently. An application intending to share an instance of this class among multiple concurrently executing threads needs to synchronize all access to the instance itself. The :class:`SynchronizedLRUDict` provides a stand-alone thread-safe version of this class. Iteration: The ordering of entries under iteration is undefined. In particular, it usually does not reflect the eviction priority order. .. py:data:: capacity Contains the current capacity of the dictionary. At any time, the dictionary holds as most as many entries as this value indicates. Changing the value of the property to a smaller number may cause entries to be removed from the dictionary. .. py:method:: clear() Removes all entries from this dictionary. Note, that currently, this method has a run-time complexity proportional to the number of entries contained in the dictionary. .. py:method:: peek(key, [, default=None]) Look up the value of `key`. This method returns the value associated with `key` in this dictionary. If no matching entry exists, the method returns `default` instead. Note, that unlike most accessor methods, this one does not affect the priority of a matching entry, i.e., looking up a value using this method does not count as "access", adn won't change the likelyhood of the entry found to be discarded or not, if the dictionary needs to make room for new entries. .. py:method:: get(key [, default=None]) Look up the value of `key`. This method returns the value associated with `key` in this dictionary. If no matching entry exists, the method returns `default` instead. If a matching entry exists, its priority is boosted by this method, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __getitem__(key) Look up the value of `key`. This method returns the value associated with `key` in this dictionary. If no matching entry exists, the method raises a `KeyError` instead. If a matching entry exists, its priority is boosted by this method, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __setitem__(key, value) Associates the value `value` to `key` in this dictionary. If there was already an entry for `key`, its value is replaced with `value`. This method boosts the entry's priority, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __delitem__(key) Deletes an entry from this dictionary. This method deletes the entry associated with `key` from this dictionary. If no matching entry exists, this method raises a `KeyError` instead. Calling this method does not affect the priorities of other entries. .. py:method:: __len__() Answers the current number of elements contained in this dictionary. .. py:method:: __contains__(key) Tests, whether this dictionary has an entry for the given `key`. If so, the entry's priority is boosted, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __iter__() See :py:meth:`iterkeys`. .. py:method:: iterkeys() This method returns a generator, which produces all the keys currently present in the dictionary. The ordering of the keys is undefined. .. py:method:: itervalues() This method returns a generator, which produces all the values currently present in the dictionary. The ordering of the values is undefined. .. py:method:: iteritems() This method returns a generator, which produces tuples of the form `key, value` for all entries in the dictionary. The ordering of the values is undefined. .. py:class:: SynchronizedLRUDict ([capacity=1024 [, lock=None]]) This class provides a thread-safe version of :class:`LRUDict`. Each instance maintains a plain `LRUDict` internally, and delegates all method calls to that object, making sure, that the calls are properly synchronized using a thread lock. If no explicit `lock` argument is provided at instance creation time, the synchronized LRU dict will create its own lock. Note, that this class does not inherit from `LRUDict`. .. py:data:: capacity Contains the current capacity of the dictionary. At any time, the dictionary holds as most as many entries as this value indicates. Changing the value of the property to a smaller number may cause entries to be removed from the dictionary. .. py:method:: clear() Removes all entries from this dictionary. Note, that currently, this method has a run-time complexity proportional to the number of entries contained in the dictionary. .. py:method:: peek(key [, default=None]) Look up the value of `key`. This method returns the value associated with `key` in this dictionary. If no matching entry exists, the method returns `default` instead. Note, that unlike most accessor methods, this one does not affect the priority of a matching entry, i.e., looking up a value using this method does not count as "access", adn won't change the likelyhood of the entry found to be discarded or not, if the dictionary needs to make room for new entries. .. py:method:: get(key [, default=None]) Look up the value of `key`. This method returns the value associated with `key` in this dictionary. If no matching entry exists, the method returns `default` instead. If a matching entry exists, its priority is boosted by this method, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __getitem__(key) Look up the value of `key`. This method returns the value associated with `key` in this dictionary. If no matching entry exists, the method raises a `KeyError` instead. If a matching entry exists, its priority is boosted by this method, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __setitem__(key, value) Associates the value `value` to `key` in this dictionary. If there was already an entry for `key`, its value is replaced with `value`. This method boosts the entry's priority, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __delitem__(key) Deletes an entry from this dictionary. This method deletes the entry associated with `key` from this dictionary. If no matching entry exists, this method raises a `KeyError` instead. Calling this method does not affect the priorities of other entries. .. py:method:: __len__() Answers the current number of elements contained in this dictionary. .. py:method:: __contains__(key) Tests, whether this dictionary has an entry for the given `key`. If so, the entry's priority is boosted, making it less likely, that the entry is removed the next time, the dictionary has to make room for new elements. .. py:method:: __iter__() See :py:meth:`iterkeys`. .. py:method:: iterkeys() This method returns a generator, which produces all the keys currently present in the dictionary. The ordering of the keys is undefined. Note, that the generator returned will traverse over a snapshot copy of the actual key set made, when the `iterkeys` method was called. .. py:method:: itervalues() This method returns a generator, which produces all the values currently present in the dictionary. The ordering of the values is undefined. Note, that the generator returned will traverse over a snapshot copy of the actual value collection made, when the `itervalues` method was called. .. py:method:: iteritems() This method returns a generator, which produces tuples of the form `key, value` for all entries in the dictionary. The ordering of the values is undefined. Note, that the generator returned will traverse over a snapshot copy of the actual items collection made, when the `iteritems` method was called. .. py:class:: AutoLRUCache(loader [, capacity=1024]) Instances of this class act as self-contained resource loaders and caches. Each instance of this class has a maximum number of elements it may contains (the cache's capacity). The `loader` must be a callable with the signature lambda key: ... where `key` will be the identifier passed to :py:meth:`load`. The loader function should return `None`, if there is no resource with the given `key`. Any other return value is taken to be the actual resource. Note, that the cache is likely to cache any value returned by this function, even `None`. The cache is careful to coalesce multiple concurrent load operations with the same key into a single operation. Note, though, that any number of load operations may started concurrently, as long as they have different keys. Thread-safety: Instances of this class are fully thread-safe, managing all required synchronization internally. Note, however, that the custom `loader` function is always called outside of any synchronization scope. In particular, it needs to perform its own synchronization if it needs one. .. py:method:: clear([discard_loads=False]) Removes all entries from this cache. This method removes all entries from this cache. If `discard_loads` is true, then all currently pending load operations will be discarded, and their results will be ignored. .. py:method:: load(key [,default=None]) Load a resource for a given `key`. This method returns the resource, whose key matches `key`, or `default` if there is no matching resource. If the resource is not currently present in the cache, the cache's resource loader function is called to load the resource from whatever is the underlying storage. If an exception is raised by the loader function, it will be re-raised by this method, wrapped in a :py:class:`CacheLoadError` exception. If a call to :py:meth:`clear` is made with `discard_loads` set to true, while the loader function is loading the resource, the method will raise a `CacheAbandonedError` exception as soon as the loader finishes loading. Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` python-darts.lib.utils.lru-0.5~git20140220/setup.py000066400000000000000000000030601247163464200217560ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Deterministic Arts Utilities # Copyright (c) 2011 Dirk Esser # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN # THE SOFTWARE. from setuptools import setup, find_packages setup( name = "darts.util.lru", version = "0.5", description='Simple dictionary with LRU behaviour', zip_safe=True, packages = find_packages(), namespace_packages=['darts', 'darts.lib', 'darts.lib.utils'], author = "Deterministic Arts", author_email = "contact@deterministic-arts.de", license = "MIT" ) python-darts.lib.utils.lru-0.5~git20140220/test.py000066400000000000000000000112231247163464200215750ustar00rootroot00000000000000 from threading import Thread, RLock, Condition from unittest import TestCase, main from darts.lib.utils.lru import AutoLRUCache, CacheLoadError, CacheAbandonedError from time import sleep from random import shuffle class AutoCacheThreadingTest(TestCase): """Tests the AutoLRUCache implementation This class adds tests for the AutoLRUCache class, which are hard to do in doctest (e.g., due to threading) Testing for race conditions and bugs stemming from concurrency is notoriously hard. This test case runs enough threads (and iterations per thread) to make sure, that there is nothing glaringly obvious wrong with the implementation, but it is unlikely to be able to find non-trivial bugs. TODO: - Need tests for the abandoning stuff (clear with parameter discard_loads=True) - Need to figure out a way to force a key to be loaded concurrently, and exercise the error handling when the loading thread encounters an exception from the loader function. All conurrently running threads loading the same key have to abort properly with the error signalled by the loader, wrapped as CacheLoadError exception. """ def test_concurrent_access(self): """Test concurrent access of AutoLRUCaches This test function tests the nominal behaviour of a cache instance, not expecting any exceptions to be raised. This test makes sure, that: - the logic, which keeps track of keys currently being loaded in a cache, works as advertised, and collapses multiple concurrent read requests for the same key into a single `loader` call. - nothing is ever evicted if the cache is large enough to accomodate all objects ever being requested from it. """ iterations_per_thread = 1000 number_of_threads = 100 key_range = range(4) loads_lock = RLock() loads = dict() start_lock = RLock() start_condition = Condition(start_lock) ready_condition = Condition(start_lock) start_now = False ready_list = list() def loader(key): with loads_lock: loads[key] = loads.get(key, 0) + 1 # Fake a complicated computation here. # This should not slow down the test too # much, as it is ever only called 4 times # at all (if the cache implementation has # no bug in the load-coalescing code), so # we should sleep for at most 4 seconds, # and expect to sleep less, since these # sleep calls are likely to happen in parallel # on different threads. sleep(1) return "R(%r)" % (key,) def shuffled(seq): copy = list(seq) shuffle(copy) return copy cache = AutoLRUCache(loader, capacity=len(key_range)) def reader(): with start_lock: while not start_now: start_condition.wait() for k in xrange(iterations_per_thread): for i in shuffled(key_range): answer = cache.load(i) self.assertEqual("R(%r)" % (i,), answer) with start_lock: ready_list.append(1) ready_condition.notifyAll() with start_lock: for k in xrange(number_of_threads): thr = Thread(target=reader) thr.start() # Ok. All threads have been started. Now, we can # send them the start signal start_now = True start_condition.notifyAll() # Must happen in a different `with` block, so that the # mutex gets released, and the threads have a chance to # read the start signal. Now, wait for all threads to # terminate. with start_lock: while len(ready_list) < number_of_threads: ready_condition.wait() # Make sure, that all keys have actually been requested # at least once. self.assertEqual(set(key_range), set(loads.iterkeys())) # The cache has a capacity such, that it can hold all # elements nominally ever requested by the readers. So, # we expect, that every requested key is loaded exactly # once (due to the cache keeping track of what it is # currently loading). for key,count in loads.iteritems(): self.assertEqual(1, count) self.assertTrue(key in key_range) if __name__ == '__main__': main()