flufl.lock-2.2.1/0000775000175000017500000000000011744120360014033 5ustar barrybarry00000000000000flufl.lock-2.2.1/flufl.lock.egg-info/0000775000175000017500000000000011744120360017564 5ustar barrybarry00000000000000flufl.lock-2.2.1/flufl.lock.egg-info/SOURCES.txt0000664000175000017500000000106711744120360021454 0ustar barrybarry00000000000000README.rst distribute_setup.py setup.cfg setup.py setup_helpers.py template.py flufl/__init__.py flufl.lock.egg-info/PKG-INFO flufl.lock.egg-info/SOURCES.txt flufl.lock.egg-info/dependency_links.txt flufl.lock.egg-info/namespace_packages.txt flufl.lock.egg-info/top_level.txt flufl/lock/NEWS.rst flufl/lock/README.rst flufl/lock/__init__.py flufl/lock/_lockfile.py flufl/lock/conf.py flufl/lock/docs/__init__.py flufl/lock/docs/using.rst flufl/lock/tests/__init__.py flufl/lock/tests/subproc.py flufl/lock/tests/test_documentation.py flufl/lock/tests/test_lockfile.pyflufl.lock-2.2.1/flufl.lock.egg-info/top_level.txt0000664000175000017500000000000611744120360022312 0ustar barrybarry00000000000000flufl flufl.lock-2.2.1/flufl.lock.egg-info/dependency_links.txt0000664000175000017500000000000111744120360023632 0ustar barrybarry00000000000000 flufl.lock-2.2.1/flufl.lock.egg-info/namespace_packages.txt0000664000175000017500000000000611744120360024113 0ustar barrybarry00000000000000flufl flufl.lock-2.2.1/flufl.lock.egg-info/PKG-INFO0000664000175000017500000001353111744120360020664 0ustar barrybarry00000000000000Metadata-Version: 1.1 Name: flufl.lock Version: 2.2.1 Summary: NFS-safe file locking with timeouts for POSIX systems Home-page: http://launchpad.net/flufl.lock Author: Barry Warsaw Author-email: barry@python.org License: LGPLv3 Download-URL: https://launchpad.net/flufl.lock/+download Description: ========== flufl.lock ========== NFS-safe file locking with timeouts for POSIX systems The ``flufl.lock`` library provides an NFS-safe file-based locking algorithm influenced by the GNU/Linux `open(2)` manpage, under the description of the `O_EXCL` option:: [...] O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful. The assumption made here is that there will be no *outside interference*, e.g. no agent external to this code will ever ``link()`` to the specific lock files used. Lock objects support lock-breaking so that you can't wedge a process forever. This is especially helpful in a web environment, but may not be appropriate for all applications. Locks have a *lifetime*, which is the maximum length of time the process expects to retain the lock. It is important to pick a good number here because other processes will not break an existing lock until the expected lifetime has expired. Too long and other processes will hang; too short and you'll end up trampling on existing process locks -- and possibly corrupting data. In a distributed (NFS) environment, you also need to make sure that your clocks are properly synchronized. License ======= This file is part of flufl.lock. flufl.lock is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, version 3 of the License. flufl.lock is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with flufl.lock. If not, see . =================== NEWS for flufl.lock =================== 2.2.1 (2012-04-19) ================== * Add classifiers to setup.py and make the long description more compatible with the Cheeseshop. * Other changes to make the Cheeseshop page look nicer. (LP: #680136) * setup_helper.py version 2.1. 2.2 (2012-01-19) ================ * Support Python 3 without the use of 2to3. * Make the documentation clear that the `flufl.test.subproc` functions are not part of the public API. (LP: #838338) * Fix claim file format in `take_possession()`. (LP: #872096) * Provide a new API for dealing with possible additional unexpected errnos while trying to read the lock file. These can happen in some NFS environments. If you want to retry the read, set the lock file's `retry_errnos` property to a sequence of errnos. If one of those errnos occurs, the read is unconditionally (and infinitely) retried. `retry_errnos` is a property which must be set to a sequence; it has a getter and a deleter too. (LP: #882261) 2.1.1 (2011-08-20) ================== * Fixed TypeError in .lock() method due to race condition in _releasetime property. Found by Stephen A. Goss. (LP: #827052) 2.1 (2010-12-22) ================ * Added lock.details. 2.0.2 (2010-12-19) ================== * Small adjustment to doctest. 2.0.1 (2010-11-27) ================== * Add missing exception to __all__. 2.0 (2010-11-26) ================ * Package renamed to flufl.lock. Earlier ======= Try `bzr log lp:flufl.lock` for details. Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+) Classifier: Operating System :: POSIX Classifier: Operating System :: MacOS :: MacOS X Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Software Development :: Libraries Classifier: Topic :: Software Development :: Libraries :: Python Modules flufl.lock-2.2.1/setup.py0000664000175000017500000000415211744114661015556 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock. # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, either version 3 of the License, or (at your option) # any later version. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . import distribute_setup distribute_setup.use_setuptools() from setup_helpers import ( description, get_version, long_description, require_python) from setuptools import setup, find_packages require_python(0x20600f0) __version__ = get_version('flufl/lock/__init__.py') setup( name='flufl.lock', version=__version__, namespace_packages=['flufl'], packages=find_packages(), include_package_data=True, maintainer='Barry Warsaw', maintainer_email='barry@python.org', description=description('README.rst'), long_description=long_description('README.rst', 'flufl/lock/NEWS.rst'), license='LGPLv3', url='http://launchpad.net/flufl.lock', download_url='https://launchpad.net/flufl.lock/+download', test_suite='flufl.lock.tests', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'License :: OSI Approved :: ' 'GNU Lesser General Public License v3 or later (LGPLv3+)', 'Operating System :: POSIX', 'Operating System :: MacOS :: MacOS X', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Topic :: Software Development :: Libraries', 'Topic :: Software Development :: Libraries :: Python Modules', ] ) flufl.lock-2.2.1/setup.cfg0000664000175000017500000000022111744120360015647 0ustar barrybarry00000000000000[build_sphinx] source_dir = flufl/lock [upload_docs] upload_dir = build/sphinx/html [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 flufl.lock-2.2.1/setup_helpers.py0000664000175000017500000001204211744113475017277 0ustar barrybarry00000000000000# Copyright (C) 2009-2012 by Barry A. Warsaw # # This file is part of flufl.lock # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, version 3 of the License. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . """setup.py helper functions.""" from __future__ import absolute_import, print_function, unicode_literals __metaclass__ = type __all__ = [ 'description', 'find_doctests', 'get_version', 'long_description', 'require_python', ] import os import re import sys DEFAULT_VERSION_RE = re.compile(r'(?P\d+\.\d+(?:\.\d+)?)') EMPTYSTRING = '' __version__ = '2.1' def require_python(minimum): """Require at least a minimum Python version. The version number is expressed in terms of `sys.hexversion`. E.g. to require a minimum of Python 2.6, use:: >>> require_python(0x206000f0) :param minimum: Minimum Python version supported. :type minimum: integer """ if sys.hexversion < minimum: hversion = hex(minimum)[2:] if len(hversion) % 2 != 0: hversion = '0' + hversion split = list(hversion) parts = [] while split: parts.append(int(''.join((split.pop(0), split.pop(0))), 16)) major, minor, micro, release = parts if release == 0xf0: print('Python {0}.{1}.{2} or better is required'.format( major, minor, micro)) else: print('Python {0}.{1}.{2} ({3}) or better is required'.format( major, minor, micro, hex(release)[2:])) sys.exit(1) def get_version(filename, pattern=None): """Extract the __version__ from a file without importing it. While you could get the __version__ by importing the module, the very act of importing can cause unintended consequences. For example, Distribute's automatic 2to3 support will break. Instead, this searches the file for a line that starts with __version__, and extract the version number by regular expression matching. By default, two or three dot-separated digits are recognized, but by passing a pattern parameter, you can recognize just about anything. Use the `version` group name to specify the match group. :param filename: The name of the file to search. :type filename: string :param pattern: Optional alternative regular expression pattern to use. :type pattern: string :return: The version that was extracted. :rtype: string """ if pattern is None: cre = DEFAULT_VERSION_RE else: cre = re.compile(pattern) with open(filename) as fp: for line in fp: if line.startswith('__version__'): mo = cre.search(line) assert mo, 'No valid __version__ string found' return mo.group('version') raise AssertionError('No __version__ assignment found') def find_doctests(start='.', extension='.rst'): """Find separate-file doctests in the package. This is useful for Distribute's automatic 2to3 conversion support. The `setup()` keyword argument `convert_2to3_doctests` requires file names, which may be difficult to track automatically as you add new doctests. :param start: Directory to start searching in (default is cwd) :type start: string :param extension: Doctest file extension (default is .txt) :type extension: string :return: The doctest files found. :rtype: list """ doctests = [] for dirpath, dirnames, filenames in os.walk(start): doctests.extend(os.path.join(dirpath, filename) for filename in filenames if filename.endswith(extension)) return doctests def long_description(*filenames): """Provide a long description.""" res = [''] for filename in filenames: with open(filename) as fp: for line in fp: res.append(' ' + line) res.append('') res.append('\n') return EMPTYSTRING.join(res) def description(filename): """Provide a short description.""" # This ends up in the Summary header for PKG-INFO and it should be a # one-liner. It will get rendered on the package page just below the # package version header but above the long_description, which ironically # gets stuff into the Description header. It should not include reST, so # pick out the first single line after the double header. with open(filename) as fp: for lineno, line in enumerate(fp): if lineno < 3: continue line = line.strip() if len(line) > 0: return line flufl.lock-2.2.1/flufl/0000775000175000017500000000000011744120360015143 5ustar barrybarry00000000000000flufl.lock-2.2.1/flufl/lock/0000775000175000017500000000000011744120360016073 5ustar barrybarry00000000000000flufl.lock-2.2.1/flufl/lock/docs/0000775000175000017500000000000011744120360017023 5ustar barrybarry00000000000000flufl.lock-2.2.1/flufl/lock/docs/using.rst0000664000175000017500000001362411706147046020720 0ustar barrybarry00000000000000============================ Using the flufl.lock library ============================ The ``flufl.lock`` package provides NFS-safe file locking with timeouts for POSIX systems. The implementation is influenced by the GNU/Linux `open(2)`_ manpage, under the description of the ``O_EXCL`` option: [...] O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful. The assumption made here is that there will be no *outside interference*, e.g. no agent external to this code will ever ``link()`` to the specific lock files used. Lock objects support lock-breaking so that you can't wedge a process forever. This is especially helpful in a web environment, but may not be appropriate for all applications. Locks have a *lifetime*, which is the maximum length of time the process expects to retain the lock. It is important to pick a good number here because other processes will not break an existing lock until the expected lifetime has expired. Too long and other processes will hang; too short and you'll end up trampling on existing process locks -- and possibly corrupting data. In a distributed (NFS) environment, you also need to make sure that your clocks are properly synchronized. Creating a lock =============== To create a lock, you must first instantiate a ``Lock`` object, specifying the path to a file that will be used to synchronize the lock. This file should not exist. :: # This function comes from the test infrastructure. >>> filename = temporary_lockfile() >>> from flufl.lock import Lock >>> lock = Lock(filename) >>> lock Locks have a default lifetime... >>> lock.lifetime datetime.timedelta(0, 15) ...which you can change. >>> from datetime import timedelta >>> lock.lifetime = timedelta(seconds=30) >>> lock.lifetime datetime.timedelta(0, 30) >>> lock.lifetime = timedelta(seconds=15) You can ask whether the lock is acquired or not. >>> lock.is_locked False Acquiring the lock is easy if no other process has already acquired it. >>> lock.lock() >>> lock.is_locked True Once you have the lock, it's easy to release it. >>> lock.unlock() >>> lock.is_locked False It is an error to attempt to acquire the lock more than once in the same process. :: >>> from flufl.lock import AlreadyLockedError >>> lock.lock() >>> try: ... lock.lock() ... except AlreadyLockedError as error: ... print(error) We already had the lock >>> lock.unlock() Lock objects also support the context manager protocol. >>> lock.is_locked False >>> with lock: ... lock.is_locked True >>> lock.is_locked False Lock acquisition blocks ======================= Trying to lock the file when the lock is unavailable (because another process has already acquired it), the lock call will block. *Note:* the `_acquire()` function is not part of the public API. :: >>> from flufl.lock.tests.subproc import _acquire >>> import time >>> t0 = time.time() >>> _acquire(filename, timedelta(seconds=5)) >>> lock.lock() >>> t1 = time.time() >>> lock.unlock() >>> t1 - t0 > 4 True Refreshing a lock ================= A process can *refresh* a lock if it realizes that it needs to hold the lock for a little longer. You cannot refresh an unlocked lock. >>> from flufl.lock import NotLockedError >>> try: ... lock.refresh() ... except NotLockedError as error: ... print(error) >> from datetime import datetime >>> lock.lifetime = timedelta(seconds=2) >>> lock.lock() >>> lock.is_locked True After the current lifetime expires, the lock is stolen from the parent process even if the parent never unlocks it. *Note:* the `_waitfor()` function is not part of the public API. >>> from flufl.lock.tests.subproc import _waitfor >>> t_broken = _waitfor(filename, lock.lifetime) >>> t_broken < 5 True >>> lock.is_locked False However, if the process holding the lock refreshes it, it will hold it can hold it for as long as it needs. >>> lock.lock() >>> lock.refresh(timedelta(seconds=5)) >>> t_broken = _waitfor(filename, lock.lifetime) >>> t_broken > 3 True >>> lock.is_locked False Lock details ============ Lock files are written with unique contents that can be queried for information about the host name the lock was acquired on, the id of the process that acquired the lock, and the path to the lock file. >>> import os, socket >>> lock.lock() >>> hostname, pid, lockfile = lock.details >>> hostname == socket.getfqdn() True >>> pid == os.getpid() True >>> lockfile == filename True >>> lock.unlock() Even if another process has acquired the lock, the details can be queried. >>> _acquire(filename, timedelta(seconds=3)) >>> lock.is_locked False >>> hostname, pid, lockfile = lock.details >>> hostname == socket.getfqdn() True >>> pid == os.getpid() False >>> lockfile == filename True However, if no process has acquired the lock, the details are unavailable. >>> lock.lock() >>> lock.unlock() >>> try: ... lock.details ... except NotLockedError as error: ... print(error) Details are unavailable .. _`open(2)`: http://manpages.ubuntu.com/manpages/dapper/en/man2/open.2.html flufl.lock-2.2.1/flufl/lock/docs/__init__.py0000664000175000017500000000000011706145124021125 0ustar barrybarry00000000000000flufl.lock-2.2.1/flufl/lock/NEWS.rst0000664000175000017500000000301711744120026017401 0ustar barrybarry00000000000000=================== NEWS for flufl.lock =================== 2.2.1 (2012-04-19) ================== * Add classifiers to setup.py and make the long description more compatible with the Cheeseshop. * Other changes to make the Cheeseshop page look nicer. (LP: #680136) * setup_helper.py version 2.1. 2.2 (2012-01-19) ================ * Support Python 3 without the use of 2to3. * Make the documentation clear that the `flufl.test.subproc` functions are not part of the public API. (LP: #838338) * Fix claim file format in `take_possession()`. (LP: #872096) * Provide a new API for dealing with possible additional unexpected errnos while trying to read the lock file. These can happen in some NFS environments. If you want to retry the read, set the lock file's `retry_errnos` property to a sequence of errnos. If one of those errnos occurs, the read is unconditionally (and infinitely) retried. `retry_errnos` is a property which must be set to a sequence; it has a getter and a deleter too. (LP: #882261) 2.1.1 (2011-08-20) ================== * Fixed TypeError in .lock() method due to race condition in _releasetime property. Found by Stephen A. Goss. (LP: #827052) 2.1 (2010-12-22) ================ * Added lock.details. 2.0.2 (2010-12-19) ================== * Small adjustment to doctest. 2.0.1 (2010-11-27) ================== * Add missing exception to __all__. 2.0 (2010-11-26) ================ * Package renamed to flufl.lock. Earlier ======= Try `bzr log lp:flufl.lock` for details. flufl.lock-2.2.1/flufl/lock/__init__.py0000664000175000017500000000205111744117731020212 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, version 3 of the License. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . """Package init.""" from __future__ import absolute_import, unicode_literals __metaclass__ = type __all__ = [ 'AlreadyLockedError', 'Lock', 'LockError', 'NotLockedError', 'TimeOutError', '__version__', ] __version__ = '2.2.1' # Public API. from flufl.lock._lockfile import ( AlreadyLockedError, Lock, LockError, NotLockedError, TimeOutError) flufl.lock-2.2.1/flufl/lock/_lockfile.py0000664000175000017500000005234511706271533020414 0ustar barrybarry00000000000000# Copyright (C) 2007-2012 by Barry A. Warsaw # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, # USA. """Portable, NFS-safe file locking with timeouts for POSIX systems. This code implements an NFS-safe file-based locking algorithm influenced by the GNU/Linux open(2) manpage, under the description of the O_EXCL option: [...] O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful. The assumption made here is that there will be no 'outside interference', e.g. no agent external to this code will ever link() to the specific lock files used. Lock objects support lock-breaking so that you can't wedge a process forever. This is especially helpful in a web environment, but may not be appropriate for all applications. Locks have a 'lifetime', which is the maximum length of time the process expects to retain the lock. It is important to pick a good number here because other processes will not break an existing lock until the expected lifetime has expired. Too long and other processes will hang; too short and you'll end up trampling on existing process locks -- and possibly corrupting data. In a distributed (NFS) environment, you also need to make sure that your clocks are properly synchronized. """ from __future__ import absolute_import, print_function, unicode_literals __metaclass__ = type __all__ = [ 'LockError', 'AlreadyLockedError', 'NotLockedError', 'Lock', ] import os import sys import time import errno import random import socket import logging import datetime DEFAULT_LOCK_LIFETIME = datetime.timedelta(seconds=15) # Allowable a bit of clock skew. CLOCK_SLOP = datetime.timedelta(seconds=10) try: MAXINT = sys.maxint except AttributeError: # Python 3. MAXINT = sys.maxsize # Details separator; also used in calculating the claim file path. Lock files # should not include this character. SEP = '|' log = logging.getLogger('flufl.lock') # Install a null handler to avoid warnings when applications don't set their # own flufl.lock logger. See http://docs.python.org/library/logging.html try: from logging import NullHandler except ImportError: # Python < 2.7. class NullHandler(logging.Handler): def emit(self, record): pass logging.getLogger('flufl.lock').addHandler(NullHandler()) # Exceptions that can be raised by this module class LockError(Exception): """Base class for all exceptions in this module.""" class AlreadyLockedError(LockError): """An attempt is made to lock an already locked object.""" class NotLockedError(LockError): """An attempt is made to unlock an object that isn't locked.""" class TimeOutError(LockError): """The timeout interval elapsed before the lock succeeded.""" class Lock: """A portable way to lock resources by way of the file system.""" def __init__(self, lockfile, lifetime=None): """Create the resource lock using the given file name and lifetime. Each process laying claim to this resource lock will create their own temporary lock file based on the path specified. An optional lifetime is the length of time that the process expects to hold the lock. :param lockfile: The full path to the lock file. :param lifetime: The expected maximum lifetime of the lock, as a timedelta. Defaults to 15 seconds. """ if lifetime is None: lifetime = DEFAULT_LOCK_LIFETIME self._lockfile = lockfile self._lifetime = lifetime self._set_claimfile() # For transferring ownership across a fork. self._owned = True # For extending the set of NFS errnos that are retried in _read(). self._retry_errnos = [] def __repr__(self): return '<%s %s [%s: %s] pid=%s at %#xx>' % ( self.__class__.__name__, self._lockfile, ('locked' if self.is_locked else 'unlocked'), self._lifetime, os.getpid(), id(self)) @property def details(self): """Details as read from the lock file. :return: A 3-tuple of hostname, process id, file name. :rtype: (str, int, str) :raises NotLockedError: if the lock is not acquired. """ try: with open(self._lockfile) as fp: filename = fp.read().strip() except IOError as error: if error.errno == errno.ENOENT: raise NotLockedError('Details are unavailable') raise # Rearrange for signature. try: lockfile, hostname, pid, random_ignored = filename.split(SEP) except ValueError: raise NotLockedError('Details are unavailable') return hostname, int(pid), lockfile @property def lifetime(self): return self._lifetime @lifetime.setter def lifetime(self, lifetime): self._lifetime = lifetime def refresh(self, lifetime=None, unconditionally=False): """Refreshes the lifetime of a locked file. Use this if you realize that you need to keep a resource locked longer than you thought. :param lifetime: If given, this sets the lock's new lifetime. This must be a datetime.timedelta. :param unconditionally: When False (the default), a `NotLockedError` is raised if an unlocked lock is refreshed. :raises NotLockedError: if the lock is not set, unless optional `unconditionally` flag is set to True. """ if lifetime is not None: self._lifetime = lifetime # Do we have the lock? As a side effect, this refreshes the lock! if not self.is_locked and not unconditionally: raise NotLockedError('{0}: {1}'.format(repr(self), self._read())) def lock(self, timeout=None): """Acquire the lock. This blocks until the lock is acquired unless optional timeout is not None, in which case a `TimeOutError` is raised when the timeout expires without lock acquisition. :param timeout: A datetime.timedelta indicating approximately how long the lock acquisition attempt should be made. None (the default) means keep trying forever. :raises AlreadyLockedError: if the lock is already acquired. :raises TimeOutError: if `timeout` is not None and the indicated time interval expires without a lock acquisition. """ if timeout is not None: timeout_time = datetime.datetime.now() + timeout # Make sure the claim file exists, and that its contents are current. self._write() # XXX This next call can fail with an EPERM. I have no idea why, but # I'm nervous about wrapping this in a try/except. It seems to be a # very rare occurrence, only happens from cron, and has only(?) been # observed on Solaris 2.6. self._touch() log.debug('laying claim: {0}'.format(self._lockfile)) # For quieting the logging output loopcount = -1 while True: loopcount += 1 # Create the hard link and test for exactly 2 links to the file. try: os.link(self._claimfile, self._lockfile) # If we got here, we know we know we got the lock, and never # had it before, so we're done. Just touch it again for the # fun of it. log.debug('got the lock: {0}'.format(self._lockfile)) self._touch() break except OSError as error: # The link failed for some reason, possibly because someone # else already has the lock (i.e. we got an EEXIST), or for # some other bizarre reason. if error.errno == errno.ENOENT: # XXX in some Linux environments, it is possible to get an # ENOENT, which is truly strange, because this means that # self._claimfile didn't exist at the time of the # os.link(), but self._write() is supposed to guarantee # that this happens! I don't honestly know why this # happens -- possibly due to weird caching file systems? # -- but for now we just say we didn't acquire the lock # and try again next time. pass elif error.errno != errno.EEXIST: # Something very bizarre happened. Clean up our state and # pass the error on up. log.exception('unexpected link') os.unlink(self._claimfile) raise elif self._linkcount != 2: # Somebody's messin' with us! Log this, and try again # later. XXX should we raise an exception? log.error('unexpected linkcount: {0:d}'.format( self._linkcount)) elif self._read() == self._claimfile: # It was us that already had the link. log.debug('already locked: {0}'.format(self._lockfile)) raise AlreadyLockedError('We already had the lock') # otherwise, someone else has the lock pass # We did not acquire the lock, because someone else already has # it. Have we timed out in our quest for the lock? if timeout is not None and timeout_time < datetime.datetime.now(): os.unlink(self._claimfile) log.error('timed out') raise TimeOutError('Could not acquire the lock') # Okay, we haven't timed out, but we didn't get the lock. Let's # find if the lock lifetime has expired. Cache the release time # to avoid race conditions. (LP: #827052) release_time = self._releasetime if (release_time != -1 and datetime.datetime.now() > release_time + CLOCK_SLOP): # Yes, so break the lock. self._break() log.error('lifetime has expired, breaking') # Okay, someone else has the lock, our claim hasn't timed out yet, # and the expected lock lifetime hasn't expired yet either. So # let's wait a while for the owner of the lock to give it up. elif not loopcount % 100: log.debug('waiting for claim: {0}'.format(self._lockfile)) self._sleep() def unlock(self, unconditionally=False): """Release the lock. :param unconditionally: When False (the default), a `NotLockedError` is raised if this is called on an unlocked lock. :raises NotLockedError: if we don't own the lock, either because of unbalanced unlock calls, or because the lock was stolen out from under us, unless optional `unconditionally` is True. """ is_locked = self.is_locked if not is_locked and not unconditionally: raise NotLockedError('Already unlocked') # If we owned the lock, remove the lockfile, relinquishing the lock. if is_locked: try: os.unlink(self._lockfile) except OSError as error: if error.errno != errno.ENOENT: raise # Remove our claim file. try: os.unlink(self._claimfile) except OSError as error: if error.errno != errno.ENOENT: raise log.debug('unlocked: {0}'.format(self._lockfile)) @property def is_locked(self): """True if we own the lock, False if we do not. Checking the status of the lock resets the lock's lifetime, which helps avoid race conditions during the lock status test. """ # Discourage breaking the lock for a while. try: self._touch() except OSError as error: if error.errno == errno.EPERM: # We can't touch the file because we're not the owner. I # don't see how we can own the lock if we're not the owner. return False else: raise # XXX Can the link count ever be > 2? if self._linkcount != 2: return False return self._read() == self._claimfile def finalize(self): """Unconditionally unlock the file.""" log.debug('finalize: {0}'.format(self._lockfile)) self.unlock(unconditionally=True) def __del__(self): log.debug('__del__: {0}'.format(self._lockfile)) if self._owned: self.finalize() # Python 2.5 context manager protocol support. def __enter__(self): self.lock() return self def __exit__(self, exc_type, exc_val, exc_tb): self.unlock() # Don't suppress any exception that might have occurred. return False def transfer_to(self, pid): """Transfer ownership of the lock to another process. Use this only if you're transfering ownership to a child process across a fork. Use at your own risk, but it should be race-condition safe. transfer_to() is called in the parent, passing in the pid of the child. take_possession() is called in the child, and blocks until the parent has transferred possession to the child. disown() is used to set the 'owned' flag to False, and it is a disgusting wart necessary to make forced lock acquisition work. :( :param pid: The process id of the child process that will take possession of the lock. """ # First touch it so it won't get broken while we're fiddling about. self._touch() # Find out current claim's file name winner = self._read() # Now twiddle ours to the given pid. self._set_claimfile(pid) # Create a hard link from the global lock file to the claim file. # This actually does things in reverse order of normal operation # because we know that lockfile exists, and claimfile better not! os.link(self._lockfile, self._claimfile) # Now update the lock file to contain a reference to the new owner self._write() # Toggle off our ownership of the file so we don't try to finalize it # in our __del__() self._owned = False # Unlink the old winner, completing the transfer. os.unlink(winner) # And do some sanity checks assert self._linkcount == 2, ( 'Unexpected link count: wanted 2, got {0:d}'.format( self._linkcount)) assert self.is_locked, 'Expected to be locked' log.debug('transferred the lock: {0}'.format(self._lockfile)) def take_possession(self): """Take possession of a lock from another process. See `transfer_to()` for more information. """ self._set_claimfile() # Wait until the linkcount is 2, indicating the parent has completed # the transfer. while self._linkcount != 2 or self._read() != self._claimfile: time.sleep(0.25) log.debug('took possession of the lock: {0}'.format(self._lockfile)) def disown(self): """Disown this lock. See `transfer_to()` for more information. """ self._owned = False def _set_claimfile(self, pid=None): """Set the _claimfile private variable.""" if pid is None: pid = os.getpid() # Calculate a hard link file name that will be used to lay claim to # the lock. We need to watch out for two Lock objects in the same # process pointing to the same lock file. Without this, if you lock # lf1 and do not lock lf2, lf2.locked() will still return True. self._claimfile = SEP.join(( self._lockfile, socket.getfqdn(), str(pid), str(random.randint(0, MAXINT)), )) def _write(self): """Write our claim file's name to the claim file.""" # Make sure it's group writable fp = open(self._claimfile, 'w') try: fp.write(self._claimfile) finally: fp.close() @property def retry_errnos(self): """The set of errno values that cause a read retry.""" return self._retry_errnos[:] @retry_errnos.setter def retry_errnos(self, errnos): self._retry_errnos = [] self._retry_errnos.extend(errnos) @retry_errnos.deleter def retry_errnos(self): self._retry_errnos = [] def _read(self): """Read the contents of our lock file. :return: The contents of the lock file or None if the lock file does not exist. """ while True: try: with open(self._lockfile) as fp: return fp.read() except EnvironmentError as error: if error.errno in self._retry_errnos: self._sleep() elif error.errno != errno.ENOENT: raise else: return None def _touch(self, filename=None): """Touch the claim file into the future. :param filename: If given, the file to touch, otherwise our claim file is touched. """ expiration_date = datetime.datetime.now() + self._lifetime t = time.mktime(expiration_date.timetuple()) try: # XXX We probably don't need to modify atime, but this is easier. os.utime(filename or self._claimfile, (t, t)) except OSError as error: if error.errno != errno.ENOENT: raise @property def _releasetime(self): """The time when the lock should be released. :return: The mtime of the file, which is when the lock should be released, or -1 if the lockfile doesn't exist. """ try: return datetime.datetime.fromtimestamp( os.stat(self._lockfile).st_mtime) except OSError as error: if error.errno == errno.ENOENT: return -1 raise @property def _linkcount(self): """The number of hard links to the lock file. :return: the number of hard links to the lock file, or -1 if the lock file doesn't exist. """ try: return os.stat(self._lockfile).st_nlink except OSError as error: if error.errno == errno.ENOENT: return -1 raise def _break(self): """Break the lock.""" # First, touch the lock file. This reduces but does not eliminate the # chance for a race condition during breaking. Two processes could # both pass the test for lock expiry in lock() before one of them gets # to touch the lock file. This shouldn't be too bad because all # they'll do in that function is delete the lock files, not claim the # lock, and we can be defensive for ENOENTs here. # # Touching the lock could fail if the process breaking the lock and # the process that claimed the lock have different owners. Don't do # that. try: self._touch(self._lockfile) except OSError as error: if error.errno != errno.EPERM: raise # Get the name of the old winner's temp file. winner = self._read() # Remove the global lockfile, which actually breaks the lock. try: os.unlink(self._lockfile) except OSError as error: if error.errno != errno.ENOENT: raise # Try to remove the old winner's claim file, since we're assuming the # winner process has hung or died. Don't worry too much if we can't # unlink their claim file -- this doesn't wreck the locking algorithm, # but will leave claim file turds laying around, a minor inconvenience. try: if winner: os.unlink(winner) except OSError as error: if error.errno != errno.ENOENT: raise def _sleep(self): """Snooze for a random amount of time.""" interval = random.random() * 2.0 + 0.01 time.sleep(interval) flufl.lock-2.2.1/flufl/lock/conf.py0000664000175000017500000001537111706272752017414 0ustar barrybarry00000000000000# -*- coding: utf-8 -*- # # flufl.lock documentation build configuration file, created by # sphinx-quickstart on Thu Jan 7 18:41:30 2010. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. from __future__ import print_function import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.append(os.path.abspath('.')) # -- General configuration ----------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc'] # Add any paths that contain templates here, relative to this directory. templates_path = ['../../_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8' # The master toctree document. master_doc = 'README' # General information about the project. project = 'flufl.lock' copyright = '2004-2012, Barry A. Warsaw' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # from flufl.lock import __version__ # The short X.Y version. version = __version__ # The full version, including alpha/beta/rc tags. release = __version__ # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of documents that shouldn't be included in the build. #unused_docs = [] # List of directories, relative to source directory, that shouldn't be searched # for source files. exclude_trees = ['_build', 'build', 'flufl.lock.egg-info', 'distribute-0.6.10'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['../../_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_use_modindex = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = '' # Output file base name for HTML help builder. htmlhelp_basename = 'flufllockdoc' # -- Options for LaTeX output -------------------------------------------------- # The paper size ('letter' or 'a4'). #latex_paper_size = 'letter' # The font size ('10pt', '11pt' or '12pt'). #latex_font_size = '10pt' # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('README.rst', 'flufllock.tex', 'flufl.lock Documentation', 'Barry A. Warsaw', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # Additional stuff for the LaTeX preamble. #latex_preamble = '' # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_use_modindex = True import errno def index_html(): cwd = os.getcwd() try: os.chdir('build/sphinx/html') try: os.unlink('index.html') except OSError as error: if error.errno != errno.ENOENT: raise os.symlink('README.html', 'index.html') print('index.html -> README.html') finally: os.chdir(cwd) import atexit atexit.register(index_html) flufl.lock-2.2.1/flufl/lock/README.rst0000664000175000017500000000430611706146031017566 0ustar barrybarry00000000000000================================== flufl.lock - An NFS-safe file lock ================================== This package is called ``flufl.lock``. It is an NFS-safe file-based lock with timeouts for POSIX systems. Requirements ============ ``flufl.lock`` requires Python 2.6 or newer, and is compatible with Python 3. Documentation ============= A `simple guide`_ to using the library is available within this package, in the form of doctests. The manual is also available online in the Cheeseshop at: http://package.python.org/flufl.lock Project details =============== The project home page is: http://launchpad.net/flufl.lock You should report bugs at: http://bugs.launchpad.net/flufl.lock You can download the latest version of the package either from the Cheeseshop: http://pypi.python.org/pypi/flufl.lock or from the Launchpad page above. Of course you can also just install it with ``pip`` or ``easy_install`` from the command line:: % sudo pip flufl.lock % sudo easy_install flufl.lock You can grab the latest development copy of the code using Bazaar, from the Launchpad home page above. See http://bazaar-vcs.org for details on the Bazaar distributed revision control system. If you have Bazaar installed, you can grab your own branch of the code like this:: bzr branch lp:flufl.lock You may contact the author via barry@python.org. Copyright ========= Copyright (C) 2004-2012 Barry A. Warsaw This file is part of flufl.lock. flufl.lock is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. flufl.lock is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with flufl.lock. If not, see . Table of Contents ================= .. toctree:: :glob: docs/using NEWS .. _`simple guide`: docs/using.html flufl.lock-2.2.1/flufl/lock/tests/0000775000175000017500000000000011744120360017235 5ustar barrybarry00000000000000flufl.lock-2.2.1/flufl/lock/tests/test_lockfile.py0000664000175000017500000000607611706273147022461 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock. # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, version 3 of the License. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . """Testing other aspects of the implementation and API.""" from __future__ import absolute_import, print_function, unicode_literals __metaclass__ = type __all__ = [ ] import os import errno import tempfile import unittest try: # Python 3 import builtins except ImportError: # Python 2 import __builtin__ as builtins from flufl.lock._lockfile import Lock, NotLockedError class TestableEnvironmentError(EnvironmentError): def __init__(self, errno): super(TestableEnvironmentError, self).__init__() self.errno = errno EMOCKEDFAILURE = 99 EOTHERMOCKEDFAILURE = 98 class ErrnoRetryTests(unittest.TestCase): def setUp(self): self._builtin_open = builtins.open self._failure_countdown = None self._retry_count = None self._errno = EMOCKEDFAILURE fd, self._lockfile = tempfile.mkstemp('.lck') os.close(fd) self._lock = Lock(self._lockfile) def tearDown(self): self._disable() try: self._lock.unlock() except NotLockedError: pass try: os.remove(self._lockfile) except OSError as error: if error.errno != errno.ENOENT: raise def _enable(self): builtins.open = self._testable_open self._failure_countdown = 3 self._retry_count = 0 def _disable(self): builtins.open = self._builtin_open def _testable_open(self, *args, **kws): if self._failure_countdown <= 0: return self._builtin_open(*args, **kws) self._failure_countdown -= 1 self._retry_count += 1 raise TestableEnvironmentError(self._errno) def test_retry_errno_api(self): self.assertEqual(self._lock.retry_errnos, []) self._lock.retry_errnos = [EMOCKEDFAILURE, EOTHERMOCKEDFAILURE] self.assertEqual(self._lock.retry_errnos, [EMOCKEDFAILURE, EOTHERMOCKEDFAILURE]) del self._lock.retry_errnos self.assertEqual(self._lock.retry_errnos, []) def test_retries(self): # Test that _read() will retry when a given errno is encountered. self._lock.lock() self._lock.retry_errnos = [self._errno] self._enable() self.assertTrue(self._lock.is_locked) # The _read() trigged by the .is_locked call should have been retried. self.assertEqual(self._retry_count, 3) flufl.lock-2.2.1/flufl/lock/tests/subproc.py0000664000175000017500000000406111706146754021302 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock. # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, version 3 of the License. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . """A helper for setting up subprocess locks.""" from __future__ import absolute_import, print_function, unicode_literals __metaclass__ = type __all__ = [ '_acquire', '_waitfor', ] import time import multiprocessing from flufl.lock import Lock def child_locker(filename, lifetime, queue): # First, acquire the file lock. with Lock(filename, lifetime): # Now inform the parent that we've acquired the lock. queue.put(True) # Keep the file lock for a while. time.sleep(lifetime.seconds - 1) def _acquire(filename, lifetime=None): """Acquire the named lock file in a subprocess.""" queue = multiprocessing.Queue() proc = multiprocessing.Process(target=child_locker, args=(filename, lifetime, queue)) proc.start() while not queue.get(): time.sleep(0.1) def child_waitfor(filename, lifetime, queue): t0 = time.time() # Try to acquire the lock. with Lock(filename, lifetime): # Tell the parent how long it took to acquire the lock. queue.put(time.time() - t0) def _waitfor(filename, lifetime): """Fire off a child that waits for a lock.""" queue = multiprocessing.Queue() proc = multiprocessing.Process(target=child_waitfor, args=(filename, lifetime, queue)) proc.start() time = queue.get() return time flufl.lock-2.2.1/flufl/lock/tests/__init__.py0000664000175000017500000000000011706145124021337 0ustar barrybarry00000000000000flufl.lock-2.2.1/flufl/lock/tests/test_documentation.py0000664000175000017500000001026011706146323023523 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock. # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, either version 3 of the License, or (at your option) # any later version. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . """Test harness for doctests.""" from __future__ import absolute_import, print_function, unicode_literals __metaclass__ = type __all__ = [ 'additional_tests', ] import os import errno import atexit import doctest import logging import tempfile import unittest from datetime import timedelta from io import StringIO from pkg_resources import ( resource_filename, resource_exists, resource_listdir, cleanup_resources) COMMASPACE = ', ' DOT = '.' DOCTEST_FLAGS = ( doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_NDIFF) # For logging debugging. log_stream = StringIO() def stop(): """Call into pdb.set_trace()""" # Do the import here so that you get the wacky special hacked pdb instead # of Python's normal pdb. import pdb pdb.set_trace() def make_temporary_lockfile(testobj): """Make a temporary lock file for the tests.""" def lockfile_creator(): fd, testobj._lockfile = tempfile.mkstemp() os.close(fd) os.remove(testobj._lockfile) return testobj._lockfile return lockfile_creator def setup(testobj): """Test setup.""" # Truncate the log. log_stream.truncate() # Note that the module has a default built-in *clock slop* of 10 seconds # to handle differences in machine clocks. Since this test is happening on # the same machine, we can bump the slop down to a more reasonable number. from flufl.lock import _lockfile testobj._slop = _lockfile.CLOCK_SLOP _lockfile.CLOCK_SLOP = timedelta(seconds=0) # Make sure future statements in our doctests match the Python code. When # run with 2to3, the future import gets removed and these names are not # defined. try: testobj.globs['absolute_import'] = absolute_import testobj.globs['print_function'] = print_function testobj.globs['unicode_literals'] = unicode_literals except NameError: pass testobj.globs['temporary_lockfile'] = make_temporary_lockfile(testobj) testobj.globs['log_stream'] = log_stream testobj.globs['stop'] = stop def teardown(testobj): """Test teardown.""" # Restore the original clock slop. from flufl.lock import _lockfile _lockfile.CLOCK_SLOP = testobj._slop try: os.remove(testobj._lockfile) except OSError as error: if error.errno != errno.ENOENT: raise except AttributeError: # lockfile_creator() was never called. pass def additional_tests(): "Run the doc tests (README.rst and docs/*, if any exist)" # Initialize logging for testing purposes. logging.basicConfig(stream=log_stream, level=logging.DEBUG, datefmt='%b %d %H:%M:%S %Y', format='%(asctime)s (%(process)d) %(message)s', ) doctest_files = [ os.path.abspath(resource_filename('flufl.lock', 'README.rst'))] if resource_exists('flufl.lock', 'docs'): for name in resource_listdir('flufl.lock', 'docs'): if name.endswith('.rst'): doctest_files.append( os.path.abspath( resource_filename('flufl.lock', 'docs/%s' % name))) kwargs = dict(module_relative=False, optionflags=DOCTEST_FLAGS, setUp=setup, tearDown=teardown, ) atexit.register(cleanup_resources) return unittest.TestSuite(( doctest.DocFileSuite(*doctest_files, **kwargs))) flufl.lock-2.2.1/flufl/__init__.py0000664000175000017500000000156611706145124017267 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock. # # flufl.lock is free software: you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation, version 3 of the License. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public # License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . # this is a namespace package try: import pkg_resources pkg_resources.declare_namespace(__name__) except ImportError: import pkgutil __path__ = pkgutil.extend_path(__path__, __name__) flufl.lock-2.2.1/distribute_setup.py0000664000175000017500000003566511706145124020025 0ustar barrybarry00000000000000#!python """Bootstrap distribute installation If you want to use setuptools in your package's setup.py, just include this file in the same directory with it, and add this to the top of your setup.py:: from distribute_setup import use_setuptools use_setuptools() If you want to require a specific version of setuptools, set a download mirror, or use an alternate download directory, you can do so by supplying the appropriate options to ``use_setuptools()``. This file can also be run as a script to install or upgrade setuptools. """ import os import sys import time import fnmatch import tempfile import tarfile from distutils import log try: from site import USER_SITE except ImportError: USER_SITE = None try: import subprocess def _python_cmd(*args): args = (sys.executable,) + args return subprocess.call(args) == 0 except ImportError: # will be used for python 2.3 def _python_cmd(*args): args = (sys.executable,) + args # quoting arguments if windows if sys.platform == 'win32': def quote(arg): if ' ' in arg: return '"%s"' % arg return arg args = [quote(arg) for arg in args] return os.spawnl(os.P_WAIT, sys.executable, *args) == 0 DEFAULT_VERSION = "0.6.10" DEFAULT_URL = "http://pypi.python.org/packages/source/d/distribute/" SETUPTOOLS_FAKED_VERSION = "0.6c11" SETUPTOOLS_PKG_INFO = """\ Metadata-Version: 1.0 Name: setuptools Version: %s Summary: xxxx Home-page: xxx Author: xxx Author-email: xxx License: xxx Description: xxx """ % SETUPTOOLS_FAKED_VERSION def _install(tarball): # extracting the tarball tmpdir = tempfile.mkdtemp() log.warn('Extracting in %s', tmpdir) old_wd = os.getcwd() try: os.chdir(tmpdir) tar = tarfile.open(tarball) _extractall(tar) tar.close() # going in the directory subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) os.chdir(subdir) log.warn('Now working in %s', subdir) # installing log.warn('Installing Distribute') if not _python_cmd('setup.py', 'install'): log.warn('Something went wrong during the installation.') log.warn('See the error message above.') finally: os.chdir(old_wd) def _build_egg(egg, tarball, to_dir): # extracting the tarball tmpdir = tempfile.mkdtemp() log.warn('Extracting in %s', tmpdir) old_wd = os.getcwd() try: os.chdir(tmpdir) tar = tarfile.open(tarball) _extractall(tar) tar.close() # going in the directory subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) os.chdir(subdir) log.warn('Now working in %s', subdir) # building an egg log.warn('Building a Distribute egg in %s', to_dir) _python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir) finally: os.chdir(old_wd) # returning the result log.warn(egg) if not os.path.exists(egg): raise IOError('Could not build the egg.') def _do_download(version, download_base, to_dir, download_delay): egg = os.path.join(to_dir, 'distribute-%s-py%d.%d.egg' % (version, sys.version_info[0], sys.version_info[1])) if not os.path.exists(egg): tarball = download_setuptools(version, download_base, to_dir, download_delay) _build_egg(egg, tarball, to_dir) sys.path.insert(0, egg) import setuptools setuptools.bootstrap_install_from = egg def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, download_delay=15, no_fake=True): # making sure we use the absolute path to_dir = os.path.abspath(to_dir) was_imported = 'pkg_resources' in sys.modules or \ 'setuptools' in sys.modules try: try: import pkg_resources if not hasattr(pkg_resources, '_distribute'): if not no_fake: _fake_setuptools() raise ImportError except ImportError: return _do_download(version, download_base, to_dir, download_delay) try: pkg_resources.require("distribute>="+version) return except pkg_resources.VersionConflict: e = sys.exc_info()[1] if was_imported: sys.stderr.write( "The required version of distribute (>=%s) is not available,\n" "and can't be installed while this script is running. Please\n" "install a more recent version first, using\n" "'easy_install -U distribute'." "\n\n(Currently using %r)\n" % (version, e.args[0])) sys.exit(2) else: del pkg_resources, sys.modules['pkg_resources'] # reload ok return _do_download(version, download_base, to_dir, download_delay) except pkg_resources.DistributionNotFound: return _do_download(version, download_base, to_dir, download_delay) finally: if not no_fake: _create_fake_setuptools_pkg_info(to_dir) def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, delay=15): """Download distribute from a specified location and return its filename `version` should be a valid distribute version number that is available as an egg for download under the `download_base` URL (which should end with a '/'). `to_dir` is the directory where the egg will be downloaded. `delay` is the number of seconds to pause before an actual download attempt. """ # making sure we use the absolute path to_dir = os.path.abspath(to_dir) try: from urllib.request import urlopen except ImportError: from urllib2 import urlopen tgz_name = "distribute-%s.tar.gz" % version url = download_base + tgz_name saveto = os.path.join(to_dir, tgz_name) src = dst = None if not os.path.exists(saveto): # Avoid repeated downloads try: log.warn("Downloading %s", url) src = urlopen(url) # Read/write all in one block, so we don't create a corrupt file # if the download is interrupted. data = src.read() dst = open(saveto, "wb") dst.write(data) finally: if src: src.close() if dst: dst.close() return os.path.realpath(saveto) def _patch_file(path, content): """Will backup the file then patch it""" existing_content = open(path).read() if existing_content == content: # already patched log.warn('Already patched.') return False log.warn('Patching...') _rename_path(path) f = open(path, 'w') try: f.write(content) finally: f.close() return True def _same_content(path, content): return open(path).read() == content def _no_sandbox(function): def __no_sandbox(*args, **kw): try: from setuptools.sandbox import DirectorySandbox def violation(*args): pass DirectorySandbox._old = DirectorySandbox._violation DirectorySandbox._violation = violation patched = True except ImportError: patched = False try: return function(*args, **kw) finally: if patched: DirectorySandbox._violation = DirectorySandbox._old del DirectorySandbox._old return __no_sandbox @_no_sandbox def _rename_path(path): new_name = path + '.OLD.%s' % time.time() log.warn('Renaming %s into %s', path, new_name) os.rename(path, new_name) return new_name def _remove_flat_installation(placeholder): if not os.path.isdir(placeholder): log.warn('Unkown installation at %s', placeholder) return False found = False for file in os.listdir(placeholder): if fnmatch.fnmatch(file, 'setuptools*.egg-info'): found = True break if not found: log.warn('Could not locate setuptools*.egg-info') return log.warn('Removing elements out of the way...') pkg_info = os.path.join(placeholder, file) if os.path.isdir(pkg_info): patched = _patch_egg_dir(pkg_info) else: patched = _patch_file(pkg_info, SETUPTOOLS_PKG_INFO) if not patched: log.warn('%s already patched.', pkg_info) return False # now let's move the files out of the way for element in ('setuptools', 'pkg_resources.py', 'site.py'): element = os.path.join(placeholder, element) if os.path.exists(element): _rename_path(element) else: log.warn('Could not find the %s element of the ' 'Setuptools distribution', element) return True def _after_install(dist): log.warn('After install bootstrap.') placeholder = dist.get_command_obj('install').install_purelib _create_fake_setuptools_pkg_info(placeholder) @_no_sandbox def _create_fake_setuptools_pkg_info(placeholder): if not placeholder or not os.path.exists(placeholder): log.warn('Could not find the install location') return pyver = '%s.%s' % (sys.version_info[0], sys.version_info[1]) setuptools_file = 'setuptools-%s-py%s.egg-info' % \ (SETUPTOOLS_FAKED_VERSION, pyver) pkg_info = os.path.join(placeholder, setuptools_file) if os.path.exists(pkg_info): log.warn('%s already exists', pkg_info) return log.warn('Creating %s', pkg_info) f = open(pkg_info, 'w') try: f.write(SETUPTOOLS_PKG_INFO) finally: f.close() pth_file = os.path.join(placeholder, 'setuptools.pth') log.warn('Creating %s', pth_file) f = open(pth_file, 'w') try: f.write(os.path.join(os.curdir, setuptools_file)) finally: f.close() def _patch_egg_dir(path): # let's check if it's already patched pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO') if os.path.exists(pkg_info): if _same_content(pkg_info, SETUPTOOLS_PKG_INFO): log.warn('%s already patched.', pkg_info) return False _rename_path(path) os.mkdir(path) os.mkdir(os.path.join(path, 'EGG-INFO')) pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO') f = open(pkg_info, 'w') try: f.write(SETUPTOOLS_PKG_INFO) finally: f.close() return True def _before_install(): log.warn('Before install bootstrap.') _fake_setuptools() def _under_prefix(location): if 'install' not in sys.argv: return True args = sys.argv[sys.argv.index('install')+1:] for index, arg in enumerate(args): for option in ('--root', '--prefix'): if arg.startswith('%s=' % option): top_dir = arg.split('root=')[-1] return location.startswith(top_dir) elif arg == option: if len(args) > index: top_dir = args[index+1] return location.startswith(top_dir) elif option == '--user' and USER_SITE is not None: return location.startswith(USER_SITE) return True def _fake_setuptools(): log.warn('Scanning installed packages') try: import pkg_resources except ImportError: # we're cool log.warn('Setuptools or Distribute does not seem to be installed.') return ws = pkg_resources.working_set try: setuptools_dist = ws.find(pkg_resources.Requirement.parse('setuptools', replacement=False)) except TypeError: # old distribute API setuptools_dist = ws.find(pkg_resources.Requirement.parse('setuptools')) if setuptools_dist is None: log.warn('No setuptools distribution found') return # detecting if it was already faked setuptools_location = setuptools_dist.location log.warn('Setuptools installation detected at %s', setuptools_location) # if --root or --preix was provided, and if # setuptools is not located in them, we don't patch it if not _under_prefix(setuptools_location): log.warn('Not patching, --root or --prefix is installing Distribute' ' in another location') return # let's see if its an egg if not setuptools_location.endswith('.egg'): log.warn('Non-egg installation') res = _remove_flat_installation(setuptools_location) if not res: return else: log.warn('Egg installation') pkg_info = os.path.join(setuptools_location, 'EGG-INFO', 'PKG-INFO') if (os.path.exists(pkg_info) and _same_content(pkg_info, SETUPTOOLS_PKG_INFO)): log.warn('Already patched.') return log.warn('Patching...') # let's create a fake egg replacing setuptools one res = _patch_egg_dir(setuptools_location) if not res: return log.warn('Patched done.') _relaunch() def _relaunch(): log.warn('Relaunching...') # we have to relaunch the process args = [sys.executable] + sys.argv sys.exit(subprocess.call(args)) def _extractall(self, path=".", members=None): """Extract all members from the archive to the current working directory and set owner, modification time and permissions on directories afterwards. `path' specifies a different directory to extract to. `members' is optional and must be a subset of the list returned by getmembers(). """ import copy import operator from tarfile import ExtractError directories = [] if members is None: members = self for tarinfo in members: if tarinfo.isdir(): # Extract directories with a safe mode. directories.append(tarinfo) tarinfo = copy.copy(tarinfo) tarinfo.mode = 448 # decimal for oct 0700 self.extract(tarinfo, path) # Reverse sort directories. if sys.version_info < (2, 4): def sorter(dir1, dir2): return cmp(dir1.name, dir2.name) directories.sort(sorter) directories.reverse() else: directories.sort(key=operator.attrgetter('name'), reverse=True) # Set correct owner, mtime and filemode on directories. for tarinfo in directories: dirpath = os.path.join(path, tarinfo.name) try: self.chown(tarinfo, dirpath) self.utime(tarinfo, dirpath) self.chmod(tarinfo, dirpath) except ExtractError: e = sys.exc_info()[1] if self.errorlevel > 1: raise else: self._dbg(1, "tarfile: %s" % e) def main(argv, version=DEFAULT_VERSION): """Install or upgrade setuptools and EasyInstall""" tarball = download_setuptools() _install(tarball) if __name__ == '__main__': main(sys.argv[1:]) flufl.lock-2.2.1/PKG-INFO0000664000175000017500000001353111744120360015133 0ustar barrybarry00000000000000Metadata-Version: 1.1 Name: flufl.lock Version: 2.2.1 Summary: NFS-safe file locking with timeouts for POSIX systems Home-page: http://launchpad.net/flufl.lock Author: Barry Warsaw Author-email: barry@python.org License: LGPLv3 Download-URL: https://launchpad.net/flufl.lock/+download Description: ========== flufl.lock ========== NFS-safe file locking with timeouts for POSIX systems The ``flufl.lock`` library provides an NFS-safe file-based locking algorithm influenced by the GNU/Linux `open(2)` manpage, under the description of the `O_EXCL` option:: [...] O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful. The assumption made here is that there will be no *outside interference*, e.g. no agent external to this code will ever ``link()`` to the specific lock files used. Lock objects support lock-breaking so that you can't wedge a process forever. This is especially helpful in a web environment, but may not be appropriate for all applications. Locks have a *lifetime*, which is the maximum length of time the process expects to retain the lock. It is important to pick a good number here because other processes will not break an existing lock until the expected lifetime has expired. Too long and other processes will hang; too short and you'll end up trampling on existing process locks -- and possibly corrupting data. In a distributed (NFS) environment, you also need to make sure that your clocks are properly synchronized. License ======= This file is part of flufl.lock. flufl.lock is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, version 3 of the License. flufl.lock is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with flufl.lock. If not, see . =================== NEWS for flufl.lock =================== 2.2.1 (2012-04-19) ================== * Add classifiers to setup.py and make the long description more compatible with the Cheeseshop. * Other changes to make the Cheeseshop page look nicer. (LP: #680136) * setup_helper.py version 2.1. 2.2 (2012-01-19) ================ * Support Python 3 without the use of 2to3. * Make the documentation clear that the `flufl.test.subproc` functions are not part of the public API. (LP: #838338) * Fix claim file format in `take_possession()`. (LP: #872096) * Provide a new API for dealing with possible additional unexpected errnos while trying to read the lock file. These can happen in some NFS environments. If you want to retry the read, set the lock file's `retry_errnos` property to a sequence of errnos. If one of those errnos occurs, the read is unconditionally (and infinitely) retried. `retry_errnos` is a property which must be set to a sequence; it has a getter and a deleter too. (LP: #882261) 2.1.1 (2011-08-20) ================== * Fixed TypeError in .lock() method due to race condition in _releasetime property. Found by Stephen A. Goss. (LP: #827052) 2.1 (2010-12-22) ================ * Added lock.details. 2.0.2 (2010-12-19) ================== * Small adjustment to doctest. 2.0.1 (2010-11-27) ================== * Add missing exception to __all__. 2.0 (2010-11-26) ================ * Package renamed to flufl.lock. Earlier ======= Try `bzr log lp:flufl.lock` for details. Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+) Classifier: Operating System :: POSIX Classifier: Operating System :: MacOS :: MacOS X Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Software Development :: Libraries Classifier: Topic :: Software Development :: Libraries :: Python Modules flufl.lock-2.2.1/template.py0000664000175000017500000000146711706145124016233 0ustar barrybarry00000000000000# Copyright (C) 2004-2012 by Barry A. Warsaw # # This file is part of flufl.lock. # # flufl.lock is free software: you can redistribute it and/or modify it under # the terms of the GNU Lesser General Public License as published by the Free # Software Foundation, version 3 of the License. # # flufl.lock is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License # for more details. # # You should have received a copy of the GNU Lesser General Public License # along with flufl.lock. If not, see . """Module contents.""" from __future__ import absolute_import, print_function, unicode_literals __metaclass__ = type __all__ = [ ] flufl.lock-2.2.1/README.rst0000664000175000017500000000436311744117707015543 0ustar barrybarry00000000000000========== flufl.lock ========== NFS-safe file locking with timeouts for POSIX systems The ``flufl.lock`` library provides an NFS-safe file-based locking algorithm influenced by the GNU/Linux `open(2)` manpage, under the description of the `O_EXCL` option:: [...] O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful. The assumption made here is that there will be no *outside interference*, e.g. no agent external to this code will ever ``link()`` to the specific lock files used. Lock objects support lock-breaking so that you can't wedge a process forever. This is especially helpful in a web environment, but may not be appropriate for all applications. Locks have a *lifetime*, which is the maximum length of time the process expects to retain the lock. It is important to pick a good number here because other processes will not break an existing lock until the expected lifetime has expired. Too long and other processes will hang; too short and you'll end up trampling on existing process locks -- and possibly corrupting data. In a distributed (NFS) environment, you also need to make sure that your clocks are properly synchronized. License ======= This file is part of flufl.lock. flufl.lock is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, version 3 of the License. flufl.lock is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with flufl.lock. If not, see .