flufl_lock-8.2.0/conftest.py 0000644 0000000 0000000 00000005621 13615410400 012764 0 ustar 00 import os
import time
import multiprocessing
from sybil import Sybil
from psutil import pid_exists
from doctest import ELLIPSIS, NORMALIZE_WHITESPACE, REPORT_NDIFF
from datetime import timedelta
from tempfile import TemporaryDirectory
from contextlib import ExitStack
from flufl.lock import SEP, Lock
from sybil.parsers.doctest import DocTestParser
from sybil.parsers.codeblock import PythonCodeBlockParser
DOCTEST_FLAGS = ELLIPSIS | NORMALIZE_WHITESPACE | REPORT_NDIFF
def child_locker(filename, lifetime, queue, extra_sleep):
# First, acquire the file lock.
with Lock(filename, lifetime):
# Now inform the parent that we've acquired the lock.
queue.put(True)
# Keep the file lock for a while.
time.sleep(lifetime.seconds + extra_sleep)
def acquire(filename, *, lifetime=None, extra_sleep=-1):
"""Acquire the named lock file in a subprocess."""
queue = multiprocessing.Queue()
proc = multiprocessing.Process(
target=child_locker,
args=(filename, timedelta(seconds=lifetime), queue, extra_sleep))
proc.start()
while not queue.get():
time.sleep(0.1)
def child_waitfor(filename, lifetime, queue):
t0 = time.time()
# Try to acquire the lock.
with Lock(filename, lifetime):
# Tell the parent how long it took to acquire the lock.
queue.put(time.time() - t0)
def waitfor(filename, lifetime):
"""Fire off a child that waits for a lock."""
queue = multiprocessing.Queue()
proc = multiprocessing.Process(
target=child_waitfor,
args=(filename, lifetime, queue))
proc.start()
return queue.get()
def simulate_process_crash(lockfile):
# We simulate an unclean crash of the process holding the lock by
# deliberately fiddling with the pid in the lock file, ensuring that no
# process with the new pid exists.
with open(lockfile) as fp:
filename = fp.read().strip()
lockfile, hostname, pid_str, random = filename.split(SEP)
pid = int(pid_str)
while pid_exists(pid):
pid += 1
with open(lockfile, 'w') as fp:
fp.write(SEP.join((lockfile, hostname, str(pid), random)))
class DoctestNamespace:
def __init__(self):
self._resources = ExitStack()
def setup(self, namespace):
namespace['acquire'] = acquire
namespace['temporary_lockfile'] = self.temporary_lockfile
namespace['waitfor'] = waitfor
namespace['simulate_process_crash'] = simulate_process_crash
def teardown(self, namespace):
self._resources.close()
def temporary_lockfile(self):
lock_dir = self._resources.enter_context(TemporaryDirectory())
return os.path.join(lock_dir, 'test.lck')
namespace = DoctestNamespace()
pytest_collect_file = Sybil(
parsers=[
DocTestParser(optionflags=DOCTEST_FLAGS),
PythonCodeBlockParser(),
],
pattern='*.rst',
setup=namespace.setup,
).pytest()
flufl_lock-8.2.0/docs/NEWS.rst 0000644 0000000 0000000 00000017405 13615410400 013026 0 ustar 00 =====================
flufl.lock change log
=====================
8.2 (2025-05-08)
================
* Add support for Python 3.13 and 3.14; remove support for Python 3.8.
* Modernize type hints.
8.1 (2024-03-30)
================
* Add support for Python 3.12. (GL#35)
* Switch to ``hatch``, replacing ``pdm`` and ``tox``. (GL#36)
* Switch to ``ruff`` from ``blue`` and ``isort``. (GL#37)
8.0.2 (2023-07-21)
==================
* Update dependencies.
* Other minor improvements and cleanups.
8.0.1 (2023-06-22)
==================
* Minor documentation fix.
8.0 (2023-06-21)
================
* Drop Python 3.7 support (GL#34)
* Added a ``claimfile`` property to ``Lock`` objects (GL#30)
* Switch to ``pdm-backend`` (GL#33)
* Use ``ruff`` for linting, since its much faster. (GL#17)
* Bump dependencies.
* More GitLab CI integration improvements.
* Make ``tox.ini`` consistent with flufl defaults.
7.1.1 (2022-09-03)
==================
* Improvements to the GitLab CI integration.
7.1 (2022-08-27)
================
* Add support for Python 3.11.
* Update to pdm 1.3.
* Update all dependencies eagerly.
7.0 (2022-01-11)
================
* Fix spurious log messages when *not* breaking the lock. (GL#29)
* Use modern package management by adopting `pdm
`_ and ``pyproject.toml``, and dropping ``setup.py``
and ``setup.cfg``.
* Build the docs with Python 3.8.
* Update to version 3.0 of `Sybil `_.
* Adopt the `Furo `_ documentation theme.
* Add a favicon and logos to the published documentation.
* Use `importlib.metadata.version()
`_
as a better way to get the package version number for the documentation.
* Drop Python 3.6 support.
* Update Windows GitLab runner to include Python 3.10.
* Update copyright years.
6.0 (2021-08-18)
================
* Added a ``default_timeout`` argument to the ``Lock`` constructor, which can
be used in the context manager syntax as well. (GL#24)
* When a ``Lock`` uses a lock file that already exists and does not appear to
be a lock file (i.e. because its contents are ill-formatted), do a better
job of not clobbering that file. (GL#25)
* Improve some QA by re-adding diff-cover, Gitlab SAST during CI, and testing
on Python 3.10 beta (except for Windows)
* The ``master`` branch is renamed to ``main``. (GL#28)
5.1 (2021-05-28)
================
* Added a ``py.typed`` file to satisfy type checkers. (GL#27)
5.0.5 (2021-02-12)
==================
* I `blue `_ it!
5.0.4 (2021-01-01)
==================
* Update copyright years.
* Include ``test/__init__.py`` and ``docs/__init__.py``.
5.0.3 (2020-10-22)
==================
* Rename top-level tests/ directory to test/ (GL#26)
5.0.2 (2020-10-21)
==================
* Minor housekeeping and cleanups.
* Add some missing licensing text.
* Don't install the ``tests`` and ``docs`` directories at the top of
``site-packages`` (GL#22)
* Fix the Windows CI tests.
* Add an index to the documentation.
5.0.1 (2020-08-21)
==================
* Reorganized docs and tests out of the code directory.
* Fix Read The Docs presentation.
5.0 (2020-08-20)
================
* **Breaking change** - The following methods have been removed:
``Lock.transfer_to()``, ``Lock.take_possession()``, ``Lock.disown()``.
These were crufty, undocumented APIs used in older versions of Mailman and
were not sustainable. (GL#21)
* Added official support for Python 3.9.
* Improvements to the documentation, including a better API reference and a
"theory of operation" page that gives more implementation technical
details. (GL#20) (GL#17)
* Boosted test coverage to 100%. (GL#18)
4.0 (2020-06-30)
================
API
---
* **Breaking change** - In ``Lock.refresh()`` and ``Lock.unlock()`` the
``unconditionally`` flag is now a keyword-only argument. (GL#13)
* **Breaking change** - Removed ``Lock.__del__()`` and ``Lock.finalize()``.
It's impossible to make ``__del__()`` work properly, and this is obsoleted
by context manager protocol support anyway. Since ``finalize()`` only
existed to help with ``__del__()`` and its functionality is identical to
``.unlock(unconditionally=True)``, this method is also removed. (GL#7)
* Added a ``Lock.expiration`` property. (GL#15)
* Added a ``Lock.lockfile`` property. (GL#16)
* Added a ``Lock.state`` property and the ``LockState`` enum. (GL#12)
* In all APIs, the ``lifetime`` parameter can now also be an integer number of
seconds, in addition to the previously allowed ``datetime.timedelta``. The
``lifetime`` property always gives you a ``datetime.timedelta``.
* The API is now properly type annotated.
* Some library-defined exceptions support exception chaining.
Behavior
--------
* Getting the ``repr()`` of a lock no longer refreshes it (GL#11)
Other
-----
* Add support for Python 3.7 and 3.8; drop support for Python 3.4 and 3.5.
* We now run the test suite on both Linux and Windows.
* The LICENSE file is now included in the sdist tarball.
* API documentation is now built automatically.
* Numerous quality improvements and modernizations.
3.2 (2017-09-03)
================
* Expose the host name used in the ``.details`` property, as a property.
(Closes #4).
3.1 (2017-07-15)
================
* Expose the ``SEP`` as a public attribute. (Closes #3)
3.0 (2017-05-31)
================
* Drop Python 2.7, add Python 3.6. (Closes #2)
* Added Windows support.
* Switch to the Apache License Version 2.0.
* Use flufl.testing for nose2 and flake8 plugins.
* Allow the claim file separator to be configurable, to support file systems
where the vertical bar is problematic. Defaults to ``^`` on Windows and
``|`` everywhere else (unchanged). (Closes #1)
2.4.1 (2015-10-29)
==================
* Fix the MANIFEST.in so that tox.ini is included in the sdist.
2.4 (2015-10-10)
================
* Drop Python 2.6 compatibility.
* Add Python 3.5 compatibility.
2.3.1 (2014-09-26)
==================
* Include MANIFEST.in in the sdist tarball, otherwise the Debian package
won't built correctly.
2.3 (2014-09-25)
================
* Fix documentation bug. (LP: #1026403)
* Catch ESTALE along with ENOENT, as NFS servers are supposed to (but don't
always) throw ESTALE instead of ENOENT. (LP: #977999)
* Purge all references to ``distribute``. (LP: #1263794)
2.2.1 (2012-04-19)
==================
* Add classifiers to setup.py and make the long description more compatible
with the Cheeseshop.
* Other changes to make the Cheeseshop page look nicer. (LP: #680136)
* setup_helper.py version 2.1.
2.2 (2012-01-19)
================
* Support Python 3 without the use of 2to3.
* Make the documentation clear that the ``flufl.test.subproc`` functions are
not part of the public API. (LP: #838338)
* Fix claim file format in ``take_possession()``. (LP: #872096)
* Provide a new API for dealing with possible additional unexpected errnos
while trying to read the lock file. These can happen in some NFS
environments. If you want to retry the read, set the lock file's
``retry_errnos`` property to a sequence of errnos. If one of those errnos
occurs, the read is unconditionally (and infinitely) retried.
``retry_errnos`` is a property which must be set to a sequence; it has a
getter and a deleter too. (LP: #882261)
2.1.1 (2011-08-20)
==================
* Fixed TypeError in .lock() method due to race condition in _releasetime
property. Found by Stephen A. Goss. (LP: #827052)
2.1 (2010-12-22)
================
* Added lock.details.
2.0.2 (2010-12-19)
==================
* Small adjustment to doctest.
2.0.1 (2010-11-27)
==================
* Add missing exception to __all__.
2.0 (2010-11-26)
================
* Package renamed to flufl.lock.
Earlier
=======
Try ``bzr log lp:flufl.lock`` for details.
flufl_lock-8.2.0/docs/__init__.py 0000644 0000000 0000000 00000000000 13615410400 013610 0 ustar 00 flufl_lock-8.2.0/docs/apiref.rst 0000644 0000000 0000000 00000000625 13615410400 013514 0 ustar 00 =============
API Reference
=============
API reference for ``flufl.lock``:
.. autoclass:: flufl.lock.Lock
:members:
.. autoclass:: flufl.lock.LockState
:members:
:member-order: bysource
Exceptions
==========
.. autoexception:: flufl.lock.LockError
.. autoexception:: flufl.lock.AlreadyLockedError
.. autoexception:: flufl.lock.NotLockedError
.. autoexception:: flufl.lock.TimeOutError
flufl_lock-8.2.0/docs/conf.py 0000644 0000000 0000000 00000015266 13615410400 013022 0 ustar 00 # -*- coding: utf-8 -*-
#
# flufl.lock documentation build configuration file, created by
# sphinx-quickstart on Thu Jan 7 18:41:30 2010.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys, os
from datetime import date
import importlib.metadata
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.append(os.path.abspath('.'))
# -- General configuration -----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
]
intersphinx_mapping = {
'python': ('https://docs.python.org/3/', None),
}
autodoc_typehints = 'both'
autoclass_content = 'both'
# Add any paths that contain templates here, relative to this directory.
templates_path = ['../../_templates']
# The suffix of source filenames.
source_suffix = {'.rst': 'restructuredtext'}
# The encoding of source files.
#source_encoding = 'utf-8'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'flufl.lock'
author = 'Barry Warsaw'
copyright = f'2004-{date.today().year}, {author}'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = importlib.metadata.version(project)
# The full version, including alpha/beta/rc tags.
release = version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of documents that shouldn't be included in the build.
#unused_docs = []
# List of directories, relative to source directory, that shouldn't be searched
# for source files.
exclude_trees = ['_build', 'build', 'flufl.lock.egg-info', 'distribute-0.6.10']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
html_theme = 'furo'
html_favicon = '_static/lock-light.svg'
html_theme_options = {
'light_logo': 'logo-light.png',
'dark_logo': 'logo-dark.png',
}
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# " v documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_use_modindex = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = ''
# Output file base name for HTML help builder.
htmlhelp_basename = 'flufllockdoc'
# -- Options for LaTeX output --------------------------------------------------
# The paper size ('letter' or 'a4').
#latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
#latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('README.rst', 'flufllock.tex', 'flufl.lock Documentation',
author, 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_use_modindex = True
flufl_lock-8.2.0/docs/index.rst 0000644 0000000 0000000 00000003452 13615410400 013356 0 ustar 00 ==================================
flufl.lock - An NFS-safe file lock
==================================
This package is called ``flufl.lock``. It is an NFS-safe file-based lock with
timeouts for POSIX systems.
Requirements
============
``flufl.lock`` requires Python 3.9 or newer.
Documentation
=============
A `simple guide`_ to using the library is available, along with a detailed
`API reference`_.
Project details
===============
* Project home: https://gitlab.com/warsaw/flufl.lock
* Report bugs at: https://gitlab.com/warsaw/flufl.lock/issues
* Code hosting: https://gitlab.com/warsaw/flufl.lock.git
* Documentation: https://flufllock.readthedocs.io/
You can install it with ``pip``::
% pip install flufl.lock
You can grab the latest development copy of the code using git. The master
repository is hosted on GitLab. If you have git installed, you can grab
your own branch of the code like this::
$ git clone https://gitlab.com/warsaw/flufl.lock.git
You can contact the author via barry@python.org.
Copyright
=========
Copyright (C) 2004-2025 Barry A. Warsaw
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Table of Contents and Index
===========================
* :ref:`genindex`
.. toctree::
:glob:
using
theory
apiref
NEWS
.. _`simple guide`: using.html
.. _`API reference`: apiref.html
flufl_lock-8.2.0/docs/theory.rst 0000644 0000000 0000000 00000022206 13615410400 013557 0 ustar 00 ===================
Technical details
===================
.. currentmodule:: flufl.lock
The :doc:`flufl.lock ` package provides NFS-safe file locking. We say
"NFS-safe" because the behavior of the locks are dictated by POSIX standards,
and are guaranteed to work even when the processes involved are running on
different machines which only share a common file system over `NFS`_.
The basic implementation of our :class:`Lock` objects is described by the
GNU/Linux `open(2)`_ manpage, under the description of the ``O_EXCL`` option:
[...] O_EXCL is broken on NFS file systems, programs which rely on it for
performing locking tasks will contain a race condition. The solution for
performing atomic file locking using a lockfile is to create a unique file
on the same fs (e.g., incorporating hostname and pid), use link(2) to make
a link to the lockfile. If link() returns 0, the lock is successful.
Otherwise, use stat(2) on the unique file to check if its link count has
increased to 2, in which case the lock is also successful.
The assumption here is that there will be no *outside interference*, e.g. no
agent external to this code will ever `link()`_ to the specific lock files
used.
Lock files and claim files
==========================
When a :class:`Lock` object is instantiated, the user names a file system
path in the constructor. This is the file that all processes will synchronize
on when attempting to acquire the lock. We call this the *lock file*. This
file should not already exist.
Locks have a *lifetime* which is the period of time that the process expects
to keep the lock, once it has been acquired. This lifetime is used to
calculate whether a lock is stale, i.e. the process that acquired the lock
exited without cleanly unlocking it. Stale locks can be broken by another
process.
It's good to use a reasonable lifetime, not too long and not too short. If
the lifetime is too long, then stale locks may not be broken for quite some
time. Too short and another process will break a lock that isn't stale. Keep
in mind though that lock lifetimes can be explicitly extended, and in some
cases will be implicitly extended. So if you're using the lock to protect a
resource for a long time, try to periodically refresh the lock after you've
acquired it, and set the lifetime to just longer than your expected refresh
period.
Internally, every :class:`Lock` object also has a *claim file*. This is a
file system path unique to the object which will be hard linked to the lock
file. The claim file isn't actually created until the process attempts to
acquire the lock. The name of the claim file is made up of several bits of
data intended to make this claim file name unique across all :class:`Lock`
object. The claim file name consists of:
* The name of the lock file
* The current hostname as returned by :func:`socket.getfqdn()`
* The current process id
* A random integer from 0 to :data:`sys.maxsize`
Once defined, the claim file name never changes. The claim file only ever
actually exists while the process is acquiring or has acquired the lock,
although it may hang around in an unlinked state if the lock has been broken.
Acquiring the lock
==================
These are the steps for acquiring a lock:
* The user defines a *timeout* period, which is the length of time in seconds
that the lock acquisition will be attempted. If the lock cannot be acquired
within this time period, a :class:`TimeOutError` is raised.
* The claim file is created. The name of the claim file is written to the
contents of the claim file. The reason for this will be evident later.
* The claim file is *touched*. Touching the file means that the file's
``atime`` and ``mtime`` are modified to point to a timestamp some time in the
future. This future time is the current time plus the lock's lifetime, and
it indicates the the time in the future after which the lock will be
considered stale.
* A `hard link`_ is created with the claim file as the source and the lock file
as the destination. This makes sense: the claim file has already been
created (see above), and if no other process has acquired the lock, then
this step creates the lock file. On success, the lock file and claim file
will point to the same physical file and the lock will be considered
acquired.
* If the hard link is successful, the lock's lifetime is immediately
refreshed to ensure it represents the lifetime plus the timestamp when the
lock was acquired. This is because if a process needed to wait some amount
of time for another process to release the lock, you want the current lock's
lifetime to be relative to the actual lock acquisition time.
There are several reasons why the hard link attempt could fail. If any of
these conditions occur, the :meth:`Lock.lock()` method will sleep for a short
amount of time and try again, assuming the timeout period hasn't expired, and
that the lock isn't considered stale.
* The *link count* of the lock file is not 2. The only way this can happen is
if some outside agent is messing with us. That weird state is logged, but
the lock acquisition attempt continues after a sleep.
* Some weird, but innocuous exception can occur, such as an ``ENOENT`` or an
``ESTALE``. NFS servers can return these errnos depending on the platform.
As above, when this happens, the lock acquisition just sleeps for a short
time and tries again.
* Any other :class:`OSError` will get logged, the claim file will be unlinked
and the exception will get re-raised immediately.
* If the contents of lock file is equal to the claim file name, then this
indicates that the current process *already* has the lock acquired, and a
:class:`AlreadyLockedError` is raised. This is why we write the claim file
name into the file.
Breaking the lock
=================
Let's say the linking operation fails because :meth:`Lock.lock()` determines
that another process has acquired the lock. The next thing that happens is a
check for lock staleness. Remember that a lock is defined to be stale if the
timestamp recorded in its ``atime`` and ``mtime`` have been exceeded. In this
case, the lock will be broken. This is also known as *stealing the lock*.
When breaking a lock, the lock file is first removed. This is technically the
act that breaks the lock. However, before removing the lock file, its
contents are read to find the other process's claim file path. This claim
file, if it exists, is also removed. This should leave the file system in a
fairly clean state, even if process that used to own the lock has exited
uncleanly.
There are a few caveats to keep in mind.
First, it's not technically possible to determine whether another active,
running process actually owns the lock. This is especially true if the locks
are acquired on different machines coordinating through a shared (e.g. NFS)
file system. However you can gain some insight into the state of a lock by
using the :data:`Lock.state` attribute and checking the :class:`LockState`
enum that's returned. This state is *not* consulted when deciding whether to
break a lock though; in that case only the lock's lifetime is considered.
There is a small race condition window present when breaking the lock. It's
possible that two processes are blocked on acquiring a stale lock that was
previously acquired by a now-defunct third process. To reduce this (it cannot
be totally eliminated), a process breaking the lock starts by *touching* the
lock, thus simulating an extension of the lock's lifetime. If the second
process looks at the expiration time, it may in fact see that the lock has
*not* yet expired, closing the race window. Ultimately, even if this fails,
the lock is still guaranteed to be acquired by only one process, because the
link creation operation is guaranteed to be atomic by POSIX.
Lifetime extension
==================
As mentioned, processes can explicitly extend the lifetime of a lock it owns,
which can be used to ensure continuous ownership of resources in long running
tasks. There are some cases where the :class:`Lock` implementation implicitly
extends the lifetime of a lock:
* When first attempting to acquire a lock
* After a lock is acquired
* When asking whether a lock is acquired through the :data:`Lock.is_locked`
property. This prevents a lock breaking race condition between the time the
property is accessed and the answer is returned.
* When a lock is about to be broken
Generally, you don't need to worry about implicit lifetime extension since the
only user-facing case is when accessing :data:`Lock.is_locked`.
Locks always have a default lifetime, but this can be set in the :class:`Lock`
constructor. It can also be optionally changed in the :meth:`Lock.refresh()`
method. In any case, the lifetime always represents the number of seconds
into the future at which the lock will expire, relative to "now". "Now" is
always calculated when the lock is refreshed, either explicitly or implicitly.
.. _`open(2)`: http://manpages.ubuntu.com/manpages/dapper/en/man2/open.2.html
.. _`link()`: https://manpages.ubuntu.com/manpages/focal/en/man2/link.2.html
.. _`NFS`: https://en.wikipedia.org/wiki/Network_File_System
.. _`hard link`: https://en.wikipedia.org/wiki/Hard_link
flufl_lock-8.2.0/docs/using.rst 0000644 0000000 0000000 00000022326 13615410400 013375 0 ustar 00 ============================
Using the flufl.lock library
============================
.. currentmodule:: flufl.lock
The :doc:`flufl.lock ` package provides safe file locking with
timeouts for POSIX and Windows systems. You can read more about how the
library works in the :doc:`technical details `.
Lock objects support lock-breaking so that you can't wedge a process forever.
Locks have a *lifetime*, which is the maximum length of time the process
expects to retain the lock. It is important to pick a good number here
because other processes will not break an existing lock until the expected
lifetime has expired. Too long and other processes will hang; too short and
you'll end up trampling on existing process locks -- and possibly corrupting
data. However, lock lifetimes can be explicitly extended, and are implicitly
extended in some cases.
In a distributed (NFS) environment, you also need to make sure that your
clocks are properly synchronized.
Creating a lock
===============
To create a lock, you must first instantiate a :class:`Lock` object,
specifying the path to a file that will be used to synchronize the lock. This
file should not already exist.
::
# This function comes from the test infrastructure.
>>> filename = temporary_lockfile()
>>> from flufl.lock import Lock
>>> lock = Lock(filename)
>>> lock
Locks have a default lifetime...
>>> lock.lifetime.seconds
15
...which you can change.
>>> from datetime import timedelta
>>> lock.lifetime = timedelta(seconds=30)
>>> lock.lifetime.seconds
30
>>> lock.lifetime = timedelta(seconds=15)
You can ask whether the lock is acquired or not.
>>> lock.is_locked
False
Acquiring the lock is easy if no other process has already acquired it.
>>> lock.lock()
>>> lock.is_locked
True
Once you have the lock, it's easy to release it.
>>> lock.unlock()
>>> lock.is_locked
False
It is an error to attempt to acquire the lock more than once in the same
process.
::
>>> from flufl.lock import AlreadyLockedError
>>> lock.lock()
>>> try:
... lock.lock()
... except AlreadyLockedError as error:
... print(error)
We already had the lock
>>> lock.unlock()
Lock objects also support the context manager protocol.
>>> lock.is_locked
False
>>> with lock:
... lock.is_locked
True
>>> lock.is_locked
False
Lock acquisition can block
==========================
When trying to lock the file when the lock is unavailable (because another
process has already acquired it), the lock call will block.
::
>>> import time
>>> t0 = time.time()
# This function comes from the test infrastructure.
>>> acquire(filename, lifetime=5)
>>> lock.lock()
>>> t1 = time.time()
>>> lock.unlock()
>>> t1 - t0 > 4
True
Refreshing a lock
=================
A process can *refresh* a lock if it realizes that it needs to hold the lock
for a little longer. You cannot refresh an unlocked lock.
>>> from flufl.lock import NotLockedError
>>> try:
... lock.refresh()
... except NotLockedError as error:
... print(error)
>> from datetime import datetime
>>> lock.lifetime = 2 # seconds
>>> lock.lock()
>>> lock.is_locked
True
After the current lifetime expires, the lock is stolen from the parent process
even if the parent never unlocks it.
::
# This function comes from the test infrastructure.
>>> t_broken = waitfor(filename, lock.lifetime)
>>> lock.is_locked
False
However, if the process holding the lock refreshes it, it will hold it can
hold it for as long as it needs.
>>> lock.lock()
>>> lock.refresh(5) # seconds
>>> t_broken = waitfor(filename, lock.lifetime)
>>> lock.is_locked
False
Time outs
=========
When attempting to acquire a lock, you can specify a timeout interval as
either an integer number of seconds, or as a :class:`datetime.timedelta`.
If the lock is not acquired within this interval, a :class:`TimeOutError` is
raised.
You can specify a default timeout interval in the :class:`Lock` constructor.
>>> from flufl.lock import TimeOutError
>>> acquire(filename, lifetime=5)
>>> try:
... with Lock(filename, default_timeout=1) as my_lock:
... pass
... except TimeOutError:
... print('Timed out, as expected')
Timed out, as expected
You can also specify a timeout interval in the :func:`Lock.lock` call. This
overrides the constructor argument.
>>> acquire(filename, lifetime=5)
>>> my_lock = Lock(filename, default_timeout=1)
>>> try:
... my_lock.lock(timeout=10)
... my_lock.is_locked
... finally:
... my_lock.unlock()
True
Lock details
============
Lock files are written with unique contents that can be queried for
information about the host name the lock was acquired on, the id of the
process that acquired the lock, and the path to the lock file.
>>> import os
>>> lock.lock()
>>> hostname, pid, lockfile = lock.details
>>> hostname == lock.hostname
True
>>> pid == os.getpid()
True
>>> lockfile == filename
True
>>> lock.unlock()
Even if another process has acquired the lock, the details can be queried.
>>> acquire(filename, lifetime=3)
>>> lock.is_locked
False
>>> hostname, pid, lockfile = lock.details
>>> hostname == lock.hostname
True
>>> pid == os.getpid()
False
>>> lockfile == filename
True
However, if no process has acquired the lock, the details are unavailable.
>>> lock.lock()
>>> lock.unlock()
>>> try:
... lock.details
... except NotLockedError as error:
... print(error)
Details are unavailable
You can also get the time at which the lock will expire.
>>> now = datetime.now()
>>> import time
>>> time.sleep(1)
>>> with lock:
... lock.refresh()
... lock.expiration > now + lock.lifetime
True
Lock state
==========
You might want to try to infer the state of the lock. This is not always
possible, but this library does try to provide some insights into the lock's
state. However, it is up to the user of the library to enforce policy based
on the lock state.
The lock state is embodied in an enumeration.
>>> from flufl.lock import LockState
The lock can be in the unlocked state.
>>> lock.state
We could own the lock, as long as it is still fresh (i.e. it hasn't expired
its lifetime yet), the state will tell us.
>>> with lock:
... lock.state
It's possible that we own the lock, but that its lifetime has expired. In
this case, another process trying to acquire the lock will break the original
lock.
>>> lock.lifetime = 1
>>> with lock:
... time.sleep(1.5)
... lock.state
It's also possible that another process once owned the lock but it exited
uncleanly. If the lock file still exists, but there is no process running
that matches the recorded pid, then the lock's state is stale.
>>> acquire(lock.lockfile, lifetime=10)
>>> simulate_process_crash(lock.lockfile)
>>> lock.state
If some other process owns the lock, we can't really infer much about it.
while we can see that there is a running process matching the pid in the lock
file, we don't know whether that process is really the one claiming the lock,
or what its intent with the lock is.
::
# This function comes from the test infrastructure.
>>> acquire(lock.lockfile, lifetime=2, extra_sleep=3)
>>> lock.state
However, once the lock has expired, we can at least report that.
>>> time.sleep(2)
>>> lock.state
Lock file separator
===================
Lock claim file names contain useful bits of information concatenated by a
*separator character*. This character is the caret (``^``) by default on
Windows and the vertical bar (``|``) by default everywhere else. You can
change this character. There are some restrictions:
* It cannot be an alphanumeric;
* It cannot appear in the host machine's fully qualified domain name
(e.g. the value of :data:`Lock.hostname`);
* It cannot appear in the lock's file name (the argument passed to the
:class:`Lock` constructor)
It may also be helpful to avoid `any reserved characters
`_ on
the file systems where you intend to run the code.
You can also get both the lock file and claim file names from the lock object.
>>> lock = Lock(filename, separator='+')
>>> lock.lock()
>>> hostname, pid, lockfile = lock.details
>>> hostname == lock.hostname
True
>>> pid == os.getpid()
True
>>> lockfile == filename
True
>>> lock.lockfile == lockfile
True
The claim file name is also stored in the contents of any acquired lock file.
>>> with open(filename) as fp:
... claimfile = fp.read().strip()
>>> lock.claimfile == claimfile
True
>>> '+' in claimfile
True
>>> lock.unlock()
flufl_lock-8.2.0/docs/_static/lock-dark.svg 0000644 0000000 0000000 00000000421 13615410400 015524 0 ustar 00 flufl_lock-8.2.0/docs/_static/lock-light.svg 0000644 0000000 0000000 00000000521 13615410400 015713 0 ustar 00 flufl_lock-8.2.0/docs/_static/logo-dark.png 0000644 0000000 0000000 00000003142 13615410400 015524 0 ustar 00 PNG
IHDR P / pHYs od tEXtSoftware www.inkscape.org< IDATx]Tef+[IJYQz}MX]DhP%HTD^^FD]E
AhjA!AW;k❕uw33.}<9 & +iKZ i^IK:)7I%!o|M0 D 6s^O :@AhzEn S-@OF\_zs8RGq9 ,*zݹ {xtO~
R jx@qTuz|)FTY=ItZ?ْnLzq*B_WM1O 31X"N7k7L#ٶ
\g71<ל+ 9,1
l+| A:2 ,ʼnzC'p.C + >Ha[!@"^H
gG|66*$ I p!v҇ʧ"Qb%b %5 蠇ΰ w S
@)6E 5 `U@g2n^ibQ&a; ۔>
!\iDDᲤ݉iw |W#nF 7$iZ!6*fPzR-IwrhqF\hqF\hqF\hqF\hqF\hqF\hqF\hqF\hqF\hi4XiBdRBSRo %|Z[۔~#9%ӊ<?٦/ SrJe(y}bZW]*i\K )U%%Fd '俼!'LRߜA '7\q#e-W_3:( a%m+l%Ż!ץKگ |?9 ie$]|lCϕ8+C .w%{\K3?$=6\4Är$}ߠZ{ޱc> !\PySR_}skj%w~ YΕCsZ[[F$}dմAnWlR2I+AsTl|Ja+Y{0i{7 IENDB` flufl_lock-8.2.0/docs/_static/logo-dark.svg 0000644 0000000 0000000 00000002371 13615410400 015542 0 ustar 00
flufl_lock-8.2.0/docs/_static/logo-light.png 0000644 0000000 0000000 00000002426 13615410400 015716 0 ustar 00 PNG
IHDR P / pHYs od tEXtSoftware www.inkscape.org< IDATx;TeߎИ d
%(0XIʠFMD B2#Rh 3"
0awEo7՝;3glfv?ޛΌƱ
wc3c%V݊:̽yZ加` O` ٣-ofY k艕mn; [GE-N՟yzJnI me]i\Ծ
TEMzVw'b+IZ7Pn÷Ce)$}+Uj oxp rJUkǙ+6u]/͕vi\\ir b,3 :Gc&IAR`$IAR`$IAR`$IAR`$IAR`$IAR`$IAR`$IAYq Wp}~^\o^Yi_bDlĻۦdc
/kAqbi6!ħX"{'NA?`K
)R|ШW B+~jF|p=vfv+Ի zY7\a̚Qf
kZY帯\&Hzc~Q5r^Hޑ )0H
)0H
)0H
)0H
)0H
)0H
)0H
)0H
)0H
-3E-'d8aFIo niGƔILaIp1^ JǢ o'2*1\n+= O^s \t)k 8r2CPƏX;Y(]ʚ.vn7*|2EK Ԧ|,KT i19͜c#J{qnV zϙ6q'n; IENDB` flufl_lock-8.2.0/docs/_static/logo-light.svg 0000644 0000000 0000000 00000002434 13615410400 015730 0 ustar 00
flufl_lock-8.2.0/src/flufl/lock/__init__.py 0000644 0000000 0000000 00000000705 13615410400 015523 0 ustar 00 from public import public as _public
from flufl.lock._lockfile import (
AlreadyLockedError,
Lock,
LockError,
LockState,
NotLockedError,
SEP,
TimeOutError,
)
__version__ = '8.2.0'
_public(
AlreadyLockedError=AlreadyLockedError,
Lock=Lock,
LockError=LockError,
LockState=LockState,
NotLockedError=NotLockedError,
SEP=SEP,
TimeOutError=TimeOutError,
__version__=__version__,
)
del _public
flufl_lock-8.2.0/src/flufl/lock/_lockfile.py 0000644 0000000 0000000 00000062416 13615410400 015722 0 ustar 00 from __future__ import annotations # noqa I001: Import block is un-sorted or un-formatted
import os
import sys
import time
import errno
import random
import socket
import logging
from contextlib import suppress
from datetime import datetime, timedelta
from enum import Enum
from logging import NullHandler
from typing import Final, Literal, Union, cast, TYPE_CHECKING
from psutil import pid_exists
from public import public
if TYPE_CHECKING:
from types import TracebackType
Interval = Union[timedelta, int]
DEFAULT_LOCK_LIFETIME: Final = timedelta(seconds=15)
# Allowable a bit of clock skew.
CLOCK_SLOP: Final = timedelta(seconds=10)
MAXINT: Final = sys.maxsize
# Details separator; also used in calculating the claim file path. Lock files
# should not include this character.
SEP: Final = '^' if sys.platform == 'win32' else '|'
public(SEP=SEP)
# LP: #977999 - catch both ENOENT and ESTALE. The latter is what an NFS
# server should return, but some Linux versions return ENOENT.
ERRORS: Final = (errno.ENOENT, errno.ESTALE)
log = logging.getLogger('flufl.lock')
# Install a null handler to avoid warnings when applications don't set their
# own flufl.lock logger. See http://docs.python.org/library/logging.html
logging.getLogger('flufl.lock').addHandler(NullHandler())
@public
class LockError(Exception):
"""Base class for all exceptions in this module."""
@public
class AlreadyLockedError(LockError):
"""An attempt is made to lock an already locked object."""
@public
class NotLockedError(LockError):
"""An attempt is made to unlock an object that isn't locked."""
@public
class TimeOutError(LockError):
"""The timeout interval elapsed before the lock succeeded."""
@public
class LockState(Enum):
"""Infer the state of the lock.
There are cases in which it is impossible to infer the state of the lock,
due to the distributed nature of the locking mechanism and environment.
However it is possible to provide some insights into the state of the
lock. Note that the policy on what to do with this information is left
entirely to the user of the library.
"""
#: There is no lock file so the lock is unlocked.
unlocked = 1
#: We own the lock and it is fresh.
ours = 2
#: We own the lock but it has expired. Another process trying
#: to acquire the lock will break it.
ours_expired = 3
#: We don't own the lock; the hostname in the details matches our
#: hostname and there is no pid running that matches pid. Therefore,
#: the lock is stale.
stale = 4
#: Some other process owns the lock (probably) but it has expired. Another
#: process trying to acquire the lock will break it.
theirs_expired = 5
#: We don't own the lock; either our hostname does not match the
#: details, or there is a process (that's not us) with a matching pid.
#: The state of the lock is therefore unknown.
unknown = 6
def _interval_to_datetime(
timeout: Interval | None = None,
) -> datetime | None:
if timeout is None:
return None
if isinstance(timeout, int):
timeout = timedelta(seconds=timeout)
return datetime.now() + timeout
@public
class Lock:
"""Portable, NFS-safe file locking with timeouts for POSIX systems.
This class implements an NFS-safe file-based locking algorithm
influenced by the GNU/Linux open(2) manpage, under the description
of the O_EXCL option:
[...] O_EXCL is broken on NFS file systems, programs which rely on it
for performing locking tasks will contain a race condition. The
solution for performing atomic file locking using a lockfile is to
create a unique file on the same fs (e.g., incorporating hostname and
pid), use link(2) to make a link to the lockfile. If link() returns
0, the lock is successful. Otherwise, use stat(2) on the unique file
to check if its link count has increased to 2, in which case the lock
is also successful.
The assumption made here is that there will be no *outside interference*,
e.g. no agent external to this code will ever link() to the specific lock
files used.
The user specifies a *lock file* in the constructor of this class. This
is whatever file system path the user wants to coordinate locks on. When
a process attempts to acquire a lock, it first writes a *claim file* which
contains unique details about the lock being acquired (e.g. the lock file
name, the hostname, the pid of the process, and a random integer). Then
it attempts to create a hard link from the claim file to the lock file.
If no other process has the lock, this hard link succeeds and the process
accquires the lock. If another process already has the lock, the hard
link will fail and the lock will not be acquired. What happens in this
and other error conditions are described in the more detailed
documentation.
Lock objects support lock-breaking so that you can't wedge a process
forever. This is especially helpful in a web environment, but may
not be appropriate for all applications.
Locks have a ``lifetime``, which is the maximum length of time the
process expects to retain the lock. It is important to pick a good
number here because other processes will not break an existing lock
until the expected lifetime has expired. Too long and other
processes will hang; too short and you'll end up trampling on
existing process locks -- and possibly corrupting data. However
locks also support extending a lock's lifetime. In a distributed
(NFS) environment, you also need to make sure that your clocks are
properly synchronized.
Each process laying claim to this resource lock will create their own
temporary lock file based on the path specified. An optional lifetime
is the length of time that the process expects to hold the lock.
:param lockfile: The full path to the lock file.
:param lifetime: The expected maximum lifetime of the lock, as a
timedelta or integer number of seconds, relative to now. Defaults
to 15 seconds.
:param separator: The separator character to use in the lock file's
file name. Defaults to the vertical bar (`|`) on POSIX systems
and caret (`^`) on Windows.
:param default_timeout: Default timeout for approximately how long the lock
acquisition attempt should be made. The value given in the `.lock()`
call always overrides this.
"""
def __init__(
self,
lockfile: str,
lifetime: Interval | None = None,
separator: str = SEP,
default_timeout: Interval | None = None,
):
"""Create the resource lock using the given file name and lifetime."""
# The hostname has to be defined before we call _set_claimfile().
self._hostname = socket.getfqdn()
if lifetime is None:
lifetime = DEFAULT_LOCK_LIFETIME
self._default_timeout = default_timeout
self._lockfile = lockfile
# Mypy does not understand that setters can accept wider types than the getter is defined to
# return. We're fine, so just ignore the complaint.
self.lifetime = lifetime # type: ignore[assignment]
# The separator must be set before we claim the lock.
self._separator = separator
self._claimfile: str
self._set_claimfile()
# For extending the set of NFS errnos that are retried in _read().
self._retry_errnos: list[int] = []
def __repr__(self) -> str:
return '<{} {} [{}: {}] pid={} at {:#x}>'.format(
self.__class__.__name__,
self._lockfile,
('locked' if self._is_locked_no_refresh() else 'unlocked'),
self._lifetime,
os.getpid(),
id(self),
)
@property
def hostname(self) -> str:
"""The current machine's host name.
:return: The current machine's hostname, as used in the `.details`
property.
"""
return self._hostname
@property
def details(self) -> tuple[str, int, str]:
"""Details as read from the lock file.
:return: A 3-tuple of hostname, process id, lock file name.
:raises NotLockedError: if the lock is not acquired.
"""
try:
with open(self._lockfile) as fp:
filename = fp.read().strip()
except OSError as error:
if error.errno in ERRORS:
raise NotLockedError('Details are unavailable') from error
raise
# Rearrange for signature.
try:
lockfile, hostname, pid, random_ignored = filename.split(self._separator)
except ValueError as error:
raise NotLockedError('Details are unavailable') from error
return hostname, int(pid), lockfile
@property
def state(self) -> LockState:
"""Infer the state of the lock."""
try:
with open(self._lockfile) as fp:
filename = fp.read().strip()
except FileNotFoundError:
return LockState.unlocked
try:
lockfile, hostname, pid_str, random_ignored = filename.split(self._separator)
pid = int(pid_str)
except (ValueError, TypeError):
# The contents of the lock file is corrupt, so we can't know
# anything about the state of the lock.
return LockState.unknown
if hostname != self._hostname:
return LockState.unknown
if pid == os.getpid():
expired = self.expiration < datetime.now()
return LockState.ours_expired if expired else LockState.ours
if pid_exists(pid):
expired = self.expiration < datetime.now()
return LockState.theirs_expired if expired else LockState.unknown
return LockState.stale
@property
def lifetime(self) -> timedelta:
"""The current lock life time."""
return self._lifetime
@lifetime.setter
def lifetime(self, lifetime: Interval) -> None:
if isinstance(lifetime, timedelta):
self._lifetime = lifetime
else:
self._lifetime = timedelta(seconds=lifetime)
def refresh(
self,
lifetime: Interval | None = None,
*,
unconditionally: bool = False,
) -> None:
"""Refresh the lifetime of a locked file.
Use this if you realize that you need to keep a resource locked longer
than you thought.
:param lifetime: If given, this sets the lock's new lifetime. It
represents the number of seconds into the future that the
lock's lifetime will expire, relative to now, or whenever it is
refreshed, either explicitly or implicitly. If not given, the
original lifetime interval is used.
:param unconditionally: When False (the default), a ``NotLockedError``
is raised if an unlocked lock is refreshed.
:raises NotLockedError: if the lock is not set, unless optional
``unconditionally`` flag is set to True.
"""
if lifetime is not None:
# Mypy does not understand that setters can accept wider types than the getter is
# defined to return. We're fine, so just ignore the complaint.
self.lifetime = lifetime # type: ignore[assignment]
# Do we have the lock? As a side effect, this refreshes the lock!
if not self.is_locked and not unconditionally:
# EM102 Exception must not use an f-string literal, assign to variable first
msg = f'{self!r}: {self._read()}'
raise NotLockedError(msg)
def lock(self, timeout: Interval | None = None) -> None:
"""Acquire the lock.
This blocks until the lock is acquired unless optional timeout is not
None, in which case a ``TimeOutError`` is raised when the timeout
expires without lock acquisition.
:param timeout: Approximately how long the lock acquisition attempt
should be made. None (the default) means keep trying forever.
:raises AlreadyLockedError: if the lock is already acquired.
:raises TimeOutError: if ``timeout`` is not None and the indicated
time interval expires without a lock acquisition.
"""
timeout_time = _interval_to_datetime(self._default_timeout if timeout is None else timeout)
# Make sure the claim file exists, and that its contents are current.
self._write()
# 2025-05-08(warsaw): Is the following comment still relevant?
#
# This next call can fail with an EPERM. I have no idea why, but I'm nervous about ignoring
# it. It seems to be a very rare occurrence, only happens from cron, and has only(?) been
# observed on Solaris 2.6.
self._touch()
log.debug('laying claim: %s', self._lockfile)
# For quieting the logging output.
loopcount = -1
while True:
loopcount += 1
# Create the hard link from the claim file to the lock file and
# test for a hard count of exactly 2 links.
try:
os.link(self._claimfile, self._lockfile)
# If we got here, we know we got the lock, and never
# had it before, so we're done. Just touch it again for the
# fun of it.
log.debug('got the lock: %s', self._lockfile)
self._touch()
break
except OSError as error:
# The link failed for some reason, possibly because someone
# else already has the lock (i.e. we got an EEXIST), or for
# some other bizarre reason.
if error.errno in ERRORS:
# 2025-05-08(warsaw): Is the following comment still relevant?
#
# In some Linux environments, it is possible to get an ENOENT, which is truly
# strange, because this means that self._claimfile didn't exist at the time of
# the os.link(), but self._write() is supposed to guarantee that this happens!
# I don't honestly know why this happens -- possibly due to weird caching file
# systems? For now we just say we didn't acquire the lock and try again next
# time.
pass
elif error.errno != errno.EEXIST:
# Something very bizarre happened. Clean up our state and
# pass the error on up.
log.exception('unexpected link')
os.unlink(self._claimfile)
raise
elif self._linkcount != 2:
# Somebody's messin' with us! Log this, and try again later. Should we raise
# an exception?
log.error('unexpected linkcount: %s', self._linkcount)
elif self._read() == self._claimfile:
# It was us that already had the link.
log.debug('already locked: %s', self._lockfile)
raise AlreadyLockedError('We already had the lock') from None
# Otherwise, someone else has the lock. The explicit pass below isn't needed but
# included for readability.
pass # noqa PIE790 Unnecessary `pass` statement
# We did not acquire the lock, because someone else already has
# it. Have we timed out in our quest for the lock?
if timeout_time is not None and timeout_time < datetime.now():
os.unlink(self._claimfile)
log.error('timed out')
raise TimeOutError('Could not acquire the lock')
# Okay, we haven't timed out, but we didn't get the lock. Let's
# find out if the lock lifetime has expired. Cache the release
# time to avoid race conditions. (LP: #827052)
release_time = self._releasetime
if release_time != -1:
now = datetime.now()
future = cast(datetime, release_time) + CLOCK_SLOP
if now > future:
# Yes, so break the lock.
log.error('lifetime has expired, breaking')
self._break()
# Okay, someone else has the lock, our claim hasn't timed out yet,
# and the expected lock lifetime hasn't expired yet either. So
# let's wait a while for the owner of the lock to give it up.
elif not loopcount % 100:
log.debug('waiting for claim: %s', self._lockfile)
self._sleep()
# 2020-06-27(bwarsaw): `unconditionally` should really be a keyword-only
# argument, but those didn't exist when this library was originally
# written, and changing that now would be a needless backward incompatible
# API break.
def unlock(self, *, unconditionally: bool = False) -> None:
"""Release the lock.
:param unconditionally: When False (the default), a ``NotLockedError``
is raised if this is called on an unlocked lock.
:raises NotLockedError: if we don't own the lock, either because of
unbalanced unlock calls, or because the lock was stolen out from
under us, unless optional ``unconditionally`` is True.
Note that successfully unlocking the lock also unlinks the claim file,
even if it is already unlocked and ``unconditionally`` is True.
"""
is_locked = self.is_locked
if not is_locked and not unconditionally:
raise NotLockedError('Already unlocked')
# If we owned the lock, remove the lockfile, relinquishing the lock.
if is_locked:
try:
os.unlink(self._lockfile)
except OSError as error:
if error.errno not in ERRORS:
raise
# Remove our claim file.
try:
os.unlink(self._claimfile)
except OSError as error:
if error.errno not in ERRORS:
raise
log.debug('unlocked: %s', self._lockfile)
def _is_locked_no_refresh(self) -> bool:
"""Don't refresh the lock, just return the status."""
# Can the link count ever be > 2?
if self._linkcount != 2:
return False
return self._read() == self._claimfile
@property
def is_locked(self) -> bool:
"""True if we own the lock, False if we do not.
Checking the status of the lock resets the lock's lifetime, which
helps avoid race conditions during the lock status test.
"""
# Discourage breaking the lock for a while.
try:
self._touch()
except PermissionError:
# We can't touch the file because we're not the owner. I don't see
# how we can own the lock if we're not the owner.
log.error('No permission to refresh the log')
return False
return self._is_locked_no_refresh()
def __enter__(self) -> Lock:
self.lock()
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> Literal[False]:
self.unlock()
# Don't suppress any exception that might have occurred.
return False
def _set_claimfile(self) -> None:
"""Set the _claimfile private variable."""
# Calculate a hard link file name that will be used to lay claim to
# the lock. We need to watch out for two Lock objects in the same
# process pointing to the same lock file. Without this, if you lock
# lf1 and do not lock lf2, lf2.locked() will still return True.
self._claimfile = self._separator.join(
(
self._lockfile,
self.hostname,
str(os.getpid()),
str(random.randint(0, MAXINT)),
)
)
def _write(self) -> None:
"""Write our claim file's name to the claim file."""
# Make sure it's group writable.
with open(self._claimfile, 'w') as fp:
fp.write(self._claimfile)
@property
def retry_errnos(self) -> list[int]:
"""The set of errno values that cause a read retry."""
return self._retry_errnos[:]
@retry_errnos.setter
def retry_errnos(self, errnos: list[int]) -> None:
self._retry_errnos = []
self._retry_errnos.extend(errnos)
@retry_errnos.deleter
def retry_errnos(self) -> None:
self._retry_errnos = []
@property
def expiration(self) -> datetime:
"""The lock's expiration time, regardless of ownership."""
return datetime.fromtimestamp(os.stat(self._lockfile).st_mtime)
@property
def lockfile(self) -> str:
"""Return the lock file name."""
return self._lockfile
@property
def claimfile(self) -> str:
"""Return the claim file name."""
return self._claimfile
def _read(self) -> str | None:
"""Read the contents of our lock file.
:return: The contents of the lock file or None if the lock file does
not exist.
"""
while True:
try:
with open(self._lockfile) as fp:
return fp.read()
except OSError as error:
if error.errno in self._retry_errnos:
self._sleep()
elif error.errno not in ERRORS:
raise
else:
return None
def _touch(self, filename: str | None = None) -> None:
"""Touch the claim file into the future.
:param filename: If given, the file to touch, otherwise our claim file
is touched.
"""
expiration_date = datetime.now() + self._lifetime
t = time.mktime(expiration_date.timetuple())
try:
# We probably don't need to modify atime, but this is easier.
os.utime(filename or self._claimfile, (t, t))
except OSError as error:
if error.errno not in ERRORS:
raise
@property
def _releasetime(self) -> int | datetime:
"""The time when the lock should be released.
:return: The mtime of the file, which is when the lock should be
released, or -1 if the lockfile doesn't exist.
"""
try:
return self.expiration
except OSError as error:
if error.errno in ERRORS:
return -1
raise
@property
def _linkcount(self) -> int:
"""The number of hard links to the lock file.
:return: the number of hard links to the lock file, or -1 if the lock
file doesn't exist.
"""
try:
return os.stat(self._lockfile).st_nlink
except OSError as error:
if error.errno in ERRORS:
return -1
raise
def _break(self) -> None:
"""Break the lock."""
# Try to read from the lock file. All we care about is that its contents have the details
# expected of any lock file. If not, then this probably isn't a lock that needs breaking,
# it's a Lock with a lock file pointing to an existing, unrelated file. Refuse to break
# that lock. All we really need to do is to log and return. If a timeout was given,
# eventually the .lock() call will timeout. However if no timeout was given, the .lock()
# will block forever.
#
# Discard the return value but keep the linter happy.
try:
_ = self.details
except NotLockedError:
log.error("lockfile exists but isn't safe to break: %s", self._lockfile)
return
# Touch the lock file. This reduces but does not eliminate the
# chance for a race condition during breaking. Two processes could
# both pass the test for lock expiry in lock() before one of them gets
# to touch the lock file. This shouldn't be too bad because all
# they'll do in that function is delete the lock files, not claim the
# lock, and we can be defensive for ENOENTs here.
#
# Touching the lock could fail if the process breaking the lock and
# the process that claimed the lock have different owners. Don't do
# that.
with suppress(PermissionError):
self._touch(self._lockfile) # pragma: nocover
# Get the name of the old winner's claim file.
winner = self._read()
# Remove the global lockfile, which actually breaks the lock.
try:
os.unlink(self._lockfile)
except OSError as error: # pragma: nocover
if error.errno not in ERRORS:
raise
# Try to remove the old winner's claim file, since we're assuming the
# winner process has hung or died. Don't worry too much if we can't
# unlink their claim file -- this doesn't wreck the locking algorithm,
# but will leave claim file turds laying around, a minor inconvenience.
try:
if winner: # pragma: nobranch
os.unlink(winner)
except OSError as error:
if error.errno not in ERRORS: # pragma: nocover
raise
def _sleep(self) -> None:
"""Snooze for a random amount of time."""
interval = random.random() * 2.0 + 0.01
time.sleep(interval)
flufl_lock-8.2.0/src/flufl/lock/py.typed 0000644 0000000 0000000 00000000000 13615410400 015075 0 ustar 00 flufl_lock-8.2.0/tests/__init__.py 0000644 0000000 0000000 00000000000 13615410400 014022 0 ustar 00 flufl_lock-8.2.0/tests/test_api.py 0000644 0000000 0000000 00000000450 13615410400 014104 0 ustar 00 import flufl.lock
def test_module_attributes_in_all():
namespace = {}
attributes = set(flufl.lock.__all__)
exec('from flufl.lock import *', namespace)
# __builtins__ is implicitly added to the namespace.
del namespace['__builtins__']
assert attributes == set(namespace)
flufl_lock-8.2.0/tests/test_lock.py 0000644 0000000 0000000 00000046724 13615410400 014301 0 ustar 00 """Testing other aspects of the implementation and API."""
import os
import re
import sys
import time
import errno
import builtins
import platform
from contextlib import contextmanager, ExitStack, suppress
from datetime import timedelta
from io import StringIO
from multiprocessing import Process, Queue
from pathlib import Path
from random import randint
from tempfile import TemporaryDirectory
from unittest.mock import patch
import pytest
from flufl.lock import Lock, LockState, NotLockedError, SEP, TimeOutError
from flufl.lock._lockfile import CLOCK_SLOP, ERRORS
EMOCKEDFAILURE = 99
EOTHERMOCKEDFAILURE = 98
ENINES = 999
@pytest.fixture
def lock():
with TemporaryDirectory() as lock_dir:
lock = Lock(os.path.join(lock_dir, 'test.lck'))
yield lock
with suppress(NotLockedError):
lock.unlock()
def child_locker(filename, queue, *, sleep=3, lifetime=15, keep=False):
with suppress(NotLockedError):
with Lock(filename, lifetime=lifetime):
queue.put(True)
time.sleep(sleep)
queue.put(True)
# The test wants us to keep the lock a little bit longer.
if keep:
queue.get()
def test_retry_errno_property(lock):
assert lock.retry_errnos == []
lock.retry_errnos = [EMOCKEDFAILURE, EOTHERMOCKEDFAILURE]
assert lock.retry_errnos == [EMOCKEDFAILURE, EOTHERMOCKEDFAILURE]
del lock.retry_errnos
assert lock.retry_errnos == []
class RetryOpen:
def __init__(self, failure_countdown=0, retry_count=0):
self.failure_countdown = failure_countdown
self.retry_count = retry_count
self._open = builtins.open
self.errno = EMOCKEDFAILURE
def __call__(self, *args, **kws):
if self.failure_countdown <= 0:
return self._open(*args, **kws)
self.failure_countdown -= 1
self.retry_count += 1
raise OSError(self.errno, 'test exception')
def test_read_retries(lock):
# Test that _read() will retry when a given expected errno is encountered.
lock.lock()
lock.retry_errnos = [EMOCKEDFAILURE]
retry_open = RetryOpen(failure_countdown=3)
with patch('builtins.open', retry_open):
# This should trigger exactly 3 retries.
assert lock.is_locked
assert retry_open.retry_count == 3
def test_read_unexpected_errors(lock):
# Test that _read() will raise when an unexpected errno is encountered.
lock.lock()
retry_open = RetryOpen(failure_countdown=3)
retry_open.errno = ENINES
with patch('builtins.open', retry_open):
with pytest.raises(OSError) as excinfo:
lock.is_locked
assert excinfo.value.errno == ENINES
def test_is_locked_permission_error(lock):
with ExitStack() as resources:
resources.enter_context(patch('os.utime', side_effect=PermissionError))
log_mock = resources.enter_context(patch('flufl.lock._lockfile.log'))
assert not lock.is_locked
log_mock.error.assert_called_once_with(
'No permission to refresh the log')
def test_nondefault_lifetime(tmpdir):
lock_file = os.path.join(tmpdir, 'test.lck')
assert Lock(lock_file, lifetime=77).lifetime.seconds == 77
def test_lockfile_repr(lock):
# Handle both POSIX and Windows paths.
assert re.match(
r'',
repr(lock))
lock.lock()
assert re.match(
r'',
repr(lock))
lock.unlock()
assert re.match(
r'',
repr(lock))
def test_lockfile_repr_does_not_refresh(lock):
with lock:
expiration = lock.expiration
time.sleep(1)
repr(lock)
assert lock.expiration == expiration
def test_details(lock):
# No details are available if the lock is not locked.
with pytest.raises(NotLockedError):
lock.details()
with lock:
hostname, pid, filename = lock.details
assert hostname == lock.hostname
assert pid == os.getpid()
assert Path(filename).name == 'test.lck'
def test_expiration(lock):
with lock:
expiration = lock.expiration
time.sleep(1)
lock.refresh()
assert lock.expiration > expiration
class FailingOpen:
def __init__(self, errno=EMOCKEDFAILURE):
self._errno = errno
def __call__(self, *args, **kws):
raise OSError(self._errno, 'test exception')
def test_details_weird_open_failure(lock):
lock.lock()
with ExitStack() as resources:
# Force open() to fail with our unexpected errno.
resources.enter_context(patch('builtins.open', FailingOpen()))
# Capture the OSError with the unexpected errno that will occur when
# .details tries to open the lock file.
error = resources.enter_context(pytest.raises(OSError))
lock.details
assert error.errno == EMOCKEDFAILURE
@contextmanager
def corrupt_open(*args, **kws):
yield StringIO('bad claim file name')
def test_details_with_corrupt_filename(lock):
lock.lock()
with patch('builtins.open', corrupt_open):
with pytest.raises(NotLockedError, match='Details are unavailable'):
lock.details
def test_lifetime_property(lock):
assert lock.lifetime.seconds == 15
lock.lifetime = timedelta(seconds=31)
assert lock.lifetime.seconds == 31
lock.lifetime = 42
assert lock.lifetime.seconds == 42
def test_refresh(lock):
with pytest.raises(NotLockedError):
lock.refresh()
# With a lifetime parameter, the lock's lifetime is set.
lock.lock()
lock.refresh(31)
assert lock.lifetime.seconds == 31
# No exception is raised when we try to refresh an unlocked lock
# unconditionally.
lock.unlock()
lock.refresh(unconditionally=True)
def test_lock_with_explicit_timeout(lock):
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
with pytest.raises(TimeOutError):
lock.lock(timeout=1)
def test_lock_with_explicit_timeout_as_timedelta(lock):
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
with pytest.raises(TimeOutError):
lock.lock(timeout=timedelta(seconds=1))
def test_lock_state_with_corrupt_lockfile(lock):
# Since we're deliberately corrupting the contents of the lock file,
# unlocking at context manager exit will not work.
with suppress(NotLockedError):
with lock:
with open(lock.lockfile, 'w') as fp:
fp.write('xxx')
assert lock.state == LockState.unknown
def test_lock_state_on_other_host(lock):
# Since we're going to corrupt the lock contents, ignore the exception
# when we leave the context manager and unlock the lock.
with suppress(NotLockedError):
with lock:
hostname, pid, lockfile = lock.details
with open(lock.lockfile, 'w') as fp:
claimfile = SEP.join((
lockfile,
# Corrupt the hostname to emulate the lock being acquired
# on some other host.
f' {hostname} ',
str(pid),
str(randint(0, sys.maxsize)),
))
fp.write(claimfile)
assert lock.state == LockState.unknown
class SymlinkErrorRaiserBase:
def __init__(self, errnos):
self.errnos = errnos
self.call_count = 0
self._os_function = None
def __call__(self, *args, **kws):
self.call_count += 1
if self.call_count > len(self.errnos):
return self._os_function(*args, **kws)
raise OSError(self.errnos[self.call_count - 1], 'test exception')
class SymlinkErrorRaiser(SymlinkErrorRaiserBase):
def __init__(self, errnos):
super().__init__(errnos)
self._os_function = os.link
def test_os_link_expected_OSError(lock):
with patch('os.link', SymlinkErrorRaiser([ENINES])):
with pytest.raises(OSError) as excinfo:
lock.lock()
assert excinfo.value.errno == ENINES
def test_os_link_unexpected_OSError(lock):
raiser = SymlinkErrorRaiser([errno.ENOENT, errno.ESTALE])
with patch('os.link', raiser):
lock.lock()
# os.link() will be called 3 time; the first two will raise exceptions
# with errnos it can handle. The third time, goes through okay.
assert raiser.call_count == 3
class FakeStat:
st_nlink = 3
class LinkCountCounter:
def __init__(self):
self.call_count = 0
self._os_stat = os.stat
def __call__(self, *args, **kws):
if self.call_count == 0:
self.call_count += 1
# Return a bogus link count. This has to be an object with an
# st_nlink attribute.
return FakeStat()
else:
# Return the real link count.
return self._os_stat(*args, **kws)
def test_unexpected_st_nlink(lock):
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
# Now we try to acquire the lock, which will fail.
linkcount = LinkCountCounter()
with patch('os.stat', linkcount):
lock.lock()
assert linkcount.call_count == 1
def test_unlock_unconditionally(lock):
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
# Try to unlock without supplying the flag; this will fail.
with pytest.raises(NotLockedError):
lock.unlock()
# Try again unconditionally. This will pass.
lock.unlock(unconditionally=True)
class SymUnlinkErrorRaiser(SymlinkErrorRaiserBase):
def __init__(self, errnos):
super().__init__(errnos)
self._os_function = os.unlink
def test_unlock_with_expected_OSError(lock):
lock.lock()
unlinker = SymUnlinkErrorRaiser([errno.ESTALE])
with patch('os.unlink', unlinker):
lock.unlock()
# os.unlink() gets called twice. The first one unlinks the lock file, but
# that results in an expected errno. The second one unlinks the claimfile.
assert unlinker.call_count == 2
def test_unlock_with_unexpected_OSError(lock):
lock.lock()
unlinker = SymUnlinkErrorRaiser([ENINES])
with patch('os.unlink', unlinker):
with pytest.raises(OSError) as excinfo:
lock.unlock()
assert excinfo.value.errno == ENINES
# os.unlink() gets called once, since the unlinking of the lockfile
# results in an unexpected errno.
assert unlinker.call_count == 1
def test_unlock_unconditionally_with_expected_OSError(lock):
unlinker = SymUnlinkErrorRaiser([errno.ESTALE])
with patch('os.unlink', unlinker):
lock.unlock(unconditionally=True)
# Since the lock was not acquired, os.unlink() should have been called
# exactly once to remove the claim file.
assert unlinker.call_count == 1
def test_unlock_unconditionally_with_unexpected_OSError(lock):
unlinker = SymUnlinkErrorRaiser([ENINES])
with patch('os.unlink', unlinker):
with pytest.raises(OSError) as excinfo:
lock.unlock(unconditionally=True)
assert excinfo.value.errno == ENINES
# Since the lock was not acquired, os.unlink() should have been called
# exactly once to remove the claim file.
assert unlinker.call_count == 1
class MtimeFailure:
def __init__(self, stat_results):
self._stat_results = stat_results
def __getattr__(self, name):
if name == 'st_mtime':
raise OSError(ENINES, 'st_mtime failure')
return getattr(self._stat_results, name)
class StatMtimeFailure:
def __init__(self):
self._os_stat = os.stat
def __call__(self, *args, **kws):
return MtimeFailure(self._os_stat(*args, **kws))
def test_releasetime_weird_failure(lock):
# _releasetime() is an internal function that returns the expiration of
# the lock, but handles error conditions. We have to basically fail to
# acquire a lock, don't time out, and the os_stat() of the lock file must
# fail with an unexpected error.
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
# Now we try to acquire the lock, which will fail.
with patch('os.stat', StatMtimeFailure()):
with pytest.raises(OSError) as excinfo:
lock.lock()
assert excinfo.value.errno == ENINES
class NlinkFailure:
def __init__(self, stat_results):
self._stat_results = stat_results
def __getattr__(self, name):
if name == 'st_nlink':
raise OSError(ENINES, 'st_nlink failure')
return getattr(self._stat_results, name)
class StatNlinkFailure:
def __init__(self):
self._os_stat = os.stat
def __call__(self, *args, **kws):
return NlinkFailure(self._os_stat(*args, **kws))
def test_linkcount_weird_failure(lock):
# _releasetime() is an internal function that returns the expiration of
# the lock, but handles error conditions. We have to basically fail to
# acquire a lock, don't time out, and the os_stat() of the lock file must
# fail with an unexpected error.
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
# Now we try to acquire the lock, which will fail.
with patch('os.stat', StatNlinkFailure()):
with pytest.raises(OSError) as excinfo:
lock.is_locked
assert excinfo.value.errno == ENINES
def test_lock_constructor_with_timeout(lock):
# Pass an optional timeout value to the constructor.
queue = Queue()
Process(target=child_locker, args=(lock.lockfile, queue)).start()
# Wait for the child process to acquire the lock.
queue.get()
with pytest.raises(TimeOutError):
with Lock(lock.lockfile, default_timeout=1):
pass
def test_lock_constructor_with_timeout_override(lock):
# Explicit timeout in the lock() call overrides constructor timeout.
queue = Queue()
Process(target=child_locker,
# Give the child lock a lifetime of 5 seconds. We'll provide a
# shorter timeout in the constructor, which should time out, but a
# longer time in the lock() call which will result in acquiring
# the lock when the lifetime of the child expires.
args=(lock.lockfile, queue), kwargs=dict(sleep=3, lifetime=5),
).start()
# Wait for the child process to acquire the lock.
queue.get()
my_lock = Lock(lock.lockfile, default_timeout=1)
try:
my_lock.lock(timeout=10)
assert my_lock.is_locked
finally:
my_lock.unlock()
@pytest.mark.parametrize('lifetime', [1, 5])
def test_use_unrelated_existing_lockfile(lock, lifetime):
# If someone gives a lock file that already exists, and that isn't a
# related lock file, then trying to lock it shouldn't destroy the existing
# file.
#
# https://gitlab.com/warsaw/flufl.lock/-/issues/25
#
# There are two cases, one where the lock's lifetime is less than the
# timeout value and one where the lifetime is greater than the timeout
# value. In both cases, the expiration time should be in the past and both
# should preserve the original (non-)lockfile.
lock.lifetime = lifetime
with open(lock.lockfile, 'w') as fp:
fp.write('save me')
# Put the lock file's release time in the past. This has to include the
# clock slop factor.
past = time.time() - lifetime - CLOCK_SLOP.seconds
os.utime(lock.lockfile, (past, past))
with pytest.raises(TimeOutError):
lock.lock(timeout=3)
with open(lock.lockfile) as fp:
assert fp.read() == 'save me'
# 2023-06-20(warsaw): On CI in Windows, we sometimes see PermissionError when
# unlocking the lock on context manager __exit__(), but only in the lock
# breaking tests. It appears to happen when we attempt to unlink the lock
# file.
#
# [WinError 32] The process cannot access the file because it is being used
# by another process
#
# Despite the '32' there, by trial and error, it seems that the errno that
# occurs is actually 13. I have no idea why the original error occurs, nor
# why we get this mysterious value 13, but at least on Windows, this allows CI
# to pass.
WINDOWS_CI_ERRNO = 13
def test_break_lock(lock, monkeypatch, capsys):
queue = Queue()
proc = Process(target=child_locker,
# The child will acquire the lock and sleep for 5 seconds, which
# is longer than lock's lifetime. Once the second boolean is
# placed in the queue, we know that the sleep has completed and
# the lock should be ready for breakage in the parent.
args=(lock.lockfile, queue),
kwargs=dict(sleep=5, lifetime=3, keep=True),
).start()
# Wait for the child process to acquire the lock.
queue.get()
child_details = lock.details
# Wait for the child to finish sleeping.
queue.get()
# Now acquire the lock in the parent. This should break the lock. See
# above for an explanation of why we have to monkeypatch ERRORS on
# Windows.
if platform.system() == 'Windows':
monkeypatch.setattr(
'flufl.lock._lockfile.ERRORS',
list(ERRORS) + [WINDOWS_CI_ERRNO])
with lock:
assert lock.is_locked
# The child no longer has the lock.
assert child_details != lock.details
# Let the child exit.
queue.put(True)
def test_break_lock_with_ununlinkable_winner(lock, monkeypatch):
queue = Queue()
proc = Process(target=child_locker,
# The child will acquire the lock and sleep for 5 seconds, which
# is longer than lock's lifetime. Once the second boolean is
# placed in the queue, we know that the sleep has completed and
# the lock should be ready for breakage in the parent.
args=(lock.lockfile, queue),
kwargs=dict(sleep=5, lifetime=3, keep=True),
).start()
# Wait for the child process to acquire the lock.
queue.get()
child_details = lock.details
# Wait for the child to finish sleeping.
queue.get()
# Now acquire the lock in the parent. This should break the lock.
#
# Count how many times os.unlink() gets called when breaking the lock.
# The fourth call (found by trial and error) should be the attempt to
# unlink the winner file in Lock._break().
unlink_count = 2
os_unlink = os.unlink
def unlink_counter(*args, **kws):
nonlocal unlink_count
unlink_count -= 1
if unlink_count > 0:
return os_unlink(*args, **kws)
raise OSError(ENINES, 'Bad Unlink')
if platform.system() == 'Windows':
monkeypatch.setattr(
'flufl.lock._lockfile.ERRORS',
list(ERRORS) + [WINDOWS_CI_ERRNO])
with patch('os.unlink', unlink_counter):
# This lock attempt will technically succeed, but it will raise an
# exception (EMOCKEDFAILURE) during the attempt to os.unlink(winner).
# It's coverage of that call that this test is actually after.
with pytest.raises(OSError) as excinfo:
lock.lock()
assert excinfo.value.errno == ENINES
# We still need to let the child exit.
queue.put(True)
flufl_lock-8.2.0/.gitignore 0000644 0000000 0000000 00000000265 13615410400 012554 0 ustar 00 build
flufl.lock.egg-info
distribute-0.6.10-py3.1.egg
distribute-0.6.10.tar.gz
dist
*.pyc
.tox
.coverage*
coverage.xml
diffcov.html
htmlcov
/__pypackages__/
/.pdm.toml
/.pdm-python
flufl_lock-8.2.0/LICENSE 0000644 0000000 0000000 00000001056 13615410400 011570 0 ustar 00 Copyright 2004-2025 Barry Warsaw
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
flufl_lock-8.2.0/README.rst 0000644 0000000 0000000 00000004123 13615410400 012250 0 ustar 00 ==========
flufl.lock
==========
NFS-safe file locking with timeouts for POSIX and Windows.
The ``flufl.lock`` library provides an NFS-safe file-based locking algorithm
influenced by the GNU/Linux ``open(2)`` manpage, under the description of the
``O_EXCL`` option.
[...] O_EXCL is broken on NFS file systems, programs which rely on it
for performing locking tasks will contain a race condition. The
solution for performing atomic file locking using a lockfile is to
create a unique file on the same fs (e.g., incorporating hostname and
pid), use link(2) to make a link to the lockfile. If link() returns
0, the lock is successful. Otherwise, use stat(2) on the unique file
to check if its link count has increased to 2, in which case the lock
is also successful.
The assumption made here is that there will be no *outside interference*,
e.g. no agent external to this code will ever ``link()`` to the specific lock
files used.
Lock objects support lock-breaking so that you can't wedge a process forever.
This is especially helpful in a web environment, but may not be appropriate
for all applications.
Locks have a *lifetime*, which is the maximum length of time the process
expects to retain the lock. It is important to pick a good number here
because other processes will not break an existing lock until the expected
lifetime has expired. Too long and other processes will hang; too short and
you'll end up trampling on existing process locks -- and possibly corrupting
data. In a distributed (NFS) environment, you also need to make sure that
your clocks are properly synchronized.
Author
======
``flufl.lock`` is Copyright (C) 2007-2025 Barry Warsaw
Licensed under the terms of the Apache License Version 2.0. See the LICENSE
file for details.
Project details
===============
* Project home: https://gitlab.com/warsaw/flufl.lock
* Report bugs at: https://gitlab.com/warsaw/flufl.lock/issues
* Code hosting: https://gitlab.com/warsaw/flufl.lock.git
* Documentation: https://flufllock.readthedocs.io/
* PyPI: https://pypi.python.org/pypi/flufl.lock
flufl_lock-8.2.0/pyproject.toml 0000644 0000000 0000000 00000012644 13615410400 013504 0 ustar 00 [project]
name = 'flufl.lock'
authors = [
{name = 'Barry Warsaw', email = 'barry@python.org'},
]
description = 'NFS-safe file locking with timeouts for POSIX and Windows'
readme = 'README.rst'
requires-python = '>=3.9'
license = {text = 'Apache-2.0'}
keywords = [
'locking',
'locks',
'lock',
]
classifiers = [
'Development Status :: 5 - Production/Stable',
'Development Status :: 6 - Mature',
'Intended Audience :: Developers',
'License :: OSI Approved :: Apache Software License',
'Operating System :: POSIX',
'Operating System :: MacOS :: MacOS X',
'Operating System :: Microsoft :: Windows',
'Programming Language :: Python',
'Programming Language :: Python :: 3',
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Python Modules',
]
dependencies = [
'atpublic',
'psutil',
]
dynamic = ['version']
[project.urls]
'Home Page' = 'https://flufllock.readthedocs.io'
'Documentation' = 'https://flufllock.readthedocs.io'
'Source Code' = 'https://gitlab.com/warsaw/flufl.lock.git'
'Bug Tracker' = 'https://gitlab.com/warsaw/flufl.lock/issues'
[tool.hatch.version]
path = 'src/flufl/lock/__init__.py'
[tool.hatch.build.targets.wheel]
packages = [
'src/flufl',
]
[tool.hatch.build.targets.sdist]
include = [
'src/flufl/',
'docs/',
'tests/',
'conftest.py',
]
excludes = [
'**/.mypy_cache',
]
[tool.hatch.envs.default.scripts]
all = [
'hatch test --all',
'hatch run qa:qa',
'hatch run docs:docs',
]
[tool.hatch.envs.hatch-test]
default-args = ['tests', 'docs']
extra-dependencies = [
'diff-cover',
'sybil',
]
[tool.hatch.envs.hatch-test.scripts]
run = [
'coverage run -m pytest{env:HATCH_TEST_ARGS:} {args}',
'coverage combine',
'coverage report',
'coverage xml',
'- diff-cover coverage.xml',
]
run-cov = 'hatch test'
cov-combine = ''
cov-report = ''
[[tool.hatch.envs.test.matrix]]
python = ['3.9', '3.10', '3.11', '3.12', '3.13', '3.14']
[tool.hatch.envs.qa]
dependencies = [
'ruff',
'mypy',
]
[tool.hatch.envs.qa.env-vars]
MODULE_NAME = '{env:MODULE_NAME:flufl.lock}'
MODULE_PATH = '{env:MODULE_PATH:src/flufl/lock}'
[tool.hatch.envs.qa.scripts]
qa = [
'hatch fmt --check src',
'mypy -p {env:MODULE_NAME}',
]
fix = [
'hatch fmt src',
]
preview = [
'hatch fmt --diff src',
]
[tool.hatch.envs.docs]
dependencies = [
'sphinx',
'furo',
]
[tool.hatch.envs.docs.scripts]
docs = [
'sphinx-build docs build/html',
]
[tool.coverage.run]
source = ['flufl.lock']
branch = true
parallel = true
[tool.coverage.report]
fail_under = 100
show_missing = true
exclude_also = [
'if TYPE_CHECKING:',
]
[tool.ruff]
line-length = 100
src = ['src']
[tool.ruff.lint.extend-per-file-ignores]
# Essentially, ignore all lint warnings in these configuration files.
'conftest.py' = [
'ARG002',
'I001',
'S101',
]
'docs/conf.py' = [
'A',
'DTZ',
'E',
'I',
'UP',
]
'src/flufl/lock/_lockfile.py' = [
# Naive datetimes are fine for our purposes.
'DTZ005', # `datetime.datetime.now()` called without a `tz` argument
'DTZ006', # `datetime.datetime.fromtimestamp()` called without a `tz` argument
'EM101', # Exception must not use a string literal, assign to variable first
# We use constant `2` for linkcount checks in places.
'PLR2004', # Magic value used in comparison
# Lock.__enter__() is annotated to reutrn `Lock` rather than `Self`. I don't want to add a
# conditional runtime dependency on typing_extensions just for this type. Once Python 3.11 is
# our minimim version, we can change the return annotation of this function to Self.
'PYI034', # `__enter__` methods in classes like `Lock` usually return `self` at runtime
# We use random.randint() for non-cryptographic purposes.
'S311', # Standard pseudo-random generators are not suitable for cryptographic purposes
'TRY003', # Avoid specifying long messages outside the exception class
'TRY400', # Use `logging.exception` instead of `logging.error`
]
[tool.ruff.format]
quote-style = 'single'
[tool.ruff.lint.pydocstyle]
convention = 'pep257'
[tool.ruff.lint.isort]
case-sensitive = true
classes = ['SEP']
forced-separate = ['docutils', 'sphinx']
known-first-party = ['flufl.lock']
length-sort-straight = true
lines-after-imports = 2
lines-between-types = 1
order-by-type = true
section-order = ['standard-library', 'third-party', 'local-folder', 'first-party']
[tool.mypy]
mypy_path = 'src'
# Disallow dynamic typing
disallow_any_generics = true
disallow_subclassing_any = true
# Untyped definitions and calls
disallow_untyped_calls = false
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = false
# None and Optional handling
no_implicit_optional = true
# Configuring warnings
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_return_any = true
warn_unreachable = true
# Miscellaneous strictness flags
implicit_reexport = false
strict_equality = true
# Configuring error messages
show_error_context = true
show_column_numbers = true
show_error_codes = true
pretty = true
show_absolute_path = true
# Miscellaneous
warn_unused_configs = true
verbosity = 0
[[tool.mypy.overrides]]
module = [
'psutil',
'pytest',
'sybil.*',
]
ignore_missing_imports = true
[build-system]
requires = ['hatchling']
build-backend = 'hatchling.build'
flufl_lock-8.2.0/PKG-INFO 0000644 0000000 0000000 00000006342 13615410400 011663 0 ustar 00 Metadata-Version: 2.4
Name: flufl.lock
Version: 8.2.0
Summary: NFS-safe file locking with timeouts for POSIX and Windows
Project-URL: Home Page, https://flufllock.readthedocs.io
Project-URL: Documentation, https://flufllock.readthedocs.io
Project-URL: Source Code, https://gitlab.com/warsaw/flufl.lock.git
Project-URL: Bug Tracker, https://gitlab.com/warsaw/flufl.lock/issues
Author-email: Barry Warsaw
License: Apache-2.0
License-File: LICENSE
Keywords: lock,locking,locks
Classifier: Development Status :: 5 - Production/Stable
Classifier: Development Status :: 6 - Mature
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Requires-Dist: atpublic
Requires-Dist: psutil
Description-Content-Type: text/x-rst
==========
flufl.lock
==========
NFS-safe file locking with timeouts for POSIX and Windows.
The ``flufl.lock`` library provides an NFS-safe file-based locking algorithm
influenced by the GNU/Linux ``open(2)`` manpage, under the description of the
``O_EXCL`` option.
[...] O_EXCL is broken on NFS file systems, programs which rely on it
for performing locking tasks will contain a race condition. The
solution for performing atomic file locking using a lockfile is to
create a unique file on the same fs (e.g., incorporating hostname and
pid), use link(2) to make a link to the lockfile. If link() returns
0, the lock is successful. Otherwise, use stat(2) on the unique file
to check if its link count has increased to 2, in which case the lock
is also successful.
The assumption made here is that there will be no *outside interference*,
e.g. no agent external to this code will ever ``link()`` to the specific lock
files used.
Lock objects support lock-breaking so that you can't wedge a process forever.
This is especially helpful in a web environment, but may not be appropriate
for all applications.
Locks have a *lifetime*, which is the maximum length of time the process
expects to retain the lock. It is important to pick a good number here
because other processes will not break an existing lock until the expected
lifetime has expired. Too long and other processes will hang; too short and
you'll end up trampling on existing process locks -- and possibly corrupting
data. In a distributed (NFS) environment, you also need to make sure that
your clocks are properly synchronized.
Author
======
``flufl.lock`` is Copyright (C) 2007-2025 Barry Warsaw
Licensed under the terms of the Apache License Version 2.0. See the LICENSE
file for details.
Project details
===============
* Project home: https://gitlab.com/warsaw/flufl.lock
* Report bugs at: https://gitlab.com/warsaw/flufl.lock/issues
* Code hosting: https://gitlab.com/warsaw/flufl.lock.git
* Documentation: https://flufllock.readthedocs.io/
* PyPI: https://pypi.python.org/pypi/flufl.lock