././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4114294
Pebble-4.6.3/ 0000755 0001750 0001750 00000000000 00000000000 013154 5 ustar 00noxdafox noxdafox ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065860.0
Pebble-4.6.3/LICENSE 0000644 0001750 0001750 00000016720 00000000000 014167 0 ustar 00noxdafox noxdafox GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc.
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.
"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.
An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.
A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".
The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.
The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:
a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license
document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:
a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.
c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1613598549.0
Pebble-4.6.3/MANIFEST.in 0000644 0001750 0001750 00000000017 00000000000 014710 0 ustar 00noxdafox noxdafox include LICENSE ././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4114294
Pebble-4.6.3/PKG-INFO 0000644 0001750 0001750 00000007115 00000000000 014255 0 ustar 00noxdafox noxdafox Metadata-Version: 1.1
Name: Pebble
Version: 4.6.3
Summary: Threading and multiprocessing eye-candy.
Home-page: https://github.com/noxdafox/pebble
Author: Matteo Cafasso
Author-email: noxdafox@gmail.com
License: LGPL
Description: Pebble
======
Pebble provides a neat API to manage threads and processes within an application.
:Source: https://github.com/noxdafox/pebble
:Documentation: https://pebble.readthedocs.io
:Download: https://pypi.org/project/Pebble/
|build badge| |docs badge|
.. |build badge| image:: https://github.com/noxdafox/pebble/actions/workflows/action.yml/badge.svg
:target: https://github.com/noxdafox/pebble/actions/workflows/action.yml
:alt: Build Status
.. |docs badge| image:: https://readthedocs.org/projects/pebble/badge/?version=latest
:target: https://pebble.readthedocs.io
:alt: Documentation Status
Examples
--------
Run a job in a separate thread and wait for its results.
.. code:: python
from pebble import concurrent
@concurrent.thread
def function(foo, bar=0):
return foo + bar
future = function(1, bar=2)
result = future.result() # blocks until results are ready
Run a function with a timeout of ten seconds and deal with errors.
.. code:: python
from pebble import concurrent
from concurrent.futures import TimeoutError
@concurrent.process(timeout=10)
def function(foo, bar=0):
return foo + bar
future = function(1, bar=2)
try:
result = future.result() # blocks until results are ready
except TimeoutError as error:
print("Function took longer than %d seconds" % error.args[1])
except Exception as error:
print("Function raised %s" % error)
print(error.traceback) # traceback of the function
Pools support workers restart, timeout for long running tasks and more.
.. code:: python
from pebble import ProcessPool
from concurrent.futures import TimeoutError
def function(foo, bar=0):
return foo + bar
def task_done(future):
try:
result = future.result() # blocks until results are ready
except TimeoutError as error:
print("Function took longer than %d seconds" % error.args[1])
except Exception as error:
print("Function raised %s" % error)
print(error.traceback) # traceback of the function
with ProcessPool(max_workers=5, max_tasks=10) as pool:
for i in range(0, 10):
future = pool.schedule(function, args=[i], timeout=3)
future.add_done_callback(task_done)
Keywords: thread process pool decorator
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4074295
Pebble-4.6.3/Pebble.egg-info/ 0000755 0001750 0001750 00000000000 00000000000 016037 5 ustar 00noxdafox noxdafox ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629838196.0
Pebble-4.6.3/Pebble.egg-info/PKG-INFO 0000644 0001750 0001750 00000007115 00000000000 017140 0 ustar 00noxdafox noxdafox Metadata-Version: 1.1
Name: Pebble
Version: 4.6.3
Summary: Threading and multiprocessing eye-candy.
Home-page: https://github.com/noxdafox/pebble
Author: Matteo Cafasso
Author-email: noxdafox@gmail.com
License: LGPL
Description: Pebble
======
Pebble provides a neat API to manage threads and processes within an application.
:Source: https://github.com/noxdafox/pebble
:Documentation: https://pebble.readthedocs.io
:Download: https://pypi.org/project/Pebble/
|build badge| |docs badge|
.. |build badge| image:: https://github.com/noxdafox/pebble/actions/workflows/action.yml/badge.svg
:target: https://github.com/noxdafox/pebble/actions/workflows/action.yml
:alt: Build Status
.. |docs badge| image:: https://readthedocs.org/projects/pebble/badge/?version=latest
:target: https://pebble.readthedocs.io
:alt: Documentation Status
Examples
--------
Run a job in a separate thread and wait for its results.
.. code:: python
from pebble import concurrent
@concurrent.thread
def function(foo, bar=0):
return foo + bar
future = function(1, bar=2)
result = future.result() # blocks until results are ready
Run a function with a timeout of ten seconds and deal with errors.
.. code:: python
from pebble import concurrent
from concurrent.futures import TimeoutError
@concurrent.process(timeout=10)
def function(foo, bar=0):
return foo + bar
future = function(1, bar=2)
try:
result = future.result() # blocks until results are ready
except TimeoutError as error:
print("Function took longer than %d seconds" % error.args[1])
except Exception as error:
print("Function raised %s" % error)
print(error.traceback) # traceback of the function
Pools support workers restart, timeout for long running tasks and more.
.. code:: python
from pebble import ProcessPool
from concurrent.futures import TimeoutError
def function(foo, bar=0):
return foo + bar
def task_done(future):
try:
result = future.result() # blocks until results are ready
except TimeoutError as error:
print("Function took longer than %d seconds" % error.args[1])
except Exception as error:
print("Function raised %s" % error)
print(error.traceback) # traceback of the function
with ProcessPool(max_workers=5, max_tasks=10) as pool:
for i in range(0, 10):
future = pool.schedule(function, args=[i], timeout=3)
future.add_done_callback(task_done)
Keywords: thread process pool decorator
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629838196.0
Pebble-4.6.3/Pebble.egg-info/SOURCES.txt 0000644 0001750 0001750 00000001405 00000000000 017723 0 ustar 00noxdafox noxdafox LICENSE
MANIFEST.in
README.rst
setup.cfg
setup.py
Pebble.egg-info/PKG-INFO
Pebble.egg-info/SOURCES.txt
Pebble.egg-info/dependency_links.txt
Pebble.egg-info/requires.txt
Pebble.egg-info/top_level.txt
pebble/__init__.py
pebble/common.py
pebble/decorators.py
pebble/functions.py
pebble/concurrent/__init__.py
pebble/concurrent/process.py
pebble/concurrent/thread.py
pebble/pool/__init__.py
pebble/pool/base_pool.py
pebble/pool/channel.py
pebble/pool/process.py
pebble/pool/thread.py
test/test_concurrent_process_fork.py
test/test_concurrent_process_forkserver.py
test/test_concurrent_process_spawn.py
test/test_concurrent_thread.py
test/test_pebble.py
test/test_process_pool_fork.py
test/test_process_pool_forkserver.py
test/test_process_pool_spawn.py
test/test_thread_pool.py ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629838196.0
Pebble-4.6.3/Pebble.egg-info/dependency_links.txt 0000644 0001750 0001750 00000000001 00000000000 022105 0 ustar 00noxdafox noxdafox
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629838196.0
Pebble-4.6.3/Pebble.egg-info/requires.txt 0000644 0001750 0001750 00000000037 00000000000 020437 0 ustar 00noxdafox noxdafox
[:python_version<'3']
futures
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629838196.0
Pebble-4.6.3/Pebble.egg-info/top_level.txt 0000644 0001750 0001750 00000000007 00000000000 020566 0 ustar 00noxdafox noxdafox pebble
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629748054.0
Pebble-4.6.3/README.rst 0000644 0001750 0001750 00000004443 00000000000 014650 0 ustar 00noxdafox noxdafox Pebble
======
Pebble provides a neat API to manage threads and processes within an application.
:Source: https://github.com/noxdafox/pebble
:Documentation: https://pebble.readthedocs.io
:Download: https://pypi.org/project/Pebble/
|build badge| |docs badge|
.. |build badge| image:: https://github.com/noxdafox/pebble/actions/workflows/action.yml/badge.svg
:target: https://github.com/noxdafox/pebble/actions/workflows/action.yml
:alt: Build Status
.. |docs badge| image:: https://readthedocs.org/projects/pebble/badge/?version=latest
:target: https://pebble.readthedocs.io
:alt: Documentation Status
Examples
--------
Run a job in a separate thread and wait for its results.
.. code:: python
from pebble import concurrent
@concurrent.thread
def function(foo, bar=0):
return foo + bar
future = function(1, bar=2)
result = future.result() # blocks until results are ready
Run a function with a timeout of ten seconds and deal with errors.
.. code:: python
from pebble import concurrent
from concurrent.futures import TimeoutError
@concurrent.process(timeout=10)
def function(foo, bar=0):
return foo + bar
future = function(1, bar=2)
try:
result = future.result() # blocks until results are ready
except TimeoutError as error:
print("Function took longer than %d seconds" % error.args[1])
except Exception as error:
print("Function raised %s" % error)
print(error.traceback) # traceback of the function
Pools support workers restart, timeout for long running tasks and more.
.. code:: python
from pebble import ProcessPool
from concurrent.futures import TimeoutError
def function(foo, bar=0):
return foo + bar
def task_done(future):
try:
result = future.result() # blocks until results are ready
except TimeoutError as error:
print("Function took longer than %d seconds" % error.args[1])
except Exception as error:
print("Function raised %s" % error)
print(error.traceback) # traceback of the function
with ProcessPool(max_workers=5, max_tasks=10) as pool:
for i in range(0, 10):
future = pool.schedule(function, args=[i], timeout=3)
future.add_done_callback(task_done)
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4074295
Pebble-4.6.3/pebble/ 0000755 0001750 0001750 00000000000 00000000000 014405 5 ustar 00noxdafox noxdafox ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629837986.0
Pebble-4.6.3/pebble/__init__.py 0000644 0001750 0001750 00000001124 00000000000 016514 0 ustar 00noxdafox noxdafox __author__ = 'Matteo Cafasso'
__version__ = '4.6.3'
__license__ = 'LGPL'
__all__ = ['waitforthreads',
'waitforqueues',
'synchronized',
'sighandler',
'ProcessFuture',
'MapFuture',
'ProcessMapFuture',
'ProcessExpired',
'ProcessPool',
'ThreadPool']
from pebble.decorators import synchronized, sighandler
from pebble.common import ProcessExpired, ProcessFuture
from pebble.functions import waitforqueues, waitforthreads
from pebble.pool import ThreadPool, ProcessPool, MapFuture, ProcessMapFuture
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065861.0
Pebble-4.6.3/pebble/common.py 0000644 0001750 0001750 00000013075 00000000000 016255 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
from __future__ import absolute_import
import os
import pickle
import signal
from threading import Thread
from traceback import format_exc
from concurrent.futures import Future
class ProcessExpired(OSError):
"""Raised when process dies unexpectedly."""
def __init__(self, msg, code=0):
super(ProcessExpired, self).__init__(msg)
self.exitcode = code
class PebbleFuture(Future):
# Same as base class, removed logline
def set_running_or_notify_cancel(self):
"""Mark the future as running or process any cancel notifications.
Should only be used by Executor implementations and unit tests.
If the future has been cancelled (cancel() was called and returned
True) then any threads waiting on the future completing (though calls
to as_completed() or wait()) are notified and False is returned.
If the future was not cancelled then it is put in the running state
(future calls to running() will return True) and True is returned.
This method should be called by Executor implementations before
executing the work associated with this future. If this method returns
False then the work should not be executed.
Returns:
False if the Future was cancelled, True otherwise.
Raises:
RuntimeError: if set_result() or set_exception() was called.
"""
with self._condition:
if self._state == CANCELLED:
self._state = CANCELLED_AND_NOTIFIED
for waiter in self._waiters:
waiter.add_cancelled(self)
return False
elif self._state == PENDING:
self._state = RUNNING
return True
else:
raise RuntimeError('Future in unexpected state')
class ProcessFuture(PebbleFuture):
def cancel(self):
"""Cancel the future.
Returns True if the future was cancelled, False otherwise. A future
cannot be cancelled if it has already completed.
"""
with self._condition:
if self._state == FINISHED:
return False
if self._state in (CANCELLED, CANCELLED_AND_NOTIFIED):
return True
self._state = CANCELLED
self._condition.notify_all()
self._invoke_callbacks()
return True
class RemoteTraceback(Exception):
"""Traceback wrapper for exceptions in remote process.
Exception.__cause__ requires a BaseException subclass.
"""
def __init__(self, traceback):
self.traceback = traceback
def __str__(self):
return self.traceback
class RemoteException(object):
"""Pickling wrapper for exceptions in remote process."""
def __init__(self, exception, traceback):
self.exception = exception
self.traceback = traceback
def __reduce__(self):
return rebuild_exception, (self.exception, self.traceback)
def rebuild_exception(exception, traceback):
exception.__cause__ = RemoteTraceback(traceback)
return exception
def launch_thread(name, function, daemon, *args, **kwargs):
thread = Thread(target=function, name=name, args=args, kwargs=kwargs)
thread.daemon = daemon
thread.start()
return thread
def launch_process(name, function, daemon, mp_context, *args, **kwargs):
process = mp_context.Process(
target=function, name=name, args=args, kwargs=kwargs)
process.daemon = daemon
process.start()
return process
def stop_process(process):
"""Does its best to stop the process."""
process.terminate()
process.join(3)
if process.is_alive() and os.name != 'nt':
try:
os.kill(process.pid, signal.SIGKILL)
process.join()
except OSError:
return
if process.is_alive():
raise RuntimeError("Unable to terminate PID %d" % os.getpid())
def execute(function, *args, **kwargs):
"""Runs the given function returning its results or exception."""
try:
return function(*args, **kwargs)
except Exception as error:
error.traceback = format_exc()
return error
def process_execute(function, *args, **kwargs):
"""Runs the given function returning its results or exception."""
try:
return function(*args, **kwargs)
except Exception as error:
error.traceback = format_exc()
return RemoteException(error, error.traceback)
def send_result(pipe, data):
"""Send result handling pickling and communication errors."""
try:
pipe.send(data)
except (pickle.PicklingError, TypeError) as error:
error.traceback = format_exc()
pipe.send(RemoteException(error, error.traceback))
SLEEP_UNIT = 0.1
# Borrowed from concurrent.futures
PENDING = 'PENDING'
RUNNING = 'RUNNING'
FINISHED = 'FINISHED'
CANCELLED = 'CANCELLED'
CANCELLED_AND_NOTIFIED = 'CANCELLED_AND_NOTIFIED'
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4074295
Pebble-4.6.3/pebble/concurrent/ 0000755 0001750 0001750 00000000000 00000000000 016567 5 ustar 00noxdafox noxdafox ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065861.0
Pebble-4.6.3/pebble/concurrent/__init__.py 0000644 0001750 0001750 00000000206 00000000000 020676 0 ustar 00noxdafox noxdafox __all__ = ['thread',
'process']
from pebble.concurrent.thread import thread
from pebble.concurrent.process import process
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1615029530.0
Pebble-4.6.3/pebble/concurrent/process.py 0000644 0001750 0001750 00000016025 00000000000 020623 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import os
import sys
import types
import signal
import multiprocessing
from itertools import count
from functools import wraps
from concurrent.futures import CancelledError, TimeoutError
from pebble.common import ProcessExpired, ProcessFuture
from pebble.common import launch_process, stop_process, SLEEP_UNIT
from pebble.common import process_execute, launch_thread, send_result
def process(*args, **kwargs):
"""Runs the decorated function in a concurrent process,
taking care of the result and error management.
Decorated functions will return a concurrent.futures.Future object
once called.
The timeout parameter will set a maximum execution time
for the decorated function. If the execution exceeds the timeout,
the process will be stopped and the Future will raise TimeoutError.
The name parameter will set the process name.
The daemon parameter controls the underlying process daemon flag.
Default is True.
The context parameter allows to provide the multiprocessing.context
object used for starting the process.
"""
timeout = kwargs.get('timeout')
name = kwargs.get('name')
daemon = kwargs.get('daemon', True)
mp_context = kwargs.get('context')
# decorator without parameters
if not kwargs and len(args) == 1 and callable(args[0]):
return _process_wrapper(args[0], timeout, name, daemon, multiprocessing)
# decorator with parameters
_validate_parameters(timeout, name, daemon, mp_context)
mp_context = mp_context if mp_context is not None else multiprocessing
# without @pie syntax
if len(args) == 1 and callable(args[0]):
return _process_wrapper(args[0], timeout, name, daemon, multiprocessing)
# with @pie syntax
def decorating_function(function):
return _process_wrapper(function, timeout, name, daemon, mp_context)
return decorating_function
def _process_wrapper(function, timeout, name, daemon, mp_context):
_register_function(function)
if hasattr(mp_context, 'get_start_method'):
start_method = mp_context.get_start_method()
else:
start_method = 'spawn' if os.name == 'nt' else 'fork'
@wraps(function)
def wrapper(*args, **kwargs):
future = ProcessFuture()
reader, writer = mp_context.Pipe(duplex=False)
if start_method != 'fork':
target = _trampoline
args = [_qualname(function), function.__module__] + list(args)
else:
target = function
worker = launch_process(
name, _function_handler, daemon, mp_context,
target, args, kwargs, (reader, writer))
writer.close()
future.set_running_or_notify_cancel()
launch_thread(name, _worker_handler, True, future, worker, reader, timeout)
return future
return wrapper
def _worker_handler(future, worker, pipe, timeout):
"""Worker lifecycle manager.
Waits for the worker to be perform its task,
collects result, runs the callback and cleans up the process.
"""
result = _get_result(future, pipe, timeout)
if isinstance(result, BaseException):
if isinstance(result, ProcessExpired):
result.exitcode = worker.exitcode
if not isinstance(result, CancelledError):
future.set_exception(result)
else:
future.set_result(result)
if worker.is_alive():
stop_process(worker)
def _function_handler(function, args, kwargs, pipe):
"""Runs the actual function in separate process and returns its result."""
signal.signal(signal.SIGINT, signal.SIG_IGN)
reader, writer = pipe
reader.close()
result = process_execute(function, *args, **kwargs)
send_result(writer, result)
def _get_result(future, pipe, timeout):
"""Waits for result and handles communication errors."""
counter = count(step=SLEEP_UNIT)
try:
while not pipe.poll(SLEEP_UNIT):
if timeout is not None and next(counter) >= timeout:
return TimeoutError('Task Timeout', timeout)
elif future.cancelled():
return CancelledError()
return pipe.recv()
except (EOFError, OSError):
return ProcessExpired('Abnormal termination')
except Exception as error:
return error
def _validate_parameters(timeout, name, daemon, mp_context):
if timeout is not None and not isinstance(timeout, (int, float)):
raise TypeError('Timeout expected to be None or integer or float')
if name is not None and not isinstance(name, str):
raise TypeError('Name expected to be None or string')
if daemon is not None and not isinstance(daemon, bool):
raise TypeError('Daemon expected to be None or bool')
if mp_context is not None and not isinstance(
mp_context, multiprocessing.context.BaseContext):
raise TypeError('Context expected to be None or multiprocessing.context')
################################################################################
# Spawn process start method handling logic
################################################################################
_registered_functions = {}
def _register_function(function):
_registered_functions[_qualname(function)] = function
return function
def _trampoline(name, module, *args, **kwargs):
"""Trampoline function for decorators.
Lookups the function between the registered ones;
if not found, forces its registering and then executes it.
"""
function = _function_lookup(name, module)
return function(*args, **kwargs)
def _function_lookup(name, module):
"""Searches the function between the registered ones.
If not found, it imports the module forcing its registration.
"""
try:
return _registered_functions[name]
except KeyError: # force function registering
__import__(module)
mod = sys.modules[module]
function = getattr(mod, name)
try:
return _registered_functions[name]
except KeyError: # decorator without @pie syntax
return _register_function(function)
def _qualname(function):
"""Returns the fully qualified domain of a function."""
try:
return function.__qualname__
except AttributeError: # Python 2
if isinstance(function, types.MethodType):
return '.'.join((function.im_class, function.__name__))
return function.__name__
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1615029565.0
Pebble-4.6.3/pebble/concurrent/thread.py 0000644 0001750 0001750 00000005247 00000000000 020420 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
from functools import wraps
from traceback import format_exc
from concurrent.futures import Future
from pebble.common import launch_thread
def thread(*args, **kwargs):
"""Runs the decorated function within a concurrent thread,
taking care of the result and error management.
Decorated functions will return a concurrent.futures.Future object
once called.
The name parameter will set the thread name.
The daemon parameter controls the underlying thread daemon flag.
Default is True.
"""
name = kwargs.get('name')
daemon = kwargs.get('daemon', True)
# decorator without parameters
if not kwargs and len(args) == 1 and callable(args[0]):
return _thread_wrapper(args[0], name, daemon)
# decorator with parameters
_validate_parameters(name, daemon)
# without @pie syntax
if len(args) == 1 and callable(args[0]):
return _thread_wrapper(args[0], name, daemon)
# with @pie syntax
def decorating_function(function):
return _thread_wrapper(function, name, daemon)
return decorating_function
def _thread_wrapper(function, name, daemon):
@wraps(function)
def wrapper(*args, **kwargs):
future = Future()
launch_thread(name, _function_handler, daemon, function, args, kwargs, future)
return future
return wrapper
def _function_handler(function, args, kwargs, future):
"""Runs the actual function in separate thread and returns its result."""
future.set_running_or_notify_cancel()
try:
result = function(*args, **kwargs)
except BaseException as error:
error.traceback = format_exc()
future.set_exception(error)
else:
future.set_result(result)
def _validate_parameters(name, daemon):
if name is not None and not isinstance(name, str):
raise TypeError('Name expected to be None or string')
if daemon is not None and not isinstance(daemon, bool):
raise TypeError('Daemon expected to be None or bool')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065861.0
Pebble-4.6.3/pebble/decorators.py 0000644 0001750 0001750 00000004432 00000000000 017127 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import signal
import threading
from functools import wraps
_synchronized_lock = threading.Lock()
def synchronized(*args):
"""A synchronized function prevents two or more callers to interleave
its execution preventing race conditions.
The synchronized decorator accepts as optional parameter a Lock, RLock or
Semaphore object which will be employed to ensure the function's atomicity.
If no synchronization object is given, a single threading.Lock will be used.
This implies that between different decorated function only one at a time
will be executed.
"""
if callable(args[0]):
return decorate_synchronized(args[0], _synchronized_lock)
else:
def wrap(function):
return decorate_synchronized(function, args[0])
return wrap
def decorate_synchronized(function, lock):
@wraps(function)
def wrapper(*args, **kwargs):
with lock:
return function(*args, **kwargs)
return wrapper
def sighandler(signals):
"""Sets the decorated function as signal handler of given *signals*.
*signals* can be either a single signal or a list/tuple
of multiple ones.
"""
def wrap(function):
set_signal_handlers(signals, function)
@wraps(function)
def wrapper(*args, **kwargs):
return function(*args, **kwargs)
return wrapper
return wrap
def set_signal_handlers(signals, function):
if isinstance(signals, (list, tuple)):
for signum in signals:
signal.signal(signum, function)
else:
signal.signal(signals, function)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065861.0
Pebble-4.6.3/pebble/functions.py 0000644 0001750 0001750 00000010102 00000000000 016761 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import threading
from time import time
from types import MethodType
_waitforthreads_lock = threading.Lock()
def waitforqueues(queues, timeout=None):
"""Waits for one or more *Queue* to be ready or until *timeout* expires.
*queues* is a list containing one or more *Queue.Queue* objects.
If *timeout* is not None the function will block
for the specified amount of seconds.
The function returns a list containing the ready *Queues*.
"""
lock = threading.Condition(threading.Lock())
prepare_queues(queues, lock)
try:
wait_queues(queues, lock, timeout)
finally:
reset_queues(queues)
return filter(lambda q: not q.empty(), queues)
def prepare_queues(queues, lock):
"""Replaces queue._put() method in order to notify the waiting Condition."""
for queue in queues:
queue._pebble_lock = lock
with queue.mutex:
queue._pebble_old_method = queue._put
queue._put = MethodType(new_method, queue)
def wait_queues(queues, lock, timeout):
with lock:
if not any(map(lambda q: not q.empty(), queues)):
lock.wait(timeout)
def reset_queues(queues):
"""Resets original queue._put() method."""
for queue in queues:
with queue.mutex:
queue._put = queue._pebble_old_method
delattr(queue, '_pebble_old_method')
delattr(queue, '_pebble_lock')
def waitforthreads(threads, timeout=None):
"""Waits for one or more *Thread* to exit or until *timeout* expires.
.. note::
Expired *Threads* are not joined by *waitforthreads*.
*threads* is a list containing one or more *threading.Thread* objects.
If *timeout* is not None the function will block
for the specified amount of seconds.
The function returns a list containing the ready *Threads*.
"""
old_function = None
lock = threading.Condition(threading.Lock())
def new_function(*args):
old_function(*args)
with lock:
lock.notify_all()
old_function = prepare_threads(new_function)
try:
wait_threads(threads, lock, timeout)
finally:
reset_threads(old_function)
return filter(lambda t: not t.is_alive(), threads)
def prepare_threads(new_function):
"""Replaces threading._get_ident() function in order to notify
the waiting Condition."""
with _waitforthreads_lock:
if hasattr(threading, 'get_ident'):
old_function = threading.get_ident
threading.get_ident = new_function
else:
old_function = threading._get_ident
threading._get_ident = new_function
return old_function
def wait_threads(threads, lock, timeout):
timestamp = time()
with lock:
while not any(map(lambda t: not t.is_alive(), threads)):
if timeout is None:
lock.wait()
elif timeout - (time() - timestamp) > 0:
lock.wait(timeout - (time() - timestamp))
else:
return
def reset_threads(old_function):
"""Resets original threading._get_ident() function."""
with _waitforthreads_lock:
if hasattr(threading, 'get_ident'):
threading.get_ident = old_function
else:
threading._get_ident = old_function
def new_method(self, *args):
self._pebble_old_method(*args)
with self._pebble_lock:
self._pebble_lock.notify_all()
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4114294
Pebble-4.6.3/pebble/pool/ 0000755 0001750 0001750 00000000000 00000000000 015356 5 ustar 00noxdafox noxdafox ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065861.0
Pebble-4.6.3/pebble/pool/__init__.py 0000644 0001750 0001750 00000000400 00000000000 017461 0 ustar 00noxdafox noxdafox __all__ = ['ThreadPool',
'ProcessPool',
'MapFuture',
'ProcessMapFuture']
from pebble.pool.thread import ThreadPool
from pebble.pool.process import ProcessPool
from pebble.pool.base_pool import MapFuture, ProcessMapFuture
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614439328.0
Pebble-4.6.3/pebble/pool/base_pool.py 0000644 0001750 0001750 00000015446 00000000000 017705 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import time
import logging
from threading import RLock
from collections import namedtuple
from itertools import chain, count, islice
from concurrent.futures import TimeoutError
try:
from queue import Queue
except ImportError:
from Queue import Queue
from pebble.common import PebbleFuture, ProcessFuture, SLEEP_UNIT
class BasePool(object):
def __init__(self, max_workers, max_tasks, initializer, initargs):
self._context = PoolContext(
max_workers, max_tasks, initializer, initargs)
self._loops = ()
self._task_counter = count()
def __enter__(self):
return self
def __exit__(self, *args):
self.close()
self.join()
@property
def active(self):
self._update_pool_state()
return self._context.state in (CLOSED, RUNNING)
def close(self):
"""Closes the Pool preventing new tasks from being accepted.
Pending tasks will be completed.
"""
self._context.state = CLOSED
def stop(self):
"""Stops the pool without performing any pending task."""
self._context.state = STOPPED
def join(self, timeout=None):
"""Joins the pool waiting until all workers exited.
If *timeout* is set, it block until all workers are done
or raises TimeoutError.
"""
if self._context.state == RUNNING:
raise RuntimeError('The Pool is still running')
if self._context.state == CLOSED:
self._wait_queue_depletion(timeout)
self.stop()
self.join()
else:
self._context.task_queue.put(None)
self._stop_pool()
def _wait_queue_depletion(self, timeout):
tick = time.time()
while self.active:
if timeout is not None and time.time() - tick > timeout:
raise TimeoutError("Tasks are still being executed")
elif self._context.task_queue.unfinished_tasks:
time.sleep(SLEEP_UNIT)
else:
return
def _check_pool_state(self):
self._update_pool_state()
if self._context.state == ERROR:
raise RuntimeError('Unexpected error within the Pool')
elif self._context.state != RUNNING:
raise RuntimeError('The Pool is not active')
def _update_pool_state(self):
if self._context.state == CREATED:
self._start_pool()
for loop in self._loops:
if not loop.is_alive():
self._context.state = ERROR
def _start_pool(self):
raise NotImplementedError("Not implemented")
def _stop_pool(self):
raise NotImplementedError("Not implemented")
class PoolContext(object):
def __init__(self, max_workers, max_tasks, initializer, initargs):
self._state = CREATED
self.state_mutex = RLock()
self.task_queue = Queue()
self.workers = max_workers
self.task_counter = count()
self.worker_parameters = Worker(max_tasks, initializer, initargs)
@property
def state(self):
return self._state
@state.setter
def state(self, state):
with self.state_mutex:
if self.alive:
self._state = state
@property
def alive(self):
return self.state not in (ERROR, STOPPED)
class Task:
def __init__(self, identifier, future, timeout, payload):
self.id = identifier
self.future = future
self.timeout = timeout
self.payload = payload
self.timestamp = 0
self.worker_id = 0
@property
def started(self):
return bool(self.timestamp > 0)
def set_running_or_notify_cancel(self):
if hasattr(self.future, 'map_future'):
if not self.future.map_future.done():
try:
self.future.map_future.set_running_or_notify_cancel()
except RuntimeError:
pass
try:
self.future.set_running_or_notify_cancel()
except RuntimeError:
pass
class MapFuture(PebbleFuture):
def __init__(self, futures):
super(MapFuture, self).__init__()
self._futures = futures
def cancel(self):
"""Cancel the future.
Returns True if any of the elements of the iterables is cancelled.
False otherwise.
"""
super(MapFuture, self).cancel()
return any(tuple(f.cancel() for f in self._futures))
class ProcessMapFuture(ProcessFuture):
def __init__(self, futures):
super(ProcessMapFuture, self).__init__()
self._futures = futures
def cancel(self):
"""Cancel the future.
Returns True if any of the elements of the iterables is cancelled.
False otherwise.
"""
super(ProcessMapFuture, self).cancel()
return any(tuple(f.cancel() for f in self._futures))
class MapResults:
def __init__(self, futures, timeout=None):
self._results = chain.from_iterable(
chunk_result(f, timeout) for f in futures)
def __iter__(self):
return self
def next(self):
result = next(self._results)
if isinstance(result, Exception):
raise result
return result
__next__ = next
def iter_chunks(chunksize, *iterables):
"""Iterates over zipped iterables in chunks."""
iterables = iter(zip(*iterables))
while 1:
chunk = tuple(islice(iterables, chunksize))
if not chunk:
return
yield chunk
def chunk_result(future, timeout):
"""Returns the results of a processed chunk."""
try:
return future.result(timeout=timeout)
except Exception as error:
return (error, )
def run_initializer(initializer, initargs):
"""Runs the Pool initializer dealing with errors."""
try:
initializer(*initargs)
return True
except Exception as error:
logging.exception(error)
return False
# Pool states
CREATED = 0
RUNNING = 1
CLOSED = 2
STOPPED = 3
ERROR = 4
Worker = namedtuple('Worker', ('max_tasks', 'initializer', 'initargs'))
TaskPayload = namedtuple('TaskPayload', ('function', 'args', 'kwargs'))
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1629836945.0
Pebble-4.6.3/pebble/pool/channel.py 0000644 0001750 0001750 00000013134 00000000000 017342 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import os
import select
from contextlib import contextmanager
class ChannelError(OSError):
"""Error occurring within the process channel."""
pass
def channels(mp_context):
read0, write0 = mp_context.Pipe(duplex=False)
read1, write1 = mp_context.Pipe(duplex=False)
return (Channel(read1, write0),
WorkerChannel(read0, write1, (read1, write0), mp_context))
class Channel(object):
def __init__(self, reader, writer):
self.reader = reader
self.writer = writer
self.poll = self._make_poll_method()
def _make_poll_method(self):
def unix_poll(timeout=None):
readonly_mask = (select.POLLIN |
select.POLLPRI |
select.POLLHUP |
select.POLLERR)
poll = select.poll()
poll.register(self.reader, readonly_mask)
# Convert from Seconds to Milliseconds
if timeout is not None:
timeout *= MILLISECONDS
try:
return bool(poll.poll(timeout))
except OSError:
raise
except select.error as err: # Python 2
error = OSError(err.args[1])
error.errno = err.args[0]
raise error
def windows_poll(timeout=None):
return self.reader.poll(timeout)
return unix_poll if os.name != 'nt' else windows_poll
def recv(self):
return self.reader.recv()
def send(self, obj):
return self.writer.send(obj)
def close(self):
self.reader.close()
self.writer.close()
class WorkerChannel(Channel):
def __init__(self, reader, writer, unused, mp_context):
super(WorkerChannel, self).__init__(reader, writer)
self.mutex = ChannelMutex(mp_context)
self.recv = self._make_recv_method()
self.send = self._make_send_method()
self.unused = unused
def __getstate__(self):
return self.reader, self.writer, self.mutex, self.unused
def __setstate__(self, state):
self.reader, self.writer, self.mutex, self.unused = state
self.poll = self._make_poll_method()
self.recv = self._make_recv_method()
self.send = self._make_send_method()
def _make_recv_method(self):
def recv():
with self.mutex.reader:
return self.reader.recv()
return recv
def _make_send_method(self):
def unix_send(obj):
with self.mutex.writer:
return self.writer.send(obj)
def windows_send(obj):
return self.writer.send(obj)
return unix_send if os.name != 'nt' else windows_send
@property
@contextmanager
def lock(self):
with self.mutex:
yield self
def initialize(self):
"""Close unused connections."""
for connection in self.unused:
connection.close()
class ChannelMutex:
def __init__(self, mp_context):
self.reader_mutex = mp_context.RLock()
self.writer_mutex = mp_context.RLock() if os.name != 'nt' else None
self.acquire = self._make_acquire_method()
self.release = self._make_release_method()
def __getstate__(self):
return self.reader_mutex, self.writer_mutex
def __setstate__(self, state):
self.reader_mutex, self.writer_mutex = state
self.acquire = self._make_acquire_method()
self.release = self._make_release_method()
def __enter__(self):
if self.acquire():
return self
raise ChannelError("Channel mutex time out")
def __exit__(self, *_):
self.release()
def _make_acquire_method(self):
def unix_acquire():
return (self.reader_mutex.acquire(timeout=LOCK_TIMEOUT) and
self.writer_mutex.acquire(timeout=LOCK_TIMEOUT))
def windows_acquire():
return self.reader_mutex.acquire(timeout=LOCK_TIMEOUT)
return unix_acquire if os.name != 'nt' else windows_acquire
def _make_release_method(self):
def unix_release():
self.reader_mutex.release()
self.writer_mutex.release()
def windows_release():
self.reader_mutex.release()
return unix_release if os.name != 'nt' else windows_release
@property
@contextmanager
def reader(self):
if self.reader_mutex.acquire(timeout=LOCK_TIMEOUT):
try:
yield self
finally:
self.reader_mutex.release()
else:
raise ChannelError("Channel mutex time out")
@property
@contextmanager
def writer(self):
if self.writer_mutex.acquire(timeout=LOCK_TIMEOUT):
try:
yield self
finally:
self.writer_mutex.release()
else:
raise ChannelError("Channel mutex time out")
MILLISECONDS = 1000
LOCK_TIMEOUT = 60
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614424439.0
Pebble-4.6.3/pebble/pool/process.py 0000644 0001750 0001750 00000040644 00000000000 017416 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import os
import time
import atexit
import pickle
import multiprocessing
from itertools import count
from collections import namedtuple
from signal import SIG_IGN, SIGINT, signal
from concurrent.futures import CancelledError, TimeoutError
try:
from concurrent.futures.process import BrokenProcessPool
except ImportError:
class BrokenProcessPool(OSError):
pass
from pebble.pool.channel import ChannelError, channels
from pebble.pool.base_pool import BasePool, Task, TaskPayload
from pebble.pool.base_pool import ProcessMapFuture, MapResults
from pebble.pool.base_pool import iter_chunks, run_initializer
from pebble.pool.base_pool import CREATED, ERROR, RUNNING, SLEEP_UNIT
from pebble.common import launch_process, stop_process
from pebble.common import ProcessExpired, ProcessFuture
from pebble.common import process_execute, launch_thread
class ProcessPool(BasePool):
"""Allows to schedule jobs within a Pool of Processes.
max_workers is an integer representing the amount of desired process workers
managed by the pool.
If max_tasks is a number greater than zero,
each worker will be restarted after performing an equal amount of tasks.
initializer must be callable, if passed, it will be called
every time a worker is started, receiving initargs as arguments.
"""
def __init__(self, max_workers=multiprocessing.cpu_count(), max_tasks=0,
initializer=None, initargs=(), context=None):
super(ProcessPool, self).__init__(
max_workers, max_tasks, initializer, initargs)
mp_context = multiprocessing if context is None else context
self._pool_manager = PoolManager(self._context, mp_context)
self._task_scheduler_loop = None
self._pool_manager_loop = None
self._message_manager_loop = None
def _start_pool(self):
with self._context.state_mutex:
if self._context.state == CREATED:
self._pool_manager.start()
self._task_scheduler_loop = launch_thread(
None, task_scheduler_loop, True, self._pool_manager)
self._pool_manager_loop = launch_thread(
None, pool_manager_loop, True, self._pool_manager)
self._message_manager_loop = launch_thread(
None, message_manager_loop, True, self._pool_manager)
self._context.state = RUNNING
def _stop_pool(self):
if self._pool_manager_loop is not None:
self._pool_manager_loop.join()
self._pool_manager.close()
if self._task_scheduler_loop is not None:
self._task_scheduler_loop.join()
if self._message_manager_loop is not None:
self._message_manager_loop.join()
self._pool_manager.stop()
def schedule(self, function, args=(), kwargs={}, timeout=None):
"""Schedules *function* to be run the Pool.
*args* and *kwargs* will be forwareded to the scheduled function
respectively as arguments and keyword arguments.
*timeout* is an integer, if expires the task will be terminated
and *Future.result()* will raise *TimeoutError*.
A *pebble.ProcessFuture* object is returned.
"""
self._check_pool_state()
future = ProcessFuture()
payload = TaskPayload(function, args, kwargs)
task = Task(next(self._task_counter), future, timeout, payload)
self._context.task_queue.put(task)
return future
def map(self, function, *iterables, **kwargs):
"""Computes the *function* using arguments from
each of the iterables. Stops when the shortest iterable is exhausted.
*timeout* is an integer, if expires the task will be terminated
and the call to next will raise *TimeoutError*.
The *timeout* is applied to each chunk of the iterable.
*chunksize* controls the size of the chunks the iterable will
be broken into before being passed to the function.
A *pebble.ProcessFuture* object is returned.
"""
self._check_pool_state()
timeout = kwargs.get('timeout')
chunksize = kwargs.get('chunksize', 1)
if chunksize < 1:
raise ValueError("chunksize must be >= 1")
futures = [self.schedule(
process_chunk, args=(function, chunk), timeout=timeout)
for chunk in iter_chunks(chunksize, *iterables)]
map_future = ProcessMapFuture(futures)
if not futures:
map_future.set_result(MapResults(futures))
return map_future
def done_map(_):
if not map_future.done():
map_future.set_result(MapResults(futures, timeout=timeout))
for future in futures:
future.add_done_callback(done_map)
setattr(future, 'map_future', map_future)
return map_future
def task_scheduler_loop(pool_manager):
context = pool_manager.context
task_queue = context.task_queue
try:
while context.alive and not global_shutdown:
task = task_queue.get()
if task is not None:
if task.future.cancelled():
task.set_running_or_notify_cancel()
task_queue.task_done()
else:
pool_manager.schedule(task)
else:
task_queue.task_done()
except BrokenProcessPool:
context.state = ERROR
def pool_manager_loop(pool_manager):
context = pool_manager.context
try:
while context.alive and not global_shutdown:
pool_manager.update_status()
time.sleep(SLEEP_UNIT)
except BrokenProcessPool:
context.state = ERROR
def message_manager_loop(pool_manager):
context = pool_manager.context
try:
while context.alive and not global_shutdown:
pool_manager.process_next_message(SLEEP_UNIT)
except BrokenProcessPool:
context.state = ERROR
class PoolManager:
"""Combines Task and Worker Managers providing a higher level one."""
def __init__(self, context, mp_context):
self.context = context
self.task_manager = TaskManager(context.task_queue.task_done)
self.worker_manager = WorkerManager(context.workers,
context.worker_parameters,
mp_context)
def start(self):
self.worker_manager.create_workers()
def close(self):
self.worker_manager.close_channels()
def stop(self):
self.worker_manager.stop_workers()
def schedule(self, task):
"""Schedules a new Task in the PoolManager."""
self.task_manager.register(task)
try:
self.worker_manager.dispatch(task)
except (pickle.PicklingError, TypeError) as error:
self.task_manager.task_problem(task.id, error)
def process_next_message(self, timeout):
"""Processes the next message coming from the workers."""
message = self.worker_manager.receive(timeout)
if isinstance(message, Acknowledgement):
self.task_manager.task_start(message.task, message.worker)
elif isinstance(message, Result):
self.task_manager.task_done(message.task, message.result)
elif isinstance(message, Problem):
self.task_manager.task_problem(message.task, message.error)
def update_status(self):
self.update_tasks()
self.update_workers()
def update_tasks(self):
"""Handles timing out Tasks."""
for task in self.task_manager.timeout_tasks():
self.task_manager.task_done(
task.id, TimeoutError("Task timeout", task.timeout))
self.worker_manager.stop_worker(task.worker_id)
for task in self.task_manager.cancelled_tasks():
self.task_manager.task_done(
task.id, CancelledError())
self.worker_manager.stop_worker(task.worker_id)
def update_workers(self):
"""Handles unexpected processes termination."""
for expiration in self.worker_manager.inspect_workers():
self.handle_worker_expiration(expiration)
self.worker_manager.create_workers()
def handle_worker_expiration(self, expiration):
worker_id, exitcode = expiration
try:
task = self.find_expired_task(worker_id)
except LookupError:
return
else:
error = ProcessExpired('Abnormal termination', code=exitcode)
self.task_manager.task_done(task.id, error)
def find_expired_task(self, worker_id):
tasks = tuple(self.task_manager.tasks.values())
running_tasks = tuple(t for t in tasks if t.worker_id != 0)
if running_tasks:
return task_worker_lookup(running_tasks, worker_id)
else:
raise BrokenProcessPool("All workers expired")
class TaskManager:
"""Manages the tasks flow within the Pool.
Tasks are registered, acknowledged and completed.
Timing out and cancelled tasks are handled as well.
"""
def __init__(self, task_done_callback):
self.tasks = {}
self.task_done_callback = task_done_callback
def register(self, task):
self.tasks[task.id] = task
def task_start(self, task_id, worker_id):
task = self.tasks[task_id]
task.worker_id = worker_id
task.timestamp = time.time()
task.set_running_or_notify_cancel()
def task_done(self, task_id, result):
"""Set the tasks result and run the callback."""
try:
task = self.tasks.pop(task_id)
except KeyError:
return # result of previously timeout Task
else:
if task.future.cancelled():
task.set_running_or_notify_cancel()
elif isinstance(result, BaseException):
task.future.set_exception(result)
else:
task.future.set_result(result)
self.task_done_callback()
def task_problem(self, task_id, error):
"""Set the task with the error it caused within the Pool."""
self.task_start(task_id, None)
self.task_done(task_id, error)
def timeout_tasks(self):
return tuple(t for t in tuple(self.tasks.values()) if self.timeout(t))
def cancelled_tasks(self):
return tuple(t for t in tuple(self.tasks.values())
if t.timestamp != 0 and t.future.cancelled())
@staticmethod
def timeout(task):
if task.timeout and task.started:
return time.time() - task.timestamp > task.timeout
else:
return False
class WorkerManager:
"""Manages the workers related mechanics within the Pool.
Maintains the workers active and encapsulates their communication logic.
"""
def __init__(self, workers, worker_parameters, mp_context):
self.workers = {}
self.workers_number = workers
self.worker_parameters = worker_parameters
self.pool_channel, self.workers_channel = channels(mp_context)
self.mp_context = mp_context
def dispatch(self, task):
try:
self.pool_channel.send(WorkerTask(task.id, task.payload))
except (pickle.PicklingError, TypeError) as error:
raise error
except (OSError, EnvironmentError, TypeError) as error:
raise BrokenProcessPool(error)
def receive(self, timeout):
try:
if self.pool_channel.poll(timeout):
return self.pool_channel.recv()
else:
return NoMessage()
except (OSError, EnvironmentError, TypeError) as error:
raise BrokenProcessPool(error)
except EOFError: # Pool shutdown
return NoMessage()
def inspect_workers(self):
"""Updates the workers status.
Returns the workers which have unexpectedly ended.
"""
workers = tuple(self.workers.values())
expired = tuple(w for w in workers if not w.is_alive())
for worker in expired:
self.workers.pop(worker.pid)
return ((w.pid, w.exitcode) for w in expired if w.exitcode != 0)
def create_workers(self):
for _ in range(self.workers_number - len(self.workers)):
self.new_worker()
def close_channels(self):
self.pool_channel.close()
self.workers_channel.close()
def stop_workers(self):
for worker_id in tuple(self.workers.keys()):
self.stop_worker(worker_id, force=True)
def new_worker(self):
try:
worker = launch_process(
WORKERS_NAME, worker_process, False, self.mp_context,
self.worker_parameters, self.workers_channel)
self.workers[worker.pid] = worker
except (OSError, EnvironmentError) as error:
raise BrokenProcessPool(error)
def stop_worker(self, worker_id, force=False):
try:
if force:
stop_process(self.workers.pop(worker_id))
else:
with self.workers_channel.lock:
stop_process(self.workers.pop(worker_id))
except ChannelError as error:
raise BrokenProcessPool(error)
except KeyError:
return # worker already expired
def worker_process(params, channel):
"""The worker process routines."""
signal(SIGINT, SIG_IGN)
channel.initialize()
if params.initializer is not None:
if not run_initializer(params.initializer, params.initargs):
os._exit(1)
try:
for task in worker_get_next_task(channel, params.max_tasks):
payload = task.payload
result = process_execute(
payload.function, *payload.args, **payload.kwargs)
send_result(channel, Result(task.id, result))
except (EnvironmentError, OSError, RuntimeError) as error:
errno = getattr(error, 'errno', 1)
os._exit(errno if isinstance(errno, int) else 1)
except EOFError as error:
os._exit(0)
def worker_get_next_task(channel, max_tasks):
counter = count()
while max_tasks == 0 or next(counter) < max_tasks:
yield fetch_task(channel)
def send_result(pipe, result):
"""Send result handling pickling and communication errors."""
try:
pipe.send(result)
except (pickle.PicklingError, TypeError) as error:
pipe.send(Problem(result.task, error))
def fetch_task(channel):
while channel.poll():
try:
return task_transaction(channel)
except RuntimeError:
continue # another worker got the task
def task_transaction(channel):
"""Ensures a task is fetched and acknowledged atomically."""
with channel.lock:
if channel.poll(0):
task = channel.recv()
channel.send(Acknowledgement(os.getpid(), task.id))
else:
raise RuntimeError("Race condition between workers")
return task
def task_worker_lookup(running_tasks, worker_id):
for task in running_tasks:
if task.worker_id == worker_id:
return task
raise LookupError("Not found")
def process_chunk(function, chunk):
"""Processes a chunk of the iterable passed to map dealing with errors."""
return [process_execute(function, *args) for args in chunk]
def interpreter_shutdown():
global global_shutdown
global_shutdown = True
workers = [p for p in multiprocessing.active_children()
if p.name == WORKERS_NAME]
for worker in workers:
stop_process(worker)
atexit.register(interpreter_shutdown)
global_shutdown = False
WORKERS_NAME = 'pebble_pool_worker'
NoMessage = namedtuple('NoMessage', ())
Result = namedtuple('Result', ('task', 'result'))
Problem = namedtuple('Problem', ('task', 'error'))
WorkerTask = namedtuple('WorkerTask', ('id', 'payload'))
Acknowledgement = namedtuple('Acknowledgement', ('worker', 'task'))
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614019999.0
Pebble-4.6.3/pebble/pool/thread.py 0000644 0001750 0001750 00000014536 00000000000 017210 0 ustar 00noxdafox noxdafox # This file is part of Pebble.
# Copyright (c) 2013-2021, Matteo Cafasso
# Pebble is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
# Pebble is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with Pebble. If not, see .
import time
from itertools import count
from multiprocessing import cpu_count
from concurrent.futures import Future
from pebble.common import execute, launch_thread
from pebble.pool.base_pool import MapFuture, MapResults
from pebble.pool.base_pool import BasePool, Task, TaskPayload
from pebble.pool.base_pool import iter_chunks, run_initializer
from pebble.pool.base_pool import CREATED, ERROR, RUNNING, SLEEP_UNIT
class ThreadPool(BasePool):
"""Allows to schedule jobs within a Pool of Threads.
max_workers is an integer representing the amount of desired process workers
managed by the pool.
If max_tasks is a number greater than zero,
each worker will be restarted after performing an equal amount of tasks.
initializer must be callable, if passed, it will be called
every time a worker is started, receiving initargs as arguments.
"""
def __init__(self, max_workers=cpu_count(), max_tasks=0,
initializer=None, initargs=()):
super(ThreadPool, self).__init__(
max_workers, max_tasks, initializer, initargs)
self._pool_manager = PoolManager(self._context)
self._pool_manager_loop = None
def _start_pool(self):
with self._context.state_mutex:
if self._context.state == CREATED:
self._pool_manager.start()
self._pool_manager_loop = launch_thread(
None, pool_manager_loop, True, self._pool_manager)
self._context.state = RUNNING
def _stop_pool(self):
if self._pool_manager_loop is not None:
self._pool_manager_loop.join()
self._pool_manager.stop()
def schedule(self, function, args=(), kwargs={}):
"""Schedules *function* to be run the Pool.
*args* and *kwargs* will be forwareded to the scheduled function
respectively as arguments and keyword arguments.
A *concurrent.futures.Future* object is returned.
"""
self._check_pool_state()
future = Future()
payload = TaskPayload(function, args, kwargs)
task = Task(next(self._task_counter), future, None, payload)
self._context.task_queue.put(task)
return future
def map(self, function, *iterables, **kwargs):
"""Returns an iterator equivalent to map(function, iterables).
*chunksize* controls the size of the chunks the iterable will
be broken into before being passed to the function. If None
the size will be controlled by the Pool.
"""
self._check_pool_state()
timeout = kwargs.get('timeout')
chunksize = kwargs.get('chunksize', 1)
if chunksize < 1:
raise ValueError("chunksize must be >= 1")
futures = [self.schedule(process_chunk, args=(function, chunk))
for chunk in iter_chunks(chunksize, *iterables)]
map_future = MapFuture(futures)
if not futures:
map_future.set_result(MapResults(futures))
return map_future
def done_map(_):
if not map_future.done():
map_future.set_result(MapResults(futures, timeout=timeout))
for future in futures:
future.add_done_callback(done_map)
setattr(future, 'map_future', map_future)
return map_future
def pool_manager_loop(pool_manager):
context = pool_manager.context
while context.alive:
pool_manager.update_status()
time.sleep(SLEEP_UNIT)
class PoolManager:
def __init__(self, context):
self.workers = []
self.context = context
def start(self):
self.create_workers()
def stop(self):
for worker in self.workers:
self.context.task_queue.put(None)
for worker in tuple(self.workers):
self.join_worker(worker)
def update_status(self):
expired = self.inspect_workers()
for worker in expired:
self.join_worker(worker)
self.create_workers()
def inspect_workers(self):
return tuple(w for w in self.workers if not w.is_alive())
def create_workers(self):
for _ in range(self.context.workers - len(self.workers)):
worker = launch_thread(None, worker_thread, True, self.context)
self.workers.append(worker)
def join_worker(self, worker):
worker.join()
self.workers.remove(worker)
def worker_thread(context):
"""The worker thread routines."""
queue = context.task_queue
parameters = context.worker_parameters
if parameters.initializer is not None:
if not run_initializer(parameters.initializer, parameters.initargs):
context.state = ERROR
return
for task in get_next_task(context, parameters.max_tasks):
execute_next_task(task)
queue.task_done()
def get_next_task(context, max_tasks):
counter = count()
queue = context.task_queue
while context.alive and (max_tasks == 0 or next(counter) < max_tasks):
task = queue.get()
if task is not None:
if task.future.cancelled():
task.set_running_or_notify_cancel()
queue.task_done()
else:
yield task
def execute_next_task(task):
payload = task.payload
task.timestamp = time.time()
task.set_running_or_notify_cancel()
result = execute(payload.function, *payload.args, **payload.kwargs)
if isinstance(result, BaseException):
task.future.set_exception(result)
else:
task.future.set_result(result)
def process_chunk(function, chunk):
"""Processes a chunk of the iterable passed to map dealing with errors."""
return [execute(function, *args) for args in chunk]
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4114294
Pebble-4.6.3/setup.cfg 0000644 0001750 0001750 00000000103 00000000000 014767 0 ustar 00noxdafox noxdafox [bdist_wheel]
universal = 1
[egg_info]
tag_build =
tag_date = 0
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065860.0
Pebble-4.6.3/setup.py 0000644 0001750 0001750 00000002343 00000000000 014670 0 ustar 00noxdafox noxdafox import os
import fileinput
from setuptools import setup, find_packages
CWD = os.path.dirname(__file__)
def package_version():
module_path = os.path.join(CWD, 'pebble', '__init__.py')
for line in fileinput.input(module_path):
if line.startswith('__version__'):
return line.split('=')[-1].strip().replace('\'', '')
setup(
name="Pebble",
version=package_version(),
author="Matteo Cafasso",
author_email="noxdafox@gmail.com",
description=("Threading and multiprocessing eye-candy."),
license="LGPL",
keywords="thread process pool decorator",
url="https://github.com/noxdafox/pebble",
packages=find_packages(exclude=["tests"]),
extras_require={":python_version<'3'": ["futures"]},
long_description=open(os.path.join(CWD, 'README.rst')).read(),
classifiers=[
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"Operating System :: OS Independent",
"Topic :: Software Development :: Libraries :: Python Modules",
"License :: OSI Approved :: " +
"GNU Library or Lesser General Public License (LGPL)"
],
)
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1629838196.4114294
Pebble-4.6.3/test/ 0000755 0001750 0001750 00000000000 00000000000 014133 5 ustar 00noxdafox noxdafox ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614893633.0
Pebble-4.6.3/test/test_concurrent_process_fork.py 0000644 0001750 0001750 00000023006 00000000000 022506 0 ustar 00noxdafox noxdafox import os
import sys
import time
import pickle
import signal
import unittest
import threading
import multiprocessing
from concurrent.futures import CancelledError, TimeoutError
from pebble import concurrent, ProcessExpired
# set start method
supported = False
mp_context = None
if sys.version_info.major > 2 and sys.version_info.minor > 3:
methods = multiprocessing.get_all_start_methods()
if 'fork' in methods:
try:
mp_context = multiprocessing.get_context('fork')
if mp_context.get_start_method() == 'fork':
supported = True
except RuntimeError: # child process
pass
else:
supported = True
def not_decorated(argument, keyword_argument=0):
return argument + keyword_argument
@concurrent.process(context=mp_context)
def decorated(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
@concurrent.process(context=mp_context)
def error_decorated():
raise RuntimeError("BOOM!")
@concurrent.process(context=mp_context)
def pickling_error_decorated():
event = threading.Event()
return event
@concurrent.process(context=mp_context)
def critical_decorated():
os._exit(123)
@concurrent.process(context=mp_context)
def decorated_cancel():
time.sleep(10)
@concurrent.process(timeout=0.1, context=mp_context)
def long_decorated():
time.sleep(10)
@concurrent.process(timeout=0.1, context=mp_context)
def sigterm_decorated():
signal.signal(signal.SIGTERM, signal.SIG_IGN)
time.sleep(10)
@concurrent.process(context=mp_context)
def name_keyword_argument(name='function_kwarg'):
return name
@concurrent.process(name='concurrent_process_name', context=mp_context)
def name_keyword_decorated():
return multiprocessing.current_process().name
@concurrent.process(name='decorator_kwarg', context=mp_context)
def name_keyword_decorated_and_argument(name='bar'):
return (multiprocessing.current_process().name, name)
@concurrent.process(daemon=False, context=mp_context)
def daemon_keyword_decorated():
return multiprocessing.current_process().daemon
class ProcessConcurrentObj:
a = 0
def __init__(self):
self.b = 1
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b
@staticmethod
@concurrent.process(context=mp_context)
def stcmethod():
return 2
class ProcessConcurrentSub1(ProcessConcurrentObj):
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a + 1
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b + 1
@staticmethod
@concurrent.process(context=mp_context)
def stcmethod():
return 2 + 1
class ProcessConcurrentSub2(ProcessConcurrentObj):
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a + 2
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b + 2
@staticmethod
@concurrent.process(context=mp_context)
def stcmethod():
return 2 + 2
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessConcurrent(unittest.TestCase):
def setUp(self):
self.results = 0
self.exception = None
self.event = threading.Event()
self.event.clear()
self.concurrentobj = ProcessConcurrentObj()
self.concurrentobj1 = ProcessConcurrentSub1()
self.concurrentobj2 = ProcessConcurrentSub2()
def callback(self, future):
try:
self.results = future.result()
except (ProcessExpired, RuntimeError, TimeoutError) as error:
self.exception = error
finally:
self.event.set()
def test_docstring(self):
"""Process Fork docstring is preserved."""
self.assertEqual(decorated.__doc__, "A docstring.")
def test_wrong_timeout(self):
"""Process Fork TypeError is raised if timeout is not number."""
with self.assertRaises(TypeError):
@concurrent.process(timeout='Foo', context=mp_context)
def function():
return
def test_class_method(self):
"""Process Fork decorated classmethods."""
future = ProcessConcurrentObj.clsmethod()
self.assertEqual(future.result(), 0)
future = ProcessConcurrentSub1.clsmethod()
self.assertEqual(future.result(), 1)
future = ProcessConcurrentSub2.clsmethod()
self.assertEqual(future.result(), 2)
def test_instance_method(self):
"""Process Fork decorated instance methods."""
future = self.concurrentobj.instmethod()
self.assertEqual(future.result(), 1)
future = self.concurrentobj1.instmethod()
self.assertEqual(future.result(), 2)
future = self.concurrentobj2.instmethod()
self.assertEqual(future.result(), 3)
def test_static_method(self):
"""Process Fork decorated static methods (Fork startmethod only)."""
future = self.concurrentobj.stcmethod()
self.assertEqual(future.result(), 2)
future = self.concurrentobj1.stcmethod()
self.assertEqual(future.result(), 3)
future = self.concurrentobj2.stcmethod()
self.assertEqual(future.result(), 4)
def test_not_decorated_results(self):
"""Process Fork results are produced."""
non_decorated = concurrent.process(not_decorated, context=mp_context)
future = non_decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results(self):
"""Process Fork results are produced."""
future = decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results_callback(self):
"""Process Fork results are forwarded to the callback."""
future = decorated(1, 1)
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertEqual(self.results, 2)
def test_error_decorated(self):
"""Process Fork errors are raised by future.result."""
future = error_decorated()
with self.assertRaises(RuntimeError):
future.result()
def test_error_decorated_callback(self):
"""Process Fork errors are forwarded to callback."""
future = error_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, RuntimeError),
msg=str(self.exception))
def test_pickling_error_decorated(self):
"""Process Fork pickling errors are raised by future.result."""
future = pickling_error_decorated()
with self.assertRaises((pickle.PicklingError, TypeError)):
future.result()
def test_timeout_decorated(self):
"""Process Fork raises TimeoutError if so."""
future = long_decorated()
with self.assertRaises(TimeoutError):
future.result()
def test_timeout_decorated_callback(self):
"""Process Fork TimeoutError is forwarded to callback."""
future = long_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, TimeoutError),
msg=str(self.exception))
def test_decorated_dead_process(self):
"""Process Fork ProcessExpired is raised if process dies."""
future = critical_decorated()
with self.assertRaises(ProcessExpired):
future.result()
def test_timeout_decorated_callback(self):
"""Process Fork ProcessExpired is forwarded to callback."""
future = critical_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, ProcessExpired),
msg=str(self.exception))
def test_cancel_decorated(self):
"""Process Fork raises CancelledError if future was cancelled."""
future = decorated_cancel()
future.cancel()
self.assertRaises(CancelledError, future.result)
@unittest.skipIf(os.name == 'nt', "Test won't run on Windows.")
def test_decorated_ignoring_sigterm(self):
"""Process Fork Concurrent ignored SIGTERM signal are handled on Unix."""
future = sigterm_decorated()
with self.assertRaises(TimeoutError):
future.result()
def test_name_keyword_argument(self):
"""name keyword can be passed to a decorated function process without name"""
f = name_keyword_argument()
fn_out = f.result()
self.assertEqual(fn_out, "function_kwarg")
def test_name_keyword_decorated(self):
"""
Check that a simple use case of the name keyword passed to the decorator works
"""
f = name_keyword_decorated()
dec_out = f.result()
self.assertEqual(dec_out, "concurrent_process_name")
def test_name_keyword_decorated_result_colision(self):
"""name kwarg is handled without modifying the function kwargs"""
f = name_keyword_decorated_and_argument(name="function_kwarg")
dec_out, fn_out = f.result()
self.assertEqual(dec_out, "decorator_kwarg")
self.assertEqual(fn_out, "function_kwarg")
def test_daemon_keyword_decorated(self):
"""Daemon keyword can be passed to a decorated function and spawns correctly."""
f = daemon_keyword_decorated()
dec_out = f.result()
self.assertEqual(dec_out, False)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614893822.0
Pebble-4.6.3/test/test_concurrent_process_forkserver.py 0000644 0001750 0001750 00000017175 00000000000 023747 0 ustar 00noxdafox noxdafox import os
import sys
import time
import pickle
import signal
import unittest
import threading
import multiprocessing
from concurrent.futures import CancelledError, TimeoutError
from pebble import concurrent, ProcessExpired
# set start method
supported = False
mp_context = None
if sys.version_info.major > 2 and sys.version_info.minor > 3:
methods = multiprocessing.get_all_start_methods()
if 'forkserver' in methods:
try:
mp_context = multiprocessing.get_context('forkserver')
if mp_context.get_start_method() == 'forkserver':
supported = True
except RuntimeError: # child process
pass
def not_decorated(argument, keyword_argument=0):
return argument + keyword_argument
@concurrent.process(context=mp_context)
def decorated(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
@concurrent.process(context=mp_context)
def error_decorated():
raise RuntimeError("BOOM!")
@concurrent.process(context=mp_context)
def pickling_error_decorated():
event = threading.Event()
return event
@concurrent.process(context=mp_context)
def critical_decorated():
os._exit(123)
@concurrent.process(context=mp_context)
def decorated_cancel():
time.sleep(10)
@concurrent.process(timeout=0.1, context=mp_context)
def long_decorated():
time.sleep(10)
@concurrent.process(timeout=0.1, context=mp_context)
def sigterm_decorated():
signal.signal(signal.SIGTERM, signal.SIG_IGN)
time.sleep(10)
@concurrent.process(daemon=False, context=mp_context)
def daemon_keyword_decorated():
return multiprocessing.current_process().daemon
class ProcessConcurrentObj:
a = 0
def __init__(self):
self.b = 1
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b
class ProcessConcurrentSub1(ProcessConcurrentObj):
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a + 1
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b + 1
class ProcessConcurrentSub2(ProcessConcurrentObj):
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a + 2
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b + 2
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessConcurrent(unittest.TestCase):
def setUp(self):
self.results = 0
self.exception = None
self.event = threading.Event()
self.event.clear()
self.concurrentobj = ProcessConcurrentObj()
self.concurrentobj1 = ProcessConcurrentSub1()
self.concurrentobj2 = ProcessConcurrentSub2()
def callback(self, future):
try:
self.results = future.result()
except (ProcessExpired, RuntimeError, TimeoutError) as error:
self.exception = error
finally:
self.event.set()
def test_docstring(self):
"""Process Forkserver docstring is preserved."""
self.assertEqual(decorated.__doc__, "A docstring.")
def test_wrong_timeout(self):
"""Process Forkserver TypeError is raised if timeout is not number."""
with self.assertRaises(TypeError):
@concurrent.process(timeout='Foo', context=mp_context)
def function():
return
def test_class_method(self):
"""Process Forkserver decorated classmethods."""
future = ProcessConcurrentObj.clsmethod()
self.assertEqual(future.result(), 0)
future = ProcessConcurrentSub1.clsmethod()
self.assertEqual(future.result(), 1)
future = ProcessConcurrentSub2.clsmethod()
self.assertEqual(future.result(), 2)
def test_instance_method(self):
"""Process Forkserver decorated instance methods."""
future = self.concurrentobj.instmethod()
self.assertEqual(future.result(), 1)
future = self.concurrentobj1.instmethod()
self.assertEqual(future.result(), 2)
future = self.concurrentobj2.instmethod()
self.assertEqual(future.result(), 3)
def test_not_decorated_results(self):
"""Process Forkserver results are produced."""
non_decorated = concurrent.process(not_decorated, context=mp_context)
future = non_decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results(self):
"""Process Forkserver results are produced."""
future = decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results_callback(self):
"""Process Forkserver results are forwarded to the callback."""
future = decorated(1, 1)
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertEqual(self.results, 2)
def test_error_decorated(self):
"""Process Forkserver errors are raised by future.result."""
future = error_decorated()
with self.assertRaises(RuntimeError):
future.result()
def test_error_decorated_callback(self):
"""Process Forkserver errors are forwarded to callback."""
future = error_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, RuntimeError),
msg=str(self.exception))
def test_pickling_error_decorated(self):
"""Process Forkserver pickling errors are raised by future.result."""
future = pickling_error_decorated()
with self.assertRaises((pickle.PicklingError, TypeError)):
future.result()
def test_timeout_decorated(self):
"""Process Forkserver raises TimeoutError if so."""
future = long_decorated()
with self.assertRaises(TimeoutError):
future.result()
def test_timeout_decorated_callback(self):
"""Process Forkserver TimeoutError is forwarded to callback."""
future = long_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, TimeoutError),
msg=str(self.exception))
def test_decorated_dead_process(self):
"""Process Forkserver ProcessExpired is raised if process dies."""
future = critical_decorated()
with self.assertRaises(ProcessExpired):
future.result()
def test_timeout_decorated_callback(self):
"""Process Forkserver ProcessExpired is forwarded to callback."""
future = critical_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, ProcessExpired),
msg=str(self.exception))
def test_cancel_decorated(self):
"""Process Forkserver raises CancelledError if future was cancelled."""
future = decorated_cancel()
future.cancel()
self.assertRaises(CancelledError, future.result)
@unittest.skipIf(os.name == 'nt', "Test won't run on Windows.")
def test_decorated_ignoring_sigterm(self):
"""Process Forkserver Concurrent ignored SIGTERM signal are handled on Unix."""
future = sigterm_decorated()
with self.assertRaises(TimeoutError):
future.result()
def test_daemon_keyword_decorated(self):
"""Daemon keyword can be passed to a decorated function and spawns correctly."""
f = daemon_keyword_decorated()
dec_out = f.result()
self.assertEqual(dec_out, False)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614893836.0
Pebble-4.6.3/test/test_concurrent_process_spawn.py 0000644 0001750 0001750 00000017036 00000000000 022703 0 ustar 00noxdafox noxdafox import os
import sys
import time
import pickle
import signal
import unittest
import threading
import multiprocessing
from concurrent.futures import CancelledError, TimeoutError
from pebble import concurrent, ProcessExpired
# set start method
supported = False
mp_context = None
if sys.version_info.major > 2 and sys.version_info.minor > 3:
methods = multiprocessing.get_all_start_methods()
if 'spawn' in methods:
try:
mp_context = multiprocessing.get_context('spawn')
if mp_context.get_start_method() == 'spawn':
supported = True
except RuntimeError: # child process
pass
def not_decorated(argument, keyword_argument=0):
return argument + keyword_argument
@concurrent.process(context=mp_context)
def decorated(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
@concurrent.process(context=mp_context)
def error_decorated():
raise RuntimeError("BOOM!")
@concurrent.process(context=mp_context)
def pickling_error_decorated():
event = threading.Event()
return event
@concurrent.process(context=mp_context)
def critical_decorated():
os._exit(123)
@concurrent.process(context=mp_context)
def decorated_cancel():
time.sleep(10)
@concurrent.process(timeout=0.1, context=mp_context)
def long_decorated():
time.sleep(10)
@concurrent.process(timeout=0.1, context=mp_context)
def sigterm_decorated():
signal.signal(signal.SIGTERM, signal.SIG_IGN)
time.sleep(10)
@concurrent.process(daemon=False, context=mp_context)
def daemon_keyword_decorated():
return multiprocessing.current_process().daemon
class ProcessConcurrentObj:
a = 0
def __init__(self):
self.b = 1
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b
class ProcessConcurrentSub1(ProcessConcurrentObj):
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a + 1
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b + 1
class ProcessConcurrentSub2(ProcessConcurrentObj):
@classmethod
@concurrent.process(context=mp_context)
def clsmethod(cls):
return cls.a + 2
@concurrent.process(context=mp_context)
def instmethod(self):
return self.b + 2
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessConcurrent(unittest.TestCase):
def setUp(self):
self.results = 0
self.exception = None
self.event = threading.Event()
self.event.clear()
self.concurrentobj = ProcessConcurrentObj()
self.concurrentobj1 = ProcessConcurrentSub1()
self.concurrentobj2 = ProcessConcurrentSub2()
def callback(self, future):
try:
self.results = future.result()
except (ProcessExpired, RuntimeError, TimeoutError) as error:
self.exception = error
finally:
self.event.set()
def test_docstring(self):
"""Process Spawn docstring is preserved."""
self.assertEqual(decorated.__doc__, "A docstring.")
def test_wrong_timeout(self):
"""Process Spawn TypeError is raised if timeout is not number."""
with self.assertRaises(TypeError):
@concurrent.process(timeout='Foo', context=mp_context)
def function():
return
def test_class_method(self):
"""Process Spawn decorated classmethods."""
future = ProcessConcurrentObj.clsmethod()
self.assertEqual(future.result(), 0)
future = ProcessConcurrentSub1.clsmethod()
self.assertEqual(future.result(), 1)
future = ProcessConcurrentSub2.clsmethod()
self.assertEqual(future.result(), 2)
def test_instance_method(self):
"""Process Spawn decorated instance methods."""
future = self.concurrentobj.instmethod()
self.assertEqual(future.result(), 1)
future = self.concurrentobj1.instmethod()
self.assertEqual(future.result(), 2)
future = self.concurrentobj2.instmethod()
self.assertEqual(future.result(), 3)
def test_not_decorated_results(self):
"""Process Spawn results are produced."""
non_decorated = concurrent.process(not_decorated, context=mp_context)
future = non_decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results(self):
"""Process Spawn results are produced."""
future = decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results_callback(self):
"""Process Spawn results are forwarded to the callback."""
future = decorated(1, 1)
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertEqual(self.results, 2)
def test_error_decorated(self):
"""Process Spawn errors are raised by future.result."""
future = error_decorated()
with self.assertRaises(RuntimeError):
future.result()
def test_error_decorated_callback(self):
"""Process Spawn errors are forwarded to callback."""
future = error_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, RuntimeError),
msg=str(self.exception))
def test_pickling_error_decorated(self):
"""Process Spawn pickling errors are raised by future.result."""
future = pickling_error_decorated()
with self.assertRaises((pickle.PicklingError, TypeError)):
future.result()
def test_timeout_decorated(self):
"""Process Spawn raises TimeoutError if so."""
future = long_decorated()
with self.assertRaises(TimeoutError):
future.result()
def test_timeout_decorated_callback(self):
"""Process Spawn TimeoutError is forwarded to callback."""
future = long_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, TimeoutError),
msg=str(self.exception))
def test_decorated_dead_process(self):
"""Process Spawn ProcessExpired is raised if process dies."""
future = critical_decorated()
with self.assertRaises(ProcessExpired):
future.result()
def test_timeout_decorated_callback(self):
"""Process Spawn ProcessExpired is forwarded to callback."""
future = critical_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, ProcessExpired),
msg=str(self.exception))
def test_cancel_decorated(self):
"""Process Spawn raises CancelledError if future was cancelled."""
future = decorated_cancel()
future.cancel()
self.assertRaises(CancelledError, future.result)
@unittest.skipIf(os.name == 'nt', "Test won't run on Windows.")
def test_decorated_ignoring_sigterm(self):
"""Process Spawn Concurrent ignored SIGTERM signal are handled on Unix."""
future = sigterm_decorated()
with self.assertRaises(TimeoutError):
future.result()
def test_daemon_keyword_decorated(self):
"""Daemon keyword can be passed to a decorated function and spawns correctly."""
f = daemon_keyword_decorated()
dec_out = f.result()
self.assertEqual(dec_out, False)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614893606.0
Pebble-4.6.3/test/test_concurrent_thread.py 0000644 0001750 0001750 00000010770 00000000000 021262 0 ustar 00noxdafox noxdafox import unittest
import threading
from pebble import concurrent
def not_decorated(argument, keyword_argument=0):
return argument + keyword_argument
@concurrent.thread
def decorated(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
@concurrent.thread
def error_decorated():
raise RuntimeError("BOOM!")
@concurrent.thread()
def name_keyword_argument(name='function_kwarg'):
return name
@concurrent.thread(name='concurrent_thread_name')
def name_keyword_decorated():
return threading.current_thread().name
@concurrent.thread(name='decorator_kwarg')
def name_keyword_decorated_and_argument(name='bar'):
return (threading.current_thread().name, name)
@concurrent.thread(daemon=False)
def daemon_keyword_decorated():
return threading.current_thread().daemon
class ThreadConcurrentObj:
a = 0
def __init__(self):
self.b = 1
@classmethod
@concurrent.thread
def clsmethod(cls):
return cls.a
@concurrent.thread
def instmethod(self):
return self.b
@staticmethod
@concurrent.thread
def stcmethod():
return 2
class TestThreadConcurrent(unittest.TestCase):
def setUp(self):
self.results = 0
self.exception = None
self.event = threading.Event()
self.event.clear()
self.concurrentobj = ThreadConcurrentObj()
def callback(self, future):
try:
self.results = future.result()
except (RuntimeError) as error:
self.exception = error
finally:
self.event.set()
def test_docstring(self):
"""Thread docstring is preserved."""
self.assertEqual(decorated.__doc__, "A docstring.")
def test_class_method(self):
"""Thread decorated classmethods."""
future = ThreadConcurrentObj.clsmethod()
self.assertEqual(future.result(), 0)
def test_instance_method(self):
"""Thread decorated instance methods."""
future = self.concurrentobj.instmethod()
self.assertEqual(future.result(), 1)
def test_static_method(self):
"""Thread decorated static methods ( startmethod only)."""
future = self.concurrentobj.stcmethod()
self.assertEqual(future.result(), 2)
def test_not_decorated_results(self):
"""Process Fork results are produced."""
non_decorated = concurrent.thread(not_decorated)
future = non_decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results(self):
"""Thread results are produced."""
future = decorated(1, 1)
self.assertEqual(future.result(), 2)
def test_decorated_results_callback(self):
"""Thread results are forwarded to the callback."""
future = decorated(1, 1)
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertEqual(self.results, 2)
def test_error_decorated(self):
"""Thread errors are raised by future.result."""
future = error_decorated()
with self.assertRaises(RuntimeError):
future.result()
def test_error_decorated_callback(self):
"""Thread errors are forwarded to callback."""
future = error_decorated()
future.add_done_callback(self.callback)
self.event.wait(timeout=1)
self.assertTrue(isinstance(self.exception, RuntimeError),
msg=str(self.exception))
def test_name_keyword_argument(self):
"""name keyword can be passed to a decorated function process without name """
f = name_keyword_argument()
fn_out = f.result()
self.assertEqual(fn_out, "function_kwarg")
def test_name_keyword_decorated(self):
"""
Check that a simple use case of the name keyword passed to the decorator works
"""
f = name_keyword_decorated()
dec_out = f.result()
self.assertEqual(dec_out, "concurrent_thread_name")
def test_name_keyword_decorated_result(self):
"""name kwarg is handled without modifying the function kwargs"""
f = name_keyword_decorated_and_argument(name="function_kwarg")
dec_out, fn_out = f.result()
self.assertEqual(dec_out, "decorator_kwarg")
self.assertEqual(fn_out, "function_kwarg")
def test_daemon_keyword_decorated(self):
"""Daemon keyword can be passed to a decorated function and spawns correctly."""
f = daemon_keyword_decorated()
dec_out = f.result()
self.assertEqual(dec_out, False)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614439376.0
Pebble-4.6.3/test/test_pebble.py 0000644 0001750 0001750 00000014465 00000000000 017007 0 ustar 00noxdafox noxdafox import os
import time
import signal
import unittest
import threading
try: # Python 2
from Queue import Queue
except: # Python 3
from queue import Queue
from pebble import decorators
from pebble.common import launch_thread
from pebble import synchronized, sighandler
from pebble import waitforthreads, waitforqueues
results = 0
semaphore = threading.Semaphore()
@synchronized
def synchronized_function():
"""A docstring."""
return decorators._synchronized_lock.acquire(False)
@synchronized(semaphore)
def custom_synchronized_function():
"""A docstring."""
return semaphore.acquire(False)
try:
from signal import SIGALRM, SIGFPE, SIGIO
@sighandler(SIGALRM)
def signal_handler(signum, frame):
"""A docstring."""
global results
results = 1
@sighandler((SIGFPE, SIGIO))
def signals_handler(signum, frame):
pass
except ImportError:
pass
def thread_function(value):
time.sleep(value)
return value
def queue_function(queues, index, value):
time.sleep(value)
queues[index].put(value)
return value
def spurious_wakeup_function(value, lock):
value = value / 2
time.sleep(value)
lock.acquire()
time.sleep(value)
return value
class TestSynchronizedDecorator(unittest.TestCase):
def test_wrapper_decorator_docstring(self):
"""Synchronized docstring of the original function is preserved."""
self.assertEqual(synchronized_function.__doc__, "A docstring.")
def test_syncronized_locked(self):
"""Synchronized Lock is acquired
during execution of decorated function."""
self.assertFalse(synchronized_function())
def test_syncronized_released(self):
"""Synchronized Lock is released
during execution of decorated function."""
synchronized_function()
self.assertTrue(decorators._synchronized_lock.acquire(False))
decorators._synchronized_lock.release()
def test_custom_syncronized_locked(self):
"""Synchronized semaphore is acquired
during execution of decorated function."""
self.assertFalse(custom_synchronized_function())
def test_custom_syncronized_released(self):
"""Synchronized semaphore is acquired
during execution of decorated function."""
custom_synchronized_function()
self.assertTrue(semaphore.acquire(False))
semaphore.release()
class TestSigHandler(unittest.TestCase):
def test_wrapper_decorator_docstring(self):
"""Sighandler docstring of the original function is preserved."""
if os.name != 'nt':
self.assertEqual(signal_handler.__doc__, "A docstring.")
def test_sighandler(self):
"""Sighandler installs SIGALRM."""
if os.name != 'nt':
self.assertEqual(signal.getsignal(signal.SIGALRM).__name__,
signal_handler.__name__)
def test_sighandler_multiple(self):
"""Sighandler installs SIGFPE and SIGIO."""
if os.name != 'nt':
self.assertEqual(signal.getsignal(signal.SIGFPE).__name__,
signals_handler.__name__)
self.assertEqual(signal.getsignal(signal.SIGIO).__name__,
signals_handler.__name__)
def test_sigalarm_sighandler(self):
"""Sighandler for SIGALARM works."""
if os.name != 'nt':
os.kill(os.getpid(), signal.SIGALRM)
time.sleep(0.1)
self.assertEqual(results, 1)
class TestWaitForThreads(unittest.TestCase):
def test_waitforthreads_single(self):
"""Waitforthreads waits for a single thread."""
thread = launch_thread(None, thread_function, True, 0.01)
self.assertEqual(list(waitforthreads([thread]))[0], thread)
def test_waitforthreads_multiple(self):
"""Waitforthreads waits for multiple threads."""
threads = []
for _ in range(5):
threads.append(launch_thread(None, thread_function, True, 0.01))
time.sleep(0.1)
self.assertEqual(list(waitforthreads(threads)), threads)
def test_waitforthreads_timeout(self):
"""Waitforthreads returns empty list if timeout."""
thread = launch_thread(None, thread_function, True, 0.1)
self.assertEqual(list(waitforthreads([thread], timeout=0.01)), [])
def test_waitforthreads_restore(self):
"""Waitforthreads get_ident is restored to original one."""
if hasattr(threading, 'get_ident'):
expected = threading.get_ident
else:
expected = threading._get_ident
thread = launch_thread(None, thread_function, True, 0)
time.sleep(0.01)
waitforthreads([thread])
if hasattr(threading, 'get_ident'):
self.assertEqual(threading.get_ident, expected)
else:
self.assertEqual(threading._get_ident, expected)
def test_waitforthreads_spurious(self):
"""Waitforthreads tolerates spurious wakeups."""
lock = threading.RLock()
thread = launch_thread(None, spurious_wakeup_function, True, 0.1, lock)
self.assertEqual(list(waitforthreads([thread])), [thread])
class TestWaitForQueues(unittest.TestCase):
def setUp(self):
self.queues = [Queue(), Queue(), Queue()]
def test_waitforqueues_single(self):
"""Waitforqueues waits for a single queue."""
launch_thread(None, queue_function, True, self.queues, 0, 0.01)
self.assertEqual(list(waitforqueues(self.queues))[0], self.queues[0])
def test_waitforqueues_multiple(self):
"""Waitforqueues waits for multiple queues."""
for index in range(3):
launch_thread(None, queue_function, True, self.queues, index, 0.01)
time.sleep(0.1)
self.assertEqual(list(waitforqueues(self.queues)), self.queues)
def test_waitforqueues_timeout(self):
"""Waitforqueues returns empty list if timeout."""
launch_thread(None, queue_function, True, self.queues, 0, 0.1)
self.assertEqual(list(waitforqueues(self.queues, timeout=0.01)), [])
def test_waitforqueues_restore(self):
"""Waitforqueues Queue object is restored to original one."""
expected = sorted(dir(self.queues[0]))
launch_thread(None, queue_function, True, self.queues, 0, 0)
waitforqueues(self.queues)
self.assertEqual(sorted(dir(self.queues[0])), expected)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614439493.0
Pebble-4.6.3/test/test_process_pool_fork.py 0000644 0001750 0001750 00000054466 00000000000 021313 0 ustar 00noxdafox noxdafox import os
import sys
import time
import pickle
import signal
import unittest
import threading
import multiprocessing
from concurrent.futures import CancelledError, TimeoutError
import pebble
from pebble import ProcessPool, ProcessExpired
from pebble.pool.base_pool import ERROR
# set start method
supported = False
mp_context = None
if sys.version_info.major > 2 and sys.version_info.minor > 3:
methods = multiprocessing.get_all_start_methods()
if 'fork' in methods:
try:
mp_context = multiprocessing.get_context('fork')
if mp_context.get_start_method() == 'fork':
supported = True
except RuntimeError: # child process
pass
else:
supported = True
initarg = 0
def initializer(value):
global initarg
initarg = value
def long_initializer():
time.sleep(60)
def broken_initializer():
raise Exception("BOOM!")
def function(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
def initializer_function():
return initarg
def error_function():
raise Exception("BOOM!")
def pickle_error_function():
return threading.Lock()
def long_function(value=1):
time.sleep(value)
return value
def pid_function():
time.sleep(0.1)
return os.getpid()
def sigterm_function():
signal.signal(signal.SIGTERM, signal.SIG_IGN)
time.sleep(10)
def suicide_function():
os._exit(1)
def process_function():
p = multiprocessing.Process(target=function, args=[1])
p.start()
p.join()
return 1
def pool_function():
pool = multiprocessing.Pool(1)
result = pool.apply(function, args=[1])
pool.close()
pool.join()
return result
def pebble_function():
with ProcessPool(max_workers=1) as pool:
f = pool.schedule(function, args=[1])
return f.result()
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessPool(unittest.TestCase):
def setUp(self):
global initarg
initarg = 0
self.event = threading.Event()
self.event.clear()
self.result = None
self.exception = None
def callback(self, future):
try:
self.result = future.result()
except Exception as error:
self.exception = error
finally:
self.event.set()
def test_process_pool_single_future(self):
"""Process Pool Fork single future."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
def test_process_pool_multiple_futures(self):
"""Process Pool Fork multiple futures."""
futures = []
with ProcessPool(max_workers=2, context=mp_context) as pool:
for _ in range(5):
futures.append(pool.schedule(function, args=[1]))
self.assertEqual(sum([f.result() for f in futures]), 5)
def test_process_pool_callback(self):
"""Process Pool Fork result is forwarded to the callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(
function, args=[1], kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
self.event.wait()
self.assertEqual(self.result, 2)
def test_process_pool_error(self):
"""Process Pool Fork errors are raised by future get."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
self.assertRaises(Exception, future.result)
def test_process_pool_error_callback(self):
"""Process Pool Fork errors are forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, Exception))
def test_process_pool_pickling_error_task(self):
"""Process Pool Fork task pickling errors
are raised by future.result."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[threading.Lock()])
self.assertRaises((pickle.PicklingError, TypeError), future.result)
def test_process_pool_pickling_error_result(self):
"""Process Pool Fork result pickling errors
are raised by future.result."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pickle_error_function)
self.assertRaises((pickle.PicklingError, TypeError), future.result)
def test_process_pool_timeout(self):
"""Process Pool Fork future raises TimeoutError if so."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function, timeout=0.1)
self.assertRaises(TimeoutError, future.result)
def test_process_pool_timeout_callback(self):
"""Process Pool Fork TimeoutError is forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function, timeout=0.1)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, TimeoutError))
def test_process_pool_cancel(self):
"""Process Pool Fork future raises CancelledError if so."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function)
time.sleep(0.1) # let the process pick up the task
self.assertTrue(future.cancel())
self.assertRaises(CancelledError, future.result)
def test_process_pool_cancel_callback(self):
"""Process Pool Fork CancelledError is forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function)
future.add_done_callback(self.callback)
time.sleep(0.1) # let the process pick up the task
self.assertTrue(future.cancel())
self.event.wait()
self.assertTrue(isinstance(self.exception, CancelledError))
@unittest.skipIf(sys.platform == 'darwin', "Not supported on MAC OS")
def test_process_pool_different_process(self):
"""Process Pool Fork multiple futures are handled by different processes."""
futures = []
with ProcessPool(max_workers=2, context=mp_context) as pool:
for _ in range(0, 5):
futures.append(pool.schedule(pid_function))
self.assertEqual(len(set([f.result() for f in futures])), 2)
def test_process_pool_future_limit(self):
"""Process Pool Fork tasks limit is honored."""
futures = []
with ProcessPool(max_workers=1, max_tasks=2, context=mp_context) as pool:
for _ in range(0, 4):
futures.append(pool.schedule(pid_function))
self.assertEqual(len(set([f.result() for f in futures])), 2)
def test_process_pool_stop_timeout(self):
"""Process Pool Fork workers are stopped if future timeout."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future1 = pool.schedule(pid_function)
pool.schedule(long_function, timeout=0.1)
future2 = pool.schedule(pid_function)
self.assertNotEqual(future1.result(), future2.result())
def test_process_pool_stop_cancel(self):
"""Process Pool Fork workers are stopped if future is cancelled."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future1 = pool.schedule(pid_function)
cancel_future = pool.schedule(long_function)
time.sleep(0.1) # let the process pick up the task
cancel_future.cancel()
future2 = pool.schedule(pid_function)
self.assertNotEqual(future1.result(), future2.result())
def test_process_pool_initializer(self):
"""Process Pool Fork initializer is correctly run."""
with ProcessPool(initializer=initializer, initargs=[1], context=mp_context) as pool:
future = pool.schedule(initializer_function)
self.assertEqual(future.result(), 1)
def test_process_pool_broken_initializer(self):
"""Process Pool Fork broken initializer is notified."""
with self.assertRaises(RuntimeError):
with ProcessPool(initializer=broken_initializer, context=mp_context) as pool:
pool.active
time.sleep(0.4)
pool.schedule(function)
def test_process_pool_running(self):
"""Process Pool Fork is active if a future is scheduled."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertTrue(pool.active)
def test_process_pool_stopped(self):
"""Process Pool Fork is not active once stopped."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertFalse(pool.active)
def test_process_pool_close_futures(self):
"""Process Pool Fork all futures are performed on close."""
futures = []
pool = ProcessPool(max_workers=1, context=mp_context)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.close()
pool.join()
map(self.assertTrue, [f.done() for f in futures])
def test_process_pool_close_stopped(self):
"""Process Pool Fork is stopped after close."""
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[1])
pool.close()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_stop_futures(self):
"""Process Pool Fork not all futures are performed on stop."""
futures = []
pool = ProcessPool(max_workers=1, context=mp_context)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.stop()
pool.join()
self.assertTrue(len([f for f in futures if not f.done()]) > 0)
def test_process_pool_stop_stopped(self):
"""Process Pool Fork is stopped after stop."""
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_stop_stopped_callback(self):
"""Process Pool Fork is stopped in callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
def stop_pool_callback(_):
pool.stop()
future = pool.schedule(function, args=[1])
future.add_done_callback(stop_pool_callback)
with self.assertRaises(RuntimeError):
for index in range(10):
time.sleep(0.1)
pool.schedule(long_function, args=[index])
self.assertFalse(pool.active)
def test_process_pool_large_data(self):
"""Process Pool Fork large data is sent on the channel."""
data = "a" * 1098 * 1024 * 50 # 50 Mb
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(
function, args=[data], kwargs={'keyword_argument': ''})
self.assertEqual(data, future.result())
def test_process_pool_stop_large_data(self):
"""Process Pool Fork is stopped if large data is sent on the channel."""
data = "a" * 1098 * 1024 * 50 # 50 Mb
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[data])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_join_workers(self):
"""Process Pool Fork no worker is running after join."""
pool = ProcessPool(max_workers=4, context=mp_context)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertEqual(len(pool._pool_manager.worker_manager.workers), 0)
def test_process_pool_join_running(self):
"""Process Pool Fork RuntimeError is raised if active pool joined."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertRaises(RuntimeError, pool.join)
def test_process_pool_join_futures_timeout(self):
"""Process Pool Fork TimeoutError is raised if join on long futures."""
pool = ProcessPool(max_workers=1, context=mp_context)
for _ in range(2):
pool.schedule(long_function)
pool.close()
self.assertRaises(TimeoutError, pool.join, 0.4)
pool.stop()
pool.join()
def test_process_pool_callback_error(self):
"""Process Pool Fork does not stop if error in callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
# sleep enough to ensure callback is run
time.sleep(0.1)
pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
def test_process_pool_exception_isolated(self):
"""Process Pool Fork an Exception does not affect other futures."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
try:
future.result()
except Exception:
pass
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
@unittest.skipIf(os.name == 'nt', "Test won't run on Windows'.")
def test_process_pool_ignoring_sigterm(self):
"""Process Pool Fork ignored SIGTERM signal are handled on Unix."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(sigterm_function, timeout=0.2)
with self.assertRaises(TimeoutError):
future.result()
def test_process_pool_expired_worker(self):
"""Process Pool Fork unexpect death of worker raises ProcessExpired."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(suicide_function)
self.assertRaises(ProcessExpired, future.result)
def test_process_pool_map(self):
"""Process Pool Fork map simple."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_empty(self):
"""Process Pool Fork map no elements."""
elements = []
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_single(self):
"""Process Pool Fork map one element."""
elements = [0]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_multi(self):
"""Process Pool Fork map multiple iterables."""
expected = (2, 4)
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, (1, 2, 3), (1, 2))
generator = future.result()
self.assertEqual(tuple(generator), expected)
def test_process_pool_map_one_chunk(self):
"""Process Pool Fork map chunksize 1."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements, chunksize=1)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_zero_chunk(self):
"""Process Pool Fork map chunksize 0."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
with self.assertRaises(ValueError):
pool.map(function, [], chunksize=0)
def test_process_pool_map_timeout(self):
"""Process Pool Fork map with timeout."""
raised = []
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, elements, timeout=0.1)
generator = future.result()
while True:
try:
next(generator)
except TimeoutError as error:
raised.append(error)
except StopIteration:
break
self.assertTrue(all((isinstance(e, TimeoutError) for e in raised)))
def test_process_pool_map_timeout_chunks(self):
"""Process Pool Fork map timeout is assigned per chunk."""
elements = [0.2]*10
with ProcessPool(max_workers=1, context=mp_context) as pool:
# it takes 1s to process a chunk
future = pool.map(
long_function, elements, chunksize=5, timeout=1.6)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_error(self):
"""Process Pool Fork errors do not stop the iteration."""
raised = None
elements = [1, 'a', 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
while True:
try:
result = next(generator)
except TypeError as error:
raised = error
except StopIteration:
break
self.assertEqual(result, 3)
self.assertTrue(isinstance(raised, TypeError))
def test_process_pool_map_cancel(self):
"""Process Pool Fork cancel iteration."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, range(5))
generator = future.result()
self.assertEqual(next(generator), 0)
future.cancel()
for _ in range(4):
with self.assertRaises(CancelledError):
next(generator)
def test_process_pool_map_broken_pool(self):
"""Process Pool Fork Broken Pool."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, elements, timeout=1)
generator = future.result()
pool._context.state = ERROR
while True:
try:
next(generator)
except TimeoutError as error:
self.assertFalse(pool.active)
future.cancel()
break
except StopIteration:
break
def test_process_pool_child_process(self):
"""Process Pool Fork worker starts process."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(process_function)
self.assertEqual(future.result(), 1)
def test_process_pool_child_pool(self):
"""Process Pool Fork worker starts multiprocessing.Pool."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pool_function)
self.assertEqual(future.result(), 1)
def test_process_pool_child_pebble(self):
"""Process Pool Fork worker starts pebble.ProcessPool."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pebble_function)
self.assertEqual(future.result(), 1)
# DEADLOCK TESTS
def broken_worker_process_tasks(_, channel):
"""Process failing in receiving new tasks."""
with channel.mutex.reader:
os._exit(1)
def broken_worker_process_result(_, channel):
"""Process failing in delivering result."""
try:
for _ in pebble.pool.process.worker_get_next_task(channel, 2):
with channel.mutex.writer:
os._exit(1)
except OSError:
os._exit(1)
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessPoolDeadlockOnNewFutures(unittest.TestCase):
def setUp(self):
self.worker_process = pebble.pool.process.worker_process
pebble.pool.process.worker_process = broken_worker_process_tasks
pebble.pool.channel.LOCK_TIMEOUT = 0.1
def tearDown(self):
pebble.pool.process.worker_process = self.worker_process
pebble.pool.channel.LOCK_TIMEOUT = 60
def test_pool_deadlock_stop(self):
"""Process Pool Fork reading deadlocks are stopping the Pool."""
with self.assertRaises(RuntimeError):
pool = pebble.ProcessPool(max_workers=1, context=mp_context)
for _ in range(10):
pool.schedule(function)
time.sleep(0.1)
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessPoolDeadlockOnResult(unittest.TestCase):
def setUp(self):
self.worker_process = pebble.pool.process.worker_process
pebble.pool.process.worker_process = broken_worker_process_result
pebble.pool.channel.LOCK_TIMEOUT = 0.1
def tearDown(self):
pebble.pool.process.worker_process = self.worker_process
pebble.pool.channel.LOCK_TIMEOUT = 60
def test_pool_deadlock(self):
"""Process Pool Fork no deadlock if writing worker dies locking channel."""
with pebble.ProcessPool(max_workers=1, context=mp_context) as pool:
with self.assertRaises(pebble.ProcessExpired):
pool.schedule(function).result()
def test_pool_deadlock_stop(self):
"""Process Pool Fork writing deadlocks are stopping the Pool."""
with self.assertRaises(RuntimeError):
pool = pebble.ProcessPool(max_workers=1, context=mp_context)
for _ in range(10):
pool.schedule(function)
time.sleep(0.1)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614439284.0
Pebble-4.6.3/test/test_process_pool_forkserver.py 0000644 0001750 0001750 00000050455 00000000000 022534 0 ustar 00noxdafox noxdafox import os
import sys
import time
import pickle
import signal
import unittest
import threading
import multiprocessing
from concurrent.futures import CancelledError, TimeoutError
import pebble
from pebble import ProcessPool, ProcessExpired
from pebble.pool.base_pool import ERROR
# set start method
supported = False
mp_context = None
if sys.version_info.major > 2 and sys.version_info.minor > 3:
methods = multiprocessing.get_all_start_methods()
if 'forkserver' in methods:
try:
mp_context = multiprocessing.get_context('forkserver')
if mp_context.get_start_method() == 'forkserver':
supported = True
else:
raise Exception(mp_context.get_start_method())
except RuntimeError: # child process
pass
initarg = 0
def initializer(value):
global initarg
initarg = value
def long_initializer():
time.sleep(60)
def broken_initializer():
raise Exception("BOOM!")
def function(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
def initializer_function():
return initarg
def error_function():
raise Exception("BOOM!")
def pickle_error_function():
return threading.Lock()
def long_function(value=1):
time.sleep(value)
return value
def pid_function():
time.sleep(0.1)
return os.getpid()
def sigterm_function():
signal.signal(signal.SIGTERM, signal.SIG_IGN)
time.sleep(10)
def suicide_function():
os._exit(1)
def process_function():
p = multiprocessing.Process(target=function, args=[1])
p.start()
p.join()
return 1
def pool_function():
pool = multiprocessing.Pool(1)
result = pool.apply(function, args=[1])
pool.close()
pool.join()
return result
def pebble_function():
with ProcessPool(max_workers=1) as pool:
f = pool.schedule(function, args=[1])
return f.result()
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessPool(unittest.TestCase):
def setUp(self):
global initarg
initarg = 0
self.event = threading.Event()
self.event.clear()
self.result = None
self.exception = None
def callback(self, future):
try:
self.result = future.result()
except Exception as error:
self.exception = error
finally:
self.event.set()
def test_process_pool_single_future(self):
"""Process Pool Forkserver single future."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
def test_process_pool_multiple_futures(self):
"""Process Pool Forkserver multiple futures."""
futures = []
with ProcessPool(max_workers=1, context=mp_context) as pool:
for _ in range(5):
futures.append(pool.schedule(function, args=[1]))
self.assertEqual(sum([f.result() for f in futures]), 5)
def test_process_pool_callback(self):
"""Process Pool Forkserver result is forwarded to the callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(
function, args=[1], kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
self.event.wait()
self.assertEqual(self.result, 2)
def test_process_pool_error(self):
"""Process Pool Forkserver errors are raised by future get."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
self.assertRaises(Exception, future.result)
def test_process_pool_error_callback(self):
"""Process Pool Forkserver errors are forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, Exception))
def test_process_pool_pickling_error_task(self):
"""Process Pool Forkserver task pickling errors
are raised by future.result."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[threading.Lock()])
self.assertRaises((pickle.PicklingError, TypeError), future.result)
def test_process_pool_pickling_error_result(self):
"""Process Pool Forkserver result pickling errors
are raised by future.result."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pickle_error_function)
self.assertRaises((pickle.PicklingError, TypeError), future.result)
def test_process_pool_timeout(self):
"""Process Pool Forkserver future raises TimeoutError if so."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function, timeout=0.1)
self.assertRaises(TimeoutError, future.result)
def test_process_pool_timeout_callback(self):
"""Process Pool Forkserver TimeoutError is forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function, timeout=0.1)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, TimeoutError))
def test_process_pool_cancel(self):
"""Process Pool Forkserver future raises CancelledError if so."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function)
time.sleep(0.1) # let the process pick up the task
self.assertTrue(future.cancel())
self.assertRaises(CancelledError, future.result)
def test_process_pool_cancel_callback(self):
"""Process Pool Forkserver CancelledError is forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function)
future.add_done_callback(self.callback)
time.sleep(0.1) # let the process pick up the task
self.assertTrue(future.cancel())
self.event.wait()
self.assertTrue(isinstance(self.exception, CancelledError))
@unittest.skipIf(sys.platform == 'darwin', "Not supported on MAC OS")
def test_process_pool_different_process(self):
"""Process Pool Forkserver multiple futures are handled by different processes."""
futures = []
with ProcessPool(max_workers=2, context=mp_context) as pool:
for _ in range(0, 5):
futures.append(pool.schedule(pid_function))
self.assertEqual(len(set([f.result() for f in futures])), 2)
def test_process_pool_future_limit(self):
"""Process Pool Forkserver tasks limit is honored."""
futures = []
with ProcessPool(max_workers=1, max_tasks=2, context=mp_context) as pool:
for _ in range(0, 4):
futures.append(pool.schedule(pid_function))
self.assertEqual(len(set([f.result() for f in futures])), 2)
def test_process_pool_stop_timeout(self):
"""Process Pool Forkserver workers are stopped if future timeout."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future1 = pool.schedule(pid_function)
pool.schedule(long_function, timeout=0.1)
future2 = pool.schedule(pid_function)
self.assertNotEqual(future1.result(), future2.result())
def test_process_pool_stop_cancel(self):
"""Process Pool Forkserver workers are stopped if future is cancelled."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future1 = pool.schedule(pid_function)
cancel_future = pool.schedule(long_function)
time.sleep(0.1) # let the process pick up the task
cancel_future.cancel()
future2 = pool.schedule(pid_function)
self.assertNotEqual(future1.result(), future2.result())
def test_process_pool_initializer(self):
"""Process Pool Forkserver initializer is correctly run."""
with ProcessPool(initializer=initializer, initargs=[1], context=mp_context) as pool:
future = pool.schedule(initializer_function)
self.assertEqual(future.result(), 1)
def test_process_pool_broken_initializer(self):
"""Process Pool Forkserver broken initializer is notified."""
with self.assertRaises(RuntimeError):
with ProcessPool(initializer=broken_initializer, context=mp_context) as pool:
pool.active
time.sleep(1)
pool.schedule(function)
def test_process_pool_running(self):
"""Process Pool Forkserver is active if a future is scheduled."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertTrue(pool.active)
def test_process_pool_stopped(self):
"""Process Pool Forkserver is not active once stopped."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertFalse(pool.active)
def test_process_pool_close_futures(self):
"""Process Pool Forkserver all futures are performed on close."""
futures = []
pool = ProcessPool(max_workers=1, context=mp_context)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.close()
pool.join()
map(self.assertTrue, [f.done() for f in futures])
def test_process_pool_close_stopped(self):
"""Process Pool Forkserver is stopped after close."""
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[1])
pool.close()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_stop_futures(self):
"""Process Pool Forkserver not all futures are performed on stop."""
futures = []
pool = ProcessPool(max_workers=1, context=mp_context)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.stop()
pool.join()
self.assertTrue(len([f for f in futures if not f.done()]) > 0)
def test_process_pool_stop_stopped(self):
"""Process Pool Forkserver is stopped after stop."""
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_stop_stopped_callback(self):
"""Process Pool Forkserver is stopped in callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
def stop_pool_callback(_):
pool.stop()
future = pool.schedule(function, args=[1])
future.add_done_callback(stop_pool_callback)
with self.assertRaises(RuntimeError):
for index in range(10):
time.sleep(0.1)
pool.schedule(long_function, args=[index])
self.assertFalse(pool.active)
def test_process_pool_large_data(self):
"""Process Pool Forkserver large data is sent on the channel."""
data = "a" * 1098 * 1024 * 50 # 50 Mb
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(
function, args=[data], kwargs={'keyword_argument': ''})
self.assertEqual(data, future.result())
def test_process_pool_stop_large_data(self):
"""Process Pool Forkserver is stopped if large data is sent on the channel."""
data = "a" * 1098 * 1024 * 50 # 50 Mb
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[data])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_join_workers(self):
"""Process Pool Forkserver no worker is running after join."""
pool = ProcessPool(max_workers=4, context=mp_context)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertEqual(len(pool._pool_manager.worker_manager.workers), 0)
def test_process_pool_join_running(self):
"""Process Pool Forkserver RuntimeError is raised if active pool joined."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertRaises(RuntimeError, pool.join)
def test_process_pool_join_futures_timeout(self):
"""Process Pool Forkserver TimeoutError is raised if join on long tasks."""
pool = ProcessPool(max_workers=1, context=mp_context)
for _ in range(2):
pool.schedule(long_function)
pool.close()
self.assertRaises(TimeoutError, pool.join, 0.4)
pool.stop()
pool.join()
def test_process_pool_callback_error(self):
"""Process Pool Forkserver does not stop if error in callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
# sleep enough to ensure callback is run
time.sleep(0.1)
pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
def test_process_pool_exception_isolated(self):
"""Process Pool Forkserver an Exception does not affect other futures."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
try:
future.result()
except Exception:
pass
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
@unittest.skipIf(os.name == 'nt', "Test won't run on Windows'.")
def test_process_pool_ignoring_sigterm(self):
"""Process Pool Forkserver ignored SIGTERM signal are handled on Unix."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(sigterm_function, timeout=0.2)
with self.assertRaises(TimeoutError):
future.result()
def test_process_pool_expired_worker(self):
"""Process Pool Forkserver unexpect death of worker raises ProcessExpired."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(suicide_function)
self.assertRaises(ProcessExpired, future.result)
def test_process_pool_map(self):
"""Process Pool Forkserver map simple."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_empty(self):
"""Process Pool Forkserver map no elements."""
elements = []
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_single(self):
"""Process Pool Forkserver map one element."""
elements = [0]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_multi(self):
"""Process Pool Forkserver map multiple iterables."""
expected = (2, 4)
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, (1, 2, 3), (1, 2))
generator = future.result()
self.assertEqual(tuple(generator), expected)
def test_process_pool_map_one_chunk(self):
"""Process Pool Forkserver map chunksize 1."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements, chunksize=1)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_zero_chunk(self):
"""Process Pool Forkserver map chunksize 0."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
with self.assertRaises(ValueError):
pool.map(function, [], chunksize=0)
def test_process_pool_map_timeout(self):
"""Process Pool Forkserver map with timeout."""
raised = []
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, elements, timeout=0.1)
generator = future.result()
while True:
try:
next(generator)
except TimeoutError as error:
raised.append(error)
except StopIteration:
break
self.assertTrue(all((isinstance(e, TimeoutError) for e in raised)))
def test_process_pool_map_timeout_chunks(self):
"""Process Pool Forkserver map timeout is assigned per chunk."""
elements = [0.1]*10
with ProcessPool(max_workers=1, context=mp_context) as pool:
# it takes 0.5s to process a chunk
future = pool.map(
long_function, elements, chunksize=5, timeout=0.8)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_error(self):
"""Process Pool Forkserver errors do not stop the iteration."""
raised = None
elements = [1, 'a', 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
while True:
try:
next(generator)
except TypeError as error:
raised = error
except StopIteration:
break
self.assertTrue(isinstance(raised, TypeError))
def test_process_pool_map_cancel(self):
"""Process Pool Forkserver cancel iteration."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, range(5))
generator = future.result()
self.assertEqual(next(generator), 0)
future.cancel()
for _ in range(4):
with self.assertRaises(CancelledError):
next(generator)
def test_process_pool_map_broken_pool(self):
"""Process Pool Forkserver Broken Pool."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, elements, timeout=1)
generator = future.result()
pool._context.state = ERROR
while True:
try:
next(generator)
except TimeoutError as error:
self.assertFalse(pool.active)
future.cancel()
break
except StopIteration:
break
def test_process_pool_child_process(self):
"""Process Pool Forkserver worker starts process."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(process_function)
self.assertEqual(future.result(), 1)
def test_process_pool_child_pool(self):
"""Process Pool Forkserver worker starts multiprocessing.Pool."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pool_function)
self.assertEqual(future.result(), 1)
def test_process_pool_child_pebble(self):
"""Process Pool Forkserver worker starts pebble.ProcessPool."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pebble_function)
self.assertEqual(future.result(), 1)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1614439264.0
Pebble-4.6.3/test/test_process_pool_spawn.py 0000644 0001750 0001750 00000047726 00000000000 021503 0 ustar 00noxdafox noxdafox import os
import sys
import time
import pickle
import signal
import unittest
import threading
import multiprocessing
from concurrent.futures import CancelledError, TimeoutError
import pebble
from pebble import ProcessPool, ProcessExpired
from pebble.pool.base_pool import ERROR
# set start method
supported = False
mp_context = None
if sys.version_info.major > 2 and sys.version_info.minor > 3:
methods = multiprocessing.get_all_start_methods()
if 'spawn' in methods:
try:
mp_context = multiprocessing.get_context('spawn')
if mp_context.get_start_method() == 'spawn':
supported = True
except RuntimeError: # child process
pass
initarg = 0
def initializer(value):
global initarg
initarg = value
def long_initializer():
time.sleep(60)
def broken_initializer():
raise Exception("BOOM!")
def function(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
def initializer_function():
return initarg
def error_function():
raise Exception("BOOM!")
def pickle_error_function():
return threading.Lock()
def long_function(value=1):
time.sleep(value)
return value
def pid_function():
time.sleep(0.1)
return os.getpid()
def sigterm_function():
signal.signal(signal.SIGTERM, signal.SIG_IGN)
time.sleep(10)
def suicide_function():
os._exit(1)
def process_function():
p = multiprocessing.Process(target=function, args=[1])
p.start()
p.join()
return 1
def pool_function():
pool = multiprocessing.Pool(1)
result = pool.apply(function, args=[1])
pool.close()
pool.join()
return result
def pebble_function():
with ProcessPool(max_workers=1) as pool:
f = pool.schedule(function, args=[1])
return f.result()
@unittest.skipIf(not supported, "Start method is not supported")
class TestProcessPool(unittest.TestCase):
def setUp(self):
global initarg
initarg = 0
self.event = threading.Event()
self.event.clear()
self.result = None
self.exception = None
def callback(self, future):
try:
self.result = future.result()
except Exception as error:
self.exception = error
finally:
self.event.set()
def test_process_pool_single_future(self):
"""Process Pool Spawn single future."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
def test_process_pool_multiple_futures(self):
"""Process Pool Spawn multiple futures."""
futures = []
with ProcessPool(max_workers=1, context=mp_context) as pool:
for _ in range(5):
futures.append(pool.schedule(function, args=[1]))
self.assertEqual(sum([f.result() for f in futures]), 5)
def test_process_pool_callback(self):
"""Process Pool Spawn result is forwarded to the callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(
function, args=[1], kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
self.event.wait()
self.assertEqual(self.result, 2)
def test_process_pool_error(self):
"""Process Pool Spawn errors are raised by future get."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
self.assertRaises(Exception, future.result)
def test_process_pool_error_callback(self):
"""Process Pool Spawn errors are forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, Exception))
def test_process_pool_pickling_error_task(self):
"""Process Pool Spawn task pickling errors
are raised by future.result."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[threading.Lock()])
self.assertRaises((pickle.PicklingError, TypeError), future.result)
def test_process_pool_pickling_error_result(self):
"""Process Pool Spawn result pickling errors
are raised by future.result."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pickle_error_function)
self.assertRaises((pickle.PicklingError, TypeError), future.result)
def test_process_pool_timeout(self):
"""Process Pool Spawn future raises TimeoutError if so."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function, timeout=0.1)
self.assertRaises(TimeoutError, future.result)
def test_process_pool_timeout_callback(self):
"""Process Pool Spawn TimeoutError is forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function, timeout=0.1)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, TimeoutError))
def test_process_pool_cancel(self):
"""Process Pool Spawn future raises CancelledError if so."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function)
time.sleep(0.1) # let the process pick up the task
self.assertTrue(future.cancel())
self.assertRaises(CancelledError, future.result)
def test_process_pool_cancel_callback(self):
"""Process Pool Spawn CancelledError is forwarded to callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(long_function)
future.add_done_callback(self.callback)
time.sleep(0.1) # let the process pick up the task
self.assertTrue(future.cancel())
self.event.wait()
self.assertTrue(isinstance(self.exception, CancelledError))
@unittest.skipIf(sys.platform == 'darwin', "Not supported on MAC OS")
def test_process_pool_different_process(self):
"""Process Pool Spawn futures are handled by different processes."""
futures = []
with ProcessPool(max_workers=2, context=mp_context) as pool:
for _ in range(0, 5):
futures.append(pool.schedule(pid_function))
self.assertEqual(len(set([f.result() for f in futures])), 2)
def test_process_pool_future_limit(self):
"""Process Pool Spawn tasks limit is honored."""
futures = []
with ProcessPool(max_workers=1, max_tasks=2, context=mp_context) as pool:
for _ in range(0, 4):
futures.append(pool.schedule(pid_function))
self.assertEqual(len(set([f.result() for f in futures])), 2)
def test_process_pool_stop_timeout(self):
"""Process Pool Spawn workers are stopped if future timeout."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future1 = pool.schedule(pid_function)
pool.schedule(long_function, timeout=0.1)
future2 = pool.schedule(pid_function)
self.assertNotEqual(future1.result(), future2.result())
def test_process_pool_stop_cancel(self):
"""Process Pool Spawn workers are stopped if future is cancelled."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future1 = pool.schedule(pid_function)
cancel_future = pool.schedule(long_function)
time.sleep(0.1) # let the process pick up the task
cancel_future.cancel()
future2 = pool.schedule(pid_function)
self.assertNotEqual(future1.result(), future2.result())
def test_process_pool_initializer(self):
"""Process Pool Spawn initializer is correctly run."""
with ProcessPool(initializer=initializer, initargs=[1], context=mp_context) as pool:
future = pool.schedule(initializer_function)
self.assertEqual(future.result(), 1)
def test_process_pool_broken_initializer(self):
"""Process Pool Spawn broken initializer is notified."""
with self.assertRaises(RuntimeError):
with ProcessPool(initializer=broken_initializer, context=mp_context) as pool:
pool.active
time.sleep(2)
pool.schedule(function)
def test_process_pool_running(self):
"""Process Pool Spawn is active if a future is scheduled."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertTrue(pool.active)
def test_process_pool_stopped(self):
"""Process Pool Spawn is not active once stopped."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertFalse(pool.active)
def test_process_pool_close_futures(self):
"""Process Pool Spawn all futures are performed on close."""
futures = []
pool = ProcessPool(max_workers=1, context=mp_context)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.close()
pool.join()
map(self.assertTrue, [f.done() for f in futures])
def test_process_pool_close_stopped(self):
"""Process Pool Spawn is stopped after close."""
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[1])
pool.close()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_stop_futures(self):
"""Process Pool Spawn not all futures are performed on stop."""
futures = []
pool = ProcessPool(max_workers=1, context=mp_context)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.stop()
pool.join()
self.assertTrue(len([f for f in futures if not f.done()]) > 0)
def test_process_pool_stop_stopped(self):
"""Process Pool Spawn is stopped after stop."""
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_stop_stopped_callback(self):
"""Process Pool Spawn is stopped in callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
def stop_pool_callback(_):
pool.stop()
future = pool.schedule(function, args=[1])
future.add_done_callback(stop_pool_callback)
with self.assertRaises(RuntimeError):
for index in range(10):
time.sleep(0.1)
pool.schedule(long_function, args=[index])
self.assertFalse(pool.active)
def test_process_pool_large_data(self):
"""Process Pool Spawn large data is sent on the channel."""
data = "a" * 1098 * 1024 * 50 # 50 Mb
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(
function, args=[data], kwargs={'keyword_argument': ''})
self.assertEqual(data, future.result())
def test_process_pool_stop_large_data(self):
"""Process Pool Spawn stopped if large data is sent on the channel."""
data = "a" * 1098 * 1024 * 50 # 50 Mb
pool = ProcessPool(max_workers=1, context=mp_context)
pool.schedule(function, args=[data])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_process_pool_join_workers(self):
"""Process Pool Spawn no worker is running after join."""
pool = ProcessPool(max_workers=4, context=mp_context)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertEqual(len(pool._pool_manager.worker_manager.workers), 0)
def test_process_pool_join_running(self):
"""Process Pool Spawn RuntimeError is raised if active pool joined."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
pool.schedule(function, args=[1])
self.assertRaises(RuntimeError, pool.join)
def test_process_pool_join_futures_timeout(self):
"""Process Pool Spawn TimeoutError is raised if join on long tasks."""
pool = ProcessPool(max_workers=1, context=mp_context)
for _ in range(2):
pool.schedule(long_function)
pool.close()
self.assertRaises(TimeoutError, pool.join, 0.4)
pool.stop()
pool.join()
def test_process_pool_callback_error(self):
"""Process Pool Spawn does not stop if error in callback."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
# sleep enough to ensure callback is run
time.sleep(0.1)
pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
def test_process_pool_exception_isolated(self):
"""Process Pool Spawn an Exception does not affect other futures."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(error_function)
try:
future.result()
except Exception:
pass
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
@unittest.skipIf(os.name == 'nt', "Test won't run on Windows'.")
def test_process_pool_ignoring_sigterm(self):
"""Process Pool Spawn ignored SIGTERM signal are handled on Unix."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(sigterm_function, timeout=0.2)
with self.assertRaises(TimeoutError):
future.result()
def test_process_pool_expired_worker(self):
"""Process Pool Spawn unexpect death of worker raises ProcessExpired."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(suicide_function)
self.assertRaises(ProcessExpired, future.result)
def test_process_pool_map(self):
"""Process Pool Spawn map simple."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_empty(self):
"""Process Pool Spawn map no elements."""
elements = []
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_single(self):
"""Process Pool Spawn map one element."""
elements = [0]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_multi(self):
"""Process Pool Spawn map multiple iterables."""
expected = (2, 4)
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, (1, 2, 3), (1, 2))
generator = future.result()
self.assertEqual(tuple(generator), expected)
def test_process_pool_map_one_chunk(self):
"""Process Pool Spawn map chunksize 1."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements, chunksize=1)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_zero_chunk(self):
"""Process Pool Spawn map chunksize 0."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
with self.assertRaises(ValueError):
pool.map(function, [], chunksize=0)
def test_process_pool_map_timeout(self):
"""Process Pool Spawn map with timeout."""
raised = []
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, elements, timeout=0.1)
generator = future.result()
while True:
try:
next(generator)
except TimeoutError as error:
raised.append(error)
except StopIteration:
break
self.assertTrue(all((isinstance(e, TimeoutError) for e in raised)))
def test_process_pool_map_timeout_chunks(self):
"""Process Pool Spawn map timeout is assigned per chunk."""
elements = [0.1]*10
with ProcessPool(max_workers=1, context=mp_context) as pool:
# it takes 0.5s to process a chunk
future = pool.map(
long_function, elements, chunksize=5, timeout=0.8)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_process_pool_map_error(self):
"""Process Pool Spawn errors do not stop the iteration."""
raised = None
elements = [1, 'a', 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(function, elements)
generator = future.result()
while True:
try:
next(generator)
except TypeError as error:
raised = error
except StopIteration:
break
self.assertTrue(isinstance(raised, TypeError))
def test_process_pool_map_cancel(self):
"""Process Pool Spawn cancel iteration."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, range(5))
generator = future.result()
self.assertEqual(next(generator), 0)
future.cancel()
for _ in range(4):
with self.assertRaises(CancelledError):
next(generator)
def test_process_pool_map_broken_pool(self):
"""Process Pool Spawn Broken Pool."""
elements = [1, 2, 3]
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.map(long_function, elements, timeout=1)
generator = future.result()
pool._context.state = ERROR
while True:
try:
next(generator)
except TimeoutError as error:
self.assertFalse(pool.active)
future.cancel()
break
except StopIteration:
break
def test_process_pool_child_process(self):
"""Process Pool Spawn worker starts process."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(process_function)
self.assertEqual(future.result(), 1)
def test_process_pool_child_pool(self):
"""Process Pool Spawn worker starts multiprocessing.Pool."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pool_function)
self.assertEqual(future.result(), 1)
def test_process_pool_child_pebble(self):
"""Process Pool Spawn worker starts pebble.ProcessPool."""
with ProcessPool(max_workers=1, context=mp_context) as pool:
future = pool.schedule(pebble_function)
self.assertEqual(future.result(), 1)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1609065860.0
Pebble-4.6.3/test/test_thread_pool.py 0000644 0001750 0001750 00000027477 00000000000 020065 0 ustar 00noxdafox noxdafox import time
import unittest
import threading
from pebble import ThreadPool
from concurrent.futures import CancelledError, TimeoutError
from pebble.pool.base_pool import ERROR
initarg = 0
def error_callback(future):
raise Exception("BOOM!")
def initializer(value):
global initarg
initarg = value
def broken_initializer():
raise Exception("BOOM!")
def function(argument, keyword_argument=0):
"""A docstring."""
return argument + keyword_argument
def initializer_function():
return initarg
def error_function():
raise Exception("BOOM!")
def long_function(value=0):
time.sleep(1)
return value
def tid_function():
time.sleep(0.1)
return threading.current_thread()
class TestThreadPool(unittest.TestCase):
def setUp(self):
global initarg
initarg = 0
self.event = threading.Event()
self.event.clear()
self.results = None
self.exception = None
def callback(self, future):
try:
self.results = future.result()
except Exception as error:
self.exception = error
finally:
self.event.set()
def test_thread_pool_single_future(self):
"""Thread Pool single future."""
with ThreadPool(max_workers=1) as pool:
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
def test_thread_pool_multiple_futures(self):
"""Thread Pool multiple futures."""
futures = []
with ThreadPool(max_workers=1) as pool:
for _ in range(5):
futures.append(pool.schedule(function, args=[1]))
self.assertEqual(sum([t.result() for t in futures]), 5)
def test_thread_pool_callback(self):
"""Thread Pool results are forwarded to the callback."""
with ThreadPool(max_workers=1) as pool:
future = pool.schedule(
function, args=[1], kwargs={'keyword_argument': 1})
future.add_done_callback(self.callback)
self.event.wait()
self.assertEqual(self.results, 2)
def test_thread_pool_error(self):
"""Thread Pool errors are raised by future get."""
with ThreadPool(max_workers=1) as pool:
future = pool.schedule(error_function)
with self.assertRaises(Exception):
future.result()
def test_thread_pool_error_callback(self):
"""Thread Pool errors are forwarded to callback."""
with ThreadPool(max_workers=1) as pool:
future = pool.schedule(error_function)
future.add_done_callback(self.callback)
self.event.wait()
self.assertTrue(isinstance(self.exception, Exception))
def test_thread_pool_cancel_callback(self):
"""Thread Pool FutureCancelled is forwarded to callback."""
with ThreadPool(max_workers=1) as pool:
pool.schedule(long_function)
future = pool.schedule(long_function)
future.add_done_callback(self.callback)
future.cancel()
self.event.wait()
self.assertTrue(isinstance(self.exception, CancelledError))
def test_thread_pool_different_thread(self):
"""Thread Pool multiple futures are handled by different threades."""
futures = []
with ThreadPool(max_workers=2) as pool:
for _ in range(0, 5):
futures.append(pool.schedule(tid_function))
self.assertEqual(len(set([t.result() for t in futures])), 2)
def test_thread_pool_tasks_limit(self):
"""Thread Pool future limit is honored."""
futures = []
with ThreadPool(max_workers=1, max_tasks=2) as pool:
for _ in range(0, 4):
futures.append(pool.schedule(tid_function))
self.assertEqual(len(set([t.result() for t in futures])), 2)
def test_thread_pool_initializer(self):
"""Thread Pool initializer is correctly run."""
with ThreadPool(initializer=initializer, initargs=[1]) as pool:
future = pool.schedule(initializer_function)
self.assertEqual(future.result(), 1)
def test_thread_pool_broken_initializer(self):
"""Thread Pool broken initializer is notified."""
with self.assertRaises(RuntimeError):
with ThreadPool(initializer=broken_initializer) as pool:
pool.active
time.sleep(0.3)
pool.schedule(function)
def test_thread_pool_running(self):
"""Thread Pool is active if a future is scheduled."""
with ThreadPool(max_workers=1) as pool:
pool.schedule(function, args=[1])
self.assertTrue(pool.active)
def test_thread_pool_stopped(self):
"""Thread Pool is not active once stopped."""
with ThreadPool(max_workers=1) as pool:
pool.schedule(function, args=[1])
self.assertFalse(pool.active)
def test_thread_pool_close_futures(self):
"""Thread Pool all futures are performed on close."""
futures = []
pool = ThreadPool(max_workers=1)
for index in range(10):
futures.append(pool.schedule(function, args=[index]))
pool.close()
pool.join()
map(self.assertTrue, [t.done() for t in futures])
def test_thread_pool_close_stopped(self):
"""Thread Pool is stopped after close."""
pool = ThreadPool(max_workers=1)
pool.schedule(function, args=[1])
pool.close()
pool.join()
self.assertFalse(pool.active)
def test_thread_pool_stop_futures(self):
"""Thread Pool not all futures are performed on stop."""
futures = []
pool = ThreadPool(max_workers=1)
for index in range(10):
futures.append(pool.schedule(long_function, args=[index]))
pool.stop()
pool.join()
self.assertTrue(len([t for t in futures if not t.done()]) > 0)
def test_thread_pool_stop_stopped(self):
"""Thread Pool is stopped after stop."""
pool = ThreadPool(max_workers=1)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertFalse(pool.active)
def test_thread_pool_stop_stopped_function(self):
"""Thread Pool is stopped in function."""
with ThreadPool(max_workers=1) as pool:
def function():
pool.stop()
pool.schedule(function)
self.assertFalse(pool.active)
def test_thread_pool_stop_stopped_callback(self):
"""Thread Pool is stopped in callback."""
with ThreadPool(max_workers=1) as pool:
def stop_pool_callback(_):
pool.stop()
future = pool.schedule(function, args=[1])
future.add_done_callback(stop_pool_callback)
with self.assertRaises(RuntimeError):
for index in range(10):
time.sleep(0.1)
pool.schedule(long_function, args=[index])
self.assertFalse(pool.active)
def test_thread_pool_join_workers(self):
"""Thread Pool no worker is running after join."""
pool = ThreadPool(max_workers=4)
pool.schedule(function, args=[1])
pool.stop()
pool.join()
self.assertEqual(len(pool._pool_manager.workers), 0)
def test_thread_pool_join_running(self):
"""Thread Pool RuntimeError is raised if active pool joined."""
with ThreadPool(max_workers=1) as pool:
pool.schedule(function, args=[1])
self.assertRaises(RuntimeError, pool.join)
def test_thread_pool_join_futures_timeout(self):
"""Thread Pool TimeoutError is raised if join on long futures."""
pool = ThreadPool(max_workers=1)
for _ in range(2):
pool.schedule(long_function)
pool.close()
self.assertRaises(TimeoutError, pool.join, 0.4)
pool.stop()
pool.join()
def test_thread_pool_exception_isolated(self):
"""Thread Pool an Exception does not affect other futures."""
with ThreadPool(max_workers=1) as pool:
future = pool.schedule(error_function)
try:
future.result()
except:
pass
future = pool.schedule(function, args=[1],
kwargs={'keyword_argument': 1})
self.assertEqual(future.result(), 2)
def test_thread_pool_map(self):
"""Thread Pool map simple."""
elements = [1, 2, 3]
with ThreadPool(max_workers=1) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_thread_pool_map_empty(self):
"""Thread Pool map no elements."""
elements = []
with ThreadPool(max_workers=1) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_thread_pool_map_single(self):
"""Thread Pool map one element."""
elements = [0]
with ThreadPool(max_workers=1) as pool:
future = pool.map(function, elements)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_thread_pool_map_multi(self):
"""Thread Pool map multiple iterables."""
expected = (2, 4)
with ThreadPool(max_workers=1) as pool:
future = pool.map(function, (1, 2, 3), (1, 2))
generator = future.result()
self.assertEqual(tuple(generator), expected)
def test_thread_pool_map_one_chunk(self):
"""Thread Pool map chunksize 1."""
elements = [1, 2, 3]
with ThreadPool(max_workers=1) as pool:
future = pool.map(function, elements, chunksize=1)
generator = future.result()
self.assertEqual(list(generator), elements)
def test_thread_pool_map_zero_chunk(self):
"""Thread Pool map chunksize 0."""
with ThreadPool(max_workers=1) as pool:
with self.assertRaises(ValueError):
pool.map(function, [], chunksize=0)
def test_thread_pool_map_error(self):
"""Thread Pool errors do not stop the iteration."""
raised = None
elements = [1, 'a', 3]
with ThreadPool(max_workers=1) as pool:
future = pool.map(function, elements)
generator = future.result()
while True:
try:
next(generator)
except TypeError as error:
raised = error
except StopIteration:
break
self.assertTrue(isinstance(raised, TypeError))
def test_thread_pool_map_cancel(self):
"""Thread Pool cancel iteration."""
with ThreadPool(max_workers=1) as pool:
future = pool.map(long_function, range(5))
generator = future.result()
self.assertEqual(next(generator), 0)
future.cancel()
# either gets computed or it gets cancelled
try:
self.assertEqual(next(generator), 1)
except CancelledError:
pass
for _ in range(3):
with self.assertRaises(CancelledError):
next(generator)
def test_thread_pool_map_broken_pool(self):
"""Thread Pool Fork Broken Pool."""
elements = [1, 2, 3]
with ThreadPool(max_workers=1) as pool:
future = pool.map(long_function, elements, timeout=1)
generator = future.result()
pool._context.state = ERROR
while True:
try:
next(generator)
except TimeoutError as error:
self.assertFalse(pool.active)
future.cancel()
break
except StopIteration:
break