aioprocessing-1.0.0/0000775000175000017500000000000013113542714013706 5ustar dandan00000000000000aioprocessing-1.0.0/README.md0000664000175000017500000001275213015471165015176 0ustar dandan00000000000000aioprocessing ============= [![Build Status](https://travis-ci.org/dano/aioprocessing.svg?branch=master)](https://travis-ci.org/dano/aioprocessing) `aioprocessing` provides asynchronous, [`asyncio`](https://docs.python.org/3/library/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) library. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`: ```python import time import asyncio import aioprocessing import multiprocessing def func(queue, event, lock, items): """ Demo worker function. This worker function runs in its own process, and uses normal blocking calls to aioprocessing objects, exactly the way you would use oridinary multiprocessing objects. """ with lock: event.set() for item in items: time.sleep(3) queue.put(item+5) queue.close() @asyncio.coroutine def example(queue, event, lock): l = [1,2,3,4,5] p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l)) p.start() while True: result = yield from queue.coro_get() if result is None: break print("Got result {}".format(result)) yield from p.coro_join() @asyncio.coroutine def example2(queue, event, lock): yield from event.coro_wait() with (yield from lock): yield from queue.coro_put(78) yield from queue.coro_put(None) # Shut down the worker if __name__ == "__main__": loop = asyncio.get_event_loop() queue = aioprocessing.AioQueue() lock = aioprocessing.AioLock() event = aioprocessing.AioEvent() tasks = [ asyncio.async(example(queue, event, lock)), asyncio.async(example2(queue, event, lock)), ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() ``` Python 3.5 syntax is supported, too. This means the `example2` function above could look like this: ```python async def example2(queue, event, lock): await event.coro_wait() async with lock: await queue.coro_put(78) await queue.coro_put(None) # Shut down the worker ``` The aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop. How does it work? ----------------- In most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does *not* re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https://docs.python.org/2/library/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See http://bugs.python.org/issue6721 for more info). The one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool.*_async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread/fork mixing caveat still applies. Each `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_*` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`. Note that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`. What parts of multiprocessing are supported? -------------------------------------------- Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods. - `Pool` - `Process` - `Pipe` - `Lock` - `RLock` - `Semaphore` - `BoundedSemaphore` - `Event` - `Condition` - `Barrier` - `connection.Connection` - `connection.Listener` - `connection.Client` - `Queue` - `JoinableQueue` - `SimpleQueue` - All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.). What versions of Python are compatible? --------------------------------------- `aioprocessing` will work out of the box on Python 3.4+, and will also work with Python 3.3 if you install the [PyPI version](https://pypi.python.org/pypi/asyncio) of `asyncio`. aioprocessing-1.0.0/MANIFEST.in0000664000175000017500000000006613044356136015452 0ustar dandan00000000000000include AUTHORS include LICENSE.txt include README.md aioprocessing-1.0.0/setup.cfg0000664000175000017500000000007313113542714015527 0ustar dandan00000000000000[egg_info] tag_svn_revision = 0 tag_date = 0 tag_build = aioprocessing-1.0.0/PKG-INFO0000664000175000017500000001677213113542714015020 0ustar dandan00000000000000Metadata-Version: 1.1 Name: aioprocessing Version: 1.0.0 Summary: A Python 3.3+ library that integrates the multiprocessing module with asyncio. Home-page: https://github.com/dano/aioprocessing Author: Dan O'Reilly Author-email: oreilldf@gmail.com License: BSD Description: aioprocessing ============= [![Build Status](https://travis-ci.org/dano/aioprocessing.svg?branch=master)](https://travis-ci.org/dano/aioprocessing) `aioprocessing` provides asynchronous, [`asyncio`](https://docs.python.org/3/library/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) library. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`: ```python import time import asyncio import aioprocessing import multiprocessing def func(queue, event, lock, items): """ Demo worker function. This worker function runs in its own process, and uses normal blocking calls to aioprocessing objects, exactly the way you would use oridinary multiprocessing objects. """ with lock: event.set() for item in items: time.sleep(3) queue.put(item+5) queue.close() @asyncio.coroutine def example(queue, event, lock): l = [1,2,3,4,5] p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l)) p.start() while True: result = yield from queue.coro_get() if result is None: break print("Got result {}".format(result)) yield from p.coro_join() @asyncio.coroutine def example2(queue, event, lock): yield from event.coro_wait() with (yield from lock): yield from queue.coro_put(78) yield from queue.coro_put(None) # Shut down the worker if __name__ == "__main__": loop = asyncio.get_event_loop() queue = aioprocessing.AioQueue() lock = aioprocessing.AioLock() event = aioprocessing.AioEvent() tasks = [ asyncio.async(example(queue, event, lock)), asyncio.async(example2(queue, event, lock)), ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() ``` Python 3.5 syntax is supported, too. This means the `example2` function above could look like this: ```python async def example2(queue, event, lock): await event.coro_wait() async with lock: await queue.coro_put(78) await queue.coro_put(None) # Shut down the worker ``` The aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop. How does it work? ----------------- In most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does *not* re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https://docs.python.org/2/library/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See http://bugs.python.org/issue6721 for more info). The one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool.*_async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread/fork mixing caveat still applies. Each `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_*` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`. Note that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`. What parts of multiprocessing are supported? -------------------------------------------- Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods. - `Pool` - `Process` - `Pipe` - `Lock` - `RLock` - `Semaphore` - `BoundedSemaphore` - `Event` - `Condition` - `Barrier` - `connection.Connection` - `connection.Listener` - `connection.Client` - `Queue` - `JoinableQueue` - `SimpleQueue` - All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.). What versions of Python are compatible? --------------------------------------- `aioprocessing` will work out of the box on Python 3.4+, and will also work with Python 3.3 if you install the [PyPI version](https://pypi.python.org/pypi/asyncio) of `asyncio`. Keywords: asyncio multiprocessing coroutine Platform: UNKNOWN Classifier: Development Status :: 3 - Alpha Classifier: Intended Audience :: Developers Classifier: Natural Language :: English Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Topic :: Software Development :: Libraries :: Python Modules aioprocessing-1.0.0/AUTHORS0000664000175000017500000000004212404213537014752 0ustar dandan00000000000000Dan O'Reilly (oreilldf@gmail.com) aioprocessing-1.0.0/aioprocessing.egg-info/0000775000175000017500000000000013113542714020245 5ustar dandan00000000000000aioprocessing-1.0.0/aioprocessing.egg-info/dependency_links.txt0000664000175000017500000000000113113542714024313 0ustar dandan00000000000000 aioprocessing-1.0.0/aioprocessing.egg-info/PKG-INFO0000664000175000017500000001677213113542714021357 0ustar dandan00000000000000Metadata-Version: 1.1 Name: aioprocessing Version: 1.0.0 Summary: A Python 3.3+ library that integrates the multiprocessing module with asyncio. Home-page: https://github.com/dano/aioprocessing Author: Dan O'Reilly Author-email: oreilldf@gmail.com License: BSD Description: aioprocessing ============= [![Build Status](https://travis-ci.org/dano/aioprocessing.svg?branch=master)](https://travis-ci.org/dano/aioprocessing) `aioprocessing` provides asynchronous, [`asyncio`](https://docs.python.org/3/library/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) library. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`: ```python import time import asyncio import aioprocessing import multiprocessing def func(queue, event, lock, items): """ Demo worker function. This worker function runs in its own process, and uses normal blocking calls to aioprocessing objects, exactly the way you would use oridinary multiprocessing objects. """ with lock: event.set() for item in items: time.sleep(3) queue.put(item+5) queue.close() @asyncio.coroutine def example(queue, event, lock): l = [1,2,3,4,5] p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l)) p.start() while True: result = yield from queue.coro_get() if result is None: break print("Got result {}".format(result)) yield from p.coro_join() @asyncio.coroutine def example2(queue, event, lock): yield from event.coro_wait() with (yield from lock): yield from queue.coro_put(78) yield from queue.coro_put(None) # Shut down the worker if __name__ == "__main__": loop = asyncio.get_event_loop() queue = aioprocessing.AioQueue() lock = aioprocessing.AioLock() event = aioprocessing.AioEvent() tasks = [ asyncio.async(example(queue, event, lock)), asyncio.async(example2(queue, event, lock)), ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() ``` Python 3.5 syntax is supported, too. This means the `example2` function above could look like this: ```python async def example2(queue, event, lock): await event.coro_wait() async with lock: await queue.coro_put(78) await queue.coro_put(None) # Shut down the worker ``` The aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop. How does it work? ----------------- In most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does *not* re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https://docs.python.org/2/library/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See http://bugs.python.org/issue6721 for more info). The one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool.*_async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread/fork mixing caveat still applies. Each `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_*` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`. Note that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`. What parts of multiprocessing are supported? -------------------------------------------- Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods. - `Pool` - `Process` - `Pipe` - `Lock` - `RLock` - `Semaphore` - `BoundedSemaphore` - `Event` - `Condition` - `Barrier` - `connection.Connection` - `connection.Listener` - `connection.Client` - `Queue` - `JoinableQueue` - `SimpleQueue` - All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.). What versions of Python are compatible? --------------------------------------- `aioprocessing` will work out of the box on Python 3.4+, and will also work with Python 3.3 if you install the [PyPI version](https://pypi.python.org/pypi/asyncio) of `asyncio`. Keywords: asyncio multiprocessing coroutine Platform: UNKNOWN Classifier: Development Status :: 3 - Alpha Classifier: Intended Audience :: Developers Classifier: Natural Language :: English Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Topic :: Software Development :: Libraries :: Python Modules aioprocessing-1.0.0/aioprocessing.egg-info/top_level.txt0000664000175000017500000000001613113542714022774 0ustar dandan00000000000000aioprocessing aioprocessing-1.0.0/aioprocessing.egg-info/not-zip-safe0000664000175000017500000000000112407210362022467 0ustar dandan00000000000000 aioprocessing-1.0.0/aioprocessing.egg-info/SOURCES.txt0000664000175000017500000000071013113542714022127 0ustar dandan00000000000000AUTHORS LICENSE.txt MANIFEST.in README.md setup.py aioprocessing/__init__.py aioprocessing/connection.py aioprocessing/executor.py aioprocessing/locks.py aioprocessing/managers.py aioprocessing/pool.py aioprocessing/process.py aioprocessing/queues.py aioprocessing/util.py aioprocessing.egg-info/PKG-INFO aioprocessing.egg-info/SOURCES.txt aioprocessing.egg-info/dependency_links.txt aioprocessing.egg-info/not-zip-safe aioprocessing.egg-info/top_level.txtaioprocessing-1.0.0/setup.py0000664000175000017500000000232513044356136015426 0ustar dandan00000000000000#import ez_setup #ez_setup.use_setuptools() from setuptools import setup, find_packages import aioprocessing with open("README.md", 'r') as f: readme = f.read() setup( name="aioprocessing", version=aioprocessing.version, packages=find_packages(exclude=["tests"]), author="Dan O'Reilly", author_email="oreilldf@gmail.com", description=" A Python 3.3+ library that integrates the multiprocessing module with asyncio.", zip_safe=False, license="BSD", keywords="asyncio multiprocessing coroutine", url="https://github.com/dano/aioprocessing", long_description=readme, classifiers=[ 'Development Status :: 3 - Alpha', 'Intended Audience :: Developers', 'Natural Language :: English', 'License :: OSI Approved :: BSD License', 'Operating System :: Microsoft :: Windows', 'Operating System :: POSIX', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Topic :: Software Development :: Libraries :: Python Modules' ], ) aioprocessing-1.0.0/LICENSE.txt0000664000175000017500000000275212404213537015537 0ustar dandan00000000000000Copyright (c) 2014, Dan O'Reilly All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of the FreeBSD Project. aioprocessing-1.0.0/aioprocessing/0000775000175000017500000000000013113542714016553 5ustar dandan00000000000000aioprocessing-1.0.0/aioprocessing/locks.py0000664000175000017500000000676513113542335020255 0ustar dandan00000000000000import asyncio from .executor import CoroBuilder from multiprocessing import (Event, Lock, RLock, BoundedSemaphore, Condition, Semaphore, Barrier) from multiprocessing.util import register_after_fork __all__ = ["AioLock", "AioRLock", "AioBarrier", "AioCondition", "AioEvent", "AioSemaphore", "AioBoundedSemaphore"] class _ContextManager: """ Context manager. This enables the following idiom for acquiring and releasing a lock around a block: with (yield from lock): """ def __init__(self, lock): self._lock = lock def __enter__(self): # We have no use for the "as ..." clause in the with # statement for locks. return None def __exit__(self, *args): try: self._lock.release() finally: self._lock = None # Crudely prevent reuse. class AioBaseLock(metaclass=CoroBuilder): pool_workers = 1 coroutines = ['acquire', 'release'] def __init__(self, *args, **kwargs): self._threaded_acquire = False def _after_fork(obj): obj._threaded_acquire = False register_after_fork(self, _after_fork) def coro_acquire(self, *args, **kwargs): """ Non-blocking acquire. We need a custom implementation here, because we need to set the _threaded_acquire attribute to True once we have the lock. This attribute is used by release() to determine whether the lock should be released in the main thread, or in the Executor thread. """ def lock_acquired(fut): if fut.result(): self._threaded_acquire = True out = self.run_in_executor(self._obj.acquire, *args, **kwargs) out.add_done_callback(lock_acquired) return out def __getstate__(self): state = super().__getstate__() state['_threaded_acquire'] = False return state def __setstate__(self, state): super().__setstate__(state) def release(self): """ Release the lock. If the lock was acquired in the same process via coro_acquire, we need to release the lock in the ThreadPoolExecutor's thread. """ if self._threaded_acquire: out = self.run_in_thread(self._obj.release) else: out = self._obj.release() self._threaded_acquire = False return out @asyncio.coroutine def __aenter__(self): yield from self.coro_acquire() return None @asyncio.coroutine def __aexit__(self, *args, **kwargs): self.release() def __enter__(self): return self._obj.__enter__() def __exit__(self, *args, **kwargs): return self._obj.__exit__(*args, **kwargs) def __iter__(self): yield from self.coro_acquire() return _ContextManager(self) class AioBaseWaiter(metaclass=CoroBuilder): pool_workers = 1 coroutines = ['wait'] class AioBarrier(AioBaseWaiter): delegate = Barrier pass class AioCondition(AioBaseLock, AioBaseWaiter): delegate = Condition pool_workers = 1 coroutines = ['wait_for', 'notify', 'notify_all'] class AioEvent(AioBaseWaiter): delegate = Event class AioLock(AioBaseLock): delegate = Lock class AioRLock(AioBaseLock): delegate = RLock class AioSemaphore(AioBaseLock): delegate = Semaphore class AioBoundedSemaphore(AioBaseLock): delegate = BoundedSemaphore aioprocessing-1.0.0/aioprocessing/executor.py0000664000175000017500000001561213015470346020772 0ustar dandan00000000000000from multiprocessing import cpu_count from functools import wraps from concurrent.futures import ThreadPoolExecutor from . import util def init_executor(func): @wraps(func) def wrapper(self, *args, **kwargs): if not hasattr(self, '_executor'): self._executor = self._get_executor() return func(self, *args, **kwargs) return wrapper class _ExecutorMixin(): """ A Mixin that provides asynchronous functionality. This mixin provides methods that allow a class to run blocking methods via asyncio in a ThreadPoolExecutor. It also provides methods that attempt to keep the object picklable despite having a non-picklable ThreadPoolExecutor as part of its state. """ pool_workers = cpu_count() @init_executor def run_in_executor(self, callback, *args, loop=None, **kwargs): """ Wraps run_in_executor so we can support kwargs. BaseEventLoop.run_in_executor does not support kwargs, so we wrap our callback in a lambda if kwargs are provided. """ return util.run_in_executor(self._executor, callback, *args, loop=loop, **kwargs) @init_executor def run_in_thread(self, callback, *args, **kwargs): """ Runs a method in an executor thread. This is used when a method must be run in a thread (e.g. to that a lock is released in the same thread it was acquired), but should be run in a blocking way. """ fut = self._executor.submit(callback, *args, **kwargs) return fut.result() def _get_executor(self): return ThreadPoolExecutor(max_workers=self.pool_workers) def __getattr__(self, attr): assert attr is not '_obj', 'Make sure that your Class has a ' \ '"delegate" assigned' if (self._obj and hasattr(self._obj, attr) and not attr.startswith("__")): return getattr(self._obj, attr) raise AttributeError(attr) def __getstate__(self): self_dict = self.__dict__.copy() self_dict['_executor'] = None return self_dict def __setstate__(self, state): self.__dict__.update(state) self._executor = self._get_executor() class CoroBuilder(type): """ Metaclass for adding coroutines to a class. This metaclass has two main roles: 1) Make _ExecutorMixin a parent of the given class 2) For every function name listed in the class attribute "coroutines", add a new instance method to the class called "coro_", which is an asyncio.coroutine that calls func_name in a ThreadPoolExecutor. Each wrapper class that uses this metaclass can define three class attributes that will influence the behavior of the metaclass: coroutines - A list of methods that should get coroutine versions in the wrapper. For example: coroutines = ['acquire', 'wait'] Will mean the class gets coro_acquire and coro_wait methods. delegate - The class object that is being wrapped. This object will be instantiated when the wrapper class is instantiated, and will be set to the `_obj` attribute of the instance. pool_workers - The number of workers in the ThreadPoolExecutor internally used by the wrapper class. This defaults to cpu_count(), but for classes that need to acquire locks, it should always be set to 1. """ def __new__(cls, clsname, bases, dct, **kwargs): coro_list = dct.get('coroutines', []) existing_coros = set() def find_existing_coros(d): for attr in d: if attr.startswith("coro_") or attr.startswith("thread_"): existing_coros.add(attr) # Determine if any bases include the coroutines attribute, or # if either this class or a base class provides an actual # implementation for a coroutine method. find_existing_coros(dct) for b in bases: b_dct = b.__dict__ coro_list.extend(b_dct.get('coroutines', [])) find_existing_coros(b_dct) # Add _ExecutorMixin to bases. if _ExecutorMixin not in bases: bases += (_ExecutorMixin,) # Add coro funcs to dct, but only if a definition # is not already provided by dct or one of our bases. for func in coro_list: coro_name = 'coro_{}'.format(func) if coro_name not in existing_coros: dct[coro_name] = cls.coro_maker(func) return super().__new__(cls, clsname, bases, dct) def __init__(cls, name, bases, dct): """ Properly initialize a coroutine wrapper class. Sets pool_workers and delegate on the class, and also adds an __init__ method to it that instantiates the delegate with the proper context. """ super().__init__(name, bases, dct) pool_workers = dct.get('pool_workers') delegate = dct.get('delegate') old_init = dct.get('__init__') # Search bases for values we care about, if we didn't # find them on the current class. for b in bases: b_dct = b.__dict__ if not pool_workers: pool_workers = b_dct.get('pool_workers') if not delegate: delegate = b_dct.get('delegate') if not old_init: old_init = b_dct.get('__init__') cls.delegate = delegate # If we found a value for pool_workers, set it. If not, # ExecutorMixin sets a default that will be used. if pool_workers: cls.pool_workers = pool_workers # Here's the __init__ we want every wrapper class to use. # It just instantiates the delegate mp object using the # correct context. @wraps(old_init) def init_func(self, *args, **kwargs): # Be sure to call the original __init__, if there # was one. if old_init: old_init(self, *args, **kwargs) # If we're wrapping a mp object, instantiate it here. # If a context was specified, we instaniate the mp class # using that context. Otherwise, we'll just use the default # context. if cls.delegate: ctx = kwargs.pop('ctx', None) if ctx: clz = getattr(ctx, cls.delegate.__name__) else: clz = cls.delegate self._obj = clz(*args, **kwargs) cls.__init__ = init_func @staticmethod def coro_maker(func): def coro_func(self, *args, loop=None, **kwargs): return self.run_in_executor(getattr(self, func), *args, loop=loop, **kwargs) return coro_func aioprocessing-1.0.0/aioprocessing/connection.py0000664000175000017500000000333412411655031021264 0ustar dandan00000000000000import multiprocessing from multiprocessing.connection import (Listener, Client, deliver_challenge, answer_challenge, wait) from .executor import CoroBuilder from .util import run_in_executor __all__ = ['AioConnection'] class AioConnection(metaclass=CoroBuilder): coroutines = ['recv', 'poll', 'send_bytes', 'recv_bytes', 'recv_bytes_into', 'send'] def __init__(self, obj): """ Initialize the AioConnection. obj - a multiprocessing.Connection object. """ super().__init__() self._obj = obj def __enter__(self): self._obj.__enter__() return self def __exit__(self, *args, **kwargs): self._obj.__exit__(*args, **kwargs) def AioClient(*args, **kwargs): """ Returns an AioConnection instance. """ conn = Client(*args, **kwargs) return AioConnection(conn) class AioListener(metaclass=CoroBuilder): delegate = Listener coroutines = ['accept'] def accept(self): conn = self._obj.accept() return AioConnection(conn) def __enter__(self): self._obj.__enter__() return self def __exit__(self, *args, **kwargs): self._obj.__exit__(*args, **kwargs) def coro_deliver_challenge(*args, **kwargs): executor = ThreadPoolExecutor(max_workers=1) return run_in_executor(executor, deliver_challenge, *args, **kwargs) def coro_answer_challenge(*args, **kwargs): executor = ThreadPoolExecutor(max_workers=1) return run_in_executor(executor, answer_challenge, *args, **kwargs) def coro_wait(*args, **kwargs): executor = ThreadPoolExecutor(max_workers=1) return run_in_executor(executor, wait, *args, **kwargs) aioprocessing-1.0.0/aioprocessing/process.py0000664000175000017500000000027412404213537020606 0ustar dandan00000000000000from multiprocessing import Process from .executor import CoroBuilder __all__ = ['AioProcess'] class AioProcess(metaclass=CoroBuilder): delegate = Process coroutines = ['join'] aioprocessing-1.0.0/aioprocessing/__init__.py0000664000175000017500000001317113113542551020666 0ustar dandan00000000000000import multiprocessing from .connection import * from .managers import * __all__ = ['AioProcess', 'AioManager', 'AioPipe', 'AioQueue', 'AioSimpleQueue', 'AioJoinableQueue', 'AioLock', 'AioRLock', 'AioCondition', 'AioPool', 'AioSemaphore', 'AioBoundedSemaphore', 'AioEvent', 'AioBarrier'] # version is a human-readable version number. # version_info is a four-tuple for programmatic comparison. The first # three numbers are the components of the version number. The fourth # is zero for an official release, positive for a development branch, # or negative for a release candidate or beta (after the base version # number has been incremented) version = "1.0.0" version_info = (1, 0, 0, 0) if hasattr(multiprocessing, 'get_context'): def _get_context(): return multiprocessing.get_context() has_context = True else: def _get_context(): return None has_context = False def AioProcess(group=None, target=None, name=None, args=(), kwargs = {}, *, daemon=None, context=None): """ Returns an asyncio-friendly version of multiprocessing.Process. Provides the following coroutines: coro_join() """ context = context if context else _get_context() from .process import AioProcess return AioProcess(group=group, target=target, name=name, args=args, kwargs=kwargs, daemon=daemon, ctx=context) def AioPool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, *, context=None): """ Returns an asyncio-friendly version of multiprocessing.Pool. Provides the following coroutines: coro_join() coro_apply() coro_map() coro_starmap() """ context = context if context else _get_context() from .pool import AioPool return AioPool(processes=processes, initializer=initializer, initargs=initargs, maxtasksperchild=maxtasksperchild, ctx=context) def AioManager(*, context=None): """ Starts and returns an asyncio-friendly mp.SyncManager. Provides the follow asyncio-friendly objects: AioQueue AioBarrier AioBoundedSemaphore AioCondition AioEvent AioLock AioRLock AioSemaphore """ context = context if context else _get_context() from .managers import AioSyncManager # For Python 3.3 support, don't always pass ctx. kwargs = {'ctx' : context} if has_context else {} m = AioSyncManager(**kwargs) m.start() return m def AioPipe(duplex=True): """ Returns a pair of AioConnection objects. """ from .connection import AioConnection conn1, conn2 = multiprocessing.Pipe(duplex=duplex) # Transform the returned connection instances into # instance of AioConnection. conn1 = AioConnection(conn1) conn2 = AioConnection(conn2) return conn1, conn2 # queues def AioQueue(maxsize=0, *, context=None): """ Returns an asyncio-friendly version of a multiprocessing.Queue Returns an AioQueue objects with the given context. If a context is not provided, the default for the platform will be used. """ context = context = context if context else _get_context() from .queues import AioQueue return AioQueue(maxsize, ctx=context) def AioJoinableQueue(maxsize=0, *, context=None): """ Returns an asyncio-friendly version of a multiprocessing.JoinableQueue Returns an AioJoinableQueue object with the given context. If a context is not provided, the default for the platform will be used. """ context = context = context if context else _get_context() from .queues import AioJoinableQueue return AioJoinableQueue(maxsize, ctx=context) def AioSimpleQueue(*, context=None): """ Returns an asyncio-friendly version of a multiprocessing.SimpleQueue Returns an AioSimpleQueue object with the given context. If a context is not provided, the default for the platform will be used. """ context = context = context if context else _get_context() from .queues import AioSimpleQueue return AioSimpleQueue(ctx=context) # locks def AioLock(*, context=None): """ Returns a non-recursive lock object. """ context = context = context if context else _get_context() from .locks import AioLock return AioLock(ctx=context) def AioRLock(*, context=None): """ Returns a recursive lock object. """ context = context = context if context else _get_context() from .locks import AioRLock return AioRLock(ctx=context) def AioCondition(lock=None, *, context=None): """ Returns a condition object. """ context = context = context if context else _get_context() from .locks import AioCondition return AioCondition(lock, ctx=context) def AioSemaphore(value=1, *, context=None): """ Returns a semaphore object. """ context = context = context if context else _get_context() from .locks import AioSemaphore return AioSemaphore(value, ctx=context) def AioBoundedSemaphore(value=1, *, context=None): """ Returns a bounded semaphore object. """ context = context = context if context else _get_context() from .locks import AioBoundedSemaphore return AioBoundedSemaphore(value, ctx=context) def AioEvent(*, context=None): """ Returns an event object. """ context = context = context if context else _get_context() from .locks import AioEvent return AioEvent(ctx=context) def AioBarrier(parties, action=None, timeout=None, *, context=None): """ Returns a barrier object. """ context = context = context if context else _get_context() from .locks import AioBarrier return AioBarrier(parties, action, timeout, ctx=context) aioprocessing-1.0.0/aioprocessing/queues.py0000664000175000017500000000203112404213537020430 0ustar dandan00000000000000import asyncio import multiprocessing from multiprocessing import Queue, SimpleQueue, JoinableQueue from .executor import CoroBuilder class AioBaseQueue(metaclass=CoroBuilder): coroutines = ['get', 'put'] class AioSimpleQueue(AioBaseQueue): """ An asyncio-friendly version of mp.SimpleQueue. Provides two asyncio.coroutines: coro_get and coro_put, which are asynchronous version of get and put, respectively. """ delegate = SimpleQueue class AioQueue(AioBaseQueue): """ An asyncio-friendly version of mp.SimpleQueue. Provides two asyncio.coroutines: coro_get and coro_put, which are asynchronous version of get and put, respectively. """ delegate = Queue class AioJoinableQueue(AioBaseQueue): """ An asyncio-friendly version of mp.JoinableQueue. Provides three asyncio.coroutines: coro_get, coro_put, and coro_join, which are asynchronous version of get put, and join, respectively. """ coroutines = ['join'] delegate = JoinableQueue aioprocessing-1.0.0/aioprocessing/pool.py0000664000175000017500000000323212407144372020101 0ustar dandan00000000000000from multiprocessing import Pool from asyncio import Future import asyncio from .executor import CoroBuilder __all__ = ['AioPool'] class AioPool(metaclass=CoroBuilder): delegate = Pool coroutines = ['join'] def _coro_func(self, funcname, *args, loop=None, **kwargs): """ Call the given function, and wrap the reuslt in a Future. funcname should be the name of a function which takes `callback` and `error_callback` keyword arguments (e.g. apply_async). """ if not loop: loop = asyncio.get_event_loop() fut = Future() def set_result(result): loop.call_soon_threadsafe(fut.set_result, result) def set_exc(exc): loop.call_soon_threadsafe(fut.set_exception, exc) func = getattr(self._obj, funcname) func(*args, callback=set_result, error_callback=set_exc, **kwargs) return fut def coro_apply(self, func, args=(), kwds={}, *, loop=None): return self._coro_func('apply_async', func, args=args, kwds=kwds, loop=loop) def coro_map(self, func, iterable, chunksize=None, *, loop=None): return self._coro_func('map_async', func, iterable, chunksize=chunksize, loop=loop) def coro_starmap(self, func, iterable, chunksize=None, *, loop=None): return self._coro_func('starmap_async', func, iterable, chunksize=chunksize, loop=loop) def __enter__(self): self._obj.__enter__() return self def __exit__(self, *args, **kwargs): self._obj.__exit__(*args, **kwargs) aioprocessing-1.0.0/aioprocessing/managers.py0000664000175000017500000001222413113542362020722 0ustar dandan00000000000000import asyncio from multiprocessing.util import register_after_fork from queue import Queue from threading import (Barrier, BoundedSemaphore, Condition, Event, Lock, RLock, Semaphore) from multiprocessing.managers import (SyncManager, MakeProxyType, BarrierProxy, EventProxy, ConditionProxy, AcquirerProxy) from aioprocessing.locks import _ContextManager from .executor import _ExecutorMixin AioBaseQueueProxy = MakeProxyType('AioQueueProxy', ( 'task_done', 'get', 'qsize', 'put', 'put_nowait', 'get_nowait', 'empty', 'join', '_qsize', 'full' )) class _AioProxyMixin(_ExecutorMixin): _obj = None def _async_call(self, method, *args, loop=None, **kwargs): return asyncio.async(self.run_in_executor(self._callmethod, method, args, kwargs, loop=loop)) class ProxyCoroBuilder(type): """ Build coroutines to proxy functions. """ def __new__(cls, clsname, bases, dct): coro_list = dct.get('coroutines', []) existing_coros = set() def find_existing_coros(d): for attr in d: if attr.startswith("coro_") or attr.startswith("thread_"): existing_coros.add(attr) # Determine if any bases include the coroutines attribute, or # if either this class or a base class provides an actual # implementation for a coroutine method. find_existing_coros(dct) for b in bases: b_dct = b.__dict__ coro_list.extend(b_dct.get('coroutines', [])) find_existing_coros(b_dct) bases += (_AioProxyMixin,) for func in coro_list: coro_name = 'coro_{}'.format(func) if coro_name not in existing_coros: dct[coro_name] = cls.coro_maker(func) return super().__new__(cls, clsname, bases, dct) @staticmethod def coro_maker(func): def coro_func(self, *args, loop=None, **kwargs): return self._async_call(func, *args, loop=loop, **kwargs) return coro_func class AioQueueProxy(AioBaseQueueProxy, metaclass=ProxyCoroBuilder): """ A Proxy object for AioQueue. Provides coroutines for calling 'get' and 'put' on the proxy. """ coroutines = ['get', 'put'] class AioAcquirerProxy(AcquirerProxy, metaclass=ProxyCoroBuilder): pool_workers = 1 coroutines = ['acquire', 'release'] def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._threaded_acquire = False def _after_fork(obj): obj._threaded_acquire = False register_after_fork(self, _after_fork) def coro_acquire(self, *args, **kwargs): """ Non-blocking acquire. We need a custom implementation here, because we need to set the _threaded_acquire attribute to True once we have the lock. This attribute is used by release() to determine whether the lock should be released in the main thread, or in the Executor thread. """ def lock_acquired(fut): if fut.result(): self._threaded_acquire = True out = self.run_in_executor(self.acquire, *args, **kwargs) out.add_done_callback(lock_acquired) return out def __getstate__(self): state = super().__getstate__() state['_threaded_acquire'] = False return state def __setstate__(self, state): super().__setstate__(state) def release(self): """ Release the lock. If the lock was acquired in the same process via coro_acquire, we need to release the lock in the ThreadPoolExecutor's thread. """ if self._threaded_acquire: out = self.run_in_thread(super().release) else: out = super().release() self._threaded_acquire = False return out @asyncio.coroutine def __aenter__(self): yield from self.coro_acquire() return None @asyncio.coroutine def __aexit__(self, *args, **kwargs): self.release() def __iter__(self): yield from self.coro_acquire() return _ContextManager(self) class AioBarrierProxy(BarrierProxy, metaclass=ProxyCoroBuilder): coroutines = ['wait'] class AioEventProxy(EventProxy, metaclass=ProxyCoroBuilder): coroutines = ['wait'] class AioConditionProxy(ConditionProxy, metaclass=ProxyCoroBuilder): coroutines = ['wait', 'wait_for'] class AioSyncManager(SyncManager): """ A mp.Manager that provides asyncio-friendly objects. """ pass AioSyncManager.register("AioQueue", Queue, AioQueueProxy) AioSyncManager.register("AioBarrier", Barrier, AioQueueProxy) AioSyncManager.register("AioBoundedSemaphore", BoundedSemaphore, AioAcquirerProxy) AioSyncManager.register("AioCondition", Condition, AioConditionProxy) AioSyncManager.register("AioEvent", Event, AioQueueProxy) AioSyncManager.register("AioLock", Lock, AioAcquirerProxy) AioSyncManager.register("AioRLock", RLock, AioAcquirerProxy) AioSyncManager.register("AioSemaphore", Semaphore, AioAcquirerProxy) aioprocessing-1.0.0/aioprocessing/util.py0000664000175000017500000000053412406677614020120 0ustar dandan00000000000000import asyncio def run_in_executor(executor, callback, *args, loop=None, **kwargs): if not loop: loop = asyncio.get_event_loop() if kwargs: return loop.run_in_executor(executor, lambda: callback(*args, **kwargs)) else: return loop.run_in_executor(executor, callback, *args)