requests-futures-0.9.7/0000755000076500000240000000000012657246066015224 5ustar rossstaff00000000000000requests-futures-0.9.7/LICENSE0000644000076500000240000000110612657245731016225 0ustar rossstaff00000000000000Copyright 2013 Ross McFarland Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. requests-futures-0.9.7/MANIFEST.in0000644000076500000240000000015212657245731016756 0ustar rossstaff00000000000000include README.rst LICENSE test_requests_futures.py requirements-python-2.7.txt requirements-python-3.txt requests-futures-0.9.7/PKG-INFO0000644000076500000240000001400612657246066016322 0ustar rossstaff00000000000000Metadata-Version: 1.1 Name: requests-futures Version: 0.9.7 Summary: Asynchronous Python HTTP for Humans. Home-page: https://github.com/ross/requests-futures Author: Ross McFarland Author-email: rwmcfa1@neces.com License: Apache License v2 Description: Asynchronous Python HTTP Requests for Humans ============================================ .. image:: https://travis-ci.org/ross/requests-futures.png?branch=master :target: https://travis-ci.org/ross/requests-futures Small add-on for the python requests_ http library. Makes use of python 3.2's `concurrent.futures`_ or the backport_ for prior versions of python. The additional API and changes are minimal and strives to avoid surprises. The following synchronous code: .. code-block:: python from requests import Session session = Session() # first requests starts and blocks until finished response_one = session.get('http://httpbin.org/get') # second request starts once first is finished response_two = session.get('http://httpbin.org/get?foo=bar') # both requests are complete print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) print('response two status: {0}'.format(response_two.status_code)) print(response_two.content) Can be translated to make use of futures, and thus be asynchronous by creating a FuturesSession and catching the returned Future in place of Response. The Response can be retrieved by calling the result method on the Future: .. code-block:: python from requests_futures.sessions import FuturesSession session = FuturesSession() # first request is started in background future_one = session.get('http://httpbin.org/get') # second requests is started immediately future_two = session.get('http://httpbin.org/get?foo=bar') # wait for the first request to complete, if it hasn't already response_one = future_one.result() print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) # wait for the second request to complete, if it hasn't already response_two = future_two.result() print('response two status: {0}'.format(response_two.status_code)) print(response_two.content) By default a ThreadPoolExecutor is created with 2 workers. If you would like to adjust that value or share a executor across multiple sessions you can provide one to the FuturesSession constructor. .. code-block:: python from concurrent.futures import ThreadPoolExecutor from requests_futures.sessions import FuturesSession session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10)) # ... As a shortcut in case of just increasing workers number you can pass `max_workers` straight to the `FuturesSession` constructor: .. code-block:: python from requests_futures.sessions import FuturesSession session = FuturesSession(max_workers=10) FutureSession will use an existing session object if supplied: .. code-block:: python from requests import session from requests_futures.sessions import FuturesSession my_session = session() future_session = FuturesSession(session=my_session) That's it. The api of requests.Session is preserved without any modifications beyond returning a Future rather than Response. As with all futures exceptions are shifted (thrown) to the future.result() call so try/except blocks should be moved there. Working in the Background ========================= There is one additional parameter to the various request functions, background_callback, which allows you to work with the Response objects in the background thread. This can be useful for shifting work out of the foreground, for a simple example take json parsing. .. code-block:: python from pprint import pprint from requests_futures.sessions import FuturesSession session = FuturesSession() def bg_cb(sess, resp): # parse the json storing the result on the response object resp.data = resp.json() future = session.get('http://httpbin.org/get', background_callback=bg_cb) # do some other stuff, send some more requests while this one works response = future.result() print('response status {0}'.format(response.status_code)) # data will have been attached to the response object in the background pprint(response.data) Installation ============ pip install requests-futures .. _`requests`: https://github.com/kennethreitz/requests .. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html .. _backport: https://pypi.python.org/pypi/futures Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: Natural Language :: English Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 requests-futures-0.9.7/README.rst0000644000076500000240000001043612657245731016715 0ustar rossstaff00000000000000Asynchronous Python HTTP Requests for Humans ============================================ .. image:: https://travis-ci.org/ross/requests-futures.png?branch=master :target: https://travis-ci.org/ross/requests-futures Small add-on for the python requests_ http library. Makes use of python 3.2's `concurrent.futures`_ or the backport_ for prior versions of python. The additional API and changes are minimal and strives to avoid surprises. The following synchronous code: .. code-block:: python from requests import Session session = Session() # first requests starts and blocks until finished response_one = session.get('http://httpbin.org/get') # second request starts once first is finished response_two = session.get('http://httpbin.org/get?foo=bar') # both requests are complete print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) print('response two status: {0}'.format(response_two.status_code)) print(response_two.content) Can be translated to make use of futures, and thus be asynchronous by creating a FuturesSession and catching the returned Future in place of Response. The Response can be retrieved by calling the result method on the Future: .. code-block:: python from requests_futures.sessions import FuturesSession session = FuturesSession() # first request is started in background future_one = session.get('http://httpbin.org/get') # second requests is started immediately future_two = session.get('http://httpbin.org/get?foo=bar') # wait for the first request to complete, if it hasn't already response_one = future_one.result() print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) # wait for the second request to complete, if it hasn't already response_two = future_two.result() print('response two status: {0}'.format(response_two.status_code)) print(response_two.content) By default a ThreadPoolExecutor is created with 2 workers. If you would like to adjust that value or share a executor across multiple sessions you can provide one to the FuturesSession constructor. .. code-block:: python from concurrent.futures import ThreadPoolExecutor from requests_futures.sessions import FuturesSession session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10)) # ... As a shortcut in case of just increasing workers number you can pass `max_workers` straight to the `FuturesSession` constructor: .. code-block:: python from requests_futures.sessions import FuturesSession session = FuturesSession(max_workers=10) FutureSession will use an existing session object if supplied: .. code-block:: python from requests import session from requests_futures.sessions import FuturesSession my_session = session() future_session = FuturesSession(session=my_session) That's it. The api of requests.Session is preserved without any modifications beyond returning a Future rather than Response. As with all futures exceptions are shifted (thrown) to the future.result() call so try/except blocks should be moved there. Working in the Background ========================= There is one additional parameter to the various request functions, background_callback, which allows you to work with the Response objects in the background thread. This can be useful for shifting work out of the foreground, for a simple example take json parsing. .. code-block:: python from pprint import pprint from requests_futures.sessions import FuturesSession session = FuturesSession() def bg_cb(sess, resp): # parse the json storing the result on the response object resp.data = resp.json() future = session.get('http://httpbin.org/get', background_callback=bg_cb) # do some other stuff, send some more requests while this one works response = future.result() print('response status {0}'.format(response.status_code)) # data will have been attached to the response object in the background pprint(response.data) Installation ============ pip install requests-futures .. _`requests`: https://github.com/kennethreitz/requests .. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html .. _backport: https://pypi.python.org/pypi/futures requests-futures-0.9.7/requests_futures/0000755000076500000240000000000012657246066020654 5ustar rossstaff00000000000000requests-futures-0.9.7/requests_futures/__init__.py0000644000076500000240000000111612657246013022754 0ustar rossstaff00000000000000# -*- coding: utf-8 -*- # Requests Futures """ async requests HTTP library ~~~~~~~~~~~~~~~~~~~~~ """ import logging __title__ = 'requests-futures' __version__ = '0.9.7' __build__ = 0x000000 __author__ = 'Ross McFarland' __license__ = 'Apache 2.0' __copyright__ = 'Copyright 2013 Ross McFarland' # Set default logging handler to avoid "No handler found" warnings. try: # Python 2.7+ from logging import NullHandler except ImportError: class NullHandler(logging.Handler): def emit(self, record): pass logging.getLogger(__name__).addHandler(NullHandler()) requests-futures-0.9.7/requests_futures/sessions.py0000644000076500000240000000540412657245731023075 0ustar rossstaff00000000000000# -*- coding: utf-8 -*- """ requests_futures ~~~~~~~~~~~~~~~~ This module provides a small add-on for the requests http library. It makes use of python 3.3's concurrent.futures or the futures backport for previous releases of python. from requests_futures import FuturesSession session = FuturesSession() # request is run in the background future = session.get('http://httpbin.org/get') # ... do other stuff ... # wait for the request to complete, if it hasn't already response = future.result() print('response status: {0}'.format(response.status_code)) print(response.content) """ from concurrent.futures import ThreadPoolExecutor from requests import Session from requests.adapters import DEFAULT_POOLSIZE, HTTPAdapter class FuturesSession(Session): def __init__(self, executor=None, max_workers=2, session=None, *args, **kwargs): """Creates a FuturesSession Notes ~~~~~ * ProcessPoolExecutor is not supported b/c Response objects are not picklable. * If you provide both `executor` and `max_workers`, the latter is ignored and provided executor is used as is. """ super(FuturesSession, self).__init__(*args, **kwargs) if executor is None: executor = ThreadPoolExecutor(max_workers=max_workers) # set connection pool size equal to max_workers if needed if max_workers > DEFAULT_POOLSIZE: adapter_kwargs = dict(pool_connections=max_workers, pool_maxsize=max_workers) self.mount('https://', HTTPAdapter(**adapter_kwargs)) self.mount('http://', HTTPAdapter(**adapter_kwargs)) self.executor = executor self.session = session def request(self, *args, **kwargs): """Maintains the existing api for Session.request. Used by all of the higher level methods, e.g. Session.get. The background_callback param allows you to do some processing on the response in the background, e.g. call resp.json() so that json parsing happens in the background thread. """ if self.session: func = sup = self.session.request else: func = sup = super(FuturesSession, self).request background_callback = kwargs.pop('background_callback', None) if background_callback: def wrap(*args_, **kwargs_): resp = sup(*args_, **kwargs_) background_callback(self, resp) return resp func = wrap return self.executor.submit(func, *args, **kwargs) def __enter__(self): return self def __exit__(self, type, value, traceback): self.executor.shutdown() requests-futures-0.9.7/requests_futures.egg-info/0000755000076500000240000000000012657246066022346 5ustar rossstaff00000000000000requests-futures-0.9.7/requests_futures.egg-info/dependency_links.txt0000644000076500000240000000000112657246066026414 0ustar rossstaff00000000000000 requests-futures-0.9.7/requests_futures.egg-info/not-zip-safe0000644000076500000240000000000112657246066024574 0ustar rossstaff00000000000000 requests-futures-0.9.7/requests_futures.egg-info/PKG-INFO0000644000076500000240000001400612657246066023444 0ustar rossstaff00000000000000Metadata-Version: 1.1 Name: requests-futures Version: 0.9.7 Summary: Asynchronous Python HTTP for Humans. Home-page: https://github.com/ross/requests-futures Author: Ross McFarland Author-email: rwmcfa1@neces.com License: Apache License v2 Description: Asynchronous Python HTTP Requests for Humans ============================================ .. image:: https://travis-ci.org/ross/requests-futures.png?branch=master :target: https://travis-ci.org/ross/requests-futures Small add-on for the python requests_ http library. Makes use of python 3.2's `concurrent.futures`_ or the backport_ for prior versions of python. The additional API and changes are minimal and strives to avoid surprises. The following synchronous code: .. code-block:: python from requests import Session session = Session() # first requests starts and blocks until finished response_one = session.get('http://httpbin.org/get') # second request starts once first is finished response_two = session.get('http://httpbin.org/get?foo=bar') # both requests are complete print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) print('response two status: {0}'.format(response_two.status_code)) print(response_two.content) Can be translated to make use of futures, and thus be asynchronous by creating a FuturesSession and catching the returned Future in place of Response. The Response can be retrieved by calling the result method on the Future: .. code-block:: python from requests_futures.sessions import FuturesSession session = FuturesSession() # first request is started in background future_one = session.get('http://httpbin.org/get') # second requests is started immediately future_two = session.get('http://httpbin.org/get?foo=bar') # wait for the first request to complete, if it hasn't already response_one = future_one.result() print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) # wait for the second request to complete, if it hasn't already response_two = future_two.result() print('response two status: {0}'.format(response_two.status_code)) print(response_two.content) By default a ThreadPoolExecutor is created with 2 workers. If you would like to adjust that value or share a executor across multiple sessions you can provide one to the FuturesSession constructor. .. code-block:: python from concurrent.futures import ThreadPoolExecutor from requests_futures.sessions import FuturesSession session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10)) # ... As a shortcut in case of just increasing workers number you can pass `max_workers` straight to the `FuturesSession` constructor: .. code-block:: python from requests_futures.sessions import FuturesSession session = FuturesSession(max_workers=10) FutureSession will use an existing session object if supplied: .. code-block:: python from requests import session from requests_futures.sessions import FuturesSession my_session = session() future_session = FuturesSession(session=my_session) That's it. The api of requests.Session is preserved without any modifications beyond returning a Future rather than Response. As with all futures exceptions are shifted (thrown) to the future.result() call so try/except blocks should be moved there. Working in the Background ========================= There is one additional parameter to the various request functions, background_callback, which allows you to work with the Response objects in the background thread. This can be useful for shifting work out of the foreground, for a simple example take json parsing. .. code-block:: python from pprint import pprint from requests_futures.sessions import FuturesSession session = FuturesSession() def bg_cb(sess, resp): # parse the json storing the result on the response object resp.data = resp.json() future = session.get('http://httpbin.org/get', background_callback=bg_cb) # do some other stuff, send some more requests while this one works response = future.result() print('response status {0}'.format(response.status_code)) # data will have been attached to the response object in the background pprint(response.data) Installation ============ pip install requests-futures .. _`requests`: https://github.com/kennethreitz/requests .. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html .. _backport: https://pypi.python.org/pypi/futures Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: Natural Language :: English Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 requests-futures-0.9.7/requests_futures.egg-info/requires.txt0000644000076500000240000000003712657246066024746 0ustar rossstaff00000000000000requests>=1.2.0 futures>=2.1.3 requests-futures-0.9.7/requests_futures.egg-info/SOURCES.txt0000644000076500000240000000063612657246066024237 0ustar rossstaff00000000000000LICENSE MANIFEST.in README.rst requirements-python-2.7.txt requirements-python-3.txt setup.py test_requests_futures.py requests_futures/__init__.py requests_futures/sessions.py requests_futures.egg-info/PKG-INFO requests_futures.egg-info/SOURCES.txt requests_futures.egg-info/dependency_links.txt requests_futures.egg-info/not-zip-safe requests_futures.egg-info/requires.txt requests_futures.egg-info/top_level.txtrequests-futures-0.9.7/requests_futures.egg-info/top_level.txt0000644000076500000240000000002112657246066025071 0ustar rossstaff00000000000000requests_futures requests-futures-0.9.7/requirements-python-2.7.txt0000644000076500000240000000003012657245731022322 0ustar rossstaff00000000000000futures requests>=1.2.0 requests-futures-0.9.7/requirements-python-3.txt0000644000076500000240000000002012657245731022155 0ustar rossstaff00000000000000requests>=1.2.0 requests-futures-0.9.7/setup.cfg0000644000076500000240000000007312657246066017045 0ustar rossstaff00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 requests-futures-0.9.7/setup.py0000644000076500000240000000277012657245731016742 0ustar rossstaff00000000000000#!/usr/bin/env python import os import sys import requests_futures try: from setuptools import setup except ImportError: from distutils.core import setup if sys.argv[-1] == 'publish': os.system('python setup.py sdist upload') sys.exit() packages = [ 'requests_futures', ] requires = [ 'requests>=1.2.0' ] if sys.version_info < (3, 2): requires.append('futures>=2.1.3') setup( name='requests-futures', version=requests_futures.__version__, description='Asynchronous Python HTTP for Humans.', long_description=open('README.rst').read(), author='Ross McFarland', author_email='rwmcfa1@neces.com', packages=packages, package_dir={'requests_futures': 'requests_futures'}, package_data={'requests_futures': ['LICENSE', 'README.rst']}, include_package_data=True, install_requires=requires, license='Apache License v2', url='https://github.com/ross/requests-futures', zip_safe=False, classifiers=( 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Natural Language :: English', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', ), ) requests-futures-0.9.7/test_requests_futures.py0000644000076500000240000001032012657245731022257 0ustar rossstaff00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """Tests for Requests.""" from concurrent.futures import Future from requests import Response, session from os import environ from requests_futures.sessions import FuturesSession from unittest import TestCase, main HTTPBIN = environ.get('HTTPBIN_URL', 'http://httpbin.org/') def httpbin(*suffix): """Returns url for HTTPBIN resource.""" return HTTPBIN + '/'.join(suffix) class RequestsTestCase(TestCase): def test_futures_session(self): # basic futures get sess = FuturesSession() future = sess.get(httpbin('get')) self.assertIsInstance(future, Future) resp = future.result() self.assertIsInstance(resp, Response) self.assertEqual(200, resp.status_code) # non-200, 404 future = sess.get(httpbin('status/404')) resp = future.result() self.assertEqual(404, resp.status_code) def cb(s, r): self.assertIsInstance(s, FuturesSession) self.assertIsInstance(r, Response) # add the parsed json data to the response r.data = r.json() future = sess.get(httpbin('get'), background_callback=cb) # this should block until complete resp = future.result() self.assertEqual(200, resp.status_code) # make sure the callback was invoked self.assertTrue(hasattr(resp, 'data')) def rasing_cb(s, r): raise Exception('boom') future = sess.get(httpbin('get'), background_callback=rasing_cb) with self.assertRaises(Exception) as cm: resp = future.result() self.assertEqual('boom', cm.exception.args[0]) def test_supplied_session(self): """ Tests the `session` keyword argument. """ requests_session = session() requests_session.headers['Foo'] = 'bar' sess = FuturesSession(session=requests_session) future = sess.get(httpbin('headers')) self.assertIsInstance(future, Future) resp = future.result() self.assertIsInstance(resp, Response) self.assertEqual(200, resp.status_code) self.assertEqual(resp.json()['headers']['Foo'], 'bar') def test_max_workers(self): """ Tests the `max_workers` shortcut. """ from concurrent.futures import ThreadPoolExecutor session = FuturesSession() self.assertEqual(session.executor._max_workers, 2) session = FuturesSession(max_workers=5) self.assertEqual(session.executor._max_workers, 5) session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10)) self.assertEqual(session.executor._max_workers, 10) session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10), max_workers=5) self.assertEqual(session.executor._max_workers, 10) def test_redirect(self): """ Tests for the ability to cleanly handle redirects. """ sess = FuturesSession() future = sess.get(httpbin('redirect-to?url=get')) self.assertIsInstance(future, Future) resp = future.result() self.assertIsInstance(resp, Response) self.assertEqual(200, resp.status_code) future = sess.get(httpbin('redirect-to?url=status/404')) resp = future.result() self.assertEqual(404, resp.status_code) def test_context(self): class FuturesSessionTestHelper(FuturesSession): def __init__(self, *args, **kwargs): super(FuturesSessionTestHelper, self).__init__(*args, **kwargs) self._exit_called = False def __exit__(self, *args, **kwargs): self._exit_called = True return super(FuturesSessionTestHelper, self).__exit__(*args, **kwargs) passout = None with FuturesSessionTestHelper() as sess: passout = sess future = sess.get(httpbin('get')) self.assertIsInstance(future, Future) resp = future.result() self.assertIsInstance(resp, Response) self.assertEqual(200, resp.status_code) self.assertTrue(passout._exit_called) if __name__ == '__main__': main()