ndg_httpsclient-0.5.1/0000755€t‰át€;ŒS0000000000013325376043017511 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/MANIFEST.in0000644€t‰át€;ŒS0000000047213152236350021244 0ustar pk23CLRC\Domain Users00000000000000# # MANIFEST.in file to enable inclusion of unit test data files and config # # NDG HTTPS Client Package # # P J Kershaw 17/01/12 # # Copyright (C) 2012 STFC # # Licence: BSD - See LICENCE file for details recursive-include ndg/httpsclient/test *.crt *.key *.sh README recursive-include documentation Makefile ndg_httpsclient-0.5.1/PKG-INFO0000644€t‰át€;ŒS0000001614213325376043020612 0ustar pk23CLRC\Domain Users00000000000000Metadata-Version: 2.1 Name: ndg_httpsclient Version: 0.5.1 Summary: Provides enhanced HTTPS support for httplib and urllib2 using PyOpenSSL Home-page: https://github.com/cedadev/ndg_httpsclient/ Author: Richard Wilkinson and Philip Kershaw Author-email: Philip.Kershaw@stfc.ac.uk License: BSD - See ndg/httpsclient/LICENCE file for details Description: A HTTPS client implementation for * ``httplib`` (Python 2), ``http.client`` (Python 3) and * ``urllib2`` (Python 2) and ``urllib`` (Python 3) ... based on PyOpenSSL. PyOpenSSL provides a more fully featured SSL implementation over the default provided with Python and importantly enables full verification of the SSL peer using ``pyasn1``. Releases ======== 0.5.1 ----- * Clean up handling for description file - pull in content from this file into setup() * Allows the nightly build to fail * Add Trove version classifiers to make it explicit what is supported * Add python_requires to help pip * Drop support for EOL Python 2.6 and 3.3 Thanks to @hugovk for contributions 0.5.0 ----- * Fix to Subject Alternative Name handling to allow for certificates with more than 64 names (max now 1024). Thanks to Matt Pegler * Fix to subjectAltName string to use byte type for correct matching * Updated SSL Context objects to default to TLS 1.2 0.4.4 ----- * Updated test certificates 0.4.3 ----- * Fix to ``ndg`` namespace package warning issue (https://github.com/cedadev/ndg_httpsclient/issues/3). ``__init__.py`` file now included in ``ndg`` directory so that there are no longer warnings with imports when using Python 2.x. Thanks to Max Mauntner for fix. * Minor fix for installation: set minimum release for ``pyasn1`` to avoid conflicts with Ubuntu install - see https://github.com/cedadev/ndg_httpsclient/issues/5 and https://github.com/cedadev/ndg_httpsclient/pull/10. ``pyasn1`` also becomes mandatory rather than optional package for install. - It required by ``cryptography`` anyway which is a dependency for ``pyOpenSSL`` from version 0.14. 0.4.2 ----- * Fix to bug in ``ndg.httpsclient.utils.open_url`` - duplicate open call. Nb. This bug and the fix DO NOT affect the ``httplib``and ``urllib2`` interfaces that this package provides. 0.4.1 ----- * Added explicit ref to Python 3 in classifier text for Python 3 checking tools. * Moved LICENSE file into package 0.4.0 ----- * Made dual compatible with Python 2 / 3. 0.3.3 ----- * Fix to add in AnotherName for ``subjectAltNames`` field - added for support for CACert issued certs (thanks to Gu1). * Fix to HTTP Basic Auth option for ``ndg.httpsclient.utils.main`` * Fix to ``ServerSSLCertVerification`` so that it can pass a function-based callback instead of using ``__call__``. In newer versions of OpenSSL (>= 0.14) the latter failed because of a request for ``__name__`` attribute. 0.3.2 ----- * Fix to SubjectAltNames support check - should only be enabled if pyasn1 is installed. * Fix to open_url: HTTP Request object was being created inside if headers is None block - now corrected to create regardless. * Added http basic auth support to script. (Thanks to Willem van Engen) 0.3.1 ----- * extended utils functions to support keyword for passing additional ``urllib2`` handlers. 0.3.0 ----- * Added ``ndg.httpsclient.utils.fetch_stream_from_url`` function and added parameter for data to post in ``open_url`` and ``fetch_*`` methods. * fix to ndg.httpsclient.utils module _should_use_proxy and open_url functions 0.2.0 ----- * added support for SSL verification with subjectAltNames using pyasn1 * fixed minor bug - SSL cert DN prefix matching 0.1.0 ----- Initial release Prerequisites ============= This has been developed and tested for Python 2.7 with pyOpenSSL 0.13 and 0.14. Version 0.4.0 tested with ``pyOpenSSL`` 0.15.1 and Python 2.7 and 3.4. ``pyasn1`` is required for correct SSL verification with ``subjectAltNames``. Installation ============ Installation can be performed using ``easy_install`` or ``pip``. Running ndg_httpclient ====================== A simple script for fetching data using HTTP or HTTPS GET from a specified URL. Parameter: ``url`` The URL of the resource to be fetched Options: ``-h, --help`` Show help message and exit. ``-c FILE, --certificate=FILE`` Certificate file - defaults to ``$HOME/credentials.pem`` ``-k FILE, --private-key=FILE`` Private key file - defaults to the certificate file ``-t DIR, --ca-certificate-dir=DIR`` Trusted CA certificate file directory. ``-d, --debug`` Print debug information - this may be useful in solving problems with HTTP or HTTPS access to a server. ``-p FILE, --post-data-file=FILE`` POST data file ``-f FILE, --fetch=FILE`` Output file ``-n, --no-verify-peer`` Skip verification of peer certificate. Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Environment :: Web Environment Classifier: Intended Audience :: End Users/Desktop Classifier: Intended Audience :: Developers Classifier: Intended Audience :: System Administrators Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: BSD License Classifier: Natural Language :: English Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: 3.6 Classifier: Topic :: Security Classifier: Topic :: Internet Classifier: Topic :: Scientific/Engineering Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: System :: Systems Administration :: Authentication/Directory Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* Description-Content-Type: text/markdown ndg_httpsclient-0.5.1/README.md0000644€t‰át€;ŒS0000001063313324350713020766 0ustar pk23CLRC\Domain Users00000000000000A HTTPS client implementation for * ``httplib`` (Python 2), ``http.client`` (Python 3) and * ``urllib2`` (Python 2) and ``urllib`` (Python 3) ... based on PyOpenSSL. PyOpenSSL provides a more fully featured SSL implementation over the default provided with Python and importantly enables full verification of the SSL peer using ``pyasn1``. Releases ======== 0.5.1 ----- * Clean up handling for description file - pull in content from this file into setup() * Allows the nightly build to fail * Add Trove version classifiers to make it explicit what is supported * Add python_requires to help pip * Drop support for EOL Python 2.6 and 3.3 Thanks to @hugovk for contributions 0.5.0 ----- * Fix to Subject Alternative Name handling to allow for certificates with more than 64 names (max now 1024). Thanks to Matt Pegler * Fix to subjectAltName string to use byte type for correct matching * Updated SSL Context objects to default to TLS 1.2 0.4.4 ----- * Updated test certificates 0.4.3 ----- * Fix to ``ndg`` namespace package warning issue (https://github.com/cedadev/ndg_httpsclient/issues/3). ``__init__.py`` file now included in ``ndg`` directory so that there are no longer warnings with imports when using Python 2.x. Thanks to Max Mauntner for fix. * Minor fix for installation: set minimum release for ``pyasn1`` to avoid conflicts with Ubuntu install - see https://github.com/cedadev/ndg_httpsclient/issues/5 and https://github.com/cedadev/ndg_httpsclient/pull/10. ``pyasn1`` also becomes mandatory rather than optional package for install. - It required by ``cryptography`` anyway which is a dependency for ``pyOpenSSL`` from version 0.14. 0.4.2 ----- * Fix to bug in ``ndg.httpsclient.utils.open_url`` - duplicate open call. Nb. This bug and the fix DO NOT affect the ``httplib``and ``urllib2`` interfaces that this package provides. 0.4.1 ----- * Added explicit ref to Python 3 in classifier text for Python 3 checking tools. * Moved LICENSE file into package 0.4.0 ----- * Made dual compatible with Python 2 / 3. 0.3.3 ----- * Fix to add in AnotherName for ``subjectAltNames`` field - added for support for CACert issued certs (thanks to Gu1). * Fix to HTTP Basic Auth option for ``ndg.httpsclient.utils.main`` * Fix to ``ServerSSLCertVerification`` so that it can pass a function-based callback instead of using ``__call__``. In newer versions of OpenSSL (>= 0.14) the latter failed because of a request for ``__name__`` attribute. 0.3.2 ----- * Fix to SubjectAltNames support check - should only be enabled if pyasn1 is installed. * Fix to open_url: HTTP Request object was being created inside if headers is None block - now corrected to create regardless. * Added http basic auth support to script. (Thanks to Willem van Engen) 0.3.1 ----- * extended utils functions to support keyword for passing additional ``urllib2`` handlers. 0.3.0 ----- * Added ``ndg.httpsclient.utils.fetch_stream_from_url`` function and added parameter for data to post in ``open_url`` and ``fetch_*`` methods. * fix to ndg.httpsclient.utils module _should_use_proxy and open_url functions 0.2.0 ----- * added support for SSL verification with subjectAltNames using pyasn1 * fixed minor bug - SSL cert DN prefix matching 0.1.0 ----- Initial release Prerequisites ============= This has been developed and tested for Python 2.7 with pyOpenSSL 0.13 and 0.14. Version 0.4.0 tested with ``pyOpenSSL`` 0.15.1 and Python 2.7 and 3.4. ``pyasn1`` is required for correct SSL verification with ``subjectAltNames``. Installation ============ Installation can be performed using ``easy_install`` or ``pip``. Running ndg_httpclient ====================== A simple script for fetching data using HTTP or HTTPS GET from a specified URL. Parameter: ``url`` The URL of the resource to be fetched Options: ``-h, --help`` Show help message and exit. ``-c FILE, --certificate=FILE`` Certificate file - defaults to ``$HOME/credentials.pem`` ``-k FILE, --private-key=FILE`` Private key file - defaults to the certificate file ``-t DIR, --ca-certificate-dir=DIR`` Trusted CA certificate file directory. ``-d, --debug`` Print debug information - this may be useful in solving problems with HTTP or HTTPS access to a server. ``-p FILE, --post-data-file=FILE`` POST data file ``-f FILE, --fetch=FILE`` Output file ``-n, --no-verify-peer`` Skip verification of peer certificate. ndg_httpsclient-0.5.1/documentation/0000755€t‰át€;ŒS0000000000013325376043022362 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/documentation/Makefile0000644€t‰át€;ŒS0000000130713152236350024015 0ustar pk23CLRC\Domain Users00000000000000# # Makefile to generate epydoc documentation for the NDG HTTPS Client Package # # @author P J Kershaw 17/01/12 # # @copyright: (C) 2012 STFC # # @license: BSD - see LICENSE file for details # # $Id$ # Generate HTML from embedded epydoc text in source code. EPYDOC=epydoc EPYDOC_INDIR=../ndg EPYDOC_OUTDIR=. EPYDOC_NAME='NDG HTTPS Client' EPYDOC_LOGFILE=epydoc.log EPYDOC_OPTS=--no-frames --include-log --graph=all -v --debug ZIP=zip ZIP_OUTFILE=./documentation.zip ZIP_INFILES=./*.* epydoc: ${EPYDOC} ${EPYDOC_INDIR} -o ${EPYDOC_OUTDIR} --name ${EPYDOC_NAME} \ ${EPYDOC_OPTS} > ${EPYDOC_LOGFILE} zip: ${ZIP} ${ZIP_OUTFILE} ${ZIP_INFILES} clean: rm -f *.txt *.html *.gif *.css *.js *.png *.log ndg_httpsclient-0.5.1/ndg/0000755€t‰át€;ŒS0000000000013325376043020261 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg/__init__.py0000644€t‰át€;ŒS0000000010113152266146022362 0ustar pk23CLRC\Domain Users00000000000000__path__ = __import__('pkgutil').extend_path(__path__, __name__) ndg_httpsclient-0.5.1/ndg/httpsclient/0000755€t‰át€;ŒS0000000000013325376043022622 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg/httpsclient/LICENSE0000644€t‰át€;ŒS0000000307613152236350023627 0ustar pk23CLRC\Domain Users00000000000000Copyright (c) 2012, Science & Technology Facilities Council (STFC) All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Science & Technology Facilities Council (STFC) nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ndg_httpsclient-0.5.1/ndg/httpsclient/__init__.py0000644€t‰át€;ŒS0000000060613152236350024727 0ustar pk23CLRC\Domain Users00000000000000"""ndg_httpsclient - PyOpenSSL utility to make a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw (STFC) and Richard Wilkinson (Tessella)" __date__ = "09/12/11" __copyright__ = "(C) 2011 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' ndg_httpsclient-0.5.1/ndg/httpsclient/https.py0000644€t‰át€;ŒS0000001141713324353023024332 0ustar pk23CLRC\Domain Users00000000000000"""ndg_httpsclient HTTPS module containing PyOpenSSL implementation of httplib.HTTPSConnection PyOpenSSL utility to make a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw (STFC)" __date__ = "09/12/11" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import logging import socket import sys if sys.version_info[0] > 2: from http.client import HTTPS_PORT from http.client import HTTPConnection from urllib.request import AbstractHTTPHandler else: from httplib import HTTPS_PORT from httplib import HTTPConnection from urllib2 import AbstractHTTPHandler from OpenSSL import SSL from ndg.httpsclient.ssl_socket import SSLSocket log = logging.getLogger(__name__) class HTTPSConnection(HTTPConnection): """This class allows communication via SSL using PyOpenSSL. It is based on httplib.HTTPSConnection, modified to use PyOpenSSL. Note: This uses the constructor inherited from HTTPConnection to allow it to be used with httplib and HTTPSContextHandler. To use the class directly with an SSL context set ssl_context after construction. @cvar default_port: default port for this class (443) @type default_port: int @cvar default_ssl_method: default SSL method used if no SSL context is explicitly set - defaults to version 2/3. @type default_ssl_method: int """ default_port = HTTPS_PORT default_ssl_method = SSL.TLSv1_2_METHOD def __init__(self, host, port=None, strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, ssl_context=None): HTTPConnection.__init__(self, host, port, strict, timeout) if not hasattr(self, 'ssl_context'): self.ssl_context = None if ssl_context is not None: if not isinstance(ssl_context, SSL.Context): raise TypeError('Expecting OpenSSL.SSL.Context type for "' 'ssl_context" keyword; got %r instead' % ssl_context) self.ssl_context = ssl_context def connect(self): """Create SSL socket and connect to peer """ if getattr(self, 'ssl_context', None): if not isinstance(self.ssl_context, SSL.Context): raise TypeError('Expecting OpenSSL.SSL.Context type for "' 'ssl_context" attribute; got %r instead' % self.ssl_context) ssl_context = self.ssl_context else: ssl_context = SSL.Context(self.__class__.default_ssl_method) sock = socket.create_connection((self.host, self.port), self.timeout) # Tunnel if using a proxy - ONLY available for Python 2.6.2 and above if getattr(self, '_tunnel_host', None): self.sock = sock self._tunnel() self.sock = SSLSocket(ssl_context, sock) # Go to client mode. self.sock.set_connect_state() def close(self): """Close socket and shut down SSL connection""" if hasattr(self.sock, "close"): self.sock.close() class HTTPSContextHandler(AbstractHTTPHandler): '''HTTPS handler that allows a SSL context to be set for the SSL connections. ''' https_request = AbstractHTTPHandler.do_request_ SSL_METHOD = SSL.TLSv1_2_METHOD def __init__(self, ssl_context, debuglevel=0): """ @param ssl_context:SSL context @type ssl_context: OpenSSL.SSL.Context @param debuglevel: debug level for HTTPSHandler @type debuglevel: int """ AbstractHTTPHandler.__init__(self, debuglevel) if ssl_context is not None: if not isinstance(ssl_context, SSL.Context): raise TypeError('Expecting OpenSSL.SSL.Context type for "' 'ssl_context" keyword; got %r instead' % ssl_context) self.ssl_context = ssl_context else: self.ssl_context = SSL.Context(self.__class__.SSL_METHOD) def https_open(self, req): """Opens HTTPS request @param req: HTTP request @return: HTTP Response object """ # Make a custom class extending HTTPSConnection, with the SSL context # set as a class variable so that it is available to the connect method. customHTTPSContextConnection = type('CustomHTTPSContextConnection', (HTTPSConnection, object), {'ssl_context': self.ssl_context}) return self.do_open(customHTTPSContextConnection, req) ndg_httpsclient-0.5.1/ndg/httpsclient/ssl_context_util.py0000644€t‰át€;ŒS0000000672013267567006026611 0ustar pk23CLRC\Domain Users00000000000000"""ndg_httpsclient SSL Context utilities module containing convenience routines for setting SSL context configuration. """ __author__ = "P J Kershaw (STFC)" __date__ = "09/12/11" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import sys if sys.version_info[0] > 2: import urllib.parse as urlparse_ else: import urlparse as urlparse_ from OpenSSL import SSL from ndg.httpsclient.ssl_peer_verification import ServerSSLCertVerification class SSlContextConfig(object): """ Holds configuration options for creating a SSL context. This is used as a template to create the contexts with specific verification callbacks. """ def __init__(self, key_file=None, cert_file=None, pem_file=None, ca_dir=None, verify_peer=False): self.key_file = key_file self.cert_file = cert_file self.pem_file = pem_file self.ca_dir = ca_dir self.verify_peer = verify_peer def make_ssl_context_from_config(ssl_config=False, url=None): return make_ssl_context(ssl_config.key_file, ssl_config.cert_file, ssl_config.pem_file, ssl_config.ca_dir, ssl_config.verify_peer, url) def make_ssl_context(key_file=None, cert_file=None, pem_file=None, ca_dir=None, verify_peer=False, url=None, method=SSL.TLSv1_2_METHOD, key_file_passphrase=None): """ Creates SSL context containing certificate and key file locations. """ ssl_context = SSL.Context(method) # Key file defaults to certificate file if present. if cert_file: ssl_context.use_certificate_file(cert_file) if key_file_passphrase: passwd_cb = lambda max_passphrase_len, set_prompt, userdata: \ key_file_passphrase ssl_context.set_passwd_cb(passwd_cb) if key_file: ssl_context.use_privatekey_file(key_file) elif cert_file: ssl_context.use_privatekey_file(cert_file) if pem_file or ca_dir: ssl_context.load_verify_locations(pem_file, ca_dir) else: ssl_context.set_default_verify_paths() # Use OS CA bundle def _callback(conn, x509, errnum, errdepth, preverify_ok): """Default certification verification callback. Performs no checks and returns the status passed in. """ return preverify_ok verify_callback = _callback if verify_peer: ssl_context.set_verify_depth(9) if url: set_peer_verification_for_url_hostname(ssl_context, url) else: ssl_context.set_verify(SSL.VERIFY_PEER, verify_callback) else: ssl_context.set_verify(SSL.VERIFY_NONE, verify_callback) return ssl_context def set_peer_verification_for_url_hostname(ssl_context, url, if_verify_enabled=False): '''Convenience routine to set peer verification callback based on ServerSSLCertVerification class''' if not if_verify_enabled or (ssl_context.get_verify_mode() & SSL.VERIFY_PEER): urlObj = urlparse_.urlparse(url) hostname = urlObj.hostname server_ssl_cert_verif = ServerSSLCertVerification(hostname=hostname) verify_callback_ = server_ssl_cert_verif.get_verify_server_cert_func() ssl_context.set_verify(SSL.VERIFY_PEER, verify_callback_) ndg_httpsclient-0.5.1/ndg/httpsclient/ssl_peer_verification.py0000644€t‰át€;ŒS0000002275013267567006027566 0ustar pk23CLRC\Domain Users00000000000000"""ndg_httpsclient - module containing SSL peer verification class. """ __author__ = "P J Kershaw (STFC)" __date__ = "09/12/11" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import re import logging log = logging.getLogger(__name__) try: from ndg.httpsclient.subj_alt_name import SubjectAltName from pyasn1.codec.der import decoder as der_decoder SUBJ_ALT_NAME_SUPPORT = True except ImportError as e: SUBJ_ALT_NAME_SUPPORT = False SUBJ_ALT_NAME_SUPPORT_MSG = ( 'SubjectAltName support is disabled - check pyasn1 package ' 'installation to enable' ) import warnings warnings.warn(SUBJ_ALT_NAME_SUPPORT_MSG) class ServerSSLCertVerification(object): """Check server identity. If hostname doesn't match, allow match of host's Distinguished Name against server DN setting""" DN_LUT = { 'commonName': 'CN', 'organisationalUnitName': 'OU', 'organisation': 'O', 'countryName': 'C', 'emailAddress': 'EMAILADDRESS', 'localityName': 'L', 'stateOrProvinceName': 'ST', 'streetAddress': 'STREET', 'domainComponent': 'DC', 'userid': 'UID' } SUBJ_ALT_NAME_EXT_NAME = b'subjectAltName' PARSER_RE_STR = '/(%s)=' % '|'.join(list(DN_LUT.keys()) + \ list(DN_LUT.values())) PARSER_RE = re.compile(PARSER_RE_STR) __slots__ = ('__hostname', '__certDN', '__subj_alt_name_match') def __init__(self, certDN=None, hostname=None, subj_alt_name_match=True): """Override parent class __init__ to enable setting of certDN setting @type certDN: string @param certDN: Set the expected Distinguished Name of the server to avoid errors matching hostnames. This is useful where the hostname is not fully qualified @type hostname: string @param hostname: hostname to match against peer certificate subjectAltNames or subject common name @type subj_alt_name_match: bool @param subj_alt_name_match: flag to enable/disable matching of hostname against peer certificate subjectAltNames. Nb. A setting of True will be ignored if the pyasn1 package is not installed """ self.__certDN = None self.__hostname = None if certDN is not None: self.certDN = certDN if hostname is not None: self.hostname = hostname if subj_alt_name_match: if not SUBJ_ALT_NAME_SUPPORT: log.warning('Overriding "subj_alt_name_match" keyword setting: ' 'peer verification with subjectAltNames is disabled') self.__subj_alt_name_match = False else: self.__subj_alt_name_match = True else: log.debug('Disabling peer verification with subject ' 'subjectAltNames!') self.__subj_alt_name_match = False def __call__(self, connection, peerCert, errorStatus, errorDepth, preverifyOK): """Verify server certificate @type connection: OpenSSL.SSL.Connection @param connection: SSL connection object @type peerCert: basestring @param peerCert: server host certificate as OpenSSL.crypto.X509 instance @type errorStatus: int @param errorStatus: error status passed from caller. This is the value returned by the OpenSSL C function X509_STORE_CTX_get_error(). Look-up x509_vfy.h in the OpenSSL source to get the meanings of the different codes. PyOpenSSL doesn't help you! @type errorDepth: int @param errorDepth: a non-negative integer representing where in the certificate chain the error occurred. If it is zero it occured in the end entity certificate, one if it is the certificate which signed the end entity certificate and so on. @type preverifyOK: int @param preverifyOK: the error status - 0 = Error, 1 = OK of the current SSL context irrespective of any verification checks done here. If this function yields an OK status, it should enforce the preverifyOK value so that any error set upstream overrides and is honoured. @rtype: int @return: status code - 0/False = Error, 1/True = OK """ if peerCert.has_expired(): # Any expired certificate in the chain should result in an error log.error('Certificate %r in peer certificate chain has expired', peerCert.get_subject()) return False elif errorDepth == 0: # Only interested in DN of last certificate in the chain - this must # match the expected Server DN setting peerCertSubj = peerCert.get_subject() peerCertDN = peerCertSubj.get_components() peerCertDN.sort() if self.certDN is None: # Check hostname against peer certificate CN field instead: if self.hostname is None: log.error('No "hostname" or "certDN" set to check peer ' 'certificate against') return False # Check for subject alternative names if self.__subj_alt_name_match: dns_names = self._get_subj_alt_name(peerCert) if self.hostname in dns_names: return preverifyOK # If no subjectAltNames, default to check of subject Common Name if peerCertSubj.commonName == self.hostname: return preverifyOK else: log.error('Peer certificate CN %r doesn\'t match the ' 'expected CN %r', peerCertSubj.commonName, self.hostname) return False else: if peerCertDN == self.certDN: return preverifyOK else: log.error('Peer certificate DN %r doesn\'t match the ' 'expected DN %r', peerCertDN, self.certDN) return False else: return preverifyOK def get_verify_server_cert_func(self): def verify_server_cert(connection, peerCert, errorStatus, errorDepth, preverifyOK): return self.__call__(connection, peerCert, errorStatus, errorDepth, preverifyOK) return verify_server_cert @classmethod def _get_subj_alt_name(cls, peer_cert): '''Extract subjectAltName DNS name settings from certificate extensions @param peer_cert: peer certificate in SSL connection. subjectAltName settings if any will be extracted from this @type peer_cert: OpenSSL.crypto.X509 ''' # Search through extensions dns_name = [] general_names = SubjectAltName() for i in range(peer_cert.get_extension_count()): ext = peer_cert.get_extension(i) ext_name = ext.get_short_name() if ext_name == cls.SUBJ_ALT_NAME_EXT_NAME: # PyOpenSSL returns extension data in ASN.1 encoded form ext_dat = ext.get_data() decoded_dat = der_decoder.decode(ext_dat, asn1Spec=general_names) for name in decoded_dat: if isinstance(name, SubjectAltName): for entry in range(len(name)): component = name.getComponentByPosition(entry) dns_name.append(str(component.getComponent())) return dns_name def _getCertDN(self): return self.__certDN def _setCertDN(self, val): if isinstance(val, str): # Allow for quoted DN certDN = val.strip('"') dnFields = self.__class__.PARSER_RE.split(certDN) if len(dnFields) < 2: raise TypeError('Error parsing DN string: "%s"' % certDN) self.__certDN = list(zip(dnFields[1::2], dnFields[2::2])) self.__certDN.sort() elif not isinstance(val, list): for i in val: if not len(i) == 2: raise TypeError('Expecting list of two element DN field, ' 'DN field value pairs for "certDN" ' 'attribute') self.__certDN = val else: raise TypeError('Expecting list or string type for "certDN" ' 'attribute') certDN = property(fget=_getCertDN, fset=_setCertDN, doc="Distinguished Name for Server Certificate") # Get/Set Property methods def _getHostname(self): return self.__hostname def _setHostname(self, val): if not isinstance(val, str): raise TypeError("Expecting string type for hostname " "attribute") self.__hostname = val hostname = property(fget=_getHostname, fset=_setHostname, doc="hostname of server") ndg_httpsclient-0.5.1/ndg/httpsclient/ssl_socket.py0000644€t‰át€;ŒS0000002220013152236350025333 0ustar pk23CLRC\Domain Users00000000000000"""PyOpenSSL utilities including HTTPSSocket class which wraps PyOpenSSL SSL connection into a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw" __date__ = "21/12/10" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' from datetime import datetime import logging import socket from io import BytesIO from OpenSSL import SSL log = logging.getLogger(__name__) class SSLSocket(object): """SSL Socket class wraps pyOpenSSL's SSL.Connection class implementing the makefile method so that it is compatible with the standard socket interface and usable with httplib. @cvar default_buf_size: default buffer size for recv operations in the makefile method @type default_buf_size: int """ default_buf_size = 8192 def __init__(self, ctx, sock=None): """Create SSL socket object @param ctx: SSL context @type ctx: OpenSSL.SSL.Context @param sock: underlying socket object @type sock: socket.socket """ if sock is not None: self.socket = sock else: self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.__ssl_conn = SSL.Connection(ctx, self.socket) self.buf_size = self.__class__.default_buf_size self._makefile_refs = 0 def __del__(self): """Close underlying socket when this object goes out of scope """ self.close() @property def buf_size(self): """Buffer size for makefile method recv() operations""" return self.__buf_size @buf_size.setter def buf_size(self, value): """Buffer size for makefile method recv() operations""" if not isinstance(value, int): raise TypeError('Expecting int type for "buf_size"; ' 'got %r instead' % type(value)) self.__buf_size = value def close(self): """Shutdown the SSL connection and call the close method of the underlying socket""" if self._makefile_refs < 1: try: self.__ssl_conn.shutdown() except (SSL.Error, SSL.SysCallError): # Make errors on shutdown non-fatal pass else: self._makefile_refs -= 1 def set_shutdown(self, mode): """Set the shutdown state of the Connection. @param mode: bit vector of either or both of SENT_SHUTDOWN and RECEIVED_SHUTDOWN """ self.__ssl_conn.set_shutdown(mode) def get_shutdown(self): """Get the shutdown state of the Connection. @return: bit vector of either or both of SENT_SHUTDOWN and RECEIVED_SHUTDOWN """ return self.__ssl_conn.get_shutdown() def bind(self, addr): """bind to the given address - calls method of the underlying socket @param addr: address/port number tuple @type addr: tuple""" self.__ssl_conn.bind(addr) def listen(self, backlog): """Listen for connections made to the socket. @param backlog: specifies the maximum number of queued connections and should be at least 1; the maximum value is system-dependent (usually 5). @param backlog: int """ self.__ssl_conn.listen(backlog) def set_accept_state(self): """Set the connection to work in server mode. The handshake will be handled automatically by read/write""" self.__ssl_conn.set_accept_state() def accept(self): """Accept an SSL connection. @return: pair (ssl, addr) where ssl is a new SSL connection object and addr is the address bound to the other end of the SSL connection. @rtype: tuple """ return self.__ssl_conn.accept() def set_connect_state(self): """Set the connection to work in client mode. The handshake will be handled automatically by read/write""" self.__ssl_conn.set_connect_state() def connect(self, addr): """Call the connect method of the underlying socket and set up SSL on the socket, using the Context object supplied to this Connection object at creation. @param addr: address/port number pair @type addr: tuple """ self.__ssl_conn.connect(addr) def shutdown(self, how): """Send the shutdown message to the Connection. @param how: for socket.socket this flag determines whether read, write or both type operations are supported. OpenSSL.SSL.Connection doesn't support this so this parameter is IGNORED @return: true if the shutdown message exchange is completed and false otherwise (in which case you call recv() or send() when the connection becomes readable/writeable. @rtype: bool """ return self.__ssl_conn.shutdown() def renegotiate(self): """Renegotiate this connection's SSL parameters.""" return self.__ssl_conn.renegotiate() def pending(self): """@return: numbers of bytes that can be safely read from the SSL buffer. @rtype: int """ return self.__ssl_conn.pending() def send(self, data, *flags_arg): """Send data to the socket. Nb. The optional flags argument is ignored. - retained for compatibility with socket.socket interface @param data: data to send down the socket @type data: string """ return self.__ssl_conn.send(data) def sendall(self, data): self.__ssl_conn.sendall(data) def recv(self, size=default_buf_size): """Receive data from the Connection. @param size: The maximum amount of data to be received at once @type size: int @return: data received. @rtype: string """ return self.__ssl_conn.recv(size) def setblocking(self, mode): """Set this connection's underlying socket blocking _mode_. @param mode: blocking mode @type mode: int """ self.__ssl_conn.setblocking(mode) def fileno(self): """ @return: file descriptor number for the underlying socket @rtype: int """ return self.__ssl_conn.fileno() def getsockopt(self, *args): """See socket.socket.getsockopt """ return self.__ssl_conn.getsockopt(*args) def setsockopt(self, *args): """See socket.socket.setsockopt @return: value of the given socket option @rtype: int/string """ return self.__ssl_conn.setsockopt(*args) def state_string(self): """Return the SSL state of this connection.""" return self.__ssl_conn.state_string() def makefile(self, *args): """Specific to Python socket API and required by httplib: convert response into a file-like object. This implementation reads using recv and copies the output into a StringIO buffer to simulate a file object for consumption by httplib Nb. Ignoring optional file open mode (StringIO is generic and will open for read and write unless a string is passed to the constructor) and buffer size - httplib set a zero buffer size which results in recv reading nothing @return: file object for data returned from socket @rtype: cStringIO.StringO """ self._makefile_refs += 1 # Optimisation _buf_size = self.buf_size i=0 stream = BytesIO() startTime = datetime.utcnow() try: dat = self.__ssl_conn.recv(_buf_size) while dat: i+=1 stream.write(dat) dat = self.__ssl_conn.recv(_buf_size) except (SSL.ZeroReturnError, SSL.SysCallError): # Connection is closed - assuming here that all is well and full # response has been received. httplib will catch an error in # incomplete content since it checks the content-length header # against the actual length of data received pass if log.getEffectiveLevel() <= logging.DEBUG: log.debug("Socket.makefile %d recv calls completed in %s", i, datetime.utcnow() - startTime) # Make sure to rewind the buffer otherwise consumers of the content will # read from the end of the buffer stream.seek(0) return stream def getsockname(self): """ @return: the socket's own address @rtype: """ return self.__ssl_conn.getsockname() def getpeername(self): """ @return: remote address to which the socket is connected """ return self.__ssl_conn.getpeername() def get_context(self): '''Retrieve the Context object associated with this Connection. ''' return self.__ssl_conn.get_context() def get_peer_certificate(self): '''Retrieve the other side's certificate (if any) ''' return self.__ssl_conn.get_peer_certificate() ndg_httpsclient-0.5.1/ndg/httpsclient/subj_alt_name.py0000644€t‰át€;ŒS0000001376513267567175026030 0ustar pk23CLRC\Domain Users00000000000000"""NDG HTTPS Client package Use pyasn1 to provide support for parsing ASN.1 formatted subjectAltName content for SSL peer verification. Code based on: http://stackoverflow.com/questions/5519958/how-do-i-parse-subjectaltname-extension-data-using-pyasn1 """ __author__ = "P J Kershaw" __date__ = "01/02/12" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' try: from pyasn1.type import univ, constraint, char, namedtype, tag except ImportError as e: import_error_msg = ('Error importing pyasn1, subjectAltName check for SSL ' 'peer verification will be disabled. Import error ' 'is: %s' % e) import warnings warnings.warn(import_error_msg) class Pyasn1ImportError(ImportError): "Raise for pyasn1 import error" raise Pyasn1ImportError(import_error_msg) MAX = 1024 class DirectoryString(univ.Choice): """ASN.1 Directory string class""" componentType = namedtype.NamedTypes( namedtype.NamedType( 'teletexString', char.TeletexString().subtype( subtypeSpec=constraint.ValueSizeConstraint(1, MAX))), namedtype.NamedType( 'printableString', char.PrintableString().subtype( subtypeSpec=constraint.ValueSizeConstraint(1, MAX))), namedtype.NamedType( 'universalString', char.UniversalString().subtype( subtypeSpec=constraint.ValueSizeConstraint(1, MAX))), namedtype.NamedType( 'utf8String', char.UTF8String().subtype( subtypeSpec=constraint.ValueSizeConstraint(1, MAX))), namedtype.NamedType( 'bmpString', char.BMPString().subtype( subtypeSpec=constraint.ValueSizeConstraint(1, MAX))), namedtype.NamedType( 'ia5String', char.IA5String().subtype( subtypeSpec=constraint.ValueSizeConstraint(1, MAX))), ) class AttributeValue(DirectoryString): """ASN.1 Attribute value""" class AttributeType(univ.ObjectIdentifier): """ASN.1 Attribute type""" class AttributeTypeAndValue(univ.Sequence): """ASN.1 Attribute type and value class""" componentType = namedtype.NamedTypes( namedtype.NamedType('type', AttributeType()), namedtype.NamedType('value', AttributeValue()), ) class RelativeDistinguishedName(univ.SetOf): '''ASN.1 Realtive distinguished name''' componentType = AttributeTypeAndValue() class RDNSequence(univ.SequenceOf): '''ASN.1 RDN sequence class''' componentType = RelativeDistinguishedName() class Name(univ.Choice): '''ASN.1 name class''' componentType = namedtype.NamedTypes( namedtype.NamedType('', RDNSequence()), ) class Extension(univ.Sequence): '''ASN.1 extension class''' componentType = namedtype.NamedTypes( namedtype.NamedType('extnID', univ.ObjectIdentifier()), namedtype.DefaultedNamedType('critical', univ.Boolean('False')), namedtype.NamedType('extnValue', univ.OctetString()), ) class Extensions(univ.SequenceOf): '''ASN.1 extensions class''' componentType = Extension() sizeSpec = univ.SequenceOf.sizeSpec + constraint.ValueSizeConstraint(1, MAX) class AnotherName(univ.Sequence): componentType = namedtype.NamedTypes( namedtype.NamedType('type-id', univ.ObjectIdentifier()), namedtype.NamedType('value', univ.Any().subtype( explicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 0))) ) class GeneralName(univ.Choice): '''ASN.1 configuration for X.509 certificate subjectAltNames fields''' componentType = namedtype.NamedTypes( namedtype.NamedType('otherName', AnotherName().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 0))), namedtype.NamedType('rfc822Name', char.IA5String().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 1))), namedtype.NamedType('dNSName', char.IA5String().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 2))), # namedtype.NamedType('x400Address', ORAddress().subtype( # implicitTag=tag.Tag(tag.tagClassContext, # tag.tagFormatSimple, 3))), namedtype.NamedType('directoryName', Name().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 4))), # namedtype.NamedType('ediPartyName', EDIPartyName().subtype( # implicitTag=tag.Tag(tag.tagClassContext, # tag.tagFormatSimple, 5))), namedtype.NamedType('uniformResourceIdentifier', char.IA5String().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 6))), namedtype.NamedType('iPAddress', univ.OctetString().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 7))), namedtype.NamedType('registeredID', univ.ObjectIdentifier().subtype( implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 8))), ) class GeneralNames(univ.SequenceOf): '''Sequence of names for ASN.1 subjectAltNames settings''' componentType = GeneralName() sizeSpec = univ.SequenceOf.sizeSpec + constraint.ValueSizeConstraint(1, MAX) class SubjectAltName(GeneralNames): '''ASN.1 implementation for subjectAltNames support''' ndg_httpsclient-0.5.1/ndg/httpsclient/test/0000755€t‰át€;ŒS0000000000013325376043023601 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg/httpsclient/test/README0000644€t‰át€;ŒS0000000120313152236350024447 0ustar pk23CLRC\Domain Users00000000000000NDG HTTPS Client Unit tests directory ===================================== The unit tests expect to connect to a simple HTTPS server listening on port 4443. An OpenSSL script is provided for this purpose in scripts/. To run, $ ./scripts/openssl_https_server.sh Unit tests ---------- Run for example, $ python ./test_urllib2.py Troubleshooting --------------- * Run the openssl script from *this* directory. * Also ensure it is has execute bits set. e.g. $ chmod 755 ./scripts/openssl_https_server.sh * You may need to set the no_proxy environment variable if you have a HTTPS proxy in place: $ export no_proxy=localhost ndg_httpsclient-0.5.1/ndg/httpsclient/test/__init__.py0000644€t‰át€;ŒS0000000216313267567006025722 0ustar pk23CLRC\Domain Users00000000000000"""Unit tests package for ndg_httpsclient PyOpenSSL utility to make a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw (STFC)" __date__ = "05/01/12" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import os import unittest class Constants(object): '''Convenience base class from which other unit tests can extend. Its sets the generic data directory path''' PORT = 4443 # PORT = 443 PORT2 = 4444 HOSTNAME = 'localhost' # HOSTNAME = 'files.pythonhosted.org' TEST_URI = 'https://%s:%d' % (HOSTNAME, PORT) TEST_URI2 = 'https://%s:%d' % (HOSTNAME, PORT2) UNITTEST_DIR = os.path.dirname(os.path.abspath(__file__)) CACERT_DIR = os.path.join(UNITTEST_DIR, 'pki', 'ca') SSL_CERT_FILENAME = 'localhost.crt' SSL_CERT_FILEPATH = os.path.join(UNITTEST_DIR, 'pki', SSL_CERT_FILENAME) SSL_PRIKEY_FILENAME = 'localhost.key' SSL_PRIKEY_FILEPATH = os.path.join(UNITTEST_DIR, 'pki', SSL_PRIKEY_FILENAME) ndg_httpsclient-0.5.1/ndg/httpsclient/test/pki/0000755€t‰át€;ŒS0000000000013325376043024364 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg/httpsclient/test/pki/ca/0000755€t‰át€;ŒS0000000000013325376043024747 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg/httpsclient/test/pki/ca/7e15277f.00000644€t‰át€;ŒS0000000221413233562056026115 0ustar pk23CLRC\Domain Users00000000000000-----BEGIN CERTIFICATE----- MIIDLjCCAhagAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRQwEgYDVQQDDAtOREcg VGVzdCBDQTEMMAoGA1UECgwDTkRHMREwDwYDVQQLDAhTZWN1cml0eTAeFw0xNjEy MTUyMTMyNDNaFw0yMTEyMTQyMTMyNDNaMDcxFDASBgNVBAMMC05ERyBUZXN0IENB MQwwCgYDVQQKDANOREcxETAPBgNVBAsMCFNlY3VyaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAv8L7md7A3NvKIYndyjHV7f1xVH6gJ8JPrecxAiVs nidsYUm8saBV/dFf1jPMQ6hdjIuxLXlopLxThkHlCes/OTxLzBFLvFQQaxLExTsX OmgeM4q20a2JLTqkmItPIYDVTYgi0EaF+I9Z1BwWGEfYr3uTizpr48rnzpXgRwrZ rjRc53zLKpeqyBs7Qmg5Jlzmk/A+UWJ2ryCWLY9KFE3uPstybYpwq7YayD6upVwc vedhh+zpXU3E9r/cnlZ4sKSP2M1QSOTb2l5XnlOU/MLODYW70uItNqs1j1XPuHYh /ikO+jmjPUtL0IE47+LcaMU6BpaOCLzcjsSn/DDMYcDB7wIDAQABo0UwQzASBgNV HRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU9S+bGqsW uQzR/yHDdHoU6afNTE0wDQYJKoZIhvcNAQELBQADggEBAF2scAHd7xjgnJqAUX0b 7f7QKjpuYHrt48tH+pFOh5j3IgdOqTgRLc/N8lEqoH4M0lKZKbBerM9FiQ4eXCwy 3Rswn3wXPvi3HJGgvZYUt8J6KY5+syHU4iuzuSHOjznC0lLZkRz6kfZHX+paqro4 1CsHhCt3ew17QMWEP6UtfZfBCg+kiEfBtSsTUIth0HgdH033PWh5v+nOzGN+3o2t ORlzpttV+0RLlsw54l51I6rCEnfKOKtEy491JUs2whUwzp9v8tG4jev3PDVWhSSS 6jaSNvW4v7SDH0pnw52ZWzaRtA/pV/dCQOlqJ87wDNPdoVGpwqtYF/7Zdw2ty4gU oSM= -----END CERTIFICATE----- ndg_httpsclient-0.5.1/ndg/httpsclient/test/pki/localhost.crt0000644€t‰át€;ŒS0000000213713233562056027070 0ustar pk23CLRC\Domain Users00000000000000-----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIBADANBgkqhkiG9w0BAQsFADA3MRQwEgYDVQQDDAtOREcg VGVzdCBDQTEMMAoGA1UECgwDTkRHMREwDwYDVQQLDAhTZWN1cml0eTAeFw0xNjEy MTYwMTA3MzJaFw0xOTEyMTYwMTA3MzJaMDUxDDAKBgNVBAoMA05ERzERMA8GA1UE CwwIU2VjdXJpdHkxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB BQADggEPADCCAQoCggEBAMOuihdNsoWgVg+DsRBc4kckQ5oiCVsx9NiRkUoCE3mu dQb6pCRhoD3vjMMt18Ajr9xNMtqTtP3lqQ4iId4bd3WaNc2L4a9698QOnpY+Z70T 0pFckNvVcC+5kXHPCFsm/jIVsjJDdPQWM0fFqPGZTwQ8ssS5P9jdB2JHchHFOCWd IYYO0jY1BEJ1AXeK4XzfU0/00s0LnleLrbEpNB9ckQaSbLU7I9o7fDr/5+WbV8tI C5OG/ZZW7jlc4t4TLpzkUatBGW3c2PKJPgtWBAbtScxbtHB4mXV6iwmp3hlLT6tx vKD+b8TFpxGPdoOfIHX6/hjMJa+kr4Z4MDHeNfHu6eUCAwEAAaMmMCQwDAYDVR0T AQH/BAIwADAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZIhvcNAQELBQADggEB AHm7NNZvti4QfopARMIcuokWHac18Ka7HBRE04WxLw9vKuYYu1m7d/0sUhk3KuBB ANOyzlt5y7ecEIAqxGe0m1yftD4Dgj33wB7kHvbIKUceKI563PM757RgDm6C70/p 3OwOCcx2FsSaKLFBdtahzv1cMriN7JpEoa1qT3MnmFyaK8hEaXxbRGwDDAXRgOix ftOkNc5w0glQ7dl9CYq2QvsnMtG327pvyo6lBTKNjFzppZ/msOagPONuAcrtLAiU wTNk5Aqlj6a+Njtq/LJFdeyYsVxwuqvVnEsU+9FJFL8PYrWpG8fA1dIXaPS+06Oi vAMwOmVKhtgKvbaGcYa6efM= -----END CERTIFICATE----- ndg_httpsclient-0.5.1/ndg/httpsclient/test/pki/localhost.key0000644€t‰át€;ŒS0000000321713233562056027070 0ustar pk23CLRC\Domain Users00000000000000-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAw66KF02yhaBWD4OxEFziRyRDmiIJWzH02JGRSgITea51Bvqk JGGgPe+Mwy3XwCOv3E0y2pO0/eWpDiIh3ht3dZo1zYvhr3r3xA6elj5nvRPSkVyQ 29VwL7mRcc8IWyb+MhWyMkN09BYzR8Wo8ZlPBDyyxLk/2N0HYkdyEcU4JZ0hhg7S NjUEQnUBd4rhfN9TT/TSzQueV4utsSk0H1yRBpJstTsj2jt8Ov/n5ZtXy0gLk4b9 llbuOVzi3hMunORRq0EZbdzY8ok+C1YEBu1JzFu0cHiZdXqLCaneGUtPq3G8oP5v xMWnEY92g58gdfr+GMwlr6SvhngwMd418e7p5QIDAQABAoIBAGgyO2PXQpU24DyY oiMVYoQBQmIDd7nwqvDa6mNfIaGsjLIvqlGZDmuMcAbKFVVBa3ZCQS70ce60aMWW 1TPgqkab28RclMsNDnt+UWGtTy32LSyr/pKgX4F6LYFNfaST6c0fkmiDVPZYD+xK ywAC43ldHM/12JKQd0sUwh/GRXR4/Y9anWR4uLcfWCRU/7JVefdUe1VWeeelCMB4 wCcv9aQeFKnKeRopg5sryMo0thXSJCrO7UTSBmyasgNlMjUFpIIlPZQrWKXBtN48 NT7yGZ4j4RpSS8jhIxZASmyj1eyn2bZEk5SFcFM0wN78lOIOyGBB50QJBSEuKvHT suFBzI0CgYEA53PCyAxHcA4wfuZJgF24VTVUpyZEjjTBRtLrZxUrIagVph5iS9f8 X8henBabaNBB/xsb17qPesCpqGJJEj2XpSKjTQdzmjCUZV8oZSLW87esXwm0Jfb6 dmDqShwlph5l31ZG27I2uTMV8HE8PAOta5rBEFsEwEZDmwtsIvwu6NsCgYEA2G+Q ZbFelFC6+aGKPTPrPVzX45IMPRE23f3euTJdEA+T06BD60LC0jQlxmQghlDUZkxs s9ihPnj2nHuaxJyR4G+/56oba5Jds2ZAiU7FNGZpBHh5J6tUligRJ8cjdRiKJheb H6imcRiD++6P2PnQUPuSnfnxo+yxDotUxtfflD8CgYA3brIn10kEvWFAA0d3Rvru 7Wbo4XFp3ZisrbdTaO57kRYeQwinwYLgLcz69S8jWz1zK7rvX+8dJyd6/I8jnzzC gBk8SZyWQSxJWxEBp+ZzWHogCOLsdMpnr65tMXCS6JTdeHxgAb65fFK002nptbfD Hb0/fY0Vv75RU9sCRgO2IQKBgQC+MALuKZdXsz0qIRZtw5LCQDSBiBPzEY5qMiym BqiadWXMP/eO4wJ5XlOXauLdxGdwYVXgD8tZL0hdp4wPmxng47H5q5QeyeZGv+KU BcDq09cSk567CiXxMNIqZyY0PlUfNf0dK8DeftFz72U+H/87OmXTJhB9oH7RrgPY G1gfVwKBgQDiCfLBoEfwOpif1XAglNdIv6J1M3rwS85txHlY8H91iiBaoKosLzdj ZTeqOn6ofsWj1NaR8OudZApZZT1gHjmDUnpzrQZvHn1gelsVwoZfXs1ZWX+J4giD Rt3aZFRc4uPVgVNLHf4kADWao5bwvbxWfx74J2LQ4QJ4rxFumeg8HA== -----END RSA PRIVATE KEY----- ndg_httpsclient-0.5.1/ndg/httpsclient/test/scripts/0000755€t‰át€;ŒS0000000000013325376043025270 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg/httpsclient/test/scripts/openssl_https_server.sh0000755€t‰át€;ŒS0000000013413152236350032112 0ustar pk23CLRC\Domain Users00000000000000#!/bin/sh openssl s_server -www -cert pki/localhost.crt -key pki/localhost.key -accept 4443 ndg_httpsclient-0.5.1/ndg/httpsclient/test/test_https.py0000644€t‰át€;ŒS0000001061513267567006026365 0ustar pk23CLRC\Domain Users00000000000000"""unit tests module for ndg.httpsclient.https.HTTPSconnection class PyOpenSSL utility to make a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw (STFC)" __date__ = "06/01/12" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import logging logging.basicConfig(level=logging.DEBUG) log = logging.getLogger(__name__) import unittest import socket from OpenSSL import SSL from ndg.httpsclient.test import Constants from ndg.httpsclient.https import HTTPSConnection from ndg.httpsclient.ssl_peer_verification import ServerSSLCertVerification class TestHTTPSConnection(unittest.TestCase): '''Test ndg HTTPS client HTTPSConnection class''' def test01_open(self): conn = HTTPSConnection(Constants.HOSTNAME, port=Constants.PORT) conn.connect() conn.request('GET', '/') resp = conn.getresponse() print('Response = %s' % resp.read()) conn.close() def test02_open_fails(self): conn = HTTPSConnection(Constants.HOSTNAME, port=Constants.PORT2) self.assertRaises(socket.error, conn.connect) def test03_ssl_verification_of_peer_fails(self): ctx = SSL.Context(SSL.TLSv1_2_METHOD) def verify_callback(conn, x509, errnum, errdepth, preverify_ok): log.debug('SSL peer certificate verification failed for %r', x509.get_subject()) return preverify_ok ctx.set_verify(SSL.VERIFY_PEER, verify_callback) ctx.set_verify_depth(9) # Set bad location - unit test dir has no CA certs to verify with ctx.load_verify_locations(None, Constants.UNITTEST_DIR) conn = HTTPSConnection(Constants.HOSTNAME, port=Constants.PORT, ssl_context=ctx) conn.connect() self.assertRaises(SSL.Error, conn.request, 'GET', '/') def test03_ssl_verification_of_peer_succeeds(self): ctx = SSL.Context(SSL.TLSv1_2_METHOD) verify_callback = lambda conn, x509, errnum, errdepth, preverify_ok: \ preverify_ok ctx.set_verify(SSL.VERIFY_PEER, verify_callback) ctx.set_verify_depth(9) # Set correct location for CA certs to verify with ctx.load_verify_locations(None, Constants.CACERT_DIR) conn = HTTPSConnection(Constants.HOSTNAME, port=Constants.PORT, ssl_context=ctx) conn.connect() conn.request('GET', '/') resp = conn.getresponse() print('Response = %s' % resp.read()) def test04_ssl_verification_with_subj_alt_name(self): ctx = SSL.Context(SSL.TLSv1_2_METHOD) verification = ServerSSLCertVerification(hostname='localhost') verify_callback = verification.get_verify_server_cert_func() ctx.set_verify(SSL.VERIFY_PEER, verify_callback) ctx.set_verify_depth(9) # Set correct location for CA certs to verify with ctx.load_verify_locations(None, Constants.CACERT_DIR) conn = HTTPSConnection(Constants.HOSTNAME, port=Constants.PORT, ssl_context=ctx) conn.connect() conn.request('GET', '/') resp = conn.getresponse() print('Response = %s' % resp.read()) def test04_ssl_verification_with_subj_common_name(self): ctx = SSL.Context(SSL.TLSv1_2_METHOD) # Explicitly set verification of peer hostname using peer certificate # subject common name verification = ServerSSLCertVerification(hostname='localhost', subj_alt_name_match=False) verify_callback = verification.get_verify_server_cert_func() ctx.set_verify(SSL.VERIFY_PEER, verify_callback) ctx.set_verify_depth(9) # Set correct location for CA certs to verify with ctx.load_verify_locations(None, Constants.CACERT_DIR) conn = HTTPSConnection(Constants.HOSTNAME, port=Constants.PORT, ssl_context=ctx) conn.connect() conn.request('GET', '/') resp = conn.getresponse() print('Response = %s' % resp.read()) if __name__ == "__main__": unittest.main()ndg_httpsclient-0.5.1/ndg/httpsclient/test/test_urllib2.py0000644€t‰át€;ŒS0000000642113267567006026576 0ustar pk23CLRC\Domain Users00000000000000"""unit tests module for ndg.httpsclient.urllib2_build_opener module PyOpenSSL utility to make a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw (STFC)" __date__ = "06/01/12" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import sys if sys.version_info[0] > 2: from urllib.error import URLError as URLError else: from urllib2 import URLError as URLError import unittest from OpenSSL import SSL from ndg.httpsclient.test import Constants from ndg.httpsclient.urllib2_build_opener import build_opener from ndg.httpsclient.ssl_peer_verification import ServerSSLCertVerification class Urllib2TestCase(unittest.TestCase): """Unit tests for urllib2 functionality""" def test01_urllib2_build_opener(self): opener = build_opener() self.assertTrue(opener) def test02_open(self): opener = build_opener() res = opener.open(Constants.TEST_URI) self.assertTrue(res) print("res = %s" % res.read()) # Skip this test for remote service as it can take a long time to timeout @unittest.skipIf(Constants.HOSTNAME != 'localhost', 'Skip non-local host') def test03_open_fails_unknown_loc(self): opener = build_opener() self.assertRaises(URLError, opener.open, Constants.TEST_URI2) def test04_open_peer_cert_verification_fails(self): # Explicitly set empty CA directory to make verification fail ctx = SSL.Context(SSL.TLSv1_2_METHOD) verify_callback = lambda conn, x509, errnum, errdepth, preverify_ok: \ preverify_ok ctx.set_verify(SSL.VERIFY_PEER, verify_callback) ctx.load_verify_locations(None, './') opener = build_opener(ssl_context=ctx) self.assertRaises(SSL.Error, opener.open, Constants.TEST_URI) def test05_open_with_subj_alt_names_verification(self): ctx = SSL.Context(SSL.TLSv1_2_METHOD) # Set wildcard hostname for subject alternative name matching - # setting a minimum of two name components for hostname split_hostname = Constants.HOSTNAME.split('.', 1) if len(split_hostname) > 1: _hostname = '*.' + split_hostname[-1] else: _hostname = Constants.HOSTNAME server_ssl_verify = ServerSSLCertVerification(hostname=_hostname) verify_callback_ = server_ssl_verify.get_verify_server_cert_func() ctx.set_verify(SSL.VERIFY_PEER, verify_callback_) # Set default verify paths if testing with peer that has corresponding # CA cert in bundle provided with the OS. In this case, load verify # locations is not needed. #ctx.set_default_verify_paths() ctx.set_verify_depth(9) # Set correct location for CA certs to verify with ctx.load_verify_locations(None, Constants.CACERT_DIR) opener = build_opener(ssl_context=ctx) res = opener.open(Constants.TEST_URI) self.assertTrue(res) print("res = %s" % res.read()) if __name__ == "__main__": unittest.main() ndg_httpsclient-0.5.1/ndg/httpsclient/test/test_utils.py0000644€t‰át€;ŒS0000000407013267567006026361 0ustar pk23CLRC\Domain Users00000000000000"""unit tests module for ndg.httpsclient.utils module PyOpenSSL utility to make a httplib-like interface suitable for use with urllib2 """ __author__ = "P J Kershaw (STFC)" __date__ = "06/01/12" __copyright__ = "(C) 2012 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import unittest import os from OpenSSL import SSL from ndg.httpsclient.test import Constants from ndg.httpsclient.utils import (Configuration, fetch_from_url, open_url, _should_use_proxy) class TestUtilsModule(unittest.TestCase): '''Test ndg.httpsclient.utils module''' def test01_configuration(self): config = Configuration(SSL.Context(SSL.TLSv1_2_METHOD), True) self.assertTrue(config.ssl_context) self.assertEqual(config.debug, True) def test02_fetch_from_url(self): config = Configuration(SSL.Context(SSL.TLSv1_2_METHOD), True) res = fetch_from_url(Constants.TEST_URI, config) self.assertTrue(res) def test03_open_url(self): config = Configuration(SSL.Context(SSL.TLSv1_2_METHOD), True) res = open_url(Constants.TEST_URI, config) self.assertEqual(res[0], 200, 'open_url for %r failed' % Constants.TEST_URI) def test04__should_use_proxy(self): if 'no_proxy' in os.environ: no_proxy = os.environ['no_proxy'] del os.environ['no_proxy'] else: no_proxy = None self.assertTrue(_should_use_proxy(Constants.TEST_URI), 'Expecting use proxy = True') os.environ['no_proxy'] = 'localhost,localhost.localdomain' self.assertFalse(_should_use_proxy(Constants.TEST_URI), 'Expecting use proxy = False') if no_proxy is not None: os.environ['no_proxy'] = no_proxy else: del os.environ['no_proxy'] if __name__ == "__main__": unittest.main()ndg_httpsclient-0.5.1/ndg/httpsclient/urllib2_build_opener.py0000644€t‰át€;ŒS0000000520113152236350027266 0ustar pk23CLRC\Domain Users00000000000000"""urllib2 style build opener integrates with HTTPSConnection class from this package. """ __author__ = "P J Kershaw" __date__ = "21/12/10" __copyright__ = "(C) 2011 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import logging import sys # Py 2 <=> 3 compatibility for class type checking if sys.version_info[0] > 2: class_type_ = type from urllib.request import (ProxyHandler, UnknownHandler, HTTPDefaultErrorHandler, FTPHandler, FileHandler, HTTPErrorProcessor, HTTPHandler, OpenerDirector, HTTPRedirectHandler) else: import types class_type_ = types.ClassType from urllib2 import (ProxyHandler, UnknownHandler, HTTPDefaultErrorHandler, FTPHandler, FileHandler, HTTPErrorProcessor, HTTPHandler, OpenerDirector, HTTPRedirectHandler) from ndg.httpsclient.https import HTTPSContextHandler log = logging.getLogger(__name__) # Copied from urllib2 with modifications for ssl def build_opener(*handlers, **kw): """Create an opener object from a list of handlers. The opener will use several default handlers, including support for HTTP and FTP. If any of the handlers passed as arguments are subclasses of the default handlers, the default handlers will not be used. """ def isclass(obj): return isinstance(obj, class_type_) or hasattr(obj, "__bases__") opener = OpenerDirector() default_classes = [ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor] check_classes = list(default_classes) check_classes.append(HTTPSContextHandler) skip = [] for klass in check_classes: for check in handlers: if isclass(check): if issubclass(check, klass): skip.append(klass) elif isinstance(check, klass): skip.append(klass) for klass in default_classes: if klass not in skip: opener.add_handler(klass()) # Pick up SSL context from keyword settings ssl_context = kw.get('ssl_context') # Add the HTTPS handler with ssl_context if HTTPSContextHandler not in skip: opener.add_handler(HTTPSContextHandler(ssl_context)) for h in handlers: if isclass(h): h = h() opener.add_handler(h) return opener ndg_httpsclient-0.5.1/ndg/httpsclient/utils.py0000644€t‰át€;ŒS0000003652313152236350024337 0ustar pk23CLRC\Domain Users00000000000000"""Utilities using NDG HTTPS Client, including a main module that can be used to fetch from a URL. """ __author__ = "R B Wilkinson" __date__ = "09/12/11" __copyright__ = "(C) 2011 Science and Technology Facilities Council" __license__ = "BSD - see LICENSE file in top-level directory" __contact__ = "Philip.Kershaw@stfc.ac.uk" __revision__ = '$Id$' import logging from optparse import OptionParser import os import sys if sys.version_info[0] > 2: import http.cookiejar as cookiejar_ import http.client as http_client_ from urllib.request import Request as Request_ from urllib.request import HTTPHandler as HTTPHandler_ from urllib.request import HTTPCookieProcessor as HTTPCookieProcessor_ from urllib.request import HTTPBasicAuthHandler as HTTPBasicAuthHandler_ from urllib.request import HTTPPasswordMgrWithDefaultRealm as \ HTTPPasswordMgrWithDefaultRealm_ from urllib.request import ProxyHandler as ProxyHandler_ from urllib.error import HTTPError as HTTPError_ import urllib.parse as urlparse_ else: import cookielib as cookiejar_ import httplib as http_client_ from urllib2 import Request as Request_ from urllib2 import HTTPHandler as HTTPHandler_ from urllib2 import HTTPCookieProcessor as HTTPCookieProcessor_ from urllib2 import HTTPBasicAuthHandler as HTTPBasicAuthHandler_ from urllib2 import HTTPPasswordMgrWithDefaultRealm as \ HTTPPasswordMgrWithDefaultRealm_ from urllib2 import ProxyHandler as ProxyHandler_ from urllib2 import HTTPError as HTTPError_ import urlparse as urlparse_ from ndg.httpsclient.urllib2_build_opener import build_opener from ndg.httpsclient.https import HTTPSContextHandler from ndg.httpsclient import ssl_context_util log = logging.getLogger(__name__) class AccumulatingHTTPCookieProcessor(HTTPCookieProcessor_): """Cookie processor that adds new cookies (instead of replacing the existing ones as HTTPCookieProcessor does) """ def http_request(self, request): """Processes cookies for a HTTP request. @param request: request to process @type request: urllib2.Request @return: request @rtype: urllib2.Request """ COOKIE_HEADER_NAME = "Cookie" tmp_request = Request_(request.get_full_url(), request.data, {}, request.origin_req_host, request.unverifiable) self.cookiejar.add_cookie_header(tmp_request) # Combine existing and new cookies. new_cookies = tmp_request.get_header(COOKIE_HEADER_NAME) if new_cookies: if request.has_header(COOKIE_HEADER_NAME): # Merge new cookies with existing ones. old_cookies = request.get_header(COOKIE_HEADER_NAME) merged_cookies = '; '.join([old_cookies, new_cookies]) request.add_unredirected_header(COOKIE_HEADER_NAME, merged_cookies) else: # No existing cookies so just set new ones. request.add_unredirected_header(COOKIE_HEADER_NAME, new_cookies) return request # Process cookies for HTTPS in the same way. https_request = http_request class URLFetchError(Exception): """Error fetching content from URL""" def fetch_from_url(url, config, data=None, handlers=None): """Returns data retrieved from a URL. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @return data retrieved from URL or None """ return_code, return_message, response = open_url(url, config, data=data, handlers=handlers) if return_code and return_code == http_client_.OK: return_data = response.read() response.close() return return_data else: raise URLFetchError(return_message) def fetch_from_url_to_file(url, config, output_file, data=None, handlers=None): """Writes data retrieved from a URL to a file. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @param output_file: output file @type output_file: basestring @return: tuple ( returned HTTP status code or 0 if an error occurred returned message boolean indicating whether access was successful) """ return_code, return_message, response = open_url(url, config, data=data, handlers=handlers) if return_code == http_client_.OK: return_data = response.read() response.close() outfile = open(output_file, "w") outfile.write(return_data) outfile.close() return return_code, return_message, return_code == http_client_.OK def fetch_stream_from_url(url, config, data=None, handlers=None): """Returns data retrieved from a URL. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @param data: HTTP POST data @type data: str @param handlers: list of custom urllib2 handlers to add to the request @type handlers: iterable @return: data retrieved from URL or None @rtype: file derived type """ return_code, return_message, response = open_url(url, config, data=data, handlers=handlers) if return_code and return_code == http_client_.OK: return response else: raise URLFetchError(return_message) def open_url(url, config, data=None, handlers=None): """Attempts to open a connection to a specified URL. @param url: URL to attempt to open @param config: SSL context configuration @type config: Configuration @param data: HTTP POST data @type data: str @param handlers: list of custom urllib2 handlers to add to the request @type handlers: iterable @return: tuple ( returned HTTP status code or 0 if an error occurred returned message or error description response object) """ debuglevel = 1 if config.debug else 0 # Set up handlers for URL opener. if config.cookie: cj = config.cookie else: cj = cookiejar_.CookieJar() # Use a cookie processor that accumulates cookies when redirects occur so # that an application can redirect for authentication and retain both any # cookies for the application and the security system (c.f., # urllib2.HTTPCookieProcessor which replaces cookies). cookie_handler = AccumulatingHTTPCookieProcessor(cj) if not handlers: handlers = [] handlers.append(cookie_handler) if config.debug: http_handler = HTTPHandler_(debuglevel=debuglevel) https_handler = HTTPSContextHandler(config.ssl_context, debuglevel=debuglevel) handlers.extend([http_handler, https_handler]) if config.http_basicauth: # currently only supports http basic auth auth_handler = HTTPBasicAuthHandler_(HTTPPasswordMgrWithDefaultRealm_()) auth_handler.add_password(realm=None, uri=url, user=config.http_basicauth[0], passwd=config.http_basicauth[1]) handlers.append(auth_handler) # Explicitly remove proxy handling if the host is one listed in the value of # the no_proxy environment variable because urllib2 does use proxy settings # set via http_proxy and https_proxy, but does not take the no_proxy value # into account. if not _should_use_proxy(url, config.no_proxy): handlers.append(ProxyHandler_({})) log.debug("Not using proxy") elif config.proxies: handlers.append(ProxyHandler_(config.proxies)) log.debug("Configuring proxies: %s" % config.proxies) opener = build_opener(*handlers, ssl_context=config.ssl_context) headers = config.headers if headers is None: headers = {} request = Request_(url, data, headers) # Open the URL and check the response. return_code = 0 return_message = '' response = None try: response = opener.open(request) return_message = response.msg return_code = response.code if log.isEnabledFor(logging.DEBUG): for index, cookie in enumerate(cj): log.debug("%s : %s", index, cookie) except HTTPError_ as exc: return_code = exc.code return_message = "Error: %s" % exc.msg if log.isEnabledFor(logging.DEBUG): log.debug("%s %s", exc.code, exc.msg) except Exception as exc: return_message = "Error: %s" % exc.__str__() if log.isEnabledFor(logging.DEBUG): import traceback log.debug(traceback.format_exc()) return (return_code, return_message, response) def _should_use_proxy(url, no_proxy=None): """Determines whether a proxy should be used to open a connection to the specified URL, based on the value of the no_proxy environment variable. @param url: URL @type url: basestring or urllib2.Request """ if no_proxy is None: no_proxy_effective = os.environ.get('no_proxy', '') else: no_proxy_effective = no_proxy urlObj = urlparse_.urlparse(_url_as_string(url)) for np in [h.strip() for h in no_proxy_effective.split(',')]: if urlObj.hostname == np: return False return True def _url_as_string(url): """Returns the URL string from a URL value that is either a string or urllib2.Request.. @param url: URL @type url: basestring or urllib2.Request @return: URL string @rtype: basestring """ if isinstance(url, Request_): return url.get_full_url() elif isinstance(url, str): return url else: raise TypeError("Expected type %r or %r" % (str, Request_)) class Configuration(object): """Connection configuration. """ def __init__(self, ssl_context, debug=False, proxies=None, no_proxy=None, cookie=None, http_basicauth=None, headers=None): """ @param ssl_context: SSL context to use with this configuration @type ssl_context: OpenSSL.SSL.Context @param debug: if True, output debugging information @type debug: bool @param proxies: proxies to use for @type proxies: dict with basestring keys and values @param no_proxy: hosts for which a proxy should not be used @type no_proxy: basestring @param cookie: cookies to set for request @type cookie: cookielib.CookieJar (python 3 - http.cookiejar) @param http_basicauth: http authentication, or None @type http_basicauth: tuple of (username,password) @param headers: http headers @type headers: dict """ self.ssl_context = ssl_context self.debug = debug self.proxies = proxies self.no_proxy = no_proxy self.cookie = cookie self.http_basicauth = http_basicauth self.headers = headers def main(): '''Utility to fetch data using HTTP or HTTPS GET from a specified URL. ''' parser = OptionParser(usage="%prog [options] url") parser.add_option("-c", "--certificate", dest="cert_file", metavar="FILE", default=os.path.expanduser("~/credentials.pem"), help="Certificate file - defaults to $HOME/credentials.pem") parser.add_option("-k", "--private-key", dest="key_file", metavar="FILE", default=None, help="Private key file - defaults to the certificate file") parser.add_option("-t", "--ca-certificate-dir", dest="ca_dir", metavar="PATH", default=None, help="Trusted CA certificate file directory") parser.add_option("-d", "--debug", action="store_true", dest="debug", default=False, help="Print debug information.") parser.add_option("-p", "--post-data-file", dest="data_file", metavar="FILE", default=None, help="POST data file") parser.add_option("-f", "--fetch", dest="output_file", metavar="FILE", default=None, help="Output file") parser.add_option("-n", "--no-verify-peer", action="store_true", dest="no_verify_peer", default=False, help="Skip verification of peer certificate.") parser.add_option("-a", "--basicauth", dest="basicauth", metavar="USER:PASSWD", default=None, help="HTTP authentication credentials") parser.add_option("--header", action="append", dest="headers", metavar="HEADER: VALUE", help="Add HTTP header to request") (options, args) = parser.parse_args() if len(args) != 1: parser.error("Incorrect number of arguments") url = args[0] if options.debug: logging.getLogger().setLevel(logging.DEBUG) if options.key_file and os.path.exists(options.key_file): key_file = options.key_file else: key_file = None if options.cert_file and os.path.exists(options.cert_file): cert_file = options.cert_file else: cert_file = None if options.ca_dir and os.path.exists(options.ca_dir): ca_dir = options.ca_dir else: ca_dir = None verify_peer = not options.no_verify_peer if options.data_file and os.path.exists(options.data_file): data_file = open(options.data_file) data = data_file.read() data_file.close() else: data = None if options.basicauth: http_basicauth = options.basicauth.split(':', 1) else: http_basicauth = None headers = {} if options.headers: for h in options.headers: key, val = h.split(':', 1) headers[key.strip()] = val.lstrip() # If a private key file is not specified, the key is assumed to be stored in # the certificate file. ssl_context = ssl_context_util.make_ssl_context(key_file, cert_file, None, ca_dir, verify_peer, url) config = Configuration(ssl_context, options.debug, http_basicauth=http_basicauth, headers=headers) if options.output_file: return_code, return_message = fetch_from_url_to_file( url, config, options.output_file, data)[:2] raise SystemExit(return_code, return_message) else: data = fetch_from_url(url, config) print(data) if __name__=='__main__': logging.basicConfig() main() ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/0000755€t‰át€;ŒS0000000000013325376043024374 5ustar pk23CLRC\Domain Users00000000000000ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/PKG-INFO0000644€t‰át€;ŒS0000001614213325376043025475 0ustar pk23CLRC\Domain Users00000000000000Metadata-Version: 2.1 Name: ndg-httpsclient Version: 0.5.1 Summary: Provides enhanced HTTPS support for httplib and urllib2 using PyOpenSSL Home-page: https://github.com/cedadev/ndg_httpsclient/ Author: Richard Wilkinson and Philip Kershaw Author-email: Philip.Kershaw@stfc.ac.uk License: BSD - See ndg/httpsclient/LICENCE file for details Description: A HTTPS client implementation for * ``httplib`` (Python 2), ``http.client`` (Python 3) and * ``urllib2`` (Python 2) and ``urllib`` (Python 3) ... based on PyOpenSSL. PyOpenSSL provides a more fully featured SSL implementation over the default provided with Python and importantly enables full verification of the SSL peer using ``pyasn1``. Releases ======== 0.5.1 ----- * Clean up handling for description file - pull in content from this file into setup() * Allows the nightly build to fail * Add Trove version classifiers to make it explicit what is supported * Add python_requires to help pip * Drop support for EOL Python 2.6 and 3.3 Thanks to @hugovk for contributions 0.5.0 ----- * Fix to Subject Alternative Name handling to allow for certificates with more than 64 names (max now 1024). Thanks to Matt Pegler * Fix to subjectAltName string to use byte type for correct matching * Updated SSL Context objects to default to TLS 1.2 0.4.4 ----- * Updated test certificates 0.4.3 ----- * Fix to ``ndg`` namespace package warning issue (https://github.com/cedadev/ndg_httpsclient/issues/3). ``__init__.py`` file now included in ``ndg`` directory so that there are no longer warnings with imports when using Python 2.x. Thanks to Max Mauntner for fix. * Minor fix for installation: set minimum release for ``pyasn1`` to avoid conflicts with Ubuntu install - see https://github.com/cedadev/ndg_httpsclient/issues/5 and https://github.com/cedadev/ndg_httpsclient/pull/10. ``pyasn1`` also becomes mandatory rather than optional package for install. - It required by ``cryptography`` anyway which is a dependency for ``pyOpenSSL`` from version 0.14. 0.4.2 ----- * Fix to bug in ``ndg.httpsclient.utils.open_url`` - duplicate open call. Nb. This bug and the fix DO NOT affect the ``httplib``and ``urllib2`` interfaces that this package provides. 0.4.1 ----- * Added explicit ref to Python 3 in classifier text for Python 3 checking tools. * Moved LICENSE file into package 0.4.0 ----- * Made dual compatible with Python 2 / 3. 0.3.3 ----- * Fix to add in AnotherName for ``subjectAltNames`` field - added for support for CACert issued certs (thanks to Gu1). * Fix to HTTP Basic Auth option for ``ndg.httpsclient.utils.main`` * Fix to ``ServerSSLCertVerification`` so that it can pass a function-based callback instead of using ``__call__``. In newer versions of OpenSSL (>= 0.14) the latter failed because of a request for ``__name__`` attribute. 0.3.2 ----- * Fix to SubjectAltNames support check - should only be enabled if pyasn1 is installed. * Fix to open_url: HTTP Request object was being created inside if headers is None block - now corrected to create regardless. * Added http basic auth support to script. (Thanks to Willem van Engen) 0.3.1 ----- * extended utils functions to support keyword for passing additional ``urllib2`` handlers. 0.3.0 ----- * Added ``ndg.httpsclient.utils.fetch_stream_from_url`` function and added parameter for data to post in ``open_url`` and ``fetch_*`` methods. * fix to ndg.httpsclient.utils module _should_use_proxy and open_url functions 0.2.0 ----- * added support for SSL verification with subjectAltNames using pyasn1 * fixed minor bug - SSL cert DN prefix matching 0.1.0 ----- Initial release Prerequisites ============= This has been developed and tested for Python 2.7 with pyOpenSSL 0.13 and 0.14. Version 0.4.0 tested with ``pyOpenSSL`` 0.15.1 and Python 2.7 and 3.4. ``pyasn1`` is required for correct SSL verification with ``subjectAltNames``. Installation ============ Installation can be performed using ``easy_install`` or ``pip``. Running ndg_httpclient ====================== A simple script for fetching data using HTTP or HTTPS GET from a specified URL. Parameter: ``url`` The URL of the resource to be fetched Options: ``-h, --help`` Show help message and exit. ``-c FILE, --certificate=FILE`` Certificate file - defaults to ``$HOME/credentials.pem`` ``-k FILE, --private-key=FILE`` Private key file - defaults to the certificate file ``-t DIR, --ca-certificate-dir=DIR`` Trusted CA certificate file directory. ``-d, --debug`` Print debug information - this may be useful in solving problems with HTTP or HTTPS access to a server. ``-p FILE, --post-data-file=FILE`` POST data file ``-f FILE, --fetch=FILE`` Output file ``-n, --no-verify-peer`` Skip verification of peer certificate. Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Environment :: Web Environment Classifier: Intended Audience :: End Users/Desktop Classifier: Intended Audience :: Developers Classifier: Intended Audience :: System Administrators Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: BSD License Classifier: Natural Language :: English Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: 3.6 Classifier: Topic :: Security Classifier: Topic :: Internet Classifier: Topic :: Scientific/Engineering Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: System :: Systems Administration :: Authentication/Directory Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* Description-Content-Type: text/markdown ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/SOURCES.txt0000644€t‰át€;ŒS0000000170313325376043026261 0ustar pk23CLRC\Domain Users00000000000000MANIFEST.in README.md setup.py documentation/Makefile ndg/__init__.py ndg/httpsclient/LICENSE ndg/httpsclient/__init__.py ndg/httpsclient/https.py ndg/httpsclient/ssl_context_util.py ndg/httpsclient/ssl_peer_verification.py ndg/httpsclient/ssl_socket.py ndg/httpsclient/subj_alt_name.py ndg/httpsclient/urllib2_build_opener.py ndg/httpsclient/utils.py ndg/httpsclient/test/README ndg/httpsclient/test/__init__.py ndg/httpsclient/test/test_https.py ndg/httpsclient/test/test_urllib2.py ndg/httpsclient/test/test_utils.py ndg/httpsclient/test/pki/localhost.crt ndg/httpsclient/test/pki/localhost.key ndg/httpsclient/test/pki/ca/7e15277f.0 ndg/httpsclient/test/scripts/openssl_https_server.sh ndg_httpsclient.egg-info/PKG-INFO ndg_httpsclient.egg-info/SOURCES.txt ndg_httpsclient.egg-info/dependency_links.txt ndg_httpsclient.egg-info/entry_points.txt ndg_httpsclient.egg-info/not-zip-safe ndg_httpsclient.egg-info/requires.txt ndg_httpsclient.egg-info/top_level.txtndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/dependency_links.txt0000644€t‰át€;ŒS0000000000113325376043030442 0ustar pk23CLRC\Domain Users00000000000000 ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/entry_points.txt0000644€t‰át€;ŒS0000000007713325376043027676 0ustar pk23CLRC\Domain Users00000000000000[console_scripts] ndg_httpclient = ndg.httpsclient.utils:main ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/not-zip-safe0000644€t‰át€;ŒS0000000000113325370350026615 0ustar pk23CLRC\Domain Users00000000000000 ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/requires.txt0000644€t‰át€;ŒS0000000003013325376043026765 0ustar pk23CLRC\Domain Users00000000000000PyOpenSSL pyasn1>=0.1.1 ndg_httpsclient-0.5.1/ndg_httpsclient.egg-info/top_level.txt0000644€t‰át€;ŒS0000000000413325376043027120 0ustar pk23CLRC\Domain Users00000000000000ndg ndg_httpsclient-0.5.1/setup.cfg0000644€t‰át€;ŒS0000000004613325376043021332 0ustar pk23CLRC\Domain Users00000000000000[egg_info] tag_build = tag_date = 0 ndg_httpsclient-0.5.1/setup.py0000644€t‰át€;ŒS0000000461713324350730021225 0ustar pk23CLRC\Domain Users00000000000000try: from setuptools import setup, find_packages except ImportError: from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages NAMESPACE_PKGS = ['ndg'] with open('README.md') as f: _long_description = f.read() setup( name='ndg_httpsclient', version="0.5.1", description='Provides enhanced HTTPS support for httplib and urllib2 using ' 'PyOpenSSL', author='Richard Wilkinson and Philip Kershaw', author_email='Philip.Kershaw@stfc.ac.uk', url='https://github.com/cedadev/ndg_httpsclient/', long_description=_long_description, long_description_content_type='text/markdown', license='BSD - See ndg/httpsclient/LICENCE file for details', packages=find_packages(), package_data={ 'ndg.httpsclient': [ 'LICENSE', 'test/README', 'test/scripts/*.sh', 'test/pki/localhost.*', 'test/pki/ca/*.0' ], }, install_requires=['PyOpenSSL', 'pyasn1>=0.1.1'], python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', 'Environment :: Web Environment', 'Intended Audience :: End Users/Desktop', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: BSD License', 'Natural Language :: English', 'Operating System :: Microsoft :: Windows', 'Operating System :: POSIX :: Linux', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Topic :: Security', 'Topic :: Internet', 'Topic :: Scientific/Engineering', 'Topic :: System :: Distributed Computing', 'Topic :: System :: Systems Administration :: Authentication/Directory', 'Topic :: Software Development :: Libraries :: Python Modules' ], zip_safe=False, entry_points={ 'console_scripts': ['ndg_httpclient = ndg.httpsclient.utils:main', ], } )