urllib3-2.0.7/CHANGES.rst0000644000000000000000000016770214513541732011646 0ustar002.0.7 (2023-10-17) ================== * Made body stripped from HTTP requests changing the request method to GET after HTTP 303 "See Other" redirect responses. 2.0.6 (2023-10-02) ================== * Added the ``Cookie`` header to the list of headers to strip from requests when redirecting to a different host. As before, different headers can be set via ``Retry.remove_headers_on_redirect``. 2.0.5 (2023-09-20) ================== - Allowed pyOpenSSL third-party module without any deprecation warning. (`#3126 `__) - Fixed default ``blocksize`` of ``HTTPConnection`` classes to match high-level classes. Previously was 8KiB, now 16KiB. (`#3066 `__) 2.0.4 (2023-07-19) ================== - Added support for union operators to ``HTTPHeaderDict`` (`#2254 `__) - Added ``BaseHTTPResponse`` to ``urllib3.__all__`` (`#3078 `__) - Fixed ``urllib3.connection.HTTPConnection`` to raise the ``http.client.connect`` audit event to have the same behavior as the standard library HTTP client (`#2757 `__) - Relied on the standard library for checking hostnames in supported PyPy releases (`#3087 `__) 2.0.3 (2023-06-07) ================== - Allowed alternative SSL libraries such as LibreSSL, while still issuing a warning as we cannot help users facing issues with implementations other than OpenSSL. (`#3020 `__) - Deprecated URLs which don't have an explicit scheme (`#2950 `_) - Fixed response decoding with Zstandard when compressed data is made of several frames. (`#3008 `__) - Fixed ``assert_hostname=False`` to correctly skip hostname check. (`#3051 `__) 2.0.2 (2023-05-03) ================== - Fixed ``HTTPResponse.stream()`` to continue yielding bytes if buffered decompressed data was still available to be read even if the underlying socket is closed. This prevents a compressed response from being truncated. (`#3009 `__) 2.0.1 (2023-04-30) ================== - Fixed a socket leak when fingerprint or hostname verifications fail. (`#2991 `__) - Fixed an error when ``HTTPResponse.read(0)`` was the first ``read`` call or when the internal response body buffer was otherwise empty. (`#2998 `__) 2.0.0 (2023-04-26) ================== Read the `v2.0 migration guide `__ for help upgrading to the latest version of urllib3. Removed ------- * Removed support for Python 2.7, 3.5, and 3.6 (`#883 `__, `#2336 `__). * Removed fallback on certificate ``commonName`` in ``match_hostname()`` function. This behavior was deprecated in May 2000 in RFC 2818. Instead only ``subjectAltName`` is used to verify the hostname by default. To enable verifying the hostname against ``commonName`` use ``SSLContext.hostname_checks_common_name = True`` (`#2113 `__). * Removed support for Python with an ``ssl`` module compiled with LibreSSL, CiscoSSL, wolfSSL, and all other OpenSSL alternatives. Python is moving to require OpenSSL with PEP 644 (`#2168 `__). * Removed support for OpenSSL versions earlier than 1.1.1 or that don't have SNI support. When an incompatible OpenSSL version is detected an ``ImportError`` is raised (`#2168 `__). * Removed the list of default ciphers for OpenSSL 1.1.1+ and SecureTransport as their own defaults are already secure (`#2082 `__). * Removed ``urllib3.contrib.appengine.AppEngineManager`` and support for Google App Engine Standard Environment (`#2044 `__). * Removed deprecated ``Retry`` options ``method_whitelist``, ``DEFAULT_REDIRECT_HEADERS_BLACKLIST`` (`#2086 `__). * Removed ``urllib3.HTTPResponse.from_httplib`` (`#2648 `__). * Removed default value of ``None`` for the ``request_context`` parameter of ``urllib3.PoolManager.connection_from_pool_key``. This change should have no effect on users as the default value of ``None`` was an invalid option and was never used (`#1897 `__). * Removed the ``urllib3.request`` module. ``urllib3.request.RequestMethods`` has been made a private API. This change was made to ensure that ``from urllib3 import request`` imported the top-level ``request()`` function instead of the ``urllib3.request`` module (`#2269 `__). * Removed support for SSLv3.0 from the ``urllib3.contrib.pyopenssl`` even when support is available from the compiled OpenSSL library (`#2233 `__). * Removed the deprecated ``urllib3.contrib.ntlmpool`` module (`#2339 `__). * Removed ``DEFAULT_CIPHERS``, ``HAS_SNI``, ``USE_DEFAULT_SSLCONTEXT_CIPHERS``, from the private module ``urllib3.util.ssl_`` (`#2168 `__). * Removed ``urllib3.exceptions.SNIMissingWarning`` (`#2168 `__). * Removed the ``_prepare_conn`` method from ``HTTPConnectionPool``. Previously this was only used to call ``HTTPSConnection.set_cert()`` by ``HTTPSConnectionPool`` (`#1985 `__). * Removed ``tls_in_tls_required`` property from ``HTTPSConnection``. This is now determined from the ``scheme`` parameter in ``HTTPConnection.set_tunnel()`` (`#1985 `__). * Removed the ``strict`` parameter/attribute from ``HTTPConnection``, ``HTTPSConnection``, ``HTTPConnectionPool``, ``HTTPSConnectionPool``, and ``HTTPResponse`` (`#2064 `__). Deprecated ---------- * Deprecated ``HTTPResponse.getheaders()`` and ``HTTPResponse.getheader()`` which will be removed in urllib3 v2.1.0. Instead use ``HTTPResponse.headers`` and ``HTTPResponse.headers.get(name, default)``. (`#1543 `__, `#2814 `__). * Deprecated ``urllib3.contrib.pyopenssl`` module which will be removed in urllib3 v2.1.0 (`#2691 `__). * Deprecated ``urllib3.contrib.securetransport`` module which will be removed in urllib3 v2.1.0 (`#2692 `__). * Deprecated ``ssl_version`` option in favor of ``ssl_minimum_version``. ``ssl_version`` will be removed in urllib3 v2.1.0 (`#2110 `__). * Deprecated the ``strict`` parameter of ``PoolManager.connection_from_context()`` as it's not longer needed in Python 3.x. It will be removed in urllib3 v2.1.0 (`#2267 `__) * Deprecated the ``NewConnectionError.pool`` attribute which will be removed in urllib3 v2.1.0 (`#2271 `__). * Deprecated ``format_header_param_html5`` and ``format_header_param`` in favor of ``format_multipart_header_param`` (`#2257 `__). * Deprecated ``RequestField.header_formatter`` parameter which will be removed in urllib3 v2.1.0 (`#2257 `__). * Deprecated ``HTTPSConnection.set_cert()`` method. Instead pass parameters to the ``HTTPSConnection`` constructor (`#1985 `__). * Deprecated ``HTTPConnection.request_chunked()`` method which will be removed in urllib3 v2.1.0. Instead pass ``chunked=True`` to ``HTTPConnection.request()`` (`#1985 `__). Added ----- * Added top-level ``urllib3.request`` function which uses a preconfigured module-global ``PoolManager`` instance (`#2150 `__). * Added the ``json`` parameter to ``urllib3.request()``, ``PoolManager.request()``, and ``ConnectionPool.request()`` methods to send JSON bodies in requests. Using this parameter will set the header ``Content-Type: application/json`` if ``Content-Type`` isn't already defined. Added support for parsing JSON response bodies with ``HTTPResponse.json()`` method (`#2243 `__). * Added type hints to the ``urllib3`` module (`#1897 `__). * Added ``ssl_minimum_version`` and ``ssl_maximum_version`` options which set ``SSLContext.minimum_version`` and ``SSLContext.maximum_version`` (`#2110 `__). * Added support for Zstandard (RFC 8878) when ``zstandard`` 1.18.0 or later is installed. Added the ``zstd`` extra which installs the ``zstandard`` package (`#1992 `__). * Added ``urllib3.response.BaseHTTPResponse`` class. All future response classes will be subclasses of ``BaseHTTPResponse`` (`#2083 `__). * Added ``FullPoolError`` which is raised when ``PoolManager(block=True)`` and a connection is returned to a full pool (`#2197 `__). * Added ``HTTPHeaderDict`` to the top-level ``urllib3`` namespace (`#2216 `__). * Added support for configuring header merging behavior with HTTPHeaderDict When using a ``HTTPHeaderDict`` to provide headers for a request, by default duplicate header values will be repeated. But if ``combine=True`` is passed into a call to ``HTTPHeaderDict.add``, then the added header value will be merged in with an existing value into a comma-separated list (``X-My-Header: foo, bar``) (`#2242 `__). * Added ``NameResolutionError`` exception when a DNS error occurs (`#2305 `__). * Added ``proxy_assert_hostname`` and ``proxy_assert_fingerprint`` kwargs to ``ProxyManager`` (`#2409 `__). * Added a configurable ``backoff_max`` parameter to the ``Retry`` class. If a custom ``backoff_max`` is provided to the ``Retry`` class, it will replace the ``Retry.DEFAULT_BACKOFF_MAX`` (`#2494 `__). * Added the ``authority`` property to the Url class as per RFC 3986 3.2. This property should be used in place of ``netloc`` for users who want to include the userinfo (auth) component of the URI (`#2520 `__). * Added the ``scheme`` parameter to ``HTTPConnection.set_tunnel`` to configure the scheme of the origin being tunnelled to (`#1985 `__). * Added the ``is_closed``, ``is_connected`` and ``has_connected_to_proxy`` properties to ``HTTPConnection`` (`#1985 `__). * Added optional ``backoff_jitter`` parameter to ``Retry``. (`#2952 `__) Changed ------- * Changed ``urllib3.response.HTTPResponse.read`` to respect the semantics of ``io.BufferedIOBase`` regardless of compression. Specifically, this method: * Only returns an empty bytes object to indicate EOF (that is, the response has been fully consumed). * Never returns more bytes than requested. * Can issue any number of system calls: zero, one or multiple. If you want each ``urllib3.response.HTTPResponse.read`` call to issue a single system call, you need to disable decompression by setting ``decode_content=False`` (`#2128 `__). * Changed ``urllib3.HTTPConnection.getresponse`` to return an instance of ``urllib3.HTTPResponse`` instead of ``http.client.HTTPResponse`` (`#2648 `__). * Changed ``ssl_version`` to instead set the corresponding ``SSLContext.minimum_version`` and ``SSLContext.maximum_version`` values. Regardless of ``ssl_version`` passed ``SSLContext`` objects are now constructed using ``ssl.PROTOCOL_TLS_CLIENT`` (`#2110 `__). * Changed default ``SSLContext.minimum_version`` to be ``TLSVersion.TLSv1_2`` in line with Python 3.10 (`#2373 `__). * Changed ``ProxyError`` to wrap any connection error (timeout, TLS, DNS) that occurs when connecting to the proxy (`#2482 `__). * Changed ``urllib3.util.create_urllib3_context`` to not override the system cipher suites with a default value. The new default will be cipher suites configured by the operating system (`#2168 `__). * Changed ``multipart/form-data`` header parameter formatting matches the WHATWG HTML Standard as of 2021-06-10. Control characters in filenames are no longer percent encoded (`#2257 `__). * Changed the error raised when connecting via HTTPS when the ``ssl`` module isn't available from ``SSLError`` to ``ImportError`` (`#2589 `__). * Changed ``HTTPConnection.request()`` to always use lowercase chunk boundaries when sending requests with ``Transfer-Encoding: chunked`` (`#2515 `__). * Changed ``enforce_content_length`` default to True, preventing silent data loss when reading streamed responses (`#2514 `__). * Changed internal implementation of ``HTTPHeaderDict`` to use ``dict`` instead of ``collections.OrderedDict`` for better performance (`#2080 `__). * Changed the ``urllib3.contrib.pyopenssl`` module to wrap ``OpenSSL.SSL.Error`` with ``ssl.SSLError`` in ``PyOpenSSLContext.load_cert_chain`` (`#2628 `__). * Changed usage of the deprecated ``socket.error`` to ``OSError`` (`#2120 `__). * Changed all parameters in the ``HTTPConnection`` and ``HTTPSConnection`` constructors to be keyword-only except ``host`` and ``port`` (`#1985 `__). * Changed ``HTTPConnection.getresponse()`` to set the socket timeout from ``HTTPConnection.timeout`` value before reading data from the socket. This previously was done manually by the ``HTTPConnectionPool`` calling ``HTTPConnection.sock.settimeout(...)`` (`#1985 `__). * Changed the ``_proxy_host`` property to ``_tunnel_host`` in ``HTTPConnectionPool`` to more closely match how the property is used (value in ``HTTPConnection.set_tunnel()``) (`#1985 `__). * Changed name of ``Retry.BACK0FF_MAX`` to be ``Retry.DEFAULT_BACKOFF_MAX``. * Changed TLS handshakes to use ``SSLContext.check_hostname`` when possible (`#2452 `__). * Changed ``server_hostname`` to behave like other parameters only used by ``HTTPSConnectionPool`` (`#2537 `__). * Changed the default ``blocksize`` to 16KB to match OpenSSL's default read amounts (`#2348 `__). * Changed ``HTTPResponse.read()`` to raise an error when calling with ``decode_content=False`` after using ``decode_content=True`` to prevent data loss (`#2800 `__). Fixed ----- * Fixed thread-safety issue where accessing a ``PoolManager`` with many distinct origins would cause connection pools to be closed while requests are in progress (`#1252 `__). * Fixed an issue where an ``HTTPConnection`` instance would erroneously reuse the socket read timeout value from reading the previous response instead of a newly configured connect timeout. Instead now if ``HTTPConnection.timeout`` is updated before sending the next request the new timeout value will be used (`#2645 `__). * Fixed ``socket.error.errno`` when raised from pyOpenSSL's ``OpenSSL.SSL.SysCallError`` (`#2118 `__). * Fixed the default value of ``HTTPSConnection.socket_options`` to match ``HTTPConnection`` (`#2213 `__). * Fixed a bug where ``headers`` would be modified by the ``remove_headers_on_redirect`` feature (`#2272 `__). * Fixed a reference cycle bug in ``urllib3.util.connection.create_connection()`` (`#2277 `__). * Fixed a socket leak if ``HTTPConnection.connect()`` fails (`#2571 `__). * Fixed ``urllib3.contrib.pyopenssl.WrappedSocket`` and ``urllib3.contrib.securetransport.WrappedSocket`` close methods (`#2970 `__) 1.26.18 (2023-10-17) ==================== * Made body stripped from HTTP requests changing the request method to GET after HTTP 303 "See Other" redirect responses. 1.26.17 (2023-10-02) ==================== * Added the ``Cookie`` header to the list of headers to strip from requests when redirecting to a different host. As before, different headers can be set via ``Retry.remove_headers_on_redirect``. (`#3139 `_) 1.26.16 (2023-05-23) ==================== * Fixed thread-safety issue where accessing a ``PoolManager`` with many distinct origins would cause connection pools to be closed while requests are in progress (`#2954 `_) 1.26.15 (2023-03-10) ==================== * Fix socket timeout value when ``HTTPConnection`` is reused (`#2645 `__) * Remove "!" character from the unreserved characters in IPv6 Zone ID parsing (`#2899 `__) * Fix IDNA handling of '\x80' byte (`#2901 `__) 1.26.14 (2023-01-11) ==================== * Fixed parsing of port 0 (zero) returning None, instead of 0. (`#2850 `__) * Removed deprecated getheaders() calls in contrib module. Fixed the type hint of ``PoolKey.key_retries`` by adding ``bool`` to the union. (`#2865 `__) 1.26.13 (2022-11-23) ==================== * Deprecated the ``HTTPResponse.getheaders()`` and ``HTTPResponse.getheader()`` methods. * Fixed an issue where parsing a URL with leading zeroes in the port would be rejected even when the port number after removing the zeroes was valid. * Fixed a deprecation warning when using cryptography v39.0.0. * Removed the ``<4`` in the ``Requires-Python`` packaging metadata field. 1.26.12 (2022-08-22) ==================== * Deprecated the `urllib3[secure]` extra and the `urllib3.contrib.pyopenssl` module. Both will be removed in v2.x. See this `GitHub issue `_ for justification and info on how to migrate. 1.26.11 (2022-07-25) ==================== * Fixed an issue where reading more than 2 GiB in a call to ``HTTPResponse.read`` would raise an ``OverflowError`` on Python 3.9 and earlier. 1.26.10 (2022-07-07) ==================== * Removed support for Python 3.5 * Fixed an issue where a ``ProxyError`` recommending configuring the proxy as HTTP instead of HTTPS could appear even when an HTTPS proxy wasn't configured. 1.26.9 (2022-03-16) =================== * Changed ``urllib3[brotli]`` extra to favor installing Brotli libraries that are still receiving updates like ``brotli`` and ``brotlicffi`` instead of ``brotlipy``. This change does not impact behavior of urllib3, only which dependencies are installed. * Fixed a socket leaking when ``HTTPSConnection.connect()`` raises an exception. * Fixed ``server_hostname`` being forwarded from ``PoolManager`` to ``HTTPConnectionPool`` when requesting an HTTP URL. Should only be forwarded when requesting an HTTPS URL. 1.26.8 (2022-01-07) =================== * Added extra message to ``urllib3.exceptions.ProxyError`` when urllib3 detects that a proxy is configured to use HTTPS but the proxy itself appears to only use HTTP. * Added a mention of the size of the connection pool when discarding a connection due to the pool being full. * Added explicit support for Python 3.11. * Deprecated the ``Retry.MAX_BACKOFF`` class property in favor of ``Retry.DEFAULT_MAX_BACKOFF`` to better match the rest of the default parameter names. ``Retry.MAX_BACKOFF`` is removed in v2.0. * Changed location of the vendored ``ssl.match_hostname`` function from ``urllib3.packages.ssl_match_hostname`` to ``urllib3.util.ssl_match_hostname`` to ensure Python 3.10+ compatibility after being repackaged by downstream distributors. * Fixed absolute imports, all imports are now relative. 1.26.7 (2021-09-22) =================== * Fixed a bug with HTTPS hostname verification involving IP addresses and lack of SNI. (Issue #2400) * Fixed a bug where IPv6 braces weren't stripped during certificate hostname matching. (Issue #2240) 1.26.6 (2021-06-25) =================== * Deprecated the ``urllib3.contrib.ntlmpool`` module. urllib3 is not able to support it properly due to `reasons listed in this issue `_. If you are a user of this module please leave a comment. * Changed ``HTTPConnection.request_chunked()`` to not erroneously emit multiple ``Transfer-Encoding`` headers in the case that one is already specified. * Fixed typo in deprecation message to recommend ``Retry.DEFAULT_ALLOWED_METHODS``. 1.26.5 (2021-05-26) =================== * Fixed deprecation warnings emitted in Python 3.10. * Updated vendored ``six`` library to 1.16.0. * Improved performance of URL parser when splitting the authority component. 1.26.4 (2021-03-15) =================== * Changed behavior of the default ``SSLContext`` when connecting to HTTPS proxy during HTTPS requests. The default ``SSLContext`` now sets ``check_hostname=True``. 1.26.3 (2021-01-26) =================== * Fixed bytes and string comparison issue with headers (Pull #2141) * Changed ``ProxySchemeUnknown`` error message to be more actionable if the user supplies a proxy URL without a scheme. (Pull #2107) 1.26.2 (2020-11-12) =================== * Fixed an issue where ``wrap_socket`` and ``CERT_REQUIRED`` wouldn't be imported properly on Python 2.7.8 and earlier (Pull #2052) 1.26.1 (2020-11-11) =================== * Fixed an issue where two ``User-Agent`` headers would be sent if a ``User-Agent`` header key is passed as ``bytes`` (Pull #2047) 1.26.0 (2020-11-10) =================== * **NOTE: urllib3 v2.0 will drop support for Python 2**. `Read more in the v2.0 Roadmap `_. * Added support for HTTPS proxies contacting HTTPS servers (Pull #1923, Pull #1806) * Deprecated negotiating TLSv1 and TLSv1.1 by default. Users that still wish to use TLS earlier than 1.2 without a deprecation warning should opt-in explicitly by setting ``ssl_version=ssl.PROTOCOL_TLSv1_1`` (Pull #2002) **Starting in urllib3 v2.0: Connections that receive a ``DeprecationWarning`` will fail** * Deprecated ``Retry`` options ``Retry.DEFAULT_METHOD_WHITELIST``, ``Retry.DEFAULT_REDIRECT_HEADERS_BLACKLIST`` and ``Retry(method_whitelist=...)`` in favor of ``Retry.DEFAULT_ALLOWED_METHODS``, ``Retry.DEFAULT_REMOVE_HEADERS_ON_REDIRECT``, and ``Retry(allowed_methods=...)`` (Pull #2000) **Starting in urllib3 v2.0: Deprecated options will be removed** * Added default ``User-Agent`` header to every request (Pull #1750) * Added ``urllib3.util.SKIP_HEADER`` for skipping ``User-Agent``, ``Accept-Encoding``, and ``Host`` headers from being automatically emitted with requests (Pull #2018) * Collapse ``transfer-encoding: chunked`` request data and framing into the same ``socket.send()`` call (Pull #1906) * Send ``http/1.1`` ALPN identifier with every TLS handshake by default (Pull #1894) * Properly terminate SecureTransport connections when CA verification fails (Pull #1977) * Don't emit an ``SNIMissingWarning`` when passing ``server_hostname=None`` to SecureTransport (Pull #1903) * Disabled requesting TLSv1.2 session tickets as they weren't being used by urllib3 (Pull #1970) * Suppress ``BrokenPipeError`` when writing request body after the server has closed the socket (Pull #1524) * Wrap ``ssl.SSLError`` that can be raised from reading a socket (e.g. "bad MAC") into an ``urllib3.exceptions.SSLError`` (Pull #1939) 1.25.11 (2020-10-19) ==================== * Fix retry backoff time parsed from ``Retry-After`` header when given in the HTTP date format. The HTTP date was parsed as the local timezone rather than accounting for the timezone in the HTTP date (typically UTC) (Pull #1932, Pull #1935, Pull #1938, Pull #1949) * Fix issue where an error would be raised when the ``SSLKEYLOGFILE`` environment variable was set to the empty string. Now ``SSLContext.keylog_file`` is not set in this situation (Pull #2016) 1.25.10 (2020-07-22) ==================== * Added support for ``SSLKEYLOGFILE`` environment variable for logging TLS session keys with use with programs like Wireshark for decrypting captured web traffic (Pull #1867) * Fixed loading of SecureTransport libraries on macOS Big Sur due to the new dynamic linker cache (Pull #1905) * Collapse chunked request bodies data and framing into one call to ``send()`` to reduce the number of TCP packets by 2-4x (Pull #1906) * Don't insert ``None`` into ``ConnectionPool`` if the pool was empty when requesting a connection (Pull #1866) * Avoid ``hasattr`` call in ``BrotliDecoder.decompress()`` (Pull #1858) 1.25.9 (2020-04-16) =================== * Added ``InvalidProxyConfigurationWarning`` which is raised when erroneously specifying an HTTPS proxy URL. urllib3 doesn't currently support connecting to HTTPS proxies but will soon be able to and we would like users to migrate properly without much breakage. See `this GitHub issue `_ for more information on how to fix your proxy config. (Pull #1851) * Drain connection after ``PoolManager`` redirect (Pull #1817) * Ensure ``load_verify_locations`` raises ``SSLError`` for all backends (Pull #1812) * Rename ``VerifiedHTTPSConnection`` to ``HTTPSConnection`` (Pull #1805) * Allow the CA certificate data to be passed as a string (Pull #1804) * Raise ``ValueError`` if method contains control characters (Pull #1800) * Add ``__repr__`` to ``Timeout`` (Pull #1795) 1.25.8 (2020-01-20) =================== * Drop support for EOL Python 3.4 (Pull #1774) * Optimize _encode_invalid_chars (Pull #1787) 1.25.7 (2019-11-11) =================== * Preserve ``chunked`` parameter on retries (Pull #1715, Pull #1734) * Allow unset ``SERVER_SOFTWARE`` in App Engine (Pull #1704, Issue #1470) * Fix issue where URL fragment was sent within the request target. (Pull #1732) * Fix issue where an empty query section in a URL would fail to parse. (Pull #1732) * Remove TLS 1.3 support in SecureTransport due to Apple removing support (Pull #1703) 1.25.6 (2019-09-24) =================== * Fix issue where tilde (``~``) characters were incorrectly percent-encoded in the path. (Pull #1692) 1.25.5 (2019-09-19) =================== * Add mitigation for BPO-37428 affecting Python <3.7.4 and OpenSSL 1.1.1+ which caused certificate verification to be enabled when using ``cert_reqs=CERT_NONE``. (Issue #1682) 1.25.4 (2019-09-19) =================== * Propagate Retry-After header settings to subsequent retries. (Pull #1607) * Fix edge case where Retry-After header was still respected even when explicitly opted out of. (Pull #1607) * Remove dependency on ``rfc3986`` for URL parsing. * Fix issue where URLs containing invalid characters within ``Url.auth`` would raise an exception instead of percent-encoding those characters. * Add support for ``HTTPResponse.auto_close = False`` which makes HTTP responses work well with BufferedReaders and other ``io`` module features. (Pull #1652) * Percent-encode invalid characters in URL for ``HTTPConnectionPool.request()`` (Pull #1673) 1.25.3 (2019-05-23) =================== * Change ``HTTPSConnection`` to load system CA certificates when ``ca_certs``, ``ca_cert_dir``, and ``ssl_context`` are unspecified. (Pull #1608, Issue #1603) * Upgrade bundled rfc3986 to v1.3.2. (Pull #1609, Issue #1605) 1.25.2 (2019-04-28) =================== * Change ``is_ipaddress`` to not detect IPvFuture addresses. (Pull #1583) * Change ``parse_url`` to percent-encode invalid characters within the path, query, and target components. (Pull #1586) 1.25.1 (2019-04-24) =================== * Add support for Google's ``Brotli`` package. (Pull #1572, Pull #1579) * Upgrade bundled rfc3986 to v1.3.1 (Pull #1578) 1.25 (2019-04-22) ================= * Require and validate certificates by default when using HTTPS (Pull #1507) * Upgraded ``urllib3.utils.parse_url()`` to be RFC 3986 compliant. (Pull #1487) * Added support for ``key_password`` for ``HTTPSConnectionPool`` to use encrypted ``key_file`` without creating your own ``SSLContext`` object. (Pull #1489) * Add TLSv1.3 support to CPython, pyOpenSSL, and SecureTransport ``SSLContext`` implementations. (Pull #1496) * Switched the default multipart header encoder from RFC 2231 to HTML 5 working draft. (Issue #303, Pull #1492) * Fixed issue where OpenSSL would block if an encrypted client private key was given and no password was given. Instead an ``SSLError`` is raised. (Pull #1489) * Added support for Brotli content encoding. It is enabled automatically if ``brotlipy`` package is installed which can be requested with ``urllib3[brotli]`` extra. (Pull #1532) * Drop ciphers using DSS key exchange from default TLS cipher suites. Improve default ciphers when using SecureTransport. (Pull #1496) * Implemented a more efficient ``HTTPResponse.__iter__()`` method. (Issue #1483) 1.24.3 (2019-05-01) =================== * Apply fix for CVE-2019-9740. (Pull #1591) 1.24.2 (2019-04-17) =================== * Don't load system certificates by default when any other ``ca_certs``, ``ca_certs_dir`` or ``ssl_context`` parameters are specified. * Remove Authorization header regardless of case when redirecting to cross-site. (Issue #1510) * Add support for IPv6 addresses in subjectAltName section of certificates. (Issue #1269) 1.24.1 (2018-11-02) =================== * Remove quadratic behavior within ``GzipDecoder.decompress()`` (Issue #1467) * Restored functionality of ``ciphers`` parameter for ``create_urllib3_context()``. (Issue #1462) 1.24 (2018-10-16) ================= * Allow key_server_hostname to be specified when initializing a PoolManager to allow custom SNI to be overridden. (Pull #1449) * Test against Python 3.7 on AppVeyor. (Pull #1453) * Early-out ipv6 checks when running on App Engine. (Pull #1450) * Change ambiguous description of backoff_factor (Pull #1436) * Add ability to handle multiple Content-Encodings (Issue #1441 and Pull #1442) * Skip DNS names that can't be idna-decoded when using pyOpenSSL (Issue #1405). * Add a server_hostname parameter to HTTPSConnection which allows for overriding the SNI hostname sent in the handshake. (Pull #1397) * Drop support for EOL Python 2.6 (Pull #1429 and Pull #1430) * Fixed bug where responses with header Content-Type: message/* erroneously raised HeaderParsingError, resulting in a warning being logged. (Pull #1439) * Move urllib3 to src/urllib3 (Pull #1409) 1.23 (2018-06-04) ================= * Allow providing a list of headers to strip from requests when redirecting to a different host. Defaults to the ``Authorization`` header. Different headers can be set via ``Retry.remove_headers_on_redirect``. (Issue #1316) * Fix ``util.selectors._fileobj_to_fd`` to accept ``long`` (Issue #1247). * Dropped Python 3.3 support. (Pull #1242) * Put the connection back in the pool when calling stream() or read_chunked() on a chunked HEAD response. (Issue #1234) * Fixed pyOpenSSL-specific ssl client authentication issue when clients attempted to auth via certificate + chain (Issue #1060) * Add the port to the connectionpool connect print (Pull #1251) * Don't use the ``uuid`` module to create multipart data boundaries. (Pull #1380) * ``read_chunked()`` on a closed response returns no chunks. (Issue #1088) * Add Python 2.6 support to ``contrib.securetransport`` (Pull #1359) * Added support for auth info in url for SOCKS proxy (Pull #1363) 1.22 (2017-07-20) ================= * Fixed missing brackets in ``HTTP CONNECT`` when connecting to IPv6 address via IPv6 proxy. (Issue #1222) * Made the connection pool retry on ``SSLError``. The original ``SSLError`` is available on ``MaxRetryError.reason``. (Issue #1112) * Drain and release connection before recursing on retry/redirect. Fixes deadlocks with a blocking connectionpool. (Issue #1167) * Fixed compatibility for cookiejar. (Issue #1229) * pyopenssl: Use vendored version of ``six``. (Issue #1231) 1.21.1 (2017-05-02) =================== * Fixed SecureTransport issue that would cause long delays in response body delivery. (Pull #1154) * Fixed regression in 1.21 that threw exceptions when users passed the ``socket_options`` flag to the ``PoolManager``. (Issue #1165) * Fixed regression in 1.21 that threw exceptions when users passed the ``assert_hostname`` or ``assert_fingerprint`` flag to the ``PoolManager``. (Pull #1157) 1.21 (2017-04-25) ================= * Improved performance of certain selector system calls on Python 3.5 and later. (Pull #1095) * Resolved issue where the PyOpenSSL backend would not wrap SysCallError exceptions appropriately when sending data. (Pull #1125) * Selectors now detects a monkey-patched select module after import for modules that patch the select module like eventlet, greenlet. (Pull #1128) * Reduced memory consumption when streaming zlib-compressed responses (as opposed to raw deflate streams). (Pull #1129) * Connection pools now use the entire request context when constructing the pool key. (Pull #1016) * ``PoolManager.connection_from_*`` methods now accept a new keyword argument, ``pool_kwargs``, which are merged with the existing ``connection_pool_kw``. (Pull #1016) * Add retry counter for ``status_forcelist``. (Issue #1147) * Added ``contrib`` module for using SecureTransport on macOS: ``urllib3.contrib.securetransport``. (Pull #1122) * urllib3 now only normalizes the case of ``http://`` and ``https://`` schemes: for schemes it does not recognise, it assumes they are case-sensitive and leaves them unchanged. (Issue #1080) 1.20 (2017-01-19) ================= * Added support for waiting for I/O using selectors other than select, improving urllib3's behaviour with large numbers of concurrent connections. (Pull #1001) * Updated the date for the system clock check. (Issue #1005) * ConnectionPools now correctly consider hostnames to be case-insensitive. (Issue #1032) * Outdated versions of PyOpenSSL now cause the PyOpenSSL contrib module to fail when it is injected, rather than at first use. (Pull #1063) * Outdated versions of cryptography now cause the PyOpenSSL contrib module to fail when it is injected, rather than at first use. (Issue #1044) * Automatically attempt to rewind a file-like body object when a request is retried or redirected. (Pull #1039) * Fix some bugs that occur when modules incautiously patch the queue module. (Pull #1061) * Prevent retries from occurring on read timeouts for which the request method was not in the method whitelist. (Issue #1059) * Changed the PyOpenSSL contrib module to lazily load idna to avoid unnecessarily bloating the memory of programs that don't need it. (Pull #1076) * Add support for IPv6 literals with zone identifiers. (Pull #1013) * Added support for socks5h:// and socks4a:// schemes when working with SOCKS proxies, and controlled remote DNS appropriately. (Issue #1035) 1.19.1 (2016-11-16) =================== * Fixed AppEngine import that didn't function on Python 3.5. (Pull #1025) 1.19 (2016-11-03) ================= * urllib3 now respects Retry-After headers on 413, 429, and 503 responses when using the default retry logic. (Pull #955) * Remove markers from setup.py to assist ancient setuptools versions. (Issue #986) * Disallow superscripts and other integerish things in URL ports. (Issue #989) * Allow urllib3's HTTPResponse.stream() method to continue to work with non-httplib underlying FPs. (Pull #990) * Empty filenames in multipart headers are now emitted as such, rather than being suppressed. (Issue #1015) * Prefer user-supplied Host headers on chunked uploads. (Issue #1009) 1.18.1 (2016-10-27) =================== * CVE-2016-9015. Users who are using urllib3 version 1.17 or 1.18 along with PyOpenSSL injection and OpenSSL 1.1.0 *must* upgrade to this version. This release fixes a vulnerability whereby urllib3 in the above configuration would silently fail to validate TLS certificates due to erroneously setting invalid flags in OpenSSL's ``SSL_CTX_set_verify`` function. These erroneous flags do not cause a problem in OpenSSL versions before 1.1.0, which interprets the presence of any flag as requesting certificate validation. There is no PR for this patch, as it was prepared for simultaneous disclosure and release. The master branch received the same fix in Pull #1010. 1.18 (2016-09-26) ================= * Fixed incorrect message for IncompleteRead exception. (Pull #973) * Accept ``iPAddress`` subject alternative name fields in TLS certificates. (Issue #258) * Fixed consistency of ``HTTPResponse.closed`` between Python 2 and 3. (Issue #977) * Fixed handling of wildcard certificates when using PyOpenSSL. (Issue #979) 1.17 (2016-09-06) ================= * Accept ``SSLContext`` objects for use in SSL/TLS negotiation. (Issue #835) * ConnectionPool debug log now includes scheme, host, and port. (Issue #897) * Substantially refactored documentation. (Issue #887) * Used URLFetch default timeout on AppEngine, rather than hardcoding our own. (Issue #858) * Normalize the scheme and host in the URL parser (Issue #833) * ``HTTPResponse`` contains the last ``Retry`` object, which now also contains retries history. (Issue #848) * Timeout can no longer be set as boolean, and must be greater than zero. (Pull #924) * Removed pyasn1 and ndg-httpsclient from dependencies used for PyOpenSSL. We now use cryptography and idna, both of which are already dependencies of PyOpenSSL. (Pull #930) * Fixed infinite loop in ``stream`` when amt=None. (Issue #928) * Try to use the operating system's certificates when we are using an ``SSLContext``. (Pull #941) * Updated cipher suite list to allow ChaCha20+Poly1305. AES-GCM is preferred to ChaCha20, but ChaCha20 is then preferred to everything else. (Pull #947) * Updated cipher suite list to remove 3DES-based cipher suites. (Pull #958) * Removed the cipher suite fallback to allow HIGH ciphers. (Pull #958) * Implemented ``length_remaining`` to determine remaining content to be read. (Pull #949) * Implemented ``enforce_content_length`` to enable exceptions when incomplete data chunks are received. (Pull #949) * Dropped connection start, dropped connection reset, redirect, forced retry, and new HTTPS connection log levels to DEBUG, from INFO. (Pull #967) 1.16 (2016-06-11) ================= * Disable IPv6 DNS when IPv6 connections are not possible. (Issue #840) * Provide ``key_fn_by_scheme`` pool keying mechanism that can be overridden. (Issue #830) * Normalize scheme and host to lowercase for pool keys, and include ``source_address``. (Issue #830) * Cleaner exception chain in Python 3 for ``_make_request``. (Issue #861) * Fixed installing ``urllib3[socks]`` extra. (Issue #864) * Fixed signature of ``ConnectionPool.close`` so it can actually safely be called by subclasses. (Issue #873) * Retain ``release_conn`` state across retries. (Issues #651, #866) * Add customizable ``HTTPConnectionPool.ResponseCls``, which defaults to ``HTTPResponse`` but can be replaced with a subclass. (Issue #879) 1.15.1 (2016-04-11) =================== * Fix packaging to include backports module. (Issue #841) 1.15 (2016-04-06) ================= * Added Retry(raise_on_status=False). (Issue #720) * Always use setuptools, no more distutils fallback. (Issue #785) * Dropped support for Python 3.2. (Issue #786) * Chunked transfer encoding when requesting with ``chunked=True``. (Issue #790) * Fixed regression with IPv6 port parsing. (Issue #801) * Append SNIMissingWarning messages to allow users to specify it in the PYTHONWARNINGS environment variable. (Issue #816) * Handle unicode headers in Py2. (Issue #818) * Log certificate when there is a hostname mismatch. (Issue #820) * Preserve order of request/response headers. (Issue #821) 1.14 (2015-12-29) ================= * contrib: SOCKS proxy support! (Issue #762) * Fixed AppEngine handling of transfer-encoding header and bug in Timeout defaults checking. (Issue #763) 1.13.1 (2015-12-18) =================== * Fixed regression in IPv6 + SSL for match_hostname. (Issue #761) 1.13 (2015-12-14) ================= * Fixed ``pip install urllib3[secure]`` on modern pip. (Issue #706) * pyopenssl: Fixed SSL3_WRITE_PENDING error. (Issue #717) * pyopenssl: Support for TLSv1.1 and TLSv1.2. (Issue #696) * Close connections more defensively on exception. (Issue #734) * Adjusted ``read_chunked`` to handle gzipped, chunk-encoded bodies without repeatedly flushing the decoder, to function better on Jython. (Issue #743) * Accept ``ca_cert_dir`` for SSL-related PoolManager configuration. (Issue #758) 1.12 (2015-09-03) ================= * Rely on ``six`` for importing ``httplib`` to work around conflicts with other Python 3 shims. (Issue #688) * Add support for directories of certificate authorities, as supported by OpenSSL. (Issue #701) * New exception: ``NewConnectionError``, raised when we fail to establish a new connection, usually ``ECONNREFUSED`` socket error. 1.11 (2015-07-21) ================= * When ``ca_certs`` is given, ``cert_reqs`` defaults to ``'CERT_REQUIRED'``. (Issue #650) * ``pip install urllib3[secure]`` will install Certifi and PyOpenSSL as dependencies. (Issue #678) * Made ``HTTPHeaderDict`` usable as a ``headers`` input value (Issues #632, #679) * Added `urllib3.contrib.appengine `_ which has an ``AppEngineManager`` for using ``URLFetch`` in a Google AppEngine environment. (Issue #664) * Dev: Added test suite for AppEngine. (Issue #631) * Fix performance regression when using PyOpenSSL. (Issue #626) * Passing incorrect scheme (e.g. ``foo://``) will raise ``ValueError`` instead of ``AssertionError`` (backwards compatible for now, but please migrate). (Issue #640) * Fix pools not getting replenished when an error occurs during a request using ``release_conn=False``. (Issue #644) * Fix pool-default headers not applying for url-encoded requests like GET. (Issue #657) * log.warning in Python 3 when headers are skipped due to parsing errors. (Issue #642) * Close and discard connections if an error occurs during read. (Issue #660) * Fix host parsing for IPv6 proxies. (Issue #668) * Separate warning type SubjectAltNameWarning, now issued once per host. (Issue #671) * Fix ``httplib.IncompleteRead`` not getting converted to ``ProtocolError`` when using ``HTTPResponse.stream()`` (Issue #674) 1.10.4 (2015-05-03) =================== * Migrate tests to Tornado 4. (Issue #594) * Append default warning configuration rather than overwrite. (Issue #603) * Fix streaming decoding regression. (Issue #595) * Fix chunked requests losing state across keep-alive connections. (Issue #599) * Fix hanging when chunked HEAD response has no body. (Issue #605) 1.10.3 (2015-04-21) =================== * Emit ``InsecurePlatformWarning`` when SSLContext object is missing. (Issue #558) * Fix regression of duplicate header keys being discarded. (Issue #563) * ``Response.stream()`` returns a generator for chunked responses. (Issue #560) * Set upper-bound timeout when waiting for a socket in PyOpenSSL. (Issue #585) * Work on platforms without `ssl` module for plain HTTP requests. (Issue #587) * Stop relying on the stdlib's default cipher list. (Issue #588) 1.10.2 (2015-02-25) =================== * Fix file descriptor leakage on retries. (Issue #548) * Removed RC4 from default cipher list. (Issue #551) * Header performance improvements. (Issue #544) * Fix PoolManager not obeying redirect retry settings. (Issue #553) 1.10.1 (2015-02-10) =================== * Pools can be used as context managers. (Issue #545) * Don't re-use connections which experienced an SSLError. (Issue #529) * Don't fail when gzip decoding an empty stream. (Issue #535) * Add sha256 support for fingerprint verification. (Issue #540) * Fixed handling of header values containing commas. (Issue #533) 1.10 (2014-12-14) ================= * Disabled SSLv3. (Issue #473) * Add ``Url.url`` property to return the composed url string. (Issue #394) * Fixed PyOpenSSL + gevent ``WantWriteError``. (Issue #412) * ``MaxRetryError.reason`` will always be an exception, not string. (Issue #481) * Fixed SSL-related timeouts not being detected as timeouts. (Issue #492) * Py3: Use ``ssl.create_default_context()`` when available. (Issue #473) * Emit ``InsecureRequestWarning`` for *every* insecure HTTPS request. (Issue #496) * Emit ``SecurityWarning`` when certificate has no ``subjectAltName``. (Issue #499) * Close and discard sockets which experienced SSL-related errors. (Issue #501) * Handle ``body`` param in ``.request(...)``. (Issue #513) * Respect timeout with HTTPS proxy. (Issue #505) * PyOpenSSL: Handle ZeroReturnError exception. (Issue #520) 1.9.1 (2014-09-13) ================== * Apply socket arguments before binding. (Issue #427) * More careful checks if fp-like object is closed. (Issue #435) * Fixed packaging issues of some development-related files not getting included. (Issue #440) * Allow performing *only* fingerprint verification. (Issue #444) * Emit ``SecurityWarning`` if system clock is waaay off. (Issue #445) * Fixed PyOpenSSL compatibility with PyPy. (Issue #450) * Fixed ``BrokenPipeError`` and ``ConnectionError`` handling in Py3. (Issue #443) 1.9 (2014-07-04) ================ * Shuffled around development-related files. If you're maintaining a distro package of urllib3, you may need to tweak things. (Issue #415) * Unverified HTTPS requests will trigger a warning on the first request. See our new `security documentation `_ for details. (Issue #426) * New retry logic and ``urllib3.util.retry.Retry`` configuration object. (Issue #326) * All raised exceptions should now wrapped in a ``urllib3.exceptions.HTTPException``-extending exception. (Issue #326) * All errors during a retry-enabled request should be wrapped in ``urllib3.exceptions.MaxRetryError``, including timeout-related exceptions which were previously exempt. Underlying error is accessible from the ``.reason`` property. (Issue #326) * ``urllib3.exceptions.ConnectionError`` renamed to ``urllib3.exceptions.ProtocolError``. (Issue #326) * Errors during response read (such as IncompleteRead) are now wrapped in ``urllib3.exceptions.ProtocolError``. (Issue #418) * Requesting an empty host will raise ``urllib3.exceptions.LocationValueError``. (Issue #417) * Catch read timeouts over SSL connections as ``urllib3.exceptions.ReadTimeoutError``. (Issue #419) * Apply socket arguments before connecting. (Issue #427) 1.8.3 (2014-06-23) ================== * Fix TLS verification when using a proxy in Python 3.4.1. (Issue #385) * Add ``disable_cache`` option to ``urllib3.util.make_headers``. (Issue #393) * Wrap ``socket.timeout`` exception with ``urllib3.exceptions.ReadTimeoutError``. (Issue #399) * Fixed proxy-related bug where connections were being reused incorrectly. (Issues #366, #369) * Added ``socket_options`` keyword parameter which allows to define ``setsockopt`` configuration of new sockets. (Issue #397) * Removed ``HTTPConnection.tcp_nodelay`` in favor of ``HTTPConnection.default_socket_options``. (Issue #397) * Fixed ``TypeError`` bug in Python 2.6.4. (Issue #411) 1.8.2 (2014-04-17) ================== * Fix ``urllib3.util`` not being included in the package. 1.8.1 (2014-04-17) ================== * Fix AppEngine bug of HTTPS requests going out as HTTP. (Issue #356) * Don't install ``dummyserver`` into ``site-packages`` as it's only needed for the test suite. (Issue #362) * Added support for specifying ``source_address``. (Issue #352) 1.8 (2014-03-04) ================ * Improved url parsing in ``urllib3.util.parse_url`` (properly parse '@' in username, and blank ports like 'hostname:'). * New ``urllib3.connection`` module which contains all the HTTPConnection objects. * Several ``urllib3.util.Timeout``-related fixes. Also changed constructor signature to a more sensible order. [Backwards incompatible] (Issues #252, #262, #263) * Use ``backports.ssl_match_hostname`` if it's installed. (Issue #274) * Added ``.tell()`` method to ``urllib3.response.HTTPResponse`` which returns the number of bytes read so far. (Issue #277) * Support for platforms without threading. (Issue #289) * Expand default-port comparison in ``HTTPConnectionPool.is_same_host`` to allow a pool with no specified port to be considered equal to to an HTTP/HTTPS url with port 80/443 explicitly provided. (Issue #305) * Improved default SSL/TLS settings to avoid vulnerabilities. (Issue #309) * Fixed ``urllib3.poolmanager.ProxyManager`` not retrying on connect errors. (Issue #310) * Disable Nagle's Algorithm on the socket for non-proxies. A subset of requests will send the entire HTTP request ~200 milliseconds faster; however, some of the resulting TCP packets will be smaller. (Issue #254) * Increased maximum number of SubjectAltNames in ``urllib3.contrib.pyopenssl`` from the default 64 to 1024 in a single certificate. (Issue #318) * Headers are now passed and stored as a custom ``urllib3.collections_.HTTPHeaderDict`` object rather than a plain ``dict``. (Issue #329, #333) * Headers no longer lose their case on Python 3. (Issue #236) * ``urllib3.contrib.pyopenssl`` now uses the operating system's default CA certificates on inject. (Issue #332) * Requests with ``retries=False`` will immediately raise any exceptions without wrapping them in ``MaxRetryError``. (Issue #348) * Fixed open socket leak with SSL-related failures. (Issue #344, #348) 1.7.1 (2013-09-25) ================== * Added granular timeout support with new ``urllib3.util.Timeout`` class. (Issue #231) * Fixed Python 3.4 support. (Issue #238) 1.7 (2013-08-14) ================ * More exceptions are now pickle-able, with tests. (Issue #174) * Fixed redirecting with relative URLs in Location header. (Issue #178) * Support for relative urls in ``Location: ...`` header. (Issue #179) * ``urllib3.response.HTTPResponse`` now inherits from ``io.IOBase`` for bonus file-like functionality. (Issue #187) * Passing ``assert_hostname=False`` when creating a HTTPSConnectionPool will skip hostname verification for SSL connections. (Issue #194) * New method ``urllib3.response.HTTPResponse.stream(...)`` which acts as a generator wrapped around ``.read(...)``. (Issue #198) * IPv6 url parsing enforces brackets around the hostname. (Issue #199) * Fixed thread race condition in ``urllib3.poolmanager.PoolManager.connection_from_host(...)`` (Issue #204) * ``ProxyManager`` requests now include non-default port in ``Host: ...`` header. (Issue #217) * Added HTTPS proxy support in ``ProxyManager``. (Issue #170 #139) * New ``RequestField`` object can be passed to the ``fields=...`` param which can specify headers. (Issue #220) * Raise ``urllib3.exceptions.ProxyError`` when connecting to proxy fails. (Issue #221) * Use international headers when posting file names. (Issue #119) * Improved IPv6 support. (Issue #203) 1.6 (2013-04-25) ================ * Contrib: Optional SNI support for Py2 using PyOpenSSL. (Issue #156) * ``ProxyManager`` automatically adds ``Host: ...`` header if not given. * Improved SSL-related code. ``cert_req`` now optionally takes a string like "REQUIRED" or "NONE". Same with ``ssl_version`` takes strings like "SSLv23" The string values reflect the suffix of the respective constant variable. (Issue #130) * Vendored ``socksipy`` now based on Anorov's fork which handles unexpectedly closed proxy connections and larger read buffers. (Issue #135) * Ensure the connection is closed if no data is received, fixes connection leak on some platforms. (Issue #133) * Added SNI support for SSL/TLS connections on Py32+. (Issue #89) * Tests fixed to be compatible with Py26 again. (Issue #125) * Added ability to choose SSL version by passing an ``ssl.PROTOCOL_*`` constant to the ``ssl_version`` parameter of ``HTTPSConnectionPool``. (Issue #109) * Allow an explicit content type to be specified when encoding file fields. (Issue #126) * Exceptions are now pickleable, with tests. (Issue #101) * Fixed default headers not getting passed in some cases. (Issue #99) * Treat "content-encoding" header value as case-insensitive, per RFC 2616 Section 3.5. (Issue #110) * "Connection Refused" SocketErrors will get retried rather than raised. (Issue #92) * Updated vendored ``six``, no longer overrides the global ``six`` module namespace. (Issue #113) * ``urllib3.exceptions.MaxRetryError`` contains a ``reason`` property holding the exception that prompted the final retry. If ``reason is None`` then it was due to a redirect. (Issue #92, #114) * Fixed ``PoolManager.urlopen()`` from not redirecting more than once. (Issue #149) * Don't assume ``Content-Type: text/plain`` for multi-part encoding parameters that are not files. (Issue #111) * Pass `strict` param down to ``httplib.HTTPConnection``. (Issue #122) * Added mechanism to verify SSL certificates by fingerprint (md5, sha1) or against an arbitrary hostname (when connecting by IP or for misconfigured servers). (Issue #140) * Streaming decompression support. (Issue #159) 1.5 (2012-08-02) ================ * Added ``urllib3.add_stderr_logger()`` for quickly enabling STDERR debug logging in urllib3. * Native full URL parsing (including auth, path, query, fragment) available in ``urllib3.util.parse_url(url)``. * Built-in redirect will switch method to 'GET' if status code is 303. (Issue #11) * ``urllib3.PoolManager`` strips the scheme and host before sending the request uri. (Issue #8) * New ``urllib3.exceptions.DecodeError`` exception for when automatic decoding, based on the Content-Type header, fails. * Fixed bug with pool depletion and leaking connections (Issue #76). Added explicit connection closing on pool eviction. Added ``urllib3.PoolManager.clear()``. * 99% -> 100% unit test coverage. 1.4 (2012-06-16) ================ * Minor AppEngine-related fixes. * Switched from ``mimetools.choose_boundary`` to ``uuid.uuid4()``. * Improved url parsing. (Issue #73) * IPv6 url support. (Issue #72) 1.3 (2012-03-25) ================ * Removed pre-1.0 deprecated API. * Refactored helpers into a ``urllib3.util`` submodule. * Fixed multipart encoding to support list-of-tuples for keys with multiple values. (Issue #48) * Fixed multiple Set-Cookie headers in response not getting merged properly in Python 3. (Issue #53) * AppEngine support with Py27. (Issue #61) * Minor ``encode_multipart_formdata`` fixes related to Python 3 strings vs bytes. 1.2.2 (2012-02-06) ================== * Fixed packaging bug of not shipping ``test-requirements.txt``. (Issue #47) 1.2.1 (2012-02-05) ================== * Fixed another bug related to when ``ssl`` module is not available. (Issue #41) * Location parsing errors now raise ``urllib3.exceptions.LocationParseError`` which inherits from ``ValueError``. 1.2 (2012-01-29) ================ * Added Python 3 support (tested on 3.2.2) * Dropped Python 2.5 support (tested on 2.6.7, 2.7.2) * Use ``select.poll`` instead of ``select.select`` for platforms that support it. * Use ``Queue.LifoQueue`` instead of ``Queue.Queue`` for more aggressive connection reusing. Configurable by overriding ``ConnectionPool.QueueCls``. * Fixed ``ImportError`` during install when ``ssl`` module is not available. (Issue #41) * Fixed ``PoolManager`` redirects between schemes (such as HTTP -> HTTPS) not completing properly. (Issue #28, uncovered by Issue #10 in v1.1) * Ported ``dummyserver`` to use ``tornado`` instead of ``webob`` + ``eventlet``. Removed extraneous unsupported dummyserver testing backends. Added socket-level tests. * More tests. Achievement Unlocked: 99% Coverage. 1.1 (2012-01-07) ================ * Refactored ``dummyserver`` to its own root namespace module (used for testing). * Added hostname verification for ``VerifiedHTTPSConnection`` by vendoring in Py32's ``ssl_match_hostname``. (Issue #25) * Fixed cross-host HTTP redirects when using ``PoolManager``. (Issue #10) * Fixed ``decode_content`` being ignored when set through ``urlopen``. (Issue #27) * Fixed timeout-related bugs. (Issues #17, #23) 1.0.2 (2011-11-04) ================== * Fixed typo in ``VerifiedHTTPSConnection`` which would only present as a bug if you're using the object manually. (Thanks pyos) * Made RecentlyUsedContainer (and consequently PoolManager) more thread-safe by wrapping the access log in a mutex. (Thanks @christer) * Made RecentlyUsedContainer more dict-like (corrected ``__delitem__`` and ``__getitem__`` behaviour), with tests. Shouldn't affect core urllib3 code. 1.0.1 (2011-10-10) ================== * Fixed a bug where the same connection would get returned into the pool twice, causing extraneous "HttpConnectionPool is full" log warnings. 1.0 (2011-10-08) ================ * Added ``PoolManager`` with LRU expiration of connections (tested and documented). * Added ``ProxyManager`` (needs tests, docs, and confirmation that it works with HTTPS proxies). * Added optional partial-read support for responses when ``preload_content=False``. You can now make requests and just read the headers without loading the content. * Made response decoding optional (default on, same as before). * Added optional explicit boundary string for ``encode_multipart_formdata``. * Convenience request methods are now inherited from ``RequestMethods``. Old helpers like ``get_url`` and ``post_url`` should be abandoned in favour of the new ``request(method, url, ...)``. * Refactored code to be even more decoupled, reusable, and extendable. * License header added to ``.py`` files. * Embiggened the documentation: Lots of Sphinx-friendly docstrings in the code and docs in ``docs/`` and on https://urllib3.readthedocs.io/. * Embettered all the things! * Started writing this file. 0.4.1 (2011-07-17) ================== * Minor bug fixes, code cleanup. 0.4 (2011-03-01) ================ * Better unicode support. * Added ``VerifiedHTTPSConnection``. * Added ``NTLMConnectionPool`` in contrib. * Minor improvements. 0.3.1 (2010-07-13) ================== * Added ``assert_host_name`` optional parameter. Now compatible with proxies. 0.3 (2009-12-10) ================ * Added HTTPS support. * Minor bug fixes. * Refactored, broken backwards compatibility with 0.2. * API to be treated as stable from this version forward. 0.2 (2008-11-17) ================ * Added unit tests. * Bug fixes. 0.1 (2008-11-16) ================ * First release. urllib3-2.0.7/dev-requirements.txt0000644000000000000000000000106514513541732014071 0ustar00coverage==7.0.4 tornado==6.2 PySocks==1.7.1 pytest==7.4.0 pytest-timeout==2.1.0 trustme==0.9.0 # We have to install at most cryptography 39.0.2 for PyPy<7.3.10 # versions of Python 3.7, 3.8, and 3.9. cryptography==39.0.2;implementation_name=="pypy" and implementation_version<"7.3.10" cryptography==41.0.4;implementation_name!="pypy" or implementation_version>="7.3.10" backports.zoneinfo==0.2.1;python_version<"3.9" towncrier==23.6.0 pytest-memray==1.4.0;python_version>="3.8" and python_version<"3.13" and sys_platform!="win32" and implementation_name=="cpython" urllib3-2.0.7/docs/Makefile0000644000000000000000000001077214513541732012426 0ustar00# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = '-W' SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " text to make text files" @echo " man to make manual pages" @echo " changes to make an overview of all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: -rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/urllib3.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/urllib3.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/urllib3" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/urllib3" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." urllib3-2.0.7/docs/advanced-usage.rst0000644000000000000000000005165614513541732014375 0ustar00Advanced Usage ============== .. currentmodule:: urllib3 Customizing Pool Behavior ------------------------- The :class:`~poolmanager.PoolManager` class automatically handles creating :class:`~connectionpool.ConnectionPool` instances for each host as needed. By default, it will keep a maximum of 10 :class:`~connectionpool.ConnectionPool` instances. If you're making requests to many different hosts it might improve performance to increase this number. .. code-block:: python import urllib3 http = urllib3.PoolManager(num_pools=50) However, keep in mind that this does increase memory and socket consumption. Similarly, the :class:`~connectionpool.ConnectionPool` class keeps a pool of individual :class:`~connection.HTTPConnection` instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use. If you are making many requests to the same host simultaneously it might improve performance to increase this number. .. code-block:: python import urllib3 http = urllib3.PoolManager(maxsize=10) # Alternatively pool = urllib3.HTTPConnectionPool("google.com", maxsize=10) The behavior of the pooling for :class:`~connectionpool.ConnectionPool` is different from :class:`~poolmanager.PoolManager`. By default, if a new request is made and there is no free connection in the pool then a new connection will be created. However, this connection will not be saved if more than ``maxsize`` connections exist. This means that ``maxsize`` does not determine the maximum number of connections that can be open to a particular host, just the maximum number of connections to keep in the pool. However, if you specify ``block=True`` then there can be at most ``maxsize`` connections open to a particular host. .. code-block:: python http = urllib3.PoolManager(maxsize=10, block=True) # Alternatively pool = urllib3.HTTPConnectionPool("google.com", maxsize=10, block=True) Any new requests will block until a connection is available from the pool. This is a great way to prevent flooding a host with too many connections in multi-threaded applications. .. _stream: .. _streaming_and_io: Streaming and I/O ----------------- When using ``preload_content=True`` (the default setting) the response body will be read immediately into memory and the HTTP connection will be released back into the pool without manual intervention. However, when dealing with large responses it's often better to stream the response content using ``preload_content=False``. Setting ``preload_content`` to ``False`` means that urllib3 will only read from the socket when data is requested. .. note:: When using ``preload_content=False``, you need to manually release the HTTP connection back to the connection pool so that it can be re-used. To ensure the HTTP connection is in a valid state before being re-used all data should be read off the wire. You can call the :meth:`~response.HTTPResponse.drain_conn` to throw away unread data still on the wire. This call isn't necessary if data has already been completely read from the response. After all data is read you can call :meth:`~response.HTTPResponse.release_conn` to release the connection into the pool. You can call the :meth:`~response.HTTPResponse.close` to close the connection, but this call doesn’t return the connection to the pool, throws away the unread data on the wire, and leaves the connection in an undefined protocol state. This is desirable if you prefer not reading data from the socket to re-using the HTTP connection. :meth:`~response.HTTPResponse.stream` lets you iterate over chunks of the response content. .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/bytes/1024", preload_content=False ) for chunk in resp.stream(32): print(chunk) # b"\x9e\xa97'\x8e\x1eT .... resp.release_conn() However, you can also treat the :class:`~response.HTTPResponse` instance as a file-like object. This allows you to do buffering: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/bytes/1024", preload_content=False ) print(resp.read(4)) # b"\x88\x1f\x8b\xe5" Calls to :meth:`~response.HTTPResponse.read()` will block until more response data is available. .. code-block:: python import io import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/bytes/1024", preload_content=False ) reader = io.BufferedReader(resp, 8) print(reader.read(4)) # b"\xbf\x9c\xd6" resp.release_conn() You can use this file-like object to do things like decode the content using :mod:`codecs`: .. code-block:: python import codecs import json import urllib3 reader = codecs.getreader("utf-8") resp = urllib3.request( "GET", "https://httpbin.org/ip", preload_content=False ) print(json.load(reader(resp))) # {"origin": "127.0.0.1"} resp.release_conn() .. _proxies: Proxies ------- You can use :class:`~poolmanager.ProxyManager` to tunnel requests through an HTTP proxy: .. code-block:: python import urllib3 proxy = urllib3.ProxyManager("https://localhost:3128/") proxy.request("GET", "https://google.com/") The usage of :class:`~poolmanager.ProxyManager` is the same as :class:`~poolmanager.PoolManager`. You can connect to a proxy using HTTP, HTTPS or SOCKS. urllib3's behavior will be different depending on the type of proxy you selected and the destination you're contacting. HTTP and HTTPS Proxies ~~~~~~~~~~~~~~~~~~~~~~ Both HTTP/HTTPS proxies support HTTP and HTTPS destinations. The only difference between them is if you need to establish a TLS connection to the proxy first. You can specify which proxy you need to contact by specifying the proper proxy scheme. (i.e ``http://`` or ``https://``) urllib3's behavior will be different depending on your proxy and destination: * HTTP proxy + HTTP destination Your request will be forwarded with the `absolute URI `_. * HTTP proxy + HTTPS destination A TCP tunnel will be established with a `HTTP CONNECT `_. Afterward a TLS connection will be established with the destination and your request will be sent. * HTTPS proxy + HTTP destination A TLS connection will be established to the proxy and later your request will be forwarded with the `absolute URI `_. * HTTPS proxy + HTTPS destination A TLS-in-TLS tunnel will be established. An initial TLS connection will be established to the proxy, then an `HTTP CONNECT `_ will be sent to establish a TCP connection to the destination and finally a second TLS connection will be established to the destination. You can customize the :class:`ssl.SSLContext` used for the proxy TLS connection through the ``proxy_ssl_context`` argument of the :class:`~poolmanager.ProxyManager` class. For HTTPS proxies we also support forwarding your requests to HTTPS destinations with an `absolute URI `_ if the ``use_forwarding_for_https`` argument is set to ``True``. We strongly recommend you **only use this option with trusted or corporate proxies** as the proxy will have full visibility of your requests. .. _https_proxy_error_http_proxy: Your proxy appears to only use HTTP and not HTTPS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you're receiving the :class:`~urllib3.exceptions.ProxyError` and it mentions your proxy only speaks HTTP and not HTTPS here's what to do to solve your issue: If you're using ``urllib3`` directly, make sure the URL you're passing into :class:`urllib3.ProxyManager` starts with ``http://`` instead of ``https://``: .. code-block:: python # Do this: http = urllib3.ProxyManager("http://...") # Not this: http = urllib3.ProxyManager("https://...") If instead you're using ``urllib3`` through another library like Requests there are multiple ways your proxy could be mis-configured. You need to figure out where the configuration isn't correct and make the fix there. Some common places to look are environment variables like ``HTTP_PROXY``, ``HTTPS_PROXY``, and ``ALL_PROXY``. Ensure that the values for all of these environment variables starts with ``http://`` and not ``https://``: .. code-block:: bash # Check your existing environment variables in bash $ env | grep "_PROXY" HTTP_PROXY=http://127.0.0.1:8888 HTTPS_PROXY=https://127.0.0.1:8888 # <--- This setting is the problem! # Make the fix in your current session and test your script $ export HTTPS_PROXY="http://127.0.0.1:8888" $ python test-proxy.py # This should now pass. # Persist your change in your shell 'profile' (~/.bashrc, ~/.profile, ~/.bash_profile, etc) # You may need to logout and log back in to ensure this works across all programs. $ vim ~/.bashrc If you're on Windows or macOS your proxy may be getting set at a system level. To check this first ensure that the above environment variables aren't set then run the following: .. code-block:: bash $ python -c 'import urllib.request; print(urllib.request.getproxies())' If the output of the above command isn't empty and looks like this: .. code-block:: python { "http": "http://127.0.0.1:8888", "https": "https://127.0.0.1:8888" # <--- This setting is the problem! } Search how to configure proxies on your operating system and change the ``https://...`` URL into ``http://``. After you make the change the return value of ``urllib.request.getproxies()`` should be: .. code-block:: python { # Everything is good here! :) "http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888" } If you still can't figure out how to configure your proxy after all these steps please `join our community Discord `_ and we'll try to help you with your issue. SOCKS Proxies ~~~~~~~~~~~~~ For SOCKS, you can use :class:`~contrib.socks.SOCKSProxyManager` to connect to SOCKS4 or SOCKS5 proxies. In order to use SOCKS proxies you will need to install `PySocks `_ or install urllib3 with the ``socks`` extra: .. code-block:: bash python -m pip install urllib3[socks] Once PySocks is installed, you can use :class:`~contrib.socks.SOCKSProxyManager`: .. code-block:: python from urllib3.contrib.socks import SOCKSProxyManager proxy = SOCKSProxyManager("socks5h://localhost:8889/") proxy.request("GET", "https://google.com/") .. note:: It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in your ``proxy_url`` to ensure that DNS resolution is done from the remote server instead of client-side when connecting to a domain name. .. _ssl_custom: .. _custom_ssl_certificates: Custom TLS Certificates ----------------------- Instead of using `certifi `_ you can provide your own certificate authority bundle. This is useful for cases where you've generated your own certificates or when you're using a private certificate authority. Just provide the full path to the certificate bundle when creating a :class:`~poolmanager.PoolManager`: .. code-block:: python import urllib3 http = urllib3.PoolManager( cert_reqs="CERT_REQUIRED", ca_certs="/path/to/your/certificate_bundle" ) resp = http.request("GET", "https://example.com") When you specify your own certificate bundle only requests that can be verified with that bundle will succeed. It's recommended to use a separate :class:`~poolmanager.PoolManager` to make requests to URLs that do not need the custom certificate. .. _sni_custom: Custom SNI Hostname ------------------- If you want to create a connection to a host over HTTPS which uses SNI, there are two places where the hostname is expected. It must be included in the Host header sent, so that the server will know which host is being requested. The hostname should also match the certificate served by the server, which is checked by urllib3. Normally, urllib3 takes care of setting and checking these values for you when you connect to a host by name. However, it's sometimes useful to set a connection's expected Host header and certificate hostname (subject), especially when you are connecting without using name resolution. For example, you could connect to a server by IP using HTTPS like so: .. code-block:: python import urllib3 pool = urllib3.HTTPSConnectionPool( "104.154.89.105", server_hostname="badssl.com" ) pool.request( "GET", "/", headers={"Host": "badssl.com"}, assert_same_host=False ) Note that when you use a connection in this way, you must specify ``assert_same_host=False``. This is useful when DNS resolution for ``example.org`` does not match the address that you would like to use. The IP may be for a private interface, or you may want to use a specific host under round-robin DNS. .. _assert_hostname: Verifying TLS against a different host -------------------------------------- If the server you're connecting to presents a different certificate than the hostname or the SNI hostname, you can use ``assert_hostname``: .. code-block:: python import urllib3 pool = urllib3.HTTPSConnectionPool( "wrong.host.badssl.com", assert_hostname="badssl.com", ) pool.request("GET", "/") .. _ssl_client: Client Certificates ------------------- You can also specify a client certificate. This is useful when both the server and the client need to verify each other's identity. Typically these certificates are issued from the same authority. To use a client certificate, provide the full path when creating a :class:`~poolmanager.PoolManager`: .. code-block:: python http = urllib3.PoolManager( cert_file="/path/to/your/client_cert.pem", cert_reqs="CERT_REQUIRED", ca_certs="/path/to/your/certificate_bundle" ) If you have an encrypted client certificate private key you can use the ``key_password`` parameter to specify a password to decrypt the key. .. code-block:: python http = urllib3.PoolManager( cert_file="/path/to/your/client_cert.pem", cert_reqs="CERT_REQUIRED", key_file="/path/to/your/client.key", key_password="keyfile_password" ) If your key isn't encrypted the ``key_password`` parameter isn't required. TLS minimum and maximum versions -------------------------------- When the configured TLS versions by urllib3 aren't compatible with the TLS versions that the server is willing to use you'll likely see an error like this one: .. code-block:: SSLError(1, '[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1124)') Starting in v2.0 by default urllib3 uses TLS 1.2 and later so servers that only support TLS 1.1 or earlier will not work by default with urllib3. To fix the issue you'll need to use the ``ssl_minimum_version`` option along with the `TLSVersion enum`_ in the standard library ``ssl`` module to configure urllib3 to accept a wider range of TLS versions. For the best security it's a good idea to set this value to the version of TLS that's being used by the server. For example if the server requires TLS 1.0 you'd configure urllib3 like so: .. code-block:: python import ssl import urllib3 http = urllib3.PoolManager( ssl_minimum_version=ssl.TLSVersion.TLSv1 ) # This request works! resp = http.request("GET", "https://tls-v1-0.badssl.com:1010") .. _TLSVersion enum: https://docs.python.org/3/library/ssl.html#ssl.TLSVersion .. _ssl_mac: .. _certificate_validation_and_mac_os_x: Certificate Validation and macOS -------------------------------- Apple-provided Python and OpenSSL libraries contain a patches that make them automatically check the system keychain's certificates. This can be surprising if you specify custom certificates and see requests unexpectedly succeed. For example, if you are specifying your own certificate for validation and the server presents a different certificate you would expect the connection to fail. However, if that server presents a certificate that is in the system keychain then the connection will succeed. `This article `_ has more in-depth analysis and explanation. .. _ssl_warnings: TLS Warnings ------------ urllib3 will issue several different warnings based on the level of certificate verification support. These warnings indicate particular situations and can be resolved in different ways. * :class:`~exceptions.InsecureRequestWarning` This happens when a request is made to an HTTPS URL without certificate verification enabled. Follow the :ref:`certificate verification ` guide to resolve this warning. .. _disable_ssl_warnings: Making unverified HTTPS requests is **strongly** discouraged, however, if you understand the risks and wish to disable these warnings, you can use :func:`~urllib3.disable_warnings`: .. code-block:: python import urllib3 urllib3.disable_warnings() Alternatively you can capture the warnings with the standard :mod:`logging` module: .. code-block:: python logging.captureWarnings(True) Finally, you can suppress the warnings at the interpreter level by setting the ``PYTHONWARNINGS`` environment variable or by using the `-W flag `_. Brotli Encoding --------------- Brotli is a compression algorithm created by Google with better compression than gzip and deflate and is supported by urllib3 if the `Brotli `_ package or `brotlicffi `_ package is installed. You may also request the package be installed via the ``urllib3[brotli]`` extra: .. code-block:: bash $ python -m pip install urllib3[brotli] Here's an example using brotli encoding via the ``Accept-Encoding`` header: .. code-block:: python import urllib3 urllib3.request( "GET", "https://www.google.com/", headers={"Accept-Encoding": "br"} ) Zstandard Encoding ------------------ `Zstandard `_ is a compression algorithm created by Facebook with better compression than brotli, gzip and deflate (see `benchmarks `_) and is supported by urllib3 if the `zstandard package `_ is installed. You may also request the package be installed via the ``urllib3[zstd]`` extra: .. code-block:: bash $ python -m pip install urllib3[zstd] .. note:: Zstandard support in urllib3 requires using v0.18.0 or later of the ``zstandard`` package. If the version installed is less than v0.18.0 then Zstandard support won't be enabled. Here's an example using zstd encoding via the ``Accept-Encoding`` header: .. code-block:: python import urllib3 urllib3.request( "GET", "https://www.facebook.com/", headers={"Accept-Encoding": "zstd"} ) Decrypting Captured TLS Sessions with Wireshark ----------------------------------------------- Python 3.8 and higher support logging of TLS pre-master secrets. With these secrets tools like `Wireshark `_ can decrypt captured network traffic. To enable this simply define environment variable `SSLKEYLOGFILE`: .. code-block:: bash export SSLKEYLOGFILE=/path/to/keylogfile.txt Then configure the key logfile in `Wireshark `_, see `Wireshark TLS Decryption `_ for instructions. Custom SSL Contexts ------------------- You can exercise fine-grained control over the urllib3 SSL configuration by providing a :class:`ssl.SSLContext ` object. For purposes of compatibility, we recommend you obtain one from :func:`~urllib3.util.create_urllib3_context`. Once you have a context object, you can mutate it to achieve whatever effect you'd like. For example, the code below loads the default SSL certificates, sets the :data:`ssl.OP_ENABLE_MIDDLEBOX_COMPAT` flag that isn't set by default, and then makes a HTTPS request: .. code-block:: python import ssl from urllib3 import PoolManager from urllib3.util import create_urllib3_context ctx = create_urllib3_context() ctx.load_default_certs() ctx.options |= ssl.OP_ENABLE_MIDDLEBOX_COMPAT with PoolManager(ssl_context=ctx) as pool: pool.request("GET", "https://www.google.com/") Note that this is different from passing an ``options`` argument to :func:`~urllib3.util.create_urllib3_context` because we don't overwrite the default options: we only add a new one. urllib3-2.0.7/docs/changelog.rst0000644000000000000000000000007314513541732013440 0ustar00========= Changelog ========= .. include:: ../CHANGES.rst urllib3-2.0.7/docs/conf.py0000644000000000000000000001011214513541732012251 0ustar00from __future__ import annotations import os import sys from datetime import date # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) sys.path.insert(0, root_path) # https://docs.readthedocs.io/en/stable/builds.html#build-environment if "READTHEDOCS" in os.environ: import glob if glob.glob("../changelog/*.*.rst"): print("-- Found changes; running towncrier --", flush=True) import subprocess subprocess.run( ["towncrier", "--yes", "--date", "not released yet"], cwd="..", check=True ) import urllib3 # -- General configuration ----------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ "sphinx.ext.autodoc", "sphinx_copybutton", "sphinx.ext.doctest", "sphinx.ext.intersphinx", "sphinxext.opengraph", ] # Open Graph metadata ogp_title = "urllib3 documentation" ogp_type = "website" ogp_social_cards = {"image": "images/logo.png", "line_color": "#F09837"} ogp_description = "urllib3 is a user-friendly HTTP client library for Python." # Test code blocks only when explicitly specified doctest_test_doctest_blocks = "" # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] # The suffix of source filenames. source_suffix = ".rst" # The master toctree document. master_doc = "index" # General information about the project. project = "urllib3" copyright = f"{date.today().year}, Andrey Petrov" # The short X.Y version. version = urllib3.__version__ # The full version, including alpha/beta/rc tags. release = version # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ["_build"] # The name of the Pygments (syntax highlighting) style to use. pygments_style = "friendly" # The base URL with a proper language and version. html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "/") # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = "furo" html_favicon = "images/favicon.png" html_static_path = ["_static"] html_theme_options = { "announcement": """ Support urllib3 on GitHub Sponsors """, "sidebar_hide_name": True, "light_logo": "banner.svg", "dark_logo": "dark-logo.svg", } intersphinx_mapping = {"python": ("https://docs.python.org/3", None)} # Show typehints as content of the function or method autodoc_typehints = "description" # Warn about all references to unknown targets nitpicky = True # Except for these ones, which we expect to point to unknown targets: nitpick_ignore = [ ("py:class", "_TYPE_SOCKS_OPTIONS"), ("py:class", "_TYPE_SOCKET_OPTIONS"), ("py:class", "_TYPE_TIMEOUT"), ("py:class", "_TYPE_FIELD_VALUE"), ("py:class", "_TYPE_BODY"), ("py:class", "_HttplibHTTPResponse"), ("py:class", "_HttplibHTTPMessage"), ("py:class", "TracebackType"), ("py:class", "Literal"), ("py:class", "email.errors.MessageDefect"), ("py:class", "MessageDefect"), ("py:class", "http.client.HTTPMessage"), ("py:class", "RequestHistory"), ("py:class", "SSLTransportType"), ("py:class", "VerifyMode"), ("py:class", "_ssl._SSLContext"), ("py:class", "urllib3._collections.HTTPHeaderDict"), ("py:class", "urllib3._collections.RecentlyUsedContainer"), ("py:class", "urllib3._request_methods.RequestMethods"), ("py:class", "urllib3.contrib.socks._TYPE_SOCKS_OPTIONS"), ("py:class", "urllib3.util.timeout._TYPE_DEFAULT"), ("py:class", "BaseHTTPConnection"), ] urllib3-2.0.7/docs/contributing.rst0000644000000000000000000002243414513541732014225 0ustar00Contributing ============ urllib3 is a community-maintained project and we happily accept contributions. If you wish to add a new feature or fix a bug: #. `Check for open issues `_ or open a fresh issue to start a discussion around a feature idea or a bug. There is a *Contributor Friendly* tag for issues that should be ideal for people who are not very familiar with the codebase yet. #. Fork the `urllib3 repository on Github `_ to start making your changes. #. Write a test which shows that the bug was fixed or that the feature works as expected. #. Format your changes with black using command `$ nox -rs format` and lint your changes using command `nox -rs lint`. #. Add a `changelog entry `__. #. Send a pull request and bug the maintainer until it gets merged and published. Setting up your development environment --------------------------------------- In order to setup the development environment all that you need is `nox `_ installed in your machine:: $ python -m pip install --user --upgrade nox Running the tests ----------------- We use some external dependencies, multiple interpreters and code coverage analysis while running test suite. Our ``noxfile.py`` handles much of this for you:: $ nox --reuse-existing-virtualenvs --sessions test-3.7 test-3.9 [ Nox will create virtualenv if needed, install the specified dependencies, and run the commands in order.] nox > Running session test-3.7 ....... ....... nox > Session test-3.7 was successful. ....... ....... nox > Running session test-3.9 ....... ....... nox > Session test-3.9 was successful. There is also a nox command for running all of our tests and multiple python versions.:: $ nox --reuse-existing-virtualenvs --sessions test Note that code coverage less than 100% is regarded as a failing run. Some platform-specific tests are skipped unless run in that platform. To make sure the code works in all of urllib3's supported platforms, you can run our ``nox`` suite:: $ nox --reuse-existing-virtualenvs --sessions test [ Nox will create virtualenv if needed, install the specified dependencies, and run the commands in order.] ....... ....... nox > Session test-3.7 was successful. nox > Session test-3.8 was successful. nox > Session test-3.9 was successful. nox > Session test-3.10 was successful. nox > Session test-pypy was successful. Our test suite `runs continuously on GitHub Actions `_ with every pull request. To run specific tests or quickly re-run without nox recreating the env, do the following:: $ nox --reuse-existing-virtualenvs --sessions test-3.8 -- pyTestArgument1 pyTestArgument2 pyTestArgumentN [ Nox will create virtualenv, install the specified dependencies, and run the commands in order.] nox > Running session test-3.8 nox > Re-using existing virtual environment at .nox/test-3-8. ....... ....... nox > Session test-3.8 was successful. After the ``--`` indicator, any arguments will be passed to pytest. To specify an exact test case the following syntax also works: ``test/dir/module_name.py::TestClassName::test_method_name`` (eg.: ``test/with_dummyserver/test_https.py::TestHTTPS::test_simple``). The following argument is another valid example to pass to pytest: ``-k test_methode_name``. These are useful when developing new test cases and there is no point re-running the entire test suite every iteration. It is also possible to further parameterize pytest for local testing. For all valid arguments, check `the pytest documentation `_. Getting paid for your contributions ----------------------------------- urllib3 has a `pool of money hosted on Open Collective `_ which the team uses to pay contributors for their work. **That could be you, too!** If you complete all tasks in an issue that is marked with the `"💰 Bounty $X00" label `_ then you're eligible to be paid for your work. - Ensure that you're able to `receive funds from Open Collective for working on OSS `_. Consider your employment contract and taxes for possible restrictions. - If an issue is already assigned to someone on GitHub then it's likely they've made substantial progress on the issue and will be given the bounty. If you're interested in bounties you should look for issues which aren't assigned to anyone. - **Don't "claim" issues or ask whether someone is already working on an issue.** Instead, focus on researching and working on the tasks in the issue. Once you have made considerable progress on the tasks in an issue we can assign your account to the issue to ensure others don't start working on it in parallel. - If you've been assigned to an issue and haven't made progress or given an update in over a week you will be unassigned from the issue to allow others a chance at solving the issue. - The amount you will be paid for the completing an issue is shown in the label (either $100, $200, $300, etc). - If you have questions about how to create an invoice on Open Collective `try reading their FAQ `_. - If you have a proposal to work on urllib3 that's not listed in the issue tracker please open an issue with your proposal and our team will discuss whether we'd pay for your work on your proposal. - If you have other questions get in contact with a maintainer in the `urllib3 Discord channel `_ or via email. - The list above isn't an exhaustive list of criteria or rules for how/when money is distributed. **The final say on whether money will be distributed is up to maintainers.** This program is an experiment so if you have positive or negative feedback on the process you can contact the maintainers through one of the above channels. Note that this program isn't a "bug bounty" program, we don't distribute funds to reporters of bugs or security vulnerabilities at this time. Running local proxies --------------------- If the feature you are developing involves a proxy, you can rely on scripts we have developed to run a proxy locally. Run an HTTP proxy locally: .. code-block:: bash $ python -m dummyserver.proxy Run an HTTPS proxy locally: .. code-block:: bash $ python -m dummyserver.https_proxy Contributing to documentation ----------------------------- You can build the docs locally using ``nox``: .. code-block:: bash $ nox -rs docs While writing documentation you should follow these guidelines: - Use the top-level ``urllib3.request()`` function for smaller code examples. For more involved examples use PoolManager, etc. - Use double quotes for all strings. (Output, Declaration etc.) - Use keyword arguments everywhere except for method and url. (ie ``http.request("GET", "https://example.com", headers={...})`` ) - Use HTTPS in URLs everywhere unless HTTP is needed. - Rules for code examples and naming variables: - ``PoolManager`` instances should be named ``http``. (ie ``http = urllib3.PoolManager(...)``) - ``ProxyManager`` instances should be named ``proxy``. - ``ConnectionPool`` instances should be named ``pool``. - ``Connection`` instances should be named ``conn``. - ``HTTPResponse`` instances should be named ``resp``. - Only use ``example.com`` or ``httpbin.org`` for example URLs - Comments within snippets should be useful, if what's being done is apparent (such as parsing JSON, making a request) then it can be skipped for that section. - Comments should always go above a code section rather than below with the exception of print statements where the comment containing the result goes below. - Imports should be their own section separated from the rest of the example with a line of whitespace. - Imports should minimized if possible. Use import urllib3 instead of from urllib3 import X. - Sort imports similarly to isort, standard library first and third-party (like urllib3) come after. - No whitespace is required between the sections as normally would be in case of isort. - Add print statements along with a comment below them showing the output, potentially compressed. - This helps users using the copy-paste button immediately see the results from a script. Releases -------- A release candidate can be created by any contributor. - Announce intent to release on Discord, see if anyone wants to include last minute changes. - Run ``towncrier build`` to update ``CHANGES.rst`` with the release notes, adjust as necessary. - Update ``urllib3/__init__.py`` with the proper version number - Commit the changes to a ``release-X.Y.Z`` branch. - Create a pull request and append ``&expand=1&template=release.md`` to the URL before submitting in order to include our release checklist in the pull request description. - Follow the checklist! urllib3-2.0.7/docs/index.rst0000644000000000000000000000737414513541732012633 0ustar00urllib3 ======= .. toctree:: :hidden: :maxdepth: 3 For Enterprise Community Discord v2-migration-guide sponsors user-guide advanced-usage reference/index contributing changelog urllib3 is a powerful, *user-friendly* HTTP client for Python. :ref:`Much of the Python ecosystem already uses ` urllib3 and you should too. urllib3 brings many critical features that are missing from the Python standard libraries: - Thread safety. - Connection pooling. - Client-side TLS/SSL verification. - File uploads with multipart encoding. - Helpers for retrying requests and dealing with HTTP redirects. - Support for gzip, deflate, brotli, and zstd encoding. - Proxy support for HTTP and SOCKS. - 100% test coverage. urllib3 is powerful and easy to use: .. code-block:: pycon >>> import urllib3 >>> resp = urllib3.request("GET", "https://httpbin.org/robots.txt") >>> resp.status 200 >>> resp.data b"User-agent: *\nDisallow: /deny\n" For Enterprise -------------- .. |tideliftlogo| image:: https://nedbatchelder.com/pix/Tidelift_Logos_RGB_Tidelift_Shorthand_On-White_small.png :width: 75 :alt: Tidelift .. list-table:: :widths: 10 100 * - |tideliftlogo|_ - Professional support for urllib3 is available as part of the `Tidelift Subscription`_. Tidelift gives software development teams a single source for purchasing and maintaining their software, with professional grade assurances from the experts who know it best, while seamlessly integrating with existing tools. .. _Tidelift Subscription: https://tidelift.com/subscription/pkg/pypi-urllib3?utm_source=pypi-urllib3&utm_medium=referral&utm_campaign=docs .. _tideliftlogo: https://tidelift.com/subscription/pkg/pypi-urllib3?utm_source=pypi-urllib3&utm_medium=referral&utm_campaign=docs |learn-more|_ |request-a-demo|_ .. |learn-more| image:: https://raw.githubusercontent.com/urllib3/urllib3/master/docs/images/learn-more-button.png :width: 49% :alt: Learn more about Tidelift Subscription .. _learn-more: https://tidelift.com/subscription/pkg/pypi-urllib3?utm_source=pypi-urllib3&utm_medium=referral&utm_campaign=docs .. |request-a-demo| image:: https://raw.githubusercontent.com/urllib3/urllib3/master/docs/images/demo-button.png :width: 49% :alt: Request a Demo for the Tidelift Subscription .. _request-a-demo: https://tidelift.com/subscription/request-a-demo?utm_source=pypi-urllib3&utm_medium=referral&utm_campaign=docs Installing ---------- urllib3 can be installed with `pip `_ .. code-block:: bash $ python -m pip install urllib3 Alternatively, you can grab the latest source code from `GitHub `_: .. code-block:: bash $ git clone https://github.com/urllib3/urllib3.git $ cd urllib3 $ pip install . Usage ----- The :doc:`user-guide` is the place to go to learn how to use the library and accomplish common tasks. The more in-depth :doc:`advanced-usage` guide is the place to go for lower-level tweaking. The :doc:`reference/index` documentation provides API-level documentation. .. _who-uses: Who uses urllib3? ----------------- `urllib3 is one of the most downloaded packages on PyPI `_ and is a dependency of many popular Python packages like `Requests `_, `Pip `_, and more! License ------- urllib3 is made available under the MIT License. For more details, see `LICENSE.txt `_. Contributing ------------ We happily welcome contributions, please see :doc:`contributing` for details. urllib3-2.0.7/docs/make.bat0000644000000000000000000001064114513541732012366 0ustar00@ECHO OFF REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set BUILDDIR=_build set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . if NOT "%PAPER%" == "" ( set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% ) if "%1" == "" goto help if "%1" == "help" ( :help echo.Please use `make ^` where ^ is one of echo. html to make standalone HTML files echo. dirhtml to make HTML files named index.html in directories echo. singlehtml to make a single large HTML file echo. pickle to make pickle files echo. json to make JSON files echo. htmlhelp to make HTML files and a HTML help project echo. qthelp to make HTML files and a qthelp project echo. devhelp to make HTML files and a Devhelp project echo. epub to make an epub echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter echo. text to make text files echo. man to make manual pages echo. changes to make an overview over all changed/added/deprecated items echo. linkcheck to check all external links for integrity echo. doctest to run all doctests embedded in the documentation if enabled goto end ) if "%1" == "clean" ( for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i del /q /s %BUILDDIR%\* goto end ) if "%1" == "html" ( %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/html. goto end ) if "%1" == "dirhtml" ( %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. goto end ) if "%1" == "singlehtml" ( %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. goto end ) if "%1" == "pickle" ( %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can process the pickle files. goto end ) if "%1" == "json" ( %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can process the JSON files. goto end ) if "%1" == "htmlhelp" ( %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can run HTML Help Workshop with the ^ .hhp project file in %BUILDDIR%/htmlhelp. goto end ) if "%1" == "qthelp" ( %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can run "qcollectiongenerator" with the ^ .qhcp project file in %BUILDDIR%/qthelp, like this: echo.^> qcollectiongenerator %BUILDDIR%\qthelp\urllib3.qhcp echo.To view the help file: echo.^> assistant -collectionFile %BUILDDIR%\qthelp\urllib3.ghc goto end ) if "%1" == "devhelp" ( %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp if errorlevel 1 exit /b 1 echo. echo.Build finished. goto end ) if "%1" == "epub" ( %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub if errorlevel 1 exit /b 1 echo. echo.Build finished. The epub file is in %BUILDDIR%/epub. goto end ) if "%1" == "latex" ( %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex if errorlevel 1 exit /b 1 echo. echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. goto end ) if "%1" == "text" ( %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text if errorlevel 1 exit /b 1 echo. echo.Build finished. The text files are in %BUILDDIR%/text. goto end ) if "%1" == "man" ( %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man if errorlevel 1 exit /b 1 echo. echo.Build finished. The manual pages are in %BUILDDIR%/man. goto end ) if "%1" == "changes" ( %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes if errorlevel 1 exit /b 1 echo. echo.The overview file is in %BUILDDIR%/changes. goto end ) if "%1" == "linkcheck" ( %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck if errorlevel 1 exit /b 1 echo. echo.Link check complete; look for any errors in the above output ^ or in %BUILDDIR%/linkcheck/output.txt. goto end ) if "%1" == "doctest" ( %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest if errorlevel 1 exit /b 1 echo. echo.Testing of doctests in the sources finished, look at the ^ results in %BUILDDIR%/doctest/output.txt. goto end ) :end urllib3-2.0.7/docs/requirements.txt0000644000000000000000000000025614513541732014246 0ustar00-r ../dev-requirements.txt # https://github.com/sphinx-doc/sphinx/issues/11662#issuecomment-1713887182 sphinx>3.0.0,<7.2.5 requests furo sphinx-copybutton sphinxext-opengraphurllib3-2.0.7/docs/sponsors.rst0000644000000000000000000000343214513541732013401 0ustar00Sponsors and Supporters ======================= Please consider sponsoring urllib3 development, especially if your company benefits from this library. Your contribution will go towards adding new features to urllib3 and making sure all functionality continues to meet our high quality standards. Sponsors and Grants ------------------- A grant for contiguous full-time development has the biggest impact for progress. Periods of 3 to 10 days allow a contributor to tackle substantial complex issues which are otherwise left to linger until somebody can't afford to not fix them. Contact `@sethmlarson `_ or `@shazow `_ to arrange a grant for a core contributor. We also welcome sponsorship in the form of time. We greatly appreciate companies who encourage employees to contribute on an ongoing basis during their work hours. Let us know and we'll be glad to add you to our sponsors list. * `Spotify `_ (June 2nd, 2022) * `GitCoin Grants `_ (2019-2020), sponsored `@sethmlarson `_ and `@pquentin `_ * `Abbott `_ (2018-2019), sponsored `@sethmlarson `_ * `Google Cloud Platform `_ (2018-2019), sponsored `@theacodes `_ * `GOVCERT.LU `_ (October 23, 2018), sponsored `@sethmlarson `_ * `Akamai `_ (2017-2018) sponsored `@haikuginger `_ * `Hewlett Packard Enterprise `_ (2016-2017) sponsored `@Lukasa `_ * `Stripe `_ (June 23, 2014) urllib3-2.0.7/docs/user-guide.rst0000644000000000000000000004036714513541732013574 0ustar00User Guide ========== .. currentmodule:: urllib3 Installing ---------- urllib3 can be installed with `pip `_ .. code-block:: bash $ python -m pip install urllib3 Making Requests --------------- First things first, import the urllib3 module: .. code-block:: python import urllib3 You'll need a :class:`~poolmanager.PoolManager` instance to make requests. This object handles all of the details of connection pooling and thread safety so that you don't have to: .. code-block:: python http = urllib3.PoolManager() To make a request use :meth:`~urllib3.PoolManager.request`: .. code-block:: python import urllib3 # Creating a PoolManager instance for sending requests. http = urllib3.PoolManager() # Sending a GET request and getting back response as HTTPResponse object. resp = http.request("GET", "https://httpbin.org/robots.txt") # Print the returned data. print(resp.data) # b"User-agent: *\nDisallow: /deny\n" ``request()`` returns a :class:`~response.HTTPResponse` object, the :ref:`response_content` section explains how to handle various responses. You can use :meth:`~urllib3.PoolManager.request` to make requests using any HTTP verb: .. code-block:: python import urllib3 http = urllib3.PoolManager() resp = http.request( "POST", "https://httpbin.org/post", fields={"hello": "world"} # Add custom form fields ) print(resp.data) # b"{\n "form": {\n "hello": "world"\n }, ... } The :ref:`request_data` section covers sending other kinds of requests data, including JSON, files, and binary data. .. note:: For quick scripts and experiments you can also use a top-level ``urllib3.request()``. It uses a module-global ``PoolManager`` instance. Because of that, its side effects could be shared across dependencies relying on it. To avoid side effects, create a new ``PoolManager`` instance and use it instead. In addition, the method does not accept the low-level ``**urlopen_kw`` keyword arguments. System CA certificates are loaded on default. .. _response_content: Response Content ---------------- The :class:`~response.HTTPResponse` object provides :attr:`~response.HTTPResponse.status`, :attr:`~response.HTTPResponse.data`, and :attr:`~response.HTTPResponse.headers` attributes: .. code-block:: python import urllib3 # Making the request (The request function returns HTTPResponse object) resp = urllib3.request("GET", "https://httpbin.org/ip") print(resp.status) # 200 print(resp.data) # b"{\n "origin": "104.232.115.37"\n}\n" print(resp.headers) # HTTPHeaderDict({"Content-Length": "32", ...}) JSON Content ~~~~~~~~~~~~ JSON content can be loaded by :meth:`~response.HTTPResponse.json` method of the response: .. code-block:: python import urllib3 resp = urllib3.request("GET", "https://httpbin.org/ip") print(resp.json()) # {"origin": "127.0.0.1"} Alternatively, Custom JSON libraries such as `orjson` can be used to encode data, retrieve data by decoding and deserializing the :attr:`~response.HTTPResponse.data` attribute of the request: .. code-block:: python import orjson import urllib3 encoded_data = orjson.dumps({"attribute": "value"}) resp = urllib3.request(method="POST", url="http://httpbin.org/post", body=encoded_data) print(orjson.loads(resp.data)["json"]) # {'attribute': 'value'} Binary Content ~~~~~~~~~~~~~~ The :attr:`~response.HTTPResponse.data` attribute of the response is always set to a byte string representing the response content: .. code-block:: python import urllib3 resp = urllib3.request("GET", "https://httpbin.org/bytes/8") print(resp.data) # b"\xaa\xa5H?\x95\xe9\x9b\x11" .. note:: For larger responses, it's sometimes better to :ref:`stream ` the response. Using io Wrappers with Response Content ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sometimes you want to use :class:`io.TextIOWrapper` or similar objects like a CSV reader directly with :class:`~response.HTTPResponse` data. Making these two interfaces play nice together requires using the :attr:`~response.HTTPResponse.auto_close` attribute by setting it to ``False``. By default HTTP responses are closed after reading all bytes, this disables that behavior: .. code-block:: python import io import urllib3 resp = urllib3.request("GET", "https://example.com", preload_content=False) resp.auto_close = False for line in io.TextIOWrapper(resp): print(line) # # # # .... # # .. _request_data: Request Data ------------ Headers ~~~~~~~ You can specify headers as a dictionary in the ``headers`` argument in :meth:`~urllib3.PoolManager.request`: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/headers", headers={ "X-Something": "value" } ) print(resp.json()["headers"]) # {"X-Something": "value", ...} Or you can use the ``HTTPHeaderDict`` class to create multi-valued HTTP headers: .. code-block:: python import urllib3 # Create an HTTPHeaderDict and add headers headers = urllib3.HTTPHeaderDict() headers.add("Accept", "application/json") headers.add("Accept", "text/plain") # Make the request using the headers resp = urllib3.request( "GET", "https://httpbin.org/headers", headers=headers ) print(resp.json()["headers"]) # {"Accept": "application/json, text/plain", ...} Cookies ~~~~~~~ Cookies are specified using the ``Cookie`` header with a string containing the ``;`` delimited key-value pairs: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/cookies", headers={ "Cookie": "session=f3efe9db; id=30" } ) print(resp.json()) # {"cookies": {"id": "30", "session": "f3efe9db"}} Note that the ``Cookie`` header will be stripped if the server redirects to a different host. Cookies provided by the server are stored in the ``Set-Cookie`` header: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/cookies/set/session/f3efe9db", redirect=False ) print(resp.headers["Set-Cookie"]) # session=f3efe9db; Path=/ Query Parameters ~~~~~~~~~~~~~~~~ For ``GET``, ``HEAD``, and ``DELETE`` requests, you can simply pass the arguments as a dictionary in the ``fields`` argument to :meth:`~urllib3.PoolManager.request`: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/get", fields={"arg": "value"} ) print(resp.json()["args"]) # {"arg": "value"} For ``POST`` and ``PUT`` requests, you need to manually encode query parameters in the URL: .. code-block:: python from urllib.parse import urlencode import urllib3 # Encode the args into url grammar. encoded_args = urlencode({"arg": "value"}) # Create a URL with args encoded. url = "https://httpbin.org/post?" + encoded_args resp = urllib3.request("POST", url) print(resp.json()["args"]) # {"arg": "value"} .. _form_data: Form Data ~~~~~~~~~ For ``PUT`` and ``POST`` requests, urllib3 will automatically form-encode the dictionary in the ``fields`` argument provided to :meth:`~urllib3.PoolManager.request`: .. code-block:: python import urllib3 resp = urllib3.request( "POST", "https://httpbin.org/post", fields={"field": "value"} ) print(resp.json()["form"]) # {"field": "value"} .. _json: JSON ~~~~ To send JSON in the body of a request, provide the data in the ``json`` argument to :meth:`~urllib3.PoolManager.request` and urllib3 will automatically encode the data using the ``json`` module with ``UTF-8`` encoding. In addition, when ``json`` is provided, the ``"Content-Type"`` in headers is set to ``"application/json"`` if not specified otherwise. .. code-block:: python import urllib3 resp = urllib3.request( "POST", "https://httpbin.org/post", json={"attribute": "value"}, headers={"Content-Type": "application/json"} ) print(resp.json()) # {'headers': {'Content-Type': 'application/json', ...}, # 'data': '{"attribute":"value"}', 'json': {'attribute': 'value'}, ...} Files & Binary Data ~~~~~~~~~~~~~~~~~~~ For uploading files using ``multipart/form-data`` encoding you can use the same approach as :ref:`form_data` and specify the file field as a tuple of ``(file_name, file_data)``: .. code-block:: python import urllib3 # Reading the text file from local storage. with open("example.txt") as fp: file_data = fp.read() # Sending the request. resp = urllib3.request( "POST", "https://httpbin.org/post", fields={ "filefield": ("example.txt", file_data), } ) print(resp.json()["files"]) # {"filefield": "..."} While specifying the filename is not strictly required, it's recommended in order to match browser behavior. You can also pass a third item in the tuple to specify the file's MIME type explicitly: .. code-block:: python resp = urllib3.request( "POST", "https://httpbin.org/post", fields={ "filefield": ("example.txt", file_data, "text/plain"), } ) For sending raw binary data simply specify the ``body`` argument. It's also recommended to set the ``Content-Type`` header: .. code-block:: python import urllib3 with open("/home/samad/example.jpg", "rb") as fp: binary_data = fp.read() resp = urllib3.request( "POST", "https://httpbin.org/post", body=binary_data, headers={"Content-Type": "image/jpeg"} ) print(resp.json()["data"]) # data:application/octet-stream;base64,... .. _ssl: Certificate Verification ------------------------ .. note:: *New in version 1.25:* HTTPS connections are now verified by default (``cert_reqs = "CERT_REQUIRED"``). While you can disable certification verification by setting ``cert_reqs = "CERT_NONE"``, it is highly recommend to leave it on. Unless otherwise specified urllib3 will try to load the default system certificate stores. The most reliable cross-platform method is to use the `certifi `_ package which provides Mozilla's root certificate bundle: .. code-block:: bash $ python -m pip install certifi Once you have certificates, you can create a :class:`~poolmanager.PoolManager` that verifies certificates when making requests: .. code-block:: python import certifi import urllib3 http = urllib3.PoolManager( cert_reqs="CERT_REQUIRED", ca_certs=certifi.where() ) The :class:`~poolmanager.PoolManager` will automatically handle certificate verification and will raise :class:`~exceptions.SSLError` if verification fails: .. code-block:: python import certifi import urllib3 http = urllib3.PoolManager( cert_reqs="CERT_REQUIRED", ca_certs=certifi.where() ) http.request("GET", "https://httpbin.org/") # (No exception) http.request("GET", "https://expired.badssl.com") # urllib3.exceptions.SSLError ... .. note:: You can use OS-provided certificates if desired. Just specify the full path to the certificate bundle as the ``ca_certs`` argument instead of ``certifi.where()``. For example, most Linux systems store the certificates at ``/etc/ssl/certs/ca-certificates.crt``. Other operating systems can be `difficult `_. Using Timeouts -------------- Timeouts allow you to control how long (in seconds) requests are allowed to run before being aborted. In simple cases, you can specify a timeout as a ``float`` to :meth:`~urllib3.PoolManager.request`: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/delay/3", timeout=4.0 ) print(type(resp)) # # This request will take more time to process than timeout. urllib3.request( "GET", "https://httpbin.org/delay/3", timeout=2.5 ) # MaxRetryError caused by ReadTimeoutError For more granular control you can use a :class:`~util.timeout.Timeout` instance which lets you specify separate connect and read timeouts: .. code-block:: python import urllib3 resp = urllib3.request( "GET", "https://httpbin.org/delay/3", timeout=urllib3.Timeout(connect=1.0) ) print(type(resp)) # urllib3.request( "GET", "https://httpbin.org/delay/3", timeout=urllib3.Timeout(connect=1.0, read=2.0) ) # MaxRetryError caused by ReadTimeoutError If you want all requests to be subject to the same timeout, you can specify the timeout at the :class:`~urllib3.poolmanager.PoolManager` level: .. code-block:: python import urllib3 http = urllib3.PoolManager(timeout=3.0) http = urllib3.PoolManager( timeout=urllib3.Timeout(connect=1.0, read=2.0) ) You still override this pool-level timeout by specifying ``timeout`` to :meth:`~urllib3.PoolManager.request`. Retrying Requests ----------------- urllib3 can automatically retry idempotent requests. This same mechanism also handles redirects. You can control the retries using the ``retries`` parameter to :meth:`~urllib3.PoolManager.request`. By default, urllib3 will retry requests 3 times and follow up to 3 redirects. To change the number of retries just specify an integer: .. code-block:: python import urllib3 urllib3.request("GET", "https://httpbin.org/ip", retries=10) To disable all retry and redirect logic specify ``retries=False``: .. code-block:: python import urllib3 urllib3.request( "GET", "https://nxdomain.example.com", retries=False ) # NewConnectionError resp = urllib3.request( "GET", "https://httpbin.org/redirect/1", retries=False ) print(resp.status) # 302 To disable redirects but keep the retrying logic, specify ``redirect=False``: .. code-block:: python resp = urllib3.request( "GET", "https://httpbin.org/redirect/1", redirect=False ) print(resp.status) # 302 For more granular control you can use a :class:`~util.retry.Retry` instance. This class allows you far greater control of how requests are retried. For example, to do a total of 3 retries, but limit to only 2 redirects: .. code-block:: python urllib3.request( "GET", "https://httpbin.org/redirect/3", retries=urllib3.Retry(3, redirect=2) ) # MaxRetryError You can also disable exceptions for too many redirects and just return the ``302`` response: .. code-block:: python resp = urllib3.request( "GET", "https://httpbin.org/redirect/3", retries=urllib3.Retry( redirect=2, raise_on_redirect=False ) ) print(resp.status) # 302 If you want all requests to be subject to the same retry policy, you can specify the retry at the :class:`~urllib3.poolmanager.PoolManager` level: .. code-block:: python import urllib3 http = urllib3.PoolManager(retries=False) http = urllib3.PoolManager( retries=urllib3.Retry(5, redirect=2) ) You still override this pool-level retry policy by specifying ``retries`` to :meth:`~urllib3.PoolManager.request`. Errors & Exceptions ------------------- urllib3 wraps lower-level exceptions, for example: .. code-block:: python import urllib3 try: urllib3.request("GET","https://nx.example.com", retries=False) except urllib3.exceptions.NewConnectionError: print("Connection failed.") # Connection failed. See :mod:`~urllib3.exceptions` for the full list of all exceptions. Logging ------- If you are using the standard library :mod:`logging` module urllib3 will emit several logs. In some cases this can be undesirable. You can use the standard logger interface to change the log level for urllib3's logger: .. code-block:: python logging.getLogger("urllib3").setLevel(logging.WARNING) urllib3-2.0.7/docs/v2-migration-guide.rst0000644000000000000000000004140714513541732015130 0ustar00v2.0 Migration Guide ==================== **urllib3 v2.0 is now available!** Read below for how to get started and what is contained in the new major release. **🚀 Migrating from 1.x to 2.0** -------------------------------- We're maintaining **functional API compatibility for most users** to make the migration an easy choice for almost everyone. Most changes are either to default configurations, supported Python versions, or internal implementation details. So unless you're in a specific situation you should notice no changes! 🎉 .. note:: If you have difficulty migrating to v2.0 or following this guide you can `open an issue on GitHub `_ or reach out in `our community Discord channel `_. Timeline for deprecations and breaking changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The 2.x initial release schedule will look like this: * urllib3 ``v2.0.0-alpha1`` will be released in November 2022. This release contains **minor breaking changes and deprecation warnings for other breaking changes**. There may be other pre-releases to address fixes before v2.0.0 is released. * urllib3 ``v2.0.0`` will be released in early 2023 after some initial integration testing against dependent packages and fixing of bug reports. * urllib3 ``v2.1.0`` will be released in the summer of 2023 with **all breaking changes being warned about in v2.0.0**. .. warning:: Please take the ``DeprecationWarnings`` you receive when migrating from v1.x to v2.0 seriously as they will become errors after 2.1.0 is released. What are the important changes? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's a short summary of which changes in urllib3 v2.0 are most important: - Python version must be **3.7 or later** (previously supported Python 2.7, 3.5, and 3.6). - Removed support for non-OpenSSL TLS libraries (like LibreSSL and wolfSSL). - Removed support for OpenSSL versions older than 1.1.1. - Removed support for Python implementations that aren't CPython or PyPy3 (previously supported Google App Engine, Jython). - Removed the ``urllib3.contrib.ntlmpool`` module. - Deprecated the ``urllib3.contrib.pyopenssl``, ``urllib3.contrib.securetransport`` modules, will be removed in v2.1.0. - Deprecated the ``urllib3[secure]`` extra, will be removed in v2.1.0. - Deprecated the ``HTTPResponse.getheaders()`` method in favor of ``HTTPResponse.headers``, will be removed in v2.1.0. - Deprecated the ``HTTPResponse.getheader(name, default)`` method in favor of ``HTTPResponse.headers.get(name, default)``, will be removed in v2.1.0. - Deprecated URLs without a scheme (ie 'https://') and will be raising an error in a future version of urllib3. - Changed the default minimum TLS version to TLS 1.2 (previously was TLS 1.0). - Removed support for verifying certificate hostnames via ``commonName``, now only ``subjectAltName`` is used. - Removed the default set of TLS ciphers, instead now urllib3 uses the list of ciphers configured by the system. For a full list of changes you can look at `the changelog `_. Migrating as a package maintainer? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you're a maintainer of a package that uses urllib3 under the hood then this section is for you. You may have already seen an issue opened from someone on our team about the upcoming release. The primary goal for migrating to urllib3 v2.x should be to ensure your package supports **both urllib3 v1.26.x and v2.0 for some time**. This is to reduce the chance that diamond dependencies are introduced into your users' dependencies which will then cause issues with them upgrading to the latest version of **your package**. The first step to supporting urllib3 v2.0 is to make sure the version v2.x not being excluded by ``install_requires``. You should ensure your package allows for both urllib3 1.26.x and 2.0 to be used: .. code-block:: python # setup.py (setuptools) setup( ... install_requires=["urllib3>=1.26,<3"] ) # pyproject.toml (hatch) [project] dependencies = [ "urllib3>=1.26,<3" ] Next you should try installing urllib3 v2.0 locally and run your test suite. .. code-block:: bash $ python -m pip install -U --pre 'urllib3>=2.0.0a1' Because there are many ``DeprecationWarnings`` you should ensure that you're able to see those warnings when running your test suite. To do so you can add the following to your test setup to ensure even ``DeprecationWarnings`` are output to the terminal: .. code-block:: bash # Set PYTHONWARNING=default to show all warnings. $ export PYTHONWARNINGS="default" # Run your test suite and look for failures. # Pytest automatically prints all warnings. $ pytest tests/ or you can opt-in within your Python code: .. code-block:: python # You can change warning filters according to the filter rules: # https://docs.python.org/3/library/warnings.html#warning-filter import warnings warnings.filterwarnings("default", category=DeprecationWarning) Any failures or deprecation warnings you receive should be fixed as urllib3 v2.1.0 will remove all deprecated features. Many deprecation warnings will make suggestions about what to do to avoid the deprecated feature. Warnings will look something like this: .. code-block:: bash DeprecationWarning: 'ssl_version' option is deprecated and will be removed in urllib3 v2.1.0. Instead use 'ssl_minimum_version' Continue removing deprecation warnings until there are no more. After this you can publish a new release of your package that supports both urllib3 v1.26.x and v2.x. .. note:: If you're not able to support both 1.26.x and v2.0 of urllib3 at the same time with your package please `open an issue on GitHub `_ or reach out in `our community Discord channel `_. Migrating as an application developer? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you're someone who writes Python but doesn't ship as a package (things like web services, data science, tools, and more) this section is for you. Python environments only allow for one version of a dependency to be installed per environment which means that **all of your dependencies using urllib3 need to support v2.0 for you to upgrade**. The best way to visualize relationships between your dependencies is using `pipdeptree `_ and ``$ pipdeptree --reverse``: .. code-block:: bash # From inside your Python environment: $ python -m pip install pipdeptree # We only care about packages requiring urllib3 $ pipdeptree --reverse | grep "requires: urllib3" - botocore==1.29.8 [requires: urllib3>=1.25.4,<2] - requests==2.28.1 [requires: urllib3>=1.21.1,<2] Reading the output from above, there are two packages which depend on urllib3: ``botocore`` and ``requests``. The versions of these two packages both require urllib3 that is less than v2.0 (ie ``<2``). Because both of these packages require urllib3 before v2.0 the new version of urllib3 can't be installed by default. There are ways to force installing the newer version of urllib3 v2.0 (ie pinning to ``urllib3==2.0.0``) which you can do to test your application. It's important to know that even if you don't upgrade all of your services to 2.x immediately you will `receive security fixes on the 1.26.x release stream <#security-fixes-for-urllib3-v1-26-x>` for some time. Security fixes for urllib3 v1.26.x ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Thanks to support from `Tidelift `_ we're able to continue supporting the v1.26.x release stream with security fixes for the foreseeable future 💖 However, upgrading is still recommended as **no new feature developments or non-critical bug fixes will be shipped to the 1.26.x release stream**. If your organization relies on urllib3 and is interested in continuing support you can learn more about the `Tidelift Subscription for Enterprise `_. **🤔 Common upgrading issues** ------------------------------- ssl module is compiled with OpenSSL 1.0.2.k-fips ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: text ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2k-fips 26 Jan 2017'. See: https://github.com/urllib3/urllib3/issues/2168 Remediation depends on your system: - **AWS Lambda**: Upgrade to the Python3.10 runtime as it uses OpenSSL 1.1.1. Alternatively, you can use a `custom Docker image `_ and ensure you use a Python build that uses OpenSSL 1.1.1 or later. - **Amazon Linux 2**: Upgrade to `Amazon Linux 2023 `_. Alternatively, you can install OpenSSL 1.1.1 on Amazon Linux 2 using ``yum install openssl11 openssl11-devel`` and then install Python with a tool like pyenv. - **Red Hat Enterpritse Linux 7 (RHEL 7)**: Upgrade to RHEL 8 or RHEL 9. - **Read the Docs**: Upgrade your `configuration file to use Ubuntu 22.04 `_ by using ``os: ubuntu-22.04`` in the ``build`` section. Feel free to use the `urllib3 configuration `_ as an inspiration. docker.errors.dockerexception: error while fetching server api version: request() got an unexpected keyword argument 'chunked' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Upgrade to ``docker==6.1.0`` that is compatible with urllib3 2.0. ImportError: cannot import name 'gaecontrib' from 'requests_toolbelt._compat' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To be compatible with urllib3 2.0, Requests Toolbelt released version 1.0.0 without Google App Engine Standard Python 2.7 support. Most users that reported this issue were using the `Pyrebase `_ library that provides an API for the Firebase API. This library is unmaintained, but `replacements exist `_. ``ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This likely happens because you're using botocore which `does not support urllib3 2.0 yet `_. The good news is that botocore explicitly declares in its dependencies that it only supports ``urllib3<2``. Make sure to use a recent pip. That way, pip will install urllib3 1.26.x until botocore starts supporting urllib3 2.0. If you're deploying to an AWS environment such as Lambda or a host using Amazon Linux 2, you'll need to explicitly pin to ``urllib3<2`` in your project to ensure urllib3 2.0 isn't brought into your environment. Otherwise, this may result in unintended side effects with the default boto3 installation. AttributeError: module 'urllib3.connectionpool' has no attribute 'VerifiedHTTPSConnection' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``VerifiedHTTPSConnection`` class has always been documented to be in the :mod:`~urllib3.connection` module. It used to be possible to import it from :mod:`~urllib3.connectionpool` but that was acccidental and is no longer possible due to a refactoring in urllib3 2.0. Note that the new name of this class is :class:`~urllib3.connection.HTTPSConnection`. It can be used starting from urllib3 1.25.9. AttributeError: 'HTTPResponse' object has no attribute 'strict' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``strict`` parameter is unneeded with Python 3 and should be removed. Pinning urllib3<2 ~~~~~~~~~~~~~~~~~ If the advice from the above sections did not help, you can pin urllib3 to 1.26.x by installing ``urllib3<2``. Please do **not** specify ``urllib3==1.26.15`` to make sure you continue getting 1.26.x updates! While urllib3 1.26.x is still supported, it won't get new features or bug fixes, just security updates. Consider opening a tracking issue to unpin urllib3 in the future to not stay on 1.26.x indefinitely. For more details on the recommended way to handle your dependencies in general, see `Semantic Versioning Will Not Save You `_. The second half even uses urllib3 2.0 as an example! **💪 User-friendly features** ----------------------------- urllib3 has always billed itself as a **user-friendly HTTP client library**. In the spirit of being even more user-friendly we've added two features which should make using urllib3 for tinkering sessions, throw-away scripts, and smaller projects a breeze! urllib3.request() ~~~~~~~~~~~~~~~~~ Previously the highest-level API available for urllib3 was a ``PoolManager``, but for many cases configuring a poolmanager is extra steps for no benefit. To make using urllib3 as simple as possible we've added a top-level function for sending requests from a global poolmanager instance: .. code-block:: python >>> import urllib3 >>> resp = urllib3.request("GET", "https://example.com") >>> resp.status 200 JSON support for requests and responses ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ JSON is everywhere – and now it's in urllib3, too! If you'd like to send JSON in a request body or deserialize a response body from JSON into Python objects you can now use the new ``json=`` parameter for requests and ``HTTPResponse.json()`` method on responses: .. code-block:: python import urllib3 # Send a request with a JSON body. # This adds 'Content-Type: application/json' by default. resp = urllib3.request( "POST", "https://example.api.com", json={"key": "value"} ) # Receive a JSON body in the response. resp = urllib3.request("GET", "https://xkcd.com/2347/info.0.json") # There's always an XKCD... resp.json() { "num": 2347, "img": "https://imgs.xkcd.com/comics/dependency.png", "title": "Dependency", ... } **✨ Optimized for Python 3.7+** -------------------------------- In v2.0 we'll be specifically targeting CPython 3.7+ and PyPy 7.0+ (compatible with CPython 3.7) and dropping support for Python versions 2.7, 3.5, and 3.6. By dropping end-of-life Python versions we're able to optimize the codebase for Python 3.7+ by using new features to improve performance and reduce the amount of code that needs to be executed in order to support legacy versions. **📜 Type-hinted APIs** ----------------------- You're finally able to run Mypy or other type-checkers on code using urllib3. This also means that for IDEs that support type hints you'll receive better suggestions from auto-complete. No more confusion with ``**kwargs``! We've also added API interfaces like ``BaseHTTPResponse`` and ``BaseHTTPConnection`` to ensure that when you're sub-classing an interface you're only using supported public APIs to ensure compatibility and minimize breakages down the road. .. note:: If you're one of the rare few who is subclassing connections or responses you should take a closer look at detailed changes in `the changelog `_. **🔠Modern security by default** --------------------------------- HTTPS requires TLS 1.2+ ~~~~~~~~~~~~~~~~~~~~~~~ Greater than 95% of websites support TLS 1.2 or above. At this point we're comfortable switching the default minimum TLS version to be 1.2 to ensure high security for users without breaking services. Dropping TLS 1.0 and 1.1 by default means you won't be vulnerable to TLS downgrade attacks if a vulnerability in TLS 1.0 or 1.1 were discovered in the future. Extra security for free! By dropping TLS 1.0 and TLS 1.1 we also tighten the list of ciphers we need to support to ensure high security for data traveling over the wire. If you still need to use TLS 1.0 or 1.1 in your application you can still upgrade to v2.0, you'll only need to set ``ssl_minimum_version`` to the proper value to continue using legacy TLS versions. Stop verifying commonName in certificates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dropping support the long deprecated ``commonName`` field on certificates in favor of only verifying ``subjectAltName`` to put us in line with browsers and other HTTP client libraries and to improve security for our users. Certificate verification via SSLContext ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default certificate verification is handled by urllib3 to support legacy Python versions, but now we can rely on Python's certificate verification instead! This should result in a speedup for verifying certificates and means that any improvements made to certificate verification in Python or OpenSSL will be immediately available. urllib3-2.0.7/docs/_static/banner.svg0000644000000000000000000000335214513541732014376 0ustar00 urllib3-2.0.7/docs/_static/banner_github.svg0000644000000000000000000000763714513541732015752 0ustar00 Layer 1 urllib3-2.0.7/docs/_static/dark-logo.svg0000644000000000000000000000343214513541732015007 0ustar00 urllib3-2.0.7/docs/images/demo-button.png0000644000000000000000000001730014513541732015170 0ustar00‰PNG  IHDRŸ‚?£d pHYs  šœ»iTXtXML:com.adobe.xmp í4ë«IDATxœíÝŒåyÀñgfg÷¼MrvL±ã/s‡œÜ¹$Çá0I{X±uˆ`T qThpD7¤U‰ÒHŠ#¡˜F‚?0‰bþˆ¡Á1‘¸¤ü8â^ï(Ê™{!¹Ø€±/õ…¬ow~ôõ:¶ÙyßÙÙÙÝÙïGB²˜Ù™÷vfßyæ÷yÆð Û6|AD Ù($Û Ùͻӭ¤ >O%‘e6 Éó;¹X—„¤«óyƒx@ïC"²Q·’.ø‰¤)è#ºtÁçÇ£i:ÀÇt+è‚Ï #j’o‰n]ðÙQC|Ú·^ê‚O 2Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bCð €Ø| 6Ÿˆ Á'bc5»@#X«6Št½¿ê2{|{Ì­AP©¾ubœaÕeî[¯Š›‹¹EâB¿Ý9>[TvóîH¶ã›çà âýß›âLíiZ;T¼¹ãròá[Cm#Õ·NÌ .«X»nzpDDDìÉQñN¼%öþ¡ö]a憤kýßå…mêÞö‚/~_Œî…U—•öíRvÌ­| +Áfå˜è•¿_{rT¼ùw_º¾ð1—®¨¹}aèŽK£˜Kú¥ë†o)×qòR|➘Zä/Šs³r.D}sÒ ¿› mˆ¢5zr²àæïZ×=<-ó}5ÔþDêì·ìïè¡Èúm4Ág™‹–‰¹h™ˆˆxko{âÉDÝAZ«7‰5p­™líŸ=ÕáYŸ¸>qßK«³Vmk`o@­ýü©c—ûÀ^qþ÷?Å=2eÛVê#ŸÓ¯“£''Þl>†5ÖÙ˱'GÅ™~®£Î£{¡˜¹¡PÁwê²OGØ"µPývï‘Þ5ôÛmŽà³ƒ™¬¤G$uÉǤ¸gk[_xÌܤ×üíéÀ:ŒÊ÷bõ]%ÅgâÑn™Kú%ý™Û"9nVï±z׈“ŸûåÑŽ>~F¶GR¯ ´nê²O‹=Þ¾}€«X¬þaqòRzþá¶îçj‘úð§BûVßU¶¦:úmTpÔÌEˤkäbôäšÝ”ºX«7I×ú-‘0"åу®õ[ÄZ½)Òí¢ÌÌ I× ßŠü¸U¤rb,ènȶÛEªw8ðhRÁF3¥r²àæï–çv€ÔÅ«ÄÈöÔõYsIÝO!‚jt¿^{g¤ÛEc|v(£{¡t|£ÙͨYzí5̬sƒ#’¾æî†î£Ó=9É\ûå†îÃ=<]×¼æ$I]2xÝÊ£Ú¤KÝÔ¿g#“sù'ëúljEcoDâè·­þaɬÿ׆îÑ!øì`F÷¶å³Vo 41=’}õ®áN:Bé+¿X×ü®Z”~ùÆn¿Õ™KúkNª²VÆó{j6«wMG ©KW×÷¹¾+#nÉŸÄÙo§rqœ“€9ŸmÈÉOˆ{ôuízæâå’Ê (×±®gòqñ ³ kGR}ëß9«²!kÉ®´ú‡Å}û`¢GÓâ8†FON{Š”³ÃÝ7^¬š(bôä$uÙ§}ÏiûÀ^e‚‰spŸ83¿Ô^«ï*ßGîáéÀÛqßz5ÐzQ©gô*•#ÛS×ï¿Ñ‚œ›Æy”P°z׈wbS$ *qö}µ¨çx¦úÖ5ìÆ°)ýv„ÇCðنܙ——™0—ôKæ³wù^L+jê °jiGF¶GÒkoÑ®çä'¤ôóû”¯3µGœ){ß±†nÓ^¸Òko÷Í©Ä&-Äq uY´ÞÜq™ßõMåwìÍæÏJŽ93[Ö+ÄþÕ”û¨åoL-»Ü÷÷âÌüºe/jõŽ^¥ú¯kÉ¿)è¹Yzª\=!uùÕÚù„éÁßœF´­j=žõŽ–ê4µß畧Ûo'ÝÎ=2)ÅgR®c^pIL­©5t›öμ´o—Ÿ¸'ð¿W˜•ÒS÷JqôåzF&+ÖÇo ÜV¼—¹x¹ryi|gÍ {|»œ|àF±'GÅžx²ã/2Öª¾¿¯XÒ¾]þŸM@⑽§Ì?r‡Ø“£Úu3Ÿ½+†5Oê’^×Èöˆù¡ËÒŽföÛ""™¿þû@ÛDs|v7?&^±à»Ü\|QŒ­©Ñ“ÓÞå†)æíLí‘ÒØå:V­ Ð Œó>¨\fZCéÙû[rÔ.nªD#gêyq&÷]ž¤Ä£Ò³÷kÏF÷ÂDdÀûõéæ¢eb.é´ UuÕ5C§úmséŠÄœ×IDðÙ!¼w~ë»ÌèVÍd}ôóÊåN~"tðaïß)ö½êv0úY7#ógêåu–‡A™Ñ“S&9ÓωW˜'?á»Nêßj@ËšÃÞ¿S;šºüêxÓ@Ρý¾ËÌ‹®´ åM‹bû:­Òo'é¼N‚O´4Ý<¶ÒóG²{ìAå~ÐÂݨ5xS³›ÐÖTsjÝc3§ç7:¯ûo#DÈVdïÛ!ÞÜqß嵌¶*÷·/ùöYA¦R¨nZÜÃÓâx»î¶µR¿¤ó:I>;E—ÿè“7÷NŒ Î\Ò¯œ3ää'"›ëçfÅ™zÞw¹‘ÉJªo]$ûê4^ñÊåVÿ°t}á;|¿u²®õ]æ|áOÿžÚã{¡62YIõ_yÛšÅ+ÌŠ=õœr £ƒ­Ê;9ç;:d*…ê¦Å9¸¯îvµZ¿]oíS4Ág0sCÊ,P÷è1¶&8ÝÅA5’SgZ}±2ο0Òýu ï„þæÆ\ºB2÷KvónI_s·X«7Œ +“sî\OÕ…º–D•v šç*¢O„kΫ¿ô]¦{äìw¼½bAœúÄ-?­Öo·zBm§¢ÔR™Kú%sõ—”ë¸o¬kÛé¡›$=þ‘iaÛ†ªÿßèzŸòsîëÿzßgmïȤxÅ‚ïÅ< «s5úŠ”/6Aê|Vœ•¨0|»ˆˆØ“£âÍ¿[wMÚ¤R•Éqòïù®œéç|k%VE‡-CÔ*¼Â¬¸Çf|o¼ÍþeÝÛŽãw„›oîKUKƒ¥.^%¶OÍOsI¿ï÷âÚê7Öjý¶qÞâH÷‡h|¶!sÙJ±ºÞ¯_/@‘y¯Xˆ¼3ˆŠªÓðŠ…†!ÞÜ11|:e]Ö6ªs¦öˆûWŸõNçJÀ”oî¸ØSÏu|?]ñþj#Lî‘Ie@–úÈç|Šˆ¸Gãû·6ú]æq±§ž«ZÈ]UÃYõBÕhjôÛ‚à³ ¥r5$©ØO¶ìH’¡˜§*óõ—QñN¼#â׉i²¶á¯¸g«t|#’ ¾Ñ½PÒƒ#’ûÀÞrÒA‹žÃ¤š³§º©t¾à|*FËÚ‘.i¦UßîT 畧}ß"”ºtuõàÓ'!È›;.n~,T{è·s>;˜{l¦mk$6*IÊ›W'Ç >Þl^æw}SÜÃÓ‘n×ê]#]·<БõüTͪG§Î+Oû~ÎÈd%ÕÛï{‘hv Bóf󾿫Êë6Ïdæ†|Q뒴¢ßFÁg‡òæŽKqÏÖf7£nªMª¼kG(Þl^æûªG÷ØLdÛ52YéZ¿¥£P37¤EV=:U+"êÚ‰sò÷ÍnA$TÙéçV1P%"©nL¢@¿ ‚Ïä›Ñ¾K»(ïf»Ô¯m«óƒÏ™Ú#óÜ!óþ‹”öí õ&•3u­ßÒöµƒ²VúNytª VÌ¥+ó=ç] \ÞîÜ+œ£¾¿£3³ÚloÍb÷ðt$ׄVë·u¥ÞÐÌùì ^± öÄ“‘=jwòâ}=’mUã8ê»ÌÈd•óµÌܘþaÿÏÿAìý;ß»ÝîEþí ùȨç—5úª¸G&Å=2)öøöòE±ÿ:1ºÞç›DúS7ËücÉI˜©ÆÈö(ç|{Å?Šµz“zšŒäÔŠ«‘xd.öÏhW¡×iæ羚0+ΡýU_iyfÕë4ÃÔö<«-!úíz)ûí¥Þ?‚ÏQÛQ5Ø Ãy9òmžÉ›W¹Ü/“SDÄ{óe±®®^‚¤Â}ó•³.°ºâÈÚz¨'O¨—/ø€H®2QgþumS¤ñÇ0(¯0{ú¦¨ôìý""§(«ïªÀ‰JæÒbôäZ~T? ]1xsѲP•DÊ )•ãЮŒlº¾ñ;¿©{Û­ò»9“óê/}ߧn^t…¸G&Å\¶²êò°µ=ÏÚVˆ~»º~[ £yxìÞ†Jc;¤°mÃYÿéÞq+J3µ÷•ËU5½Â¬”ÆÕ‡Ìgï:{{Šò#"úz¨ž&ø4>P_@ {=œw²þೕÙãÛÅß.'¾U Û6~DŸZþñZ×kzpDÙ‘¹ù1±'ý'Ó«îœEÊ#A“ Tó‹ŒLV;í\ÖªêzŽ_¨i{Íbæ†Äê–®õ[$}ÍÝ¡FÔÌÅËÕ+„HÀje©þë´çjÔ¬kcÝ_Xéµwjß¹^Ëï¹ÕšµÞˆÚžQôÛ*©¾uÚã잎|DÑ!Û=a¼Ù¼”öíò}ÝšˆHú3·Éü#wÄרì±Ë£†Š ozpDÌÅË¥ôóûª>f)={¿¤.¾¢®ùrºÄ¥39S{Ä[{‹o[Óƒ#¾ežÎeæ†ôl›Ì[Ë\ý¥Óÿ¶z׈ջFœü„8¯×”ùj­Ú¨}µ¬óú¯êmfK;³^ã¹ÜÃÓ2ÿØWëÚ®™’®õէ䙬¤úÖEšÜÖª’ºüê@YþÅŸÝC‹šÇ90*Ö'®t£UmÏj¢è·Ïed{ĺM;â)¢ˆAs|&=¾]Yž¦òø=l½OsÙJ±"Jîp^yÚ·ô ³Rzö’¾]¹Tn@R÷±ìïè¡÷xÅŸÝ']7|«¶vå'j¾ð:SÏ+û¥‡nsÙJßr-©¾ubþÅGµ¬{x:ô¼µ8Ž¡ßèm*7P$‡o{rT¼ùwÅ™|¼êE(Õ·NR—®Öž¼˜6“¹¤_X…©ÑXÎ’ö/Kæ÷~ðF rnç](©(íÛÉ\ϸú¾z¨j~¾gßÕöôkGý¶È©>ñ‚K×.íÛ•È~ I>ªøÌC¾£"å;N÷Cuħƒ‡¸o½ªì,œ©=R:ÿBåˆn…Õ»F¤wvÔPÇ›;.¥Ÿ×>Jb¿ôSIõ]©¼ã¯|waÚXzñ'uöÜvD¡Ú14²=b}âzíg+• ÇW%Šï¤¥>ò9åò°5C/ú^Ø+ÙÉq_Ì£<7EDì{#{ÁFœ}_=T5?+¢¬íéÛŽ&ôÛQg4s>ÊÍiçÜœ[ç²ÕÙãÛõõL#âÍ/¿‚´ŽLIo6/öÄ“ hÕŸØö¶Å|&](Ù“£mñÔJ—hää'Bgô:ÓꤓvL<:“}`¯”žº·Ù͈ªægE\™àqöÛN~¢£Žs;cä3Átsn*u.ÛéM&¥§îïĦÐ#dÚýŒï 5aosñòHGn*Üc3b=ùvÁ;ñ–xÅBÃP÷ðt[ǵP½QDÄym<ô>Ü#“â›ñ}´oõ]Õ¶£Ix»[;8ùð­ÍnÂiqôÛöähbû€$bä3Á‚Ôþ´ú‡Û*û]äT±ñ'¶6´ÌPzõƺÊ"©øÄ=âä'¢iÐ)î±)þøŸÛ¦v½§œ|àÆÀo'ªk“£u'Û´U±w¯Xˆl>¦ªr‚ѽ0’2Yqròrò‡_éÈÀ³5ªßöæŽËü[ <Û ÁgÂéjŠ´ßãw‘òc¥ùGîhXPct/k0ÜÜ#‘rZÛIKûvÉü#w´Mày&{|{9ÛY@î›IüEÇ\Ò/æÒ¾ËużkáL>®\n­ –ìÑlö䨜üáW¤øÄ=$´˜(ûm¯XÒ¾]ròá[9Ý&éxìÞŠ¿øž,¸ù»¾Ëî…’¾æ+Syÿw­Ù"å‘ã¼ú?jì÷탡G–ìý;ÅÞ¿S¬Õ›j~ï›çà mûÈó\öþ"§F¢¬Õ›Äèz_MÇLäTÆò[¯vÄG›h¤™«Y ¯0+N~Â÷ülVâ‘N¥RB§œI¦ßVeÅ£}žçù.,lÛà¿haÖª¾ïø>33zrÒuã½¾sê¼b¡ü>ñˆ/¸F¶GYp>êò+íÀÌ ù¿J4`mTí+h¿Ö—ݼÛP-'øDÇKõ­SÖ¢sÍ´UQ~šI|2çϙڣ|ý¦¹h™¤×Þc‹H.‚O@ʯßTeaZýÃ’ê[c‹H&Ž€SJÿõ ˜]á»Ü8ÿÂ[@2|§¸G&#yï3ðÇcwĆà±!ø@l>‚OĆà±!ø@l>‚OĆà±!ø@l>‚OĆà±!ø@l>‚OĆà±!ø@l>‚OĆà±!ø@ltÁç\,­@u+è‚Ï#5ɧuÁçK5É÷‚n]ð¹+šv ìÒ­  >‘™Hš€$û”cG%eð™Ý¼»("ÿU‹Xwe7ïž×­¤-µ”ݼûQÙI“D[OÅŒZAë|~]Dþ½þö ¡¾'åX1Ãó¼À[.lÛp½ˆÜ+"—ÖÞ.$Èk"òõìæÝÕò¡š‚O‘¶ ¹AD®‘+DäC"’®i#h7%)'½("‹È£§òƒjRsðy®Â¶ ¡>€ö’ݼ»îφ>€ ‚&¡| 6Ÿˆ Á'bóÿBÏtðû'7IEND®B`‚urllib3-2.0.7/docs/images/favicon.png0000644000000000000000000000131214513541732014354 0ustar00‰PNG  IHDRóÿa„iCCPICC profile(‘}‘=HÃP…OÓJU*vqÈP,ˆŠ8jŠP!Ô ­:˜¼ôš4$).Ž‚kÁÁŸÅªƒ‹³®®‚ øâäè¤è"%Þ—ZÄxáñ>λçðÞ}€Ð¨0Í šn›édBÌæVÅð+zB2³Œ9IJÁ·¾î©ê.γüûþ¬>5o1 Ï2ô‰7ˆ§7mƒó>q”•d•øœx̤ ?r]ñøsÑegFÍLzž8J,;Xé`V25â)☪é”/d=V9oqÖ*5Öº'a$¯¯,sÖ0’XÄ$ˆPPC؈Ӯ“b!Mç ÿë—È¥« FŽT¡Avýàð{¶VarÂKŠ$€®Çù»@³î8ßÇŽÓ<‚ÏÀ•ÞöWÀÌ'éõ¶;ú·‹ë¶¦ì—;Àà“!›²+i …ð~Fß”nÞ5on­sœ>šUê88F‹”½îóîîιýÛÓšß™ÅrPXïbKGDÆ;·(¬u pHYs.#.#x¥?vtIMEä  µô%tEXtCommentCreated with GIMPW¢IDAT8Ë“Ñ Â0 D_‹+V`¦hgá—!€!úË,í À ¨¥æ¤ÄJHÌ}žO§ó%næ'ú‰=Àc”WÈÎëàt9n­4¤‘ã#T SÆ P2(ÍTq¸]ýc…îÌ+ ½–æî¬é¯´ÆLú)mðýy5 ¼ˆ æÍÜBû¹…Å$ê¼äÒuV”+kÍðKmZÿ&±t Ê IEND®B`‚urllib3-2.0.7/docs/images/learn-more-button.png0000644000000000000000000001412214513541732016304 0ustar00‰PNG  IHDRŸ‚?£d pHYs  šœ³iTXtXML:com.adobe.xmp Ö©¤EIDATxœíÝolÇyÇñg÷öŽ¹6L@²6B™€}’«ˆ€\’pJ+ª)õŸüG¨ ®äIaGA¡"… ¨ˆ§/$ !z!”içEäP€Àv"•.„ÚráØtTV Ò0@š¶t6ÂH®e‰ˆØæÄ»ÛÛ¾ èÐÊíìÞîìpïôýìÞîènr÷ãìÌ3–çy˜`¯upó |ÀÂ'Œ!|ÀÂ'Œqâ^ 46¬£hùÑñȯµ-µTΉÈÈõÿ>+"ëD$¹hÿ+"ó"rVDŽŠÈ¿åGÇË^¤¡ðYÞ%"ß‘;½ZÊÛ"ò•üèøOyQ¨ðYΈÈ?‰È?DkZÔ·Eäó£ãn˜“ÃÎù$x ž•Œø•0'Ž|–ƆÿRDžÙ(´¶ÝùÑñç‚NR†Ï닋ΉHƆ õüRDÖ-B ªó¹[žv›ˆì :)(|Žhi n#A'…Ï?ÔÓÜ>tBPøü”¦† õu>Û45­/p×Ë ð hCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„Oã¬u7'g`HÛÇë«N6ܦ>‘JùÑqßc•‰#R=ûÌšÜ[oñŠ\{ú±Ø×±»û¤m÷·”ç¸Å))?ÿµØ÷Z¡ëý©]ž÷Üiñ~õž¸3ǵµÃ[¼"K?Þ/^i!R»¬Ž‚|ì¯ÿÙ÷xil8ÒuÃûÞêè?AÿÎÕjædé¹/ǺŸˆH¦w§Ø·n§oGà¹ÙÁ©ÎžïÒymÿŸO{ÿÕI×÷  áhb™» >§Ð/VGA¼…¢…gwõˆÝÕ#""ÞöG¥:õ‚–Ñ.«½S²¾_kàN«½SìÂÔŠ‘¯‘ùôŸhl‘š³e¯8ý÷‹•Ë7þÚM[E6m瞇µõ’ê¿ÀÍ€9Ÿ@“²ò’Y?ê\“A# +——ìàˆ´}î{bub_/Sè_~¤ÛÂ2¿ÿG±^ïônÓÔvaHÚ>÷=ÉŽD ž«­ô‘}á_Å. ij¡ºû/ÐêŸ@“ÊlÚúÝDÐÐÁîꑶ‘¯kùwîy¸¥ƒ@fý€XùŽH¯µ»ûÄjïÔÜ¢r¶ì•¶‡žøptP«½SÚzB²Û×z]tö_ •>&•Ù0úÜ•Ç´ÍÀj‘¯Ç¿N./¹OhhQ:Y¹¼Øw|6Òk3“ýc$»ýñçl&ÅéÛ!¹‡¾™è=¢ÐÕVFøšÝÝ'öº ½ÆÙ¼È#-¬öNq¶ì}»«'•#dºdîÜíu½÷jnÉo8[ö†ZP¤C¦Ð/Ùûž4r¯Fèê¿@«bÁз8%µKï¬u3"\e ýbå;"¯#ìûcßr‡d ýÊsœþûÅ>»½Nß©½.ÒŠä´‹ò™fzwÆž©ºvØOÕ*öFVÅ;›¶Šwu¯–Å>ié¿iùž’BøP›#Ñ2OaE¹ÊôíJtEn#ïÝÝ'¹öûÎ=\y¬¬#4f·?*µ÷fR·â_‡F?Ó¨£¥A¬|‡d·?xž[œ’ÊK‡”¡Ì9.îŒHõÔq†ö-¯xWÈŽˆûæ˱?ß´ôß´|ÏIá±;Ðdœ=¾#W^¹$•SÇü_›¢…Gµ‹ÓRþÙ”çØ·nÐr¯Vžÿ™Ùð™ÐçZù±oût"íp†öŽ¨VN“òó_ =蕤òâ)Ÿx*ðÜÜŸý]¨kêb²ÿ­†ð 4ÕB#wæ5q§úOÛ£ZqB¼rÉ÷¸}ËíÚîewõ4õ<<¿÷Éîê»»/Ô5TTŸC«£8:Y9u,ò¨»;s\*G”çØë6ïÛ&û/ÐJŸ@±: Ê…FîÜ«â•Ä-Nùž·>¤nÞ¿ð=fµÿžÖ{eGBµ´qÏŸõ=fß~w¨k(ÿpQ\?ˆó¡<î§bO÷¨ž}Fª³'•ç¬Eß6ÙVAøšˆªX|íò¼Ô.N‹ˆˆûö¤ÿ5bÔ‡l¹ö7å¿¿ö‹×}GÙÂL§PýáR»0'ÞÕ÷#·-hr嵧#_{µêÄ÷•#7{ßšáh"Nÿý¾ÇÜs§ó¿gŽûþH[¹¼dúvio[dm¿ã{È[ü@ûíV¶ßl6ÞµEßÑÉ0Ó)T¸¸çNEn—Ýݧœë駴-ôòJ âμæ{&T"çƹžªéF©$É. )wÀ©]z7ÒuUÓD®o¿Ù„ó?Ý·~î{,葳ßgî•KâΞˆÜ¦ GþªQø(ܹW•ÇM.òIªÿ­ŽRK@²CŸ—ìÐçc_§46ÜðkT%rÜâÔo­ vç^õ­“¸²Heå1ýZ°»û$÷Ç_TžS{ÿ\¤kWßXSªZŒÙÁ©½{fM߃FÕŠâ-~±nyŸÌú©úÔü´»û|C’{þl¬ZªVÛï*×Þù¯È×®{½‹Óâ•K¾ˆYŸ¸Eëýü$Ù×ò{0ð 4«£  RõF—j§¥vyÞ7tdîzP{ð²{6‹ÓöñàóBéöÊ¥XÁ¥òÒ!±ÿêróÜûeéÇû-¼¯[uæÕº…ÜUu%U›¨FSÃP…=¯\Jä½õ/‹åÓ¯­OD_䓦þ ´2Â'ÐTóõT?rî¹ÓþáS1RU¦Ðø£Vuê…XmóJ RþCÒ¶û[¾çXíâ í“Ê‹"ßÇ4÷Í—}wÊܹ¥~øôYä-^‘Zq"V{,ÅœGYŠ^¾IÅ»úˆ_øÌ)Ú MýheÌùš€j5³ê±©ûæ˾¯³ryÉlJç~ïµËóZvbª]œVÝYÞž1Ó»3ö½LñŠR»0W÷ØÊv›«Ù…!ßGÔÕõüɸ’Zpã-ý:‘ëꢫÿ­Šð ¤œ]R>:V=6UuÝǵâ-^‘òñƒÚ®W<¸)»ýQ±: Úî™4Õêô+¨"©þ8Ñ!©:—ÊÑÖ5¦»ÿ­ˆð ¤œ³Ùt2ÌcSUP±×mLUÑõÚåyY:ö í{°W^:$ÞâßãͶý¦;{·”ÖêUíV¾C2ëêžW»0§å}VŽB¶©·ÛŒJ5¯Ó+¯Ý¨hRýh5Ìù৤véc÷³òÊ9h^ù×%ƒ‚V#g6n[óß^¹$Õ©{Tfþ§ÝÕ#ÙíKõõO¤ :y¥qÏŸ­»¥åêJªí4ãÔöüH[®^ò=fåòbižW,"bµw)Úc¾¶¦îþkú{0ð 4 6ÿ†TÏ>cì~AÅàí®eÁP÷è½W*¯|'Ö5â¨L1òž®Ìÿô[¬#"âôíoéÿo‹î[?÷ÝOݾýn©]œ»gsÝãqk{~äZï—ß ü¨‚ŠÚ«Âp’迦¿gÓŸ@Š…Ù61.+—g`–»z?ÄÙûžô I""¢´.ÕÉÃerTá4MT5?ÞmâNõýwÆ­íù‘v¼{FDñžù­ÀJU6J$zmM‘ô÷_ U0çH© …F:%¹ð(h?îìàˆÑÅ>Aó?›‰ßjõ -DãÖö\m¥è»ŸL¡_Ûçkå;”ûÈ'Q[3mýh„O ¥‚¶KÔÉ^·1±P¯´ Õÿþ‰ò“‹}¼Ò‚T&[㑦jµºß¨§ŽÚž¿ÕÅV®""Ù{¿ å>ÎÐ>õ³GtW¤­ÿ­€ð ¤j•rRT…ì㪞}FYòÉîê1º×º;s\ªÓzæ<®¥ RZõ$QÛ3h‘V¦Ðûóuö¨‹ÞÝÕÒÖfGøR(Ó·K9“§ÿþD¯_þÏï*gGŒ–}ª¼ò©]ž7v¿¤4ºj=‰ÚžÞBQª³'•çdG"´LïÎÀ½Îkæ´è®–¶þ 43)´ºVãjædé¹/Gº®]’¶‡ê?"´ryÉôîÔº8d5o¡¸Ú<û§ûdéG_Jäþõ””¶Gú:¹³'ĹçáPÿ]µ=ë©N|_2ë”íÈŽˆ}ËËónC<·òâ í ñ ‡q¥±ÿÍŠð‰¦c÷l'æ SïWïE Y:î½Â}óåºAÀîîS–OŠSŸQµBZDÿÊäU'‹Ó»Í÷þ+/MmMè-¥òÊ%·ãoÜ/ ªšŸ7ÒUÛÓ¯aÞËL¡_2óC©ÎžïÒùºU2½;žuƒ8}ᶭœ:f¤°»©þkâ{XK„O4L¡_Y*'ŒêìÉH!KǽWÔþç­º? ™»T¾.n}F÷üßõ••ÉIþX•ößÑW‘åѱڻgŒ¾wgŽKµ “FªšŸ+tÖöômÇÌq©|òS¡ÊU9›¶ŠlÚø8=Huö¤Ñ}ÔMô_ß3ÀZbÎ'"A ÜâTìÕ¼îœzÁI’ D–G_ƒææð/”„fŸÿ¹<¢­.•ÄJðzª“‡?_]Üâ”T^<`ä^+ÒØfÃÈ'"ªíEDÜ·'cߣvqZj—ç}í;½ÛI šhµwJvûãFw^jöùŸ×ž~l­›ð¡Ê‹Ä»º7Ñ‚ýÕék¶3Wû/ÐLùRDUìÝ+—´ÍÇtÏö=fµwŠ]Òr?aj':};Œ®^™ÿ =ª“‡eéùƒÚG”½Å+²ôüÁ5 viì¿@3!|)aw÷‰½n£ïñ BÞp§*;›“ŸÿT;QÄüãËV©ÿ™µâ„,ýèKË ‚»…á•KR9uL®=ýX¢%•ÂJcÿšáH‰À…Fs5á•Ä-Nù·Eã–ˆ*Aåq¬öNÉÞ÷dâíX­Ùç¦Quò°\{ê)Ÿxªáp_=)•‰#rí©GŒ., #ýh–çy¾KcÃþˆÁØ#âSR(mA@xùÑqKuœG€5Q¯Æ'€ÖÇcwCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„OCø€1„O>´­ tBPø¼¨©!h}Ù1(|¾®©!h}§ƒN ŸÇô´7cA'…ÏgEd^KSÐÊ~)ËÙQI>ó£ãeù{]-@ËÚŸ_ :)°ÔR~tüY9¨¥IhE¯gÆ@aë|~UDþ%z{Т¾+ËY1Ëó¼ÐW. ?,"DäÎÆÛ€ò¶ˆ|5?:þ\#/j(|Šˆ”Ɔs"²[Dv‰ÈÝ"r›ˆdºšME–‘£"òìõõA i8|Þ¨46ëõh.ùÑñȯ>€°Â.8b#|ÀÂ'Œ!|À˜ÿ^/¬+¾–IEND®B`‚urllib3-2.0.7/docs/images/logo.png0000644000000000000000000000416514513541732013700 0ustar00‰PNG  IHDR«§ ÓxsBIT|dˆtEXtSoftwaregnome-screenshotï¿>IDATxœíØ1«ÖeÇáû''Ó E0ƒ’œjHih\œZZóE„ƒÐ4EÐAS´åAKSÐP ‘ Eæ©ǧWÃóðÿ¨×õ~|·Ïͽ2›ÍfÂV§óˆyb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰ykË:¼ó÷ͱ³usYçYõ#Ǧž0fÛ·Æìö©g0Çêæá1VwMºaûîqcëI70ß¾ÍýcÏúÞ…Þ\Z¬~ÿä£ñÛ‡ï,ë< rêëkSOÛ×¾·®\žzsì{í³±ºïȤ~üù»qùƒ7&ÝÀ|/\gNŸ[èM߀ä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX·6õ€ûuôÐÓãâ…KSÏ`ŽgŸZøM±ögNŸ›zð @žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä­M=à~ýtý‡ññçoO=ƒ9Ο}}<⥅Þ+àñÏ­­ñýÕo§žÁgOŸ[øM߀ä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä‰yb@žX'Vä­-ëð/¿:v9¶¬óõ æ¹·3õa¾È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €¼µe~ìø3ãñ^\Öy"«ÇÚÑÓSÏ`Ž•]ëSO»w¯CO™zsìYß»ð›+³Ùl¶ð«°@¾È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òÄ €<± O¬È+òþTx\¢¬ÌIEND®B`‚urllib3-2.0.7/docs/images/logo.svg0000644000000000000000000000100414513541732013700 0ustar00 urllib3-2.0.7/docs/reference/index.rst0000644000000000000000000000034214513541732014555 0ustar00API Reference ============= .. toctree:: urllib3.request urllib3.poolmanager urllib3.connectionpool urllib3.connection urllib3.exceptions urllib3.response urllib3.fields urllib3.util contrib/index urllib3-2.0.7/docs/reference/urllib3.connection.rst0000644000000000000000000000054014513541732017160 0ustar00Connections =========== .. automodule:: urllib3.connection .. autoclass:: urllib3.connection.HTTPConnection :members: :exclude-members: putrequest :show-inheritance: .. autoclass:: urllib3.connection.HTTPSConnection :members: :show-inheritance: .. autoclass:: urllib3.connection.ProxyConfig :members: :show-inheritance: urllib3-2.0.7/docs/reference/urllib3.connectionpool.rst0000644000000000000000000000070014513541732020050 0ustar00Connection Pools ================ .. automodule:: urllib3.connectionpool .. autoclass:: urllib3.HTTPConnectionPool :members: :undoc-members: :show-inheritance: .. autoclass:: urllib3.HTTPSConnectionPool :members: :undoc-members: :show-inheritance: .. autoclass:: urllib3.connectionpool.ConnectionPool :members: :undoc-members: :show-inheritance: .. autofunction:: urllib3.connectionpool.connection_from_url urllib3-2.0.7/docs/reference/urllib3.exceptions.rst0000644000000000000000000000027114513541732017203 0ustar00Exceptions and Warnings ======================= .. automodule:: urllib3.exceptions :members: :undoc-members: :show-inheritance: .. autofunction:: urllib3.disable_warnings urllib3-2.0.7/docs/reference/urllib3.fields.rst0000644000000000000000000000053614513541732016274 0ustar00Fields and Multipart Forms ========================== Fields ------ .. automodule:: urllib3.fields :members: :undoc-members: :show-inheritance: Multipart Forms --------------- .. autofunction:: urllib3.encode_multipart_formdata .. autofunction:: urllib3.filepost.choose_boundary .. autofunction:: urllib3.filepost.iter_field_objects urllib3-2.0.7/docs/reference/urllib3.poolmanager.rst0000644000000000000000000000052214513541732017325 0ustar00Pool Manager ============ .. autoclass:: urllib3.PoolManager :members: :undoc-members: :show-inheritance: :inherited-members: .. autoclass:: urllib3.ProxyManager :members: :undoc-members: :show-inheritance: .. autoclass:: urllib3.poolmanager.PoolKey :members: :undoc-members: :show-inheritance: urllib3-2.0.7/docs/reference/urllib3.request.rst0000644000000000000000000000010714513541732016510 0ustar00urllib3.request() ================= .. autofunction:: urllib3.request urllib3-2.0.7/docs/reference/urllib3.response.rst0000644000000000000000000000142414513541732016661 0ustar00Response and Decoders ===================== Response -------- .. autoclass:: urllib3.response.BaseHTTPResponse :members: :undoc-members: :show-inheritance: .. autoclass:: urllib3.response.HTTPResponse :members: :undoc-members: :show-inheritance: :inherited-members: json .. autoattribute:: auto_close .. autoattribute:: status .. autoattribute:: headers Decoders -------- Decoder classes are used for transforming compressed HTTP bodies using the ``Content-Encoding`` into their uncompressed binary representation. .. autoclass:: urllib3.response.BrotliDecoder .. autoclass:: urllib3.response.DeflateDecoder .. autoclass:: urllib3.response.GzipDecoder .. autoclass:: urllib3.response.ZstdDecoder .. autoclass:: urllib3.response.MultiDecoder urllib3-2.0.7/docs/reference/urllib3.util.rst0000644000000000000000000000105314513541732015776 0ustar00Utilities ========= Useful methods for working with :mod:`http.client`, completely decoupled from code specific to **urllib3**. At the very core, just like its predecessors, urllib3 is built on top of :mod:`http.client` -- the lowest level HTTP library included in the Python standard library. To aid the limited functionality of the :mod:`http.client` module, urllib3 provides various helper methods which are used with the higher level components but can also be used independently. .. automodule:: urllib3.util :members: :show-inheritance: urllib3-2.0.7/docs/reference/contrib/index.rst0000644000000000000000000000035114513541732016215 0ustar00Third-Party Modules =================== These modules implement various extra features, that may not be ready for prime time or that require optional third-party dependencies. .. toctree:: pyopenssl securetransport socks urllib3-2.0.7/docs/reference/contrib/pyopenssl.rst0000644000000000000000000000017014513541732017141 0ustar00PyOpenSSL ========= .. automodule:: urllib3.contrib.pyopenssl :members: :undoc-members: :show-inheritance: urllib3-2.0.7/docs/reference/contrib/securetransport.rst0000644000000000000000000000262714513541732020361 0ustar00macOS SecureTransport ===================== .. warning:: DEPRECATED: This module is deprecated and will be removed in urllib3 v2.1.0. Read more in this `issue `_. `SecureTranport `_ support for urllib3 via ctypes. This makes platform-native TLS available to urllib3 users on macOS without the use of a compiler. This is an important feature because the Python Package Index is moving to become a TLSv1.2-or-higher server, and the default OpenSSL that ships with macOS is not capable of doing TLSv1.2. The only way to resolve this is to give macOS users an alternative solution to the problem, and that solution is to use SecureTransport. We use ctypes here because this solution must not require a compiler. That's because Pip is not allowed to require a compiler either. This code is a bastardised version of the code found in Will Bond's `oscrypto `_ library. An enormous debt is owed to him for blazing this trail for us. For that reason, this code should be considered to be covered both by urllib3's license and by `oscrypto's `_. To use this module, simply import and inject it: .. code-block:: python import urllib3.contrib.securetransport urllib3.contrib.securetransport.inject_into_urllib3() urllib3-2.0.7/docs/reference/contrib/socks.rst0000644000000000000000000000017414513541732016233 0ustar00SOCKS Proxies ============= .. automodule:: urllib3.contrib.socks :members: :undoc-members: :show-inheritance: urllib3-2.0.7/dummyserver/__init__.py0000644000000000000000000000000014513541732014510 0ustar00urllib3-2.0.7/dummyserver/handlers.py0000644000000000000000000003214014513541732014563 0ustar00from __future__ import annotations import collections import contextlib import gzip import json import logging import sys import typing import zlib from datetime import datetime, timedelta, timezone from http.client import responses from io import BytesIO from urllib.parse import urlsplit from tornado import httputil from tornado.web import RequestHandler from urllib3.util.util import to_str log = logging.getLogger(__name__) class Response: def __init__( self, body: str | bytes | typing.Sequence[str | bytes] = "", status: str = "200 OK", headers: typing.Sequence[tuple[str, str | bytes]] | None = None, json: typing.Any | None = None, ) -> None: self.body = body self.status = status if json is not None: self.headers = headers or [("Content-type", "application/json")] self.body = json else: self.headers = headers or [("Content-type", "text/plain")] def __call__(self, request_handler: RequestHandler) -> None: status, reason = self.status.split(" ", 1) request_handler.set_status(int(status), reason) for header, value in self.headers: request_handler.add_header(header, value) if isinstance(self.body, str): request_handler.write(self.body.encode()) elif isinstance(self.body, bytes): request_handler.write(self.body) # chunked else: for item in self.body: if not isinstance(item, bytes): item = item.encode("utf8") request_handler.write(item) request_handler.flush() RETRY_TEST_NAMES: dict[str, int] = collections.defaultdict(int) def request_params(request: httputil.HTTPServerRequest) -> dict[str, bytes]: params = {} for k, v in request.arguments.items(): params[k] = next(iter(v)) return params class TestingApp(RequestHandler): """ Simple app that performs various operations, useful for testing an HTTP library. Given any path, it will attempt to load a corresponding local method if it exists. Status code 200 indicates success, 400 indicates failure. Each method has its own conditions for success/failure. """ def get(self) -> None: """Handle GET requests""" self._call_method() def post(self) -> None: """Handle POST requests""" self._call_method() def put(self) -> None: """Handle PUT requests""" self._call_method() def options(self) -> None: """Handle OPTIONS requests""" self._call_method() def head(self) -> None: """Handle HEAD requests""" self._call_method() def _call_method(self) -> None: """Call the correct method in this class based on the incoming URI""" req = self.request path = req.path[:] if not path.startswith("/"): path = urlsplit(path).path target = path[1:].split("/", 1)[0] method = getattr(self, target, self.index) resp = method(req) if dict(resp.headers).get("Connection") == "close": # FIXME: Can we kill the connection somehow? pass resp(self) def index(self, _request: httputil.HTTPServerRequest) -> Response: "Render simple message" return Response("Dummy server!") def certificate(self, request: httputil.HTTPServerRequest) -> Response: """Return the requester's certificate.""" cert = request.get_ssl_certificate() assert isinstance(cert, dict) subject = {} if cert is not None: subject = {k: v for (k, v) in [y for z in cert["subject"] for y in z]} return Response(json.dumps(subject)) def alpn_protocol(self, request: httputil.HTTPServerRequest) -> Response: """Return the selected ALPN protocol.""" assert request.connection is not None proto = request.connection.stream.socket.selected_alpn_protocol() # type: ignore[attr-defined] return Response(proto.encode("utf8") if proto is not None else "") def source_address(self, request: httputil.HTTPServerRequest) -> Response: """Return the requester's IP address.""" return Response(request.remote_ip) # type: ignore[arg-type] def set_up(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) test_type = params.get("test_type") test_id = params.get("test_id") if test_id: print(f"\nNew test {test_type!r}: {test_id!r}") else: print(f"\nNew test {test_type!r}") return Response("Dummy server is ready!") def specific_method(self, request: httputil.HTTPServerRequest) -> Response: "Confirm that the request matches the desired method type" params = request_params(request) method = params.get("method") method_str = method.decode() if method else None if request.method != method_str: return Response( f"Wrong method: {method_str} != {request.method}", status="400 Bad Request", ) return Response() def upload(self, request: httputil.HTTPServerRequest) -> Response: "Confirm that the uploaded file conforms to specification" params = request_params(request) # FIXME: This is a huge broken mess param = params.get("upload_param", b"myfile").decode("ascii") filename = params.get("upload_filename", b"").decode("utf-8") size = int(params.get("upload_size", "0")) files_ = request.files.get(param) assert files_ is not None if len(files_) != 1: return Response( f"Expected 1 file for '{param}', not {len(files_)}", status="400 Bad Request", ) file_ = files_[0] data = file_["body"] if int(size) != len(data): return Response( f"Wrong size: {int(size)} != {len(data)}", status="400 Bad Request" ) got_filename = file_["filename"] if isinstance(got_filename, bytes): got_filename = got_filename.decode("utf-8") # Tornado can leave the trailing \n in place on the filename. if filename != got_filename: return Response( f"Wrong filename: {filename} != {file_.filename}", status="400 Bad Request", ) return Response() def redirect(self, request: httputil.HTTPServerRequest) -> Response: # type: ignore[override] "Perform a redirect to ``target``" params = request_params(request) target = params.get("target", "/") status = params.get("status", b"303 See Other").decode("latin-1") if len(status) == 3: status = f"{status} Redirect" headers = [("Location", target)] return Response(status=status, headers=headers) def not_found(self, request: httputil.HTTPServerRequest) -> Response: return Response("Not found", status="404 Not Found") def multi_redirect(self, request: httputil.HTTPServerRequest) -> Response: "Performs a redirect chain based on ``redirect_codes``" params = request_params(request) codes = params.get("redirect_codes", b"200").decode("utf-8") head, tail = codes.split(",", 1) if "," in codes else (codes, None) assert head is not None status = f"{head} {responses[int(head)]}" if not tail: return Response("Done redirecting", status=status) headers = [("Location", f"/multi_redirect?redirect_codes={tail}")] return Response(status=status, headers=headers) def keepalive(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) if params.get("close", b"0") == b"1": headers = [("Connection", "close")] return Response("Closing", headers=headers) headers = [("Connection", "keep-alive")] return Response("Keeping alive", headers=headers) def echo_params(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) echod = sorted((to_str(k), to_str(v)) for k, v in params.items()) return Response(repr(echod)) def echo(self, request: httputil.HTTPServerRequest) -> Response: "Echo back the params" if request.method == "GET": return Response(request.query) return Response(request.body) def echo_json(self, request: httputil.HTTPServerRequest) -> Response: "Echo back the JSON" return Response(json=request.body, headers=list(request.headers.items())) def echo_uri(self, request: httputil.HTTPServerRequest) -> Response: "Echo back the requested URI" assert request.uri is not None return Response(request.uri) def encodingrequest(self, request: httputil.HTTPServerRequest) -> Response: "Check for UA accepting gzip/deflate encoding" data = b"hello, world!" encoding = request.headers.get("Accept-Encoding", "") headers = None if encoding == "gzip": headers = [("Content-Encoding", "gzip")] file_ = BytesIO() with contextlib.closing( gzip.GzipFile("", mode="w", fileobj=file_) ) as zipfile: zipfile.write(data) data = file_.getvalue() elif encoding == "deflate": headers = [("Content-Encoding", "deflate")] data = zlib.compress(data) elif encoding == "garbage-gzip": headers = [("Content-Encoding", "gzip")] data = b"garbage" elif encoding == "garbage-deflate": headers = [("Content-Encoding", "deflate")] data = b"garbage" return Response(data, headers=headers) def headers(self, request: httputil.HTTPServerRequest) -> Response: return Response(json.dumps(dict(request.headers))) def headers_and_params(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) return Response( json.dumps({"headers": dict(request.headers), "params": params}) ) def multi_headers(self, request: httputil.HTTPServerRequest) -> Response: return Response(json.dumps({"headers": list(request.headers.get_all())})) def successful_retry(self, request: httputil.HTTPServerRequest) -> Response: """Handler which will return an error and then success It's not currently very flexible as the number of retries is hard-coded. """ test_name = request.headers.get("test-name", None) if not test_name: return Response("test-name header not set", status="400 Bad Request") RETRY_TEST_NAMES[test_name] += 1 if RETRY_TEST_NAMES[test_name] >= 2: return Response("Retry successful!") else: return Response("need to keep retrying!", status="418 I'm A Teapot") def chunked(self, request: httputil.HTTPServerRequest) -> Response: return Response(["123"] * 4) def chunked_gzip(self, request: httputil.HTTPServerRequest) -> Response: chunks = [] compressor = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) for uncompressed in [b"123"] * 4: chunks.append(compressor.compress(uncompressed)) chunks.append(compressor.flush()) return Response(chunks, headers=[("Content-Encoding", "gzip")]) def nbytes(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) length = int(params["length"]) data = b"1" * length return Response(data, headers=[("Content-Type", "application/octet-stream")]) def status(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) status = params.get("status", b"200 OK").decode("latin-1") return Response(status=status) def retry_after(self, request: httputil.HTTPServerRequest) -> Response: params = request_params(request) if datetime.now() - self.application.last_req < timedelta(seconds=1): # type: ignore[attr-defined] status = params.get("status", b"429 Too Many Requests") return Response( status=status.decode("utf-8"), headers=[("Retry-After", "1")] ) self.application.last_req = datetime.now() # type: ignore[attr-defined] return Response(status="200 OK") def redirect_after(self, request: httputil.HTTPServerRequest) -> Response: "Perform a redirect to ``target``" params = request_params(request) date = params.get("date") if date: retry_after = str( httputil.format_timestamp( datetime.fromtimestamp(float(date), tz=timezone.utc) ) ) else: retry_after = "1" target = params.get("target", "/") headers = [("Location", target), ("Retry-After", retry_after)] return Response(status="303 See Other", headers=headers) def shutdown(self, request: httputil.HTTPServerRequest) -> typing.NoReturn: sys.exit() urllib3-2.0.7/dummyserver/https_proxy.py0000755000000000000000000000225614513541732015376 0ustar00#!/usr/bin/env python from __future__ import annotations import sys import typing import tornado.httpserver import tornado.ioloop import tornado.web from dummyserver.proxy import ProxyHandler from dummyserver.server import DEFAULT_CERTS, ssl_options_to_context def run_proxy(port: int, certs: dict[str, typing.Any] = DEFAULT_CERTS) -> None: """ Run proxy on the specified port using the provided certs. Example usage: python -m dummyserver.https_proxy You'll need to ensure you have access to certain packages such as trustme, tornado, urllib3. """ upstream_ca_certs = certs.get("ca_certs") app = tornado.web.Application( [(r".*", ProxyHandler)], upstream_ca_certs=upstream_ca_certs ) ssl_opts = ssl_options_to_context(**certs) http_server = tornado.httpserver.HTTPServer(app, ssl_options=ssl_opts) http_server.listen(port) ioloop = tornado.ioloop.IOLoop.instance() try: ioloop.start() except KeyboardInterrupt: ioloop.stop() if __name__ == "__main__": port = 8443 if len(sys.argv) > 1: port = int(sys.argv[1]) print(f"Starting HTTPS proxy on port {port}") run_proxy(port) urllib3-2.0.7/dummyserver/proxy.py0000755000000000000000000001074614513541732014157 0ustar00#!/usr/bin/env python # # Simple asynchronous HTTP proxy with tunnelling (CONNECT). # # GET/POST proxying based on # http://groups.google.com/group/python-tornado/msg/7bea08e7a049cf26 # # Copyright (C) 2012 Senko Rasic # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN # THE SOFTWARE. from __future__ import annotations import socket import ssl import sys import tornado.gen import tornado.httpclient import tornado.httpserver import tornado.ioloop import tornado.iostream import tornado.web __all__ = ["ProxyHandler", "run_proxy"] class ProxyHandler(tornado.web.RequestHandler): SUPPORTED_METHODS = ["GET", "POST", "CONNECT"] # type: ignore[assignment] async def get(self) -> None: upstream_ca_certs = self.application.settings.get("upstream_ca_certs", None) ssl_options = None if upstream_ca_certs: ssl_options = ssl.create_default_context(cafile=upstream_ca_certs) assert self.request.uri is not None assert self.request.method is not None req = tornado.httpclient.HTTPRequest( url=self.request.uri, method=self.request.method, body=self.request.body, headers=self.request.headers, follow_redirects=False, allow_nonstandard_methods=True, ssl_options=ssl_options, ) client = tornado.httpclient.AsyncHTTPClient() response = await client.fetch(req, raise_error=False) self.set_status(response.code) for header in ( "Date", "Cache-Control", "Server", "Content-Type", "Location", ): v = response.headers.get(header) if v: self.set_header(header, v) if response.body: self.write(response.body) await self.finish() async def post(self) -> None: await self.get() async def connect(self) -> None: assert self.request.uri is not None host, port = self.request.uri.split(":") assert self.request.connection is not None client: tornado.iostream.IOStream = self.request.connection.stream # type: ignore[attr-defined] async def start_forward( reader: tornado.iostream.IOStream, writer: tornado.iostream.IOStream ) -> None: while True: try: data = await reader.read_bytes(4096, partial=True) except tornado.iostream.StreamClosedError: break if not data: break writer.write(data) writer.close() s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) upstream = tornado.iostream.IOStream(s) await upstream.connect((host, int(port))) client.write(b"HTTP/1.0 200 Connection established\r\n\r\n") fu1 = start_forward(client, upstream) fu2 = start_forward(upstream, client) await tornado.gen.multi([fu1, fu2]) def run_proxy(port: int, start_ioloop: bool = True) -> None: """ Run proxy on the specified port. If start_ioloop is True (default), the tornado IOLoop will be started immediately. """ app = tornado.web.Application([(r".*", ProxyHandler)]) app.listen(port) ioloop = tornado.ioloop.IOLoop.instance() if start_ioloop: ioloop.start() if __name__ == "__main__": port = 8888 if len(sys.argv) > 1: port = int(sys.argv[1]) print(f"Starting HTTP proxy on port {port}") run_proxy(port) urllib3-2.0.7/dummyserver/server.py0000755000000000000000000002315714513541732014304 0ustar00#!/usr/bin/env python """ Dummy server used for unit testing. """ from __future__ import annotations import asyncio import concurrent.futures import contextlib import errno import logging import os import socket import ssl import sys import threading import typing import warnings from collections.abc import Coroutine, Generator from datetime import datetime import tornado.httpserver import tornado.ioloop import tornado.netutil import tornado.web import trustme from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives import serialization from urllib3.exceptions import HTTPWarning from urllib3.util import ALPN_PROTOCOLS, resolve_cert_reqs, resolve_ssl_version if typing.TYPE_CHECKING: from typing_extensions import ParamSpec P = ParamSpec("P") log = logging.getLogger(__name__) CERTS_PATH = os.path.join(os.path.dirname(__file__), "certs") DEFAULT_CERTS: dict[str, typing.Any] = { "certfile": os.path.join(CERTS_PATH, "server.crt"), "keyfile": os.path.join(CERTS_PATH, "server.key"), "cert_reqs": ssl.CERT_OPTIONAL, "ca_certs": os.path.join(CERTS_PATH, "cacert.pem"), "alpn_protocols": ALPN_PROTOCOLS, } DEFAULT_CA = os.path.join(CERTS_PATH, "cacert.pem") DEFAULT_CA_KEY = os.path.join(CERTS_PATH, "cacert.key") def _resolves_to_ipv6(host: str) -> bool: """Returns True if the system resolves host to an IPv6 address by default.""" resolves_to_ipv6 = False try: for res in socket.getaddrinfo(host, None, socket.AF_UNSPEC): af, _, _, _, _ = res if af == socket.AF_INET6: resolves_to_ipv6 = True except socket.gaierror: pass return resolves_to_ipv6 def _has_ipv6(host: str) -> bool: """Returns True if the system can bind an IPv6 address.""" sock = None has_ipv6 = False if socket.has_ipv6: # has_ipv6 returns true if cPython was compiled with IPv6 support. # It does not tell us if the system has IPv6 support enabled. To # determine that we must bind to an IPv6 address. # https://github.com/urllib3/urllib3/pull/611 # https://bugs.python.org/issue658327 try: sock = socket.socket(socket.AF_INET6) sock.bind((host, 0)) has_ipv6 = _resolves_to_ipv6("localhost") except Exception: pass if sock: sock.close() return has_ipv6 # Some systems may have IPv6 support but DNS may not be configured # properly. We can not count that localhost will resolve to ::1 on all # systems. See https://github.com/urllib3/urllib3/pull/611 and # https://bugs.python.org/issue18792 HAS_IPV6_AND_DNS = _has_ipv6("localhost") HAS_IPV6 = _has_ipv6("::1") # Different types of servers we have: class NoIPv6Warning(HTTPWarning): "IPv6 is not available" class SocketServerThread(threading.Thread): """ :param socket_handler: Callable which receives a socket argument for one request. :param ready_event: Event which gets set when the socket handler is ready to receive requests. """ USE_IPV6 = HAS_IPV6_AND_DNS def __init__( self, socket_handler: typing.Callable[[socket.socket], None], host: str = "localhost", ready_event: threading.Event | None = None, ) -> None: super().__init__() self.daemon = True self.socket_handler = socket_handler self.host = host self.ready_event = ready_event def _start_server(self) -> None: if self.USE_IPV6: sock = socket.socket(socket.AF_INET6) else: warnings.warn("No IPv6 support. Falling back to IPv4.", NoIPv6Warning) sock = socket.socket(socket.AF_INET) if sys.platform != "win32": sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.bind((self.host, 0)) self.port = sock.getsockname()[1] # Once listen() returns, the server socket is ready sock.listen(1) if self.ready_event: self.ready_event.set() self.socket_handler(sock) sock.close() def run(self) -> None: self._start_server() def ssl_options_to_context( # type: ignore[no-untyped-def] keyfile=None, certfile=None, server_side=None, cert_reqs=None, ssl_version: str | int | None = None, ca_certs=None, do_handshake_on_connect=None, suppress_ragged_eofs=None, ciphers=None, alpn_protocols=None, ) -> ssl.SSLContext: """Return an equivalent SSLContext based on ssl.wrap_socket args.""" ssl_version = resolve_ssl_version(ssl_version) cert_none = resolve_cert_reqs("CERT_NONE") if cert_reqs is None: cert_reqs = cert_none else: cert_reqs = resolve_cert_reqs(cert_reqs) ctx = ssl.SSLContext(ssl_version) ctx.load_cert_chain(certfile, keyfile) ctx.verify_mode = cert_reqs if ctx.verify_mode != cert_none: ctx.load_verify_locations(cafile=ca_certs) if alpn_protocols and hasattr(ctx, "set_alpn_protocols"): try: ctx.set_alpn_protocols(alpn_protocols) except NotImplementedError: pass return ctx def run_tornado_app( app: tornado.web.Application, certs: dict[str, typing.Any] | None, scheme: str, host: str, ) -> tuple[tornado.httpserver.HTTPServer, int]: # We can't use fromtimestamp(0) because of CPython issue 29097, so we'll # just construct the datetime object directly. app.last_req = datetime(1970, 1, 1) # type: ignore[attr-defined] if scheme == "https": assert certs is not None ssl_opts = ssl_options_to_context(**certs) http_server = tornado.httpserver.HTTPServer(app, ssl_options=ssl_opts) else: http_server = tornado.httpserver.HTTPServer(app) # When we request a socket with host localhost and port zero (None in Python), then # Tornado gets a free IPv4 port and requests that same port in IPv6. But that port # could easily be taken with IPv6, especially in crowded CI environments. For this # reason we put bind_sockets in a retry loop. Full details: # * https://github.com/urllib3/urllib3/issues/2171 # * https://github.com/tornadoweb/tornado/issues/1860 for i in range(10): try: sockets = tornado.netutil.bind_sockets(None, address=host) # type: ignore[arg-type] except OSError as e: if e.errno == errno.EADDRINUSE: # TODO this should be a warning if there's a way for pytest to print it print( f"Retrying bind_sockets({host}) after EADDRINUSE", file=sys.stderr ) continue break port = sockets[0].getsockname()[1] http_server.add_sockets(sockets) return http_server, port def get_unreachable_address() -> tuple[str, int]: # reserved as per rfc2606 return ("something.invalid", 54321) def encrypt_key_pem(private_key_pem: trustme.Blob, password: bytes) -> trustme.Blob: private_key = serialization.load_pem_private_key( private_key_pem.bytes(), password=None, backend=default_backend() ) encrypted_key = private_key.private_bytes( serialization.Encoding.PEM, serialization.PrivateFormat.TraditionalOpenSSL, serialization.BestAvailableEncryption(password), ) return trustme.Blob(encrypted_key) R = typing.TypeVar("R") def _run_and_close_tornado( async_fn: typing.Callable[P, Coroutine[typing.Any, typing.Any, R]], *args: P.args, **kwargs: P.kwargs, ) -> R: tornado_loop = None async def inner_fn() -> R: nonlocal tornado_loop tornado_loop = tornado.ioloop.IOLoop.current() return await async_fn(*args, **kwargs) try: return asyncio.run(inner_fn()) finally: tornado_loop.close(all_fds=True) # type: ignore[union-attr] @contextlib.contextmanager def run_loop_in_thread() -> Generator[tornado.ioloop.IOLoop, None, None]: loop_started: concurrent.futures.Future[ tuple[tornado.ioloop.IOLoop, asyncio.Event] ] = concurrent.futures.Future() with concurrent.futures.ThreadPoolExecutor( 1, thread_name_prefix="test IOLoop" ) as tpe: async def run() -> None: io_loop = tornado.ioloop.IOLoop.current() stop_event = asyncio.Event() loop_started.set_result((io_loop, stop_event)) await stop_event.wait() # run asyncio.run in a thread and collect exceptions from *either* # the loop failing to start, or failing to close ran = tpe.submit(_run_and_close_tornado, run) # type: ignore[arg-type] for f in concurrent.futures.as_completed((loop_started, ran)): # type: ignore[misc] if f is loop_started: io_loop, stop_event = loop_started.result() try: yield io_loop finally: io_loop.add_callback(stop_event.set) elif f is ran: # if this is the first iteration the loop failed to start # if it's the second iteration the loop has finished or # the loop failed to close and we need to raise the exception ran.result() return def main() -> int: # For debugging dummyserver itself - python -m dummyserver.server from .handlers import TestingApp host = "127.0.0.1" async def amain() -> int: app = tornado.web.Application([(r".*", TestingApp)]) server, port = run_tornado_app(app, None, "http", host) print(f"Listening on http://{host}:{port}") await asyncio.Event().wait() return 0 return asyncio.run(amain()) if __name__ == "__main__": sys.exit(main()) urllib3-2.0.7/dummyserver/testcase.py0000644000000000000000000002564314513541732014610 0ustar00from __future__ import annotations import asyncio import contextlib import socket import ssl import threading import typing import pytest from tornado import httpserver, ioloop, web from dummyserver.handlers import TestingApp from dummyserver.proxy import ProxyHandler from dummyserver.server import ( DEFAULT_CERTS, HAS_IPV6, SocketServerThread, run_loop_in_thread, run_tornado_app, ) from urllib3.connection import HTTPConnection from urllib3.util.ssltransport import SSLTransport def consume_socket( sock: SSLTransport | socket.socket, chunks: int = 65536 ) -> bytearray: consumed = bytearray() while True: b = sock.recv(chunks) assert isinstance(b, bytes) consumed += b if b.endswith(b"\r\n\r\n"): break return consumed class SocketDummyServerTestCase: """ A simple socket-based server is created for this class that is good for exactly one request. """ scheme = "http" host = "localhost" server_thread: typing.ClassVar[SocketServerThread] port: typing.ClassVar[int] tmpdir: typing.ClassVar[str] ca_path: typing.ClassVar[str] cert_combined_path: typing.ClassVar[str] cert_path: typing.ClassVar[str] key_path: typing.ClassVar[str] password_key_path: typing.ClassVar[str] server_context: typing.ClassVar[ssl.SSLContext] client_context: typing.ClassVar[ssl.SSLContext] proxy_server: typing.ClassVar[SocketDummyServerTestCase] @classmethod def _start_server( cls, socket_handler: typing.Callable[[socket.socket], None] ) -> None: ready_event = threading.Event() cls.server_thread = SocketServerThread( socket_handler=socket_handler, ready_event=ready_event, host=cls.host ) cls.server_thread.start() ready_event.wait(5) if not ready_event.is_set(): raise Exception("most likely failed to start server") cls.port = cls.server_thread.port @classmethod def start_response_handler( cls, response: bytes, num: int = 1, block_send: threading.Event | None = None ) -> threading.Event: ready_event = threading.Event() def socket_handler(listener: socket.socket) -> None: for _ in range(num): ready_event.set() sock = listener.accept()[0] consume_socket(sock) if block_send: block_send.wait() block_send.clear() sock.send(response) sock.close() cls._start_server(socket_handler) return ready_event @classmethod def start_basic_handler( cls, num: int = 1, block_send: threading.Event | None = None ) -> threading.Event: return cls.start_response_handler( b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n", num, block_send, ) @classmethod def teardown_class(cls) -> None: if hasattr(cls, "server_thread"): cls.server_thread.join(0.1) def assert_header_received( self, received_headers: typing.Iterable[bytes], header_name: str, expected_value: str | None = None, ) -> None: header_name_bytes = header_name.encode("ascii") if expected_value is None: expected_value_bytes = None else: expected_value_bytes = expected_value.encode("ascii") header_titles = [] for header in received_headers: key, value = header.split(b": ") header_titles.append(key) if key == header_name_bytes and expected_value_bytes is not None: assert value == expected_value_bytes assert header_name_bytes in header_titles class IPV4SocketDummyServerTestCase(SocketDummyServerTestCase): @classmethod def _start_server( cls, socket_handler: typing.Callable[[socket.socket], None] ) -> None: ready_event = threading.Event() cls.server_thread = SocketServerThread( socket_handler=socket_handler, ready_event=ready_event, host=cls.host ) cls.server_thread.USE_IPV6 = False cls.server_thread.start() ready_event.wait(5) if not ready_event.is_set(): raise Exception("most likely failed to start server") cls.port = cls.server_thread.port class HTTPDummyServerTestCase: """A simple HTTP server that runs when your test class runs Have your test class inherit from this one, and then a simple server will start when your tests run, and automatically shut down when they complete. For examples of what test requests you can send to the server, see the TestingApp in dummyserver/handlers.py. """ scheme = "http" host = "localhost" host_alt = "127.0.0.1" # Some tests need two hosts certs = DEFAULT_CERTS base_url: typing.ClassVar[str] base_url_alt: typing.ClassVar[str] io_loop: typing.ClassVar[ioloop.IOLoop] server: typing.ClassVar[httpserver.HTTPServer] port: typing.ClassVar[int] server_thread: typing.ClassVar[threading.Thread] _stack: typing.ClassVar[contextlib.ExitStack] @classmethod def _start_server(cls) -> None: with contextlib.ExitStack() as stack: io_loop = stack.enter_context(run_loop_in_thread()) async def run_app() -> None: app = web.Application([(r".*", TestingApp)]) cls.server, cls.port = run_tornado_app( app, cls.certs, cls.scheme, cls.host ) asyncio.run_coroutine_threadsafe(run_app(), io_loop.asyncio_loop).result() # type: ignore[attr-defined] cls._stack = stack.pop_all() @classmethod def _stop_server(cls) -> None: cls._stack.close() @classmethod def setup_class(cls) -> None: cls._start_server() @classmethod def teardown_class(cls) -> None: cls._stop_server() class HTTPSDummyServerTestCase(HTTPDummyServerTestCase): scheme = "https" host = "localhost" certs = DEFAULT_CERTS certs_dir = "" bad_ca_path = "" class HTTPDummyProxyTestCase: io_loop: typing.ClassVar[ioloop.IOLoop] http_host: typing.ClassVar[str] = "localhost" http_host_alt: typing.ClassVar[str] = "127.0.0.1" http_server: typing.ClassVar[httpserver.HTTPServer] http_port: typing.ClassVar[int] http_url: typing.ClassVar[str] http_url_alt: typing.ClassVar[str] https_host: typing.ClassVar[str] = "localhost" https_host_alt: typing.ClassVar[str] = "127.0.0.1" https_certs: typing.ClassVar[dict[str, typing.Any]] = DEFAULT_CERTS https_server: typing.ClassVar[httpserver.HTTPServer] https_port: typing.ClassVar[int] https_url: typing.ClassVar[str] https_url_alt: typing.ClassVar[str] proxy_host: typing.ClassVar[str] = "localhost" proxy_host_alt: typing.ClassVar[str] = "127.0.0.1" proxy_server: typing.ClassVar[httpserver.HTTPServer] proxy_port: typing.ClassVar[int] proxy_url: typing.ClassVar[str] https_proxy_server: typing.ClassVar[httpserver.HTTPServer] https_proxy_port: typing.ClassVar[int] https_proxy_url: typing.ClassVar[str] certs_dir: typing.ClassVar[str] = "" bad_ca_path: typing.ClassVar[str] = "" server_thread: typing.ClassVar[threading.Thread] _stack: typing.ClassVar[contextlib.ExitStack] @classmethod def setup_class(cls) -> None: with contextlib.ExitStack() as stack: io_loop = stack.enter_context(run_loop_in_thread()) async def run_app() -> None: app = web.Application([(r".*", TestingApp)]) cls.http_server, cls.http_port = run_tornado_app( app, None, "http", cls.http_host ) app = web.Application([(r".*", TestingApp)]) cls.https_server, cls.https_port = run_tornado_app( app, cls.https_certs, "https", cls.http_host ) app = web.Application([(r".*", ProxyHandler)]) cls.proxy_server, cls.proxy_port = run_tornado_app( app, None, "http", cls.proxy_host ) upstream_ca_certs = cls.https_certs.get("ca_certs") app = web.Application( [(r".*", ProxyHandler)], upstream_ca_certs=upstream_ca_certs ) cls.https_proxy_server, cls.https_proxy_port = run_tornado_app( app, cls.https_certs, "https", cls.proxy_host ) asyncio.run_coroutine_threadsafe(run_app(), io_loop.asyncio_loop).result() # type: ignore[attr-defined] cls._stack = stack.pop_all() @classmethod def teardown_class(cls) -> None: cls._stack.close() @pytest.mark.skipif(not HAS_IPV6, reason="IPv6 not available") class IPv6HTTPDummyServerTestCase(HTTPDummyServerTestCase): host = "::1" @pytest.mark.skipif(not HAS_IPV6, reason="IPv6 not available") class IPv6HTTPDummyProxyTestCase(HTTPDummyProxyTestCase): http_host = "localhost" http_host_alt = "127.0.0.1" https_host = "localhost" https_host_alt = "127.0.0.1" https_certs = DEFAULT_CERTS proxy_host = "::1" proxy_host_alt = "127.0.0.1" class ConnectionMarker: """ Marks an HTTP(S)Connection's socket after a request was made. Helps a test server understand when a client finished a request, without implementing a complete HTTP server. """ MARK_FORMAT = b"$#MARK%04x*!" @classmethod @contextlib.contextmanager def mark( cls, monkeypatch: pytest.MonkeyPatch ) -> typing.Generator[None, None, None]: """ Mark connections under in that context. """ orig_request = HTTPConnection.request def call_and_mark( target: typing.Callable[..., None] ) -> typing.Callable[..., None]: def part( self: HTTPConnection, *args: typing.Any, **kwargs: typing.Any ) -> None: target(self, *args, **kwargs) self.sock.sendall(cls._get_socket_mark(self.sock, False)) return part with monkeypatch.context() as m: m.setattr(HTTPConnection, "request", call_and_mark(orig_request)) yield @classmethod def consume_request(cls, sock: socket.socket, chunks: int = 65536) -> bytearray: """ Consume a socket until after the HTTP request is sent. """ consumed = bytearray() mark = cls._get_socket_mark(sock, True) while True: b = sock.recv(chunks) if not b: break consumed += b if consumed.endswith(mark): break return consumed @classmethod def _get_socket_mark(cls, sock: socket.socket, server: bool) -> bytes: if server: port = sock.getpeername()[1] else: port = sock.getsockname()[1] return cls.MARK_FORMAT % (port,) urllib3-2.0.7/dummyserver/certs/README.rst0000644000000000000000000000077714513541732015233 0ustar00Generating new certificates --------------------------- Here's how you can regenerate the certificates:: import trustme ca = trustme.CA() server_cert = ca.issue_cert("localhost") ca.cert_pem.write_to_path("cacert.pem") ca.private_key_pem.write_to_path("cacert.key") server_cert.cert_chain_pems[0].write_to_path("server.crt") server_cert.private_key_pem.write_to_path("server.key") This will break a number of tests: you will need to update the relevant fingerprints and hashes. urllib3-2.0.7/dummyserver/certs/cacert.key0000644000000000000000000000321714513541732015507 0ustar00-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAn330NEISY7w+GeZR2jh8Of1x2CtpenWRKuuD2u4FOaN0G1SN pm6Owum6xzhv93jmj+tZrp9kYvC9HcvGrgzH9yP566pLLfY2SEEAJYNNeVdqegY5 9W1wa6vEDN5UYruVEbymgPHoItiYhfndgEDbagPN5dhrdNvaRNI2c5zMmBDlzzhC G7AZbXuthTp6OwTVye71f2lnNhKi6rzWtre/ji88fq8Qm406l29O1RTzmNttN5BZ nPWU9v4GnCYKXdY6BN1Ub6z8C9hna/oRgBqa0Zbv7kEuAuqhsFQeuFCfBhQm7NdP d/7Kh6LS+VIiCu3AOccZHOuFOjtHb/KyndmgCwIDAQABAoIBAGQg9wc308O5kmNA LXMKszLU4nwMBRRUaua/JPB1LeKZs3LVCnjKP+YuRox76g87X8RKxOrUNnnHGXNz UzBB5ehKNcS2DKy2Pi3uYOEsJZ9gOgCRmCF0q3dtRo+tpNy3V0bjYMTjGhGGWXsC +wRhs15DNShvTkb3H3jFYFoEvo1YUKsvImBWJGwDbdDMfMZv4aeBWXlOrF+2fwt2 TM3G1o8xzEEWBB4FLZBW+tq2zfUSa1KwqqyQ4ZIqXepjQcN6nNfuHADA+nxuruVV LPUhz4ZmsBEnJ7CL9zWJkLUw/al9/6Q14tleRmiZTTztqAlFgZUpNhaKSzVdsIc/ Xz3+OgECgYEAzgNu7eFJAOzq+ZVFRrrbA7tu+f318FTCaADJ1kFgAJyj6b9yOGan LNL4TfXVjzgqtfQ4AIVqEHwXO7yS+YgTprvgzqRxqNRZ6ikuo2IPkIwXIAXZAlwd JsWLPBXOlOFW6LHvhYxjY2xF+A9y4AbuZ3UDRUQ+tp226VfEaeY80+ECgYEAxjDV cJqeBO06YRVGmcXfAYwRLJGT4hvIZeiSbxX/kJ0rx+cYLT/XZbAguJYQ5ZK2lkeA YneXYDlSTxmBxHxiWwWe3mcmctdE4Jbw8oIZ8a49a4KE/F2ojC4gmisIt3/OqGOw C4e/pDCE/QV64LWdazgUWHPGoVEmZx9/oMm/MWsCgYEAsLtlSJFB7ZdRpTcXLSxT gwoilDf36mrsNAipHjMLRrsaKwbf197If72k4kyJHspSabHO8TOC4A10aPzHIWZJ ZXo7y0prbyhs0mLt7Z/MNnbXx9L8bffT0lUZszwJ8tK1mf47utfK05opFDs8k0+e 6gYJ/jwjiMoYBmoSx76KZEECgYBagJxHAmQcbdQV1yhZOhFe3H5PMt8sBnHZj32m +o2slQkUDQRuTVPoHKikgeqPWxLDxzzqOiBHEYXzlvs6JW6okAV/G+1jzcenI2Y9 54k/YsirWnut3nsEIGBE5lfhq5xMKtGOQlwR9xITlLgK+wQ6nO41ghD3Q15dAvY+ D0KepwKBgQChHvbyzw0t76J2gLxUSyuG7VsId651bpqTYUsbSDFlRo4g8UbBAkHd fdv5BOon3ALJFreSK+a78es0kpiLwrS2SqG/y3mb9aUoLpCVB1haDmmP4Rn4AYXz OCfUkusuSoXOR8CMjqkXYl5QjeJAUAt9GTsZnXIbOQKbaZwkeV0HEg== -----END RSA PRIVATE KEY----- urllib3-2.0.7/dummyserver/certs/cacert.pem0000644000000000000000000000237114513541732015500 0ustar00-----BEGIN CERTIFICATE----- MIIDfzCCAmegAwIBAgIUVGEi+7bkaGRIPoQrp9zFFAT5JYQwDQYJKoZIhvcNAQEL BQAwQDEXMBUGA1UECgwOdHJ1c3RtZSB2MC41LjMxJTAjBgNVBAsMHFRlc3Rpbmcg Q0EgIzdEUWJWOTBzV2xZSEstY0wwHhcNMDAwMTAxMDAwMDAwWhcNMzgwMTAxMDAw MDAwWjBAMRcwFQYDVQQKDA50cnVzdG1lIHYwLjUuMzElMCMGA1UECwwcVGVzdGlu ZyBDQSAjN0RRYlY5MHNXbFlISy1jTDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAJ999DRCEmO8PhnmUdo4fDn9cdgraXp1kSrrg9ruBTmjdBtUjaZujsLp usc4b/d45o/rWa6fZGLwvR3Lxq4Mx/cj+euqSy32NkhBACWDTXlXanoGOfVtcGur xAzeVGK7lRG8poDx6CLYmIX53YBA22oDzeXYa3Tb2kTSNnOczJgQ5c84QhuwGW17 rYU6ejsE1cnu9X9pZzYSouq81ra3v44vPH6vEJuNOpdvTtUU85jbbTeQWZz1lPb+ BpwmCl3WOgTdVG+s/AvYZ2v6EYAamtGW7+5BLgLqobBUHrhQnwYUJuzXT3f+yoei 0vlSIgrtwDnHGRzrhTo7R2/ysp3ZoAsCAwEAAaNxMG8wHQYDVR0OBBYEFHWf39Hn rdChKjsOBoBGn1U+0VgxMBIGA1UdEwEB/wQIMAYBAf8CAQkwDgYDVR0PAQH/BAQD AgEGMCoGA1UdJQEB/wQgMB4GCCsGAQUFBwMCBggrBgEFBQcDAQYIKwYBBQUHAwMw DQYJKoZIhvcNAQELBQADggEBAIAylnWX2WTB+mrVVpE2W8i0HollTMJIPJA9Jq3Q /t2uPjXDEAVAcBmQju8qy2tHpamvzpQseVm3EF3UFNlGxwOGKsTzU4J45qOJITZk eLRAcWNEt6cgqj8ml8PuMHU7oDnp7pP6VPe5KQH1a0FYQnDNEwg7MyX+GjnXeRwd re6y9nMC+XKCYUAd1/nQcrZdnSsws1M5lzXir2vuyyN9EUkf2xMMKA2E1s0f+5he 3eNghAXtZw616ITBoMb7ckG6a0+YobbiQ0tKgB8D3MG2544Gx6xhCXf7pX4q4g// 1nTPeYFsBDyqEOEhcW1o9/MSSbjpUJC+QUmCb2Y1wYeum+w= -----END CERTIFICATE----- urllib3-2.0.7/dummyserver/certs/server.crt0000644000000000000000000000236114513541732015553 0ustar00-----BEGIN CERTIFICATE----- MIIDeTCCAmGgAwIBAgIUQeadxkH6YMoSecB2rNbFEr1u1kUwDQYJKoZIhvcNAQEL BQAwQDEXMBUGA1UECgwOdHJ1c3RtZSB2MC41LjMxJTAjBgNVBAsMHFRlc3Rpbmcg Q0EgIzdEUWJWOTBzV2xZSEstY0wwHhcNMDAwMTAxMDAwMDAwWhcNMzgwMTAxMDAw MDAwWjBCMRcwFQYDVQQKDA50cnVzdG1lIHYwLjUuMzEnMCUGA1UECwweVGVzdGlu ZyBjZXJ0ICMtSGRsMnMyTEYyeVp0NDFOMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEArRLZX+5dyCh4N7q90sH2Q4Ea6QLK8OfoUQPWtpzAtINDUAdfSXCC /qYTtGeSCGjB4W0LfvRTI8afHoD/M+YpaCRnx7T1sy1taA2rnGrEVXEHalVP+RI4 t4ZWtX56aez2M0Fs6o4MtzAuP6fKgSdWzIvOmtCxqn0Zf2KbfEHnQylsy2LgPa/x Lg50fbZ195+h4EAB3d2/jqaeFGGhN+7zrrv4+L1eeW3bzOkvPEkTNepq3Gy/8r5e 0i2icEnM+eBfl8NYgQ1toJYvDIy5Qi2TRzaFxBVmqUOc8EFtHpL7E9YLbTTW15xd oLVLdXI5igGxkwPYoeiiAJWxIsC/hL1RRQIDAQABo2kwZzAdBgNVHQ4EFgQUMU6+ uwNmL8TxLwjrj7jzzlwDPiowDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBR1n9/R 563QoSo7DgaARp9VPtFYMTAXBgNVHREBAf8EDTALgglsb2NhbGhvc3QwDQYJKoZI hvcNAQELBQADggEBAJ6w5neQKw+/efA/I3IHzt8GaSHQ/YehMHx8GxCViJUmLg6P Vf74k856Knvh7IsVaqF1uRi6qQaFPik6CwtBCj7/ZftdseOCDljd+8EWyQ+ZWie7 +tzMIdQWZxYSdR9Ov42VD++a6oWJtfJhWV5eyDit99FFK31/M1ZXoceiDS5AsIG6 wfsxrFj1qV9pLNSIlfrnycYhYx7avVJTf+2mfZgTO9Tx+VPapkZrfCnP/2jpN39u zblFFjP9Ir0QqBw7MXjVX+Y1HkQ2TQnEeSsp1HuFRIZYx72Cttnckv1Lxcx/HiQB oebTDYiRfxOAEeIMgIhX88Jca8vNIRcXDeGK9mU= -----END CERTIFICATE----- urllib3-2.0.7/dummyserver/certs/server.key0000644000000000000000000000321714513541732015554 0ustar00-----BEGIN RSA PRIVATE KEY----- MIIEpQIBAAKCAQEArRLZX+5dyCh4N7q90sH2Q4Ea6QLK8OfoUQPWtpzAtINDUAdf SXCC/qYTtGeSCGjB4W0LfvRTI8afHoD/M+YpaCRnx7T1sy1taA2rnGrEVXEHalVP +RI4t4ZWtX56aez2M0Fs6o4MtzAuP6fKgSdWzIvOmtCxqn0Zf2KbfEHnQylsy2Lg Pa/xLg50fbZ195+h4EAB3d2/jqaeFGGhN+7zrrv4+L1eeW3bzOkvPEkTNepq3Gy/ 8r5e0i2icEnM+eBfl8NYgQ1toJYvDIy5Qi2TRzaFxBVmqUOc8EFtHpL7E9YLbTTW 15xdoLVLdXI5igGxkwPYoeiiAJWxIsC/hL1RRQIDAQABAoIBAQCZ/62f6G9WDHx7 yhPhlmjTw+r37l45YYCbpbjFoFDvzeR1LzogFJbak1fxLD8KcHwjY23ZNvlLWg53 i/yIZ4Hsgog9cM0283LoJVHPykiMZhhdCzAvxYDl/AjnUXUHD6w6CzsoseCql5pv VZOgvCpFsxjRNGUB+HJZoJoNRG7MmHaY638pGHGMiVbskT9Ww3emtMLdTKy1rQcj 9XO/4mlaBGD79wYxy5Hlysbh2UYuQRv8XN5V46Uugk6sC/i2G7VC8KkqPQ2pM+rA LaeWSuN9dfBwiKcHtJssMP95ilsXsjqh3NoVuFODDXHv3u+nBAxtg2TnLZFkDZXH FvxPJu8BAoGBANwWWzvl/dnTtVDFbhF/61v3AVZV4vVpokXRJKny05CZPZWROTc4 LXMVw9kxeecNdo0yR4jn1yyNUmKnQpTmpsR9Yo9NYH+Z1CLxswpc7ILfVRZBK6bK cCG43lM5xZprG6FXhqkHN2u9z5Y8/PuaMzC8iVs402/gakgPKmn8OjdhAoGBAMlQ mmrx24n9YY/dOn55XC5V/iN3Z6mIsHThnDIU515bwLwZVG7chOLSiWHAh4JzUH+v bV3NnlE1jhf5ln0WAadCtIeVprJG6msNTQlbTMTTV5kVNdbfYX6sFQEI+hC1LCiV yJtuNIa4P5W/PtoC3FWUlcAH8C91S/M4CeZZ0HhlAoGBAIxflgE2SBrO9S53PiTb OfqGKMwwK3nrzhxJsODUiCwKEUV8Qsn9gr+MekXlUKMV6y9Tily/wnYgDRPvKoBe PK/GaT6NU6cPLka7cj6B1jgCyfpPxs+y/qIDj4n1pxs+hXj6omDcwXRutCBW9eRk DZJgLhuIuxL4R9F+GsdOoLMBAoGAKQn1cLe9OXQd32YJ9p5m3EtLc49z4murDSiw 3sTEJcgukinXvIHX1SV2PCczeLRpRJ5OfUDddVCllt2agAVscNx4UOuA//bU8t3T RoUGMVmkEeDxCMyg42HRJlTeJWnJhryCGK1up8gHrk8+UNMkd43CuVLk88fFo99Y pUzJ4sECgYEAvBDTo3k3sD18qV6p6tQwy+MVjvQb9V81GHP18tYcVKta3LkkqUFa 3qSyVxi7gl3JtynG7NJ7+GDx6zxW2xUR72NTcJwWvesLI+1orM288pyNDVw9MJ/j AyVFnW5SEYEqdizTnQxL+rQB4CyeHfwZx2/1/Qr0ezLGUJv51lnk4mQ= -----END RSA PRIVATE KEY----- urllib3-2.0.7/src/urllib3/__init__.py0000644000000000000000000001227314513541732014310 0ustar00""" Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more """ from __future__ import annotations # Set default logging handler to avoid "No handler found" warnings. import logging import typing import warnings from logging import NullHandler from . import exceptions from ._base_connection import _TYPE_BODY from ._collections import HTTPHeaderDict from ._version import __version__ from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url from .filepost import _TYPE_FIELDS, encode_multipart_formdata from .poolmanager import PoolManager, ProxyManager, proxy_from_url from .response import BaseHTTPResponse, HTTPResponse from .util.request import make_headers from .util.retry import Retry from .util.timeout import Timeout # Ensure that Python is compiled with OpenSSL 1.1.1+ # If the 'ssl' module isn't available at all that's # fine, we only care if the module is available. try: import ssl except ImportError: pass else: if not ssl.OPENSSL_VERSION.startswith("OpenSSL "): # Defensive: warnings.warn( "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. " "See: https://github.com/urllib3/urllib3/issues/3020", exceptions.NotOpenSSLWarning, ) elif ssl.OPENSSL_VERSION_INFO < (1, 1, 1): # Defensive: raise ImportError( "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. " "See: https://github.com/urllib3/urllib3/issues/2168" ) # === NOTE TO REPACKAGERS AND VENDORS === # Please delete this block, this logic is only # for urllib3 being distributed via PyPI. # See: https://github.com/urllib3/urllib3/issues/2680 try: import urllib3_secure_extra # type: ignore # noqa: F401 except ModuleNotFoundError: pass else: warnings.warn( "'urllib3[secure]' extra is deprecated and will be removed " "in urllib3 v2.1.0. Read more in this issue: " "https://github.com/urllib3/urllib3/issues/2680", category=DeprecationWarning, stacklevel=2, ) __author__ = "Andrey Petrov (andrey.petrov@shazow.net)" __license__ = "MIT" __version__ = __version__ __all__ = ( "HTTPConnectionPool", "HTTPHeaderDict", "HTTPSConnectionPool", "PoolManager", "ProxyManager", "HTTPResponse", "Retry", "Timeout", "add_stderr_logger", "connection_from_url", "disable_warnings", "encode_multipart_formdata", "make_headers", "proxy_from_url", "request", "BaseHTTPResponse", ) logging.getLogger(__name__).addHandler(NullHandler()) def add_stderr_logger( level: int = logging.DEBUG, ) -> logging.StreamHandler[typing.TextIO]: """ Helper for quickly adding a StreamHandler to the logger. Useful for debugging. Returns the handler after adding it. """ # This method needs to be in this __init__.py to get the __name__ correct # even if urllib3 is vendored within another package. logger = logging.getLogger(__name__) handler = logging.StreamHandler() handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s")) logger.addHandler(handler) logger.setLevel(level) logger.debug("Added a stderr logging handler to logger: %s", __name__) return handler # ... Clean up. del NullHandler # All warning filters *must* be appended unless you're really certain that they # shouldn't be: otherwise, it's very hard for users to use most Python # mechanisms to silence them. # SecurityWarning's always go off by default. warnings.simplefilter("always", exceptions.SecurityWarning, append=True) # InsecurePlatformWarning's don't vary between requests, so we keep it default. warnings.simplefilter("default", exceptions.InsecurePlatformWarning, append=True) def disable_warnings(category: type[Warning] = exceptions.HTTPWarning) -> None: """ Helper for quickly disabling all urllib3 warnings. """ warnings.simplefilter("ignore", category) _DEFAULT_POOL = PoolManager() def request( method: str, url: str, *, body: _TYPE_BODY | None = None, fields: _TYPE_FIELDS | None = None, headers: typing.Mapping[str, str] | None = None, preload_content: bool | None = True, decode_content: bool | None = True, redirect: bool | None = True, retries: Retry | bool | int | None = None, timeout: Timeout | float | int | None = 3, json: typing.Any | None = None, ) -> BaseHTTPResponse: """ A convenience, top-level request method. It uses a module-global ``PoolManager`` instance. Therefore, its side effects could be shared across dependencies relying on it. To avoid side effects create a new ``PoolManager`` instance and use it instead. The method does not accept low-level ``**urlopen_kw`` keyword arguments. """ return _DEFAULT_POOL.request( method, url, body=body, fields=fields, headers=headers, preload_content=preload_content, decode_content=decode_content, redirect=redirect, retries=retries, timeout=timeout, json=json, ) urllib3-2.0.7/src/urllib3/_base_connection.py0000644000000000000000000001302414513541732016034 0ustar00from __future__ import annotations import typing from .util.connection import _TYPE_SOCKET_OPTIONS from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT from .util.url import Url _TYPE_BODY = typing.Union[bytes, typing.IO[typing.Any], typing.Iterable[bytes], str] class ProxyConfig(typing.NamedTuple): ssl_context: ssl.SSLContext | None use_forwarding_for_https: bool assert_hostname: None | str | Literal[False] assert_fingerprint: str | None class _ResponseOptions(typing.NamedTuple): # TODO: Remove this in favor of a better # HTTP request/response lifecycle tracking. request_method: str request_url: str preload_content: bool decode_content: bool enforce_content_length: bool if typing.TYPE_CHECKING: import ssl from typing_extensions import Literal, Protocol from .response import BaseHTTPResponse class BaseHTTPConnection(Protocol): default_port: typing.ClassVar[int] default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS] host: str port: int timeout: None | ( float ) # Instance doesn't store _DEFAULT_TIMEOUT, must be resolved. blocksize: int source_address: tuple[str, int] | None socket_options: _TYPE_SOCKET_OPTIONS | None proxy: Url | None proxy_config: ProxyConfig | None is_verified: bool proxy_is_verified: bool | None def __init__( self, host: str, port: int | None = None, *, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, source_address: tuple[str, int] | None = None, blocksize: int = 8192, socket_options: _TYPE_SOCKET_OPTIONS | None = ..., proxy: Url | None = None, proxy_config: ProxyConfig | None = None, ) -> None: ... def set_tunnel( self, host: str, port: int | None = None, headers: typing.Mapping[str, str] | None = None, scheme: str = "http", ) -> None: ... def connect(self) -> None: ... def request( self, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, # We know *at least* botocore is depending on the order of the # first 3 parameters so to be safe we only mark the later ones # as keyword-only to ensure we have space to extend. *, chunked: bool = False, preload_content: bool = True, decode_content: bool = True, enforce_content_length: bool = True, ) -> None: ... def getresponse(self) -> BaseHTTPResponse: ... def close(self) -> None: ... @property def is_closed(self) -> bool: """Whether the connection either is brand new or has been previously closed. If this property is True then both ``is_connected`` and ``has_connected_to_proxy`` properties must be False. """ @property def is_connected(self) -> bool: """Whether the connection is actively connected to any origin (proxy or target)""" @property def has_connected_to_proxy(self) -> bool: """Whether the connection has successfully connected to its proxy. This returns False if no proxy is in use. Used to determine whether errors are coming from the proxy layer or from tunnelling to the target origin. """ class BaseHTTPSConnection(BaseHTTPConnection, Protocol): default_port: typing.ClassVar[int] default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS] # Certificate verification methods cert_reqs: int | str | None assert_hostname: None | str | Literal[False] assert_fingerprint: str | None ssl_context: ssl.SSLContext | None # Trusted CAs ca_certs: str | None ca_cert_dir: str | None ca_cert_data: None | str | bytes # TLS version ssl_minimum_version: int | None ssl_maximum_version: int | None ssl_version: int | str | None # Deprecated # Client certificates cert_file: str | None key_file: str | None key_password: str | None def __init__( self, host: str, port: int | None = None, *, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, source_address: tuple[str, int] | None = None, blocksize: int = 16384, socket_options: _TYPE_SOCKET_OPTIONS | None = ..., proxy: Url | None = None, proxy_config: ProxyConfig | None = None, cert_reqs: int | str | None = None, assert_hostname: None | str | Literal[False] = None, assert_fingerprint: str | None = None, server_hostname: str | None = None, ssl_context: ssl.SSLContext | None = None, ca_certs: str | None = None, ca_cert_dir: str | None = None, ca_cert_data: None | str | bytes = None, ssl_minimum_version: int | None = None, ssl_maximum_version: int | None = None, ssl_version: int | str | None = None, # Deprecated cert_file: str | None = None, key_file: str | None = None, key_password: str | None = None, ) -> None: ... urllib3-2.0.7/src/urllib3/_collections.py0000644000000000000000000004176014513541732015231 0ustar00from __future__ import annotations import typing from collections import OrderedDict from enum import Enum, auto from threading import RLock if typing.TYPE_CHECKING: # We can only import Protocol if TYPE_CHECKING because it's a development # dependency, and is not available at runtime. from typing_extensions import Protocol, Self class HasGettableStringKeys(Protocol): def keys(self) -> typing.Iterator[str]: ... def __getitem__(self, key: str) -> str: ... __all__ = ["RecentlyUsedContainer", "HTTPHeaderDict"] # Key type _KT = typing.TypeVar("_KT") # Value type _VT = typing.TypeVar("_VT") # Default type _DT = typing.TypeVar("_DT") ValidHTTPHeaderSource = typing.Union[ "HTTPHeaderDict", typing.Mapping[str, str], typing.Iterable[typing.Tuple[str, str]], "HasGettableStringKeys", ] class _Sentinel(Enum): not_passed = auto() def ensure_can_construct_http_header_dict( potential: object, ) -> ValidHTTPHeaderSource | None: if isinstance(potential, HTTPHeaderDict): return potential elif isinstance(potential, typing.Mapping): # Full runtime checking of the contents of a Mapping is expensive, so for the # purposes of typechecking, we assume that any Mapping is the right shape. return typing.cast(typing.Mapping[str, str], potential) elif isinstance(potential, typing.Iterable): # Similarly to Mapping, full runtime checking of the contents of an Iterable is # expensive, so for the purposes of typechecking, we assume that any Iterable # is the right shape. return typing.cast(typing.Iterable[typing.Tuple[str, str]], potential) elif hasattr(potential, "keys") and hasattr(potential, "__getitem__"): return typing.cast("HasGettableStringKeys", potential) else: return None class RecentlyUsedContainer(typing.Generic[_KT, _VT], typing.MutableMapping[_KT, _VT]): """ Provides a thread-safe dict-like container which maintains up to ``maxsize`` keys while throwing away the least-recently-used keys beyond ``maxsize``. :param maxsize: Maximum number of recent elements to retain. :param dispose_func: Every time an item is evicted from the container, ``dispose_func(value)`` is called. Callback which will get called """ _container: typing.OrderedDict[_KT, _VT] _maxsize: int dispose_func: typing.Callable[[_VT], None] | None lock: RLock def __init__( self, maxsize: int = 10, dispose_func: typing.Callable[[_VT], None] | None = None, ) -> None: super().__init__() self._maxsize = maxsize self.dispose_func = dispose_func self._container = OrderedDict() self.lock = RLock() def __getitem__(self, key: _KT) -> _VT: # Re-insert the item, moving it to the end of the eviction line. with self.lock: item = self._container.pop(key) self._container[key] = item return item def __setitem__(self, key: _KT, value: _VT) -> None: evicted_item = None with self.lock: # Possibly evict the existing value of 'key' try: # If the key exists, we'll overwrite it, which won't change the # size of the pool. Because accessing a key should move it to # the end of the eviction line, we pop it out first. evicted_item = key, self._container.pop(key) self._container[key] = value except KeyError: # When the key does not exist, we insert the value first so that # evicting works in all cases, including when self._maxsize is 0 self._container[key] = value if len(self._container) > self._maxsize: # If we didn't evict an existing value, and we've hit our maximum # size, then we have to evict the least recently used item from # the beginning of the container. evicted_item = self._container.popitem(last=False) # After releasing the lock on the pool, dispose of any evicted value. if evicted_item is not None and self.dispose_func: _, evicted_value = evicted_item self.dispose_func(evicted_value) def __delitem__(self, key: _KT) -> None: with self.lock: value = self._container.pop(key) if self.dispose_func: self.dispose_func(value) def __len__(self) -> int: with self.lock: return len(self._container) def __iter__(self) -> typing.NoReturn: raise NotImplementedError( "Iteration over this class is unlikely to be threadsafe." ) def clear(self) -> None: with self.lock: # Copy pointers to all values, then wipe the mapping values = list(self._container.values()) self._container.clear() if self.dispose_func: for value in values: self.dispose_func(value) def keys(self) -> set[_KT]: # type: ignore[override] with self.lock: return set(self._container.keys()) class HTTPHeaderDictItemView(typing.Set[typing.Tuple[str, str]]): """ HTTPHeaderDict is unusual for a Mapping[str, str] in that it has two modes of address. If we directly try to get an item with a particular name, we will get a string back that is the concatenated version of all the values: >>> d['X-Header-Name'] 'Value1, Value2, Value3' However, if we iterate over an HTTPHeaderDict's items, we will optionally combine these values based on whether combine=True was called when building up the dictionary >>> d = HTTPHeaderDict({"A": "1", "B": "foo"}) >>> d.add("A", "2", combine=True) >>> d.add("B", "bar") >>> list(d.items()) [ ('A', '1, 2'), ('B', 'foo'), ('B', 'bar'), ] This class conforms to the interface required by the MutableMapping ABC while also giving us the nonstandard iteration behavior we want; items with duplicate keys, ordered by time of first insertion. """ _headers: HTTPHeaderDict def __init__(self, headers: HTTPHeaderDict) -> None: self._headers = headers def __len__(self) -> int: return len(list(self._headers.iteritems())) def __iter__(self) -> typing.Iterator[tuple[str, str]]: return self._headers.iteritems() def __contains__(self, item: object) -> bool: if isinstance(item, tuple) and len(item) == 2: passed_key, passed_val = item if isinstance(passed_key, str) and isinstance(passed_val, str): return self._headers._has_value_for_header(passed_key, passed_val) return False class HTTPHeaderDict(typing.MutableMapping[str, str]): """ :param headers: An iterable of field-value pairs. Must not contain multiple field names when compared case-insensitively. :param kwargs: Additional field-value pairs to pass in to ``dict.update``. A ``dict`` like container for storing HTTP Headers. Field names are stored and compared case-insensitively in compliance with RFC 7230. Iteration provides the first case-sensitive key seen for each case-insensitive pair. Using ``__setitem__`` syntax overwrites fields that compare equal case-insensitively in order to maintain ``dict``'s api. For fields that compare equal, instead create a new ``HTTPHeaderDict`` and use ``.add`` in a loop. If multiple fields that are equal case-insensitively are passed to the constructor or ``.update``, the behavior is undefined and some will be lost. >>> headers = HTTPHeaderDict() >>> headers.add('Set-Cookie', 'foo=bar') >>> headers.add('set-cookie', 'baz=quxx') >>> headers['content-length'] = '7' >>> headers['SET-cookie'] 'foo=bar, baz=quxx' >>> headers['Content-Length'] '7' """ _container: typing.MutableMapping[str, list[str]] def __init__(self, headers: ValidHTTPHeaderSource | None = None, **kwargs: str): super().__init__() self._container = {} # 'dict' is insert-ordered in Python 3.7+ if headers is not None: if isinstance(headers, HTTPHeaderDict): self._copy_from(headers) else: self.extend(headers) if kwargs: self.extend(kwargs) def __setitem__(self, key: str, val: str) -> None: # avoid a bytes/str comparison by decoding before httplib if isinstance(key, bytes): key = key.decode("latin-1") self._container[key.lower()] = [key, val] def __getitem__(self, key: str) -> str: val = self._container[key.lower()] return ", ".join(val[1:]) def __delitem__(self, key: str) -> None: del self._container[key.lower()] def __contains__(self, key: object) -> bool: if isinstance(key, str): return key.lower() in self._container return False def setdefault(self, key: str, default: str = "") -> str: return super().setdefault(key, default) def __eq__(self, other: object) -> bool: maybe_constructable = ensure_can_construct_http_header_dict(other) if maybe_constructable is None: return False else: other_as_http_header_dict = type(self)(maybe_constructable) return {k.lower(): v for k, v in self.itermerged()} == { k.lower(): v for k, v in other_as_http_header_dict.itermerged() } def __ne__(self, other: object) -> bool: return not self.__eq__(other) def __len__(self) -> int: return len(self._container) def __iter__(self) -> typing.Iterator[str]: # Only provide the originally cased names for vals in self._container.values(): yield vals[0] def discard(self, key: str) -> None: try: del self[key] except KeyError: pass def add(self, key: str, val: str, *, combine: bool = False) -> None: """Adds a (name, value) pair, doesn't overwrite the value if it already exists. If this is called with combine=True, instead of adding a new header value as a distinct item during iteration, this will instead append the value to any existing header value with a comma. If no existing header value exists for the key, then the value will simply be added, ignoring the combine parameter. >>> headers = HTTPHeaderDict(foo='bar') >>> headers.add('Foo', 'baz') >>> headers['foo'] 'bar, baz' >>> list(headers.items()) [('foo', 'bar'), ('foo', 'baz')] >>> headers.add('foo', 'quz', combine=True) >>> list(headers.items()) [('foo', 'bar, baz, quz')] """ # avoid a bytes/str comparison by decoding before httplib if isinstance(key, bytes): key = key.decode("latin-1") key_lower = key.lower() new_vals = [key, val] # Keep the common case aka no item present as fast as possible vals = self._container.setdefault(key_lower, new_vals) if new_vals is not vals: # if there are values here, then there is at least the initial # key/value pair assert len(vals) >= 2 if combine: vals[-1] = vals[-1] + ", " + val else: vals.append(val) def extend(self, *args: ValidHTTPHeaderSource, **kwargs: str) -> None: """Generic import function for any type of header-like object. Adapted version of MutableMapping.update in order to insert items with self.add instead of self.__setitem__ """ if len(args) > 1: raise TypeError( f"extend() takes at most 1 positional arguments ({len(args)} given)" ) other = args[0] if len(args) >= 1 else () if isinstance(other, HTTPHeaderDict): for key, val in other.iteritems(): self.add(key, val) elif isinstance(other, typing.Mapping): for key, val in other.items(): self.add(key, val) elif isinstance(other, typing.Iterable): other = typing.cast(typing.Iterable[typing.Tuple[str, str]], other) for key, value in other: self.add(key, value) elif hasattr(other, "keys") and hasattr(other, "__getitem__"): # THIS IS NOT A TYPESAFE BRANCH # In this branch, the object has a `keys` attr but is not a Mapping or any of # the other types indicated in the method signature. We do some stuff with # it as though it partially implements the Mapping interface, but we're not # doing that stuff safely AT ALL. for key in other.keys(): self.add(key, other[key]) for key, value in kwargs.items(): self.add(key, value) @typing.overload def getlist(self, key: str) -> list[str]: ... @typing.overload def getlist(self, key: str, default: _DT) -> list[str] | _DT: ... def getlist( self, key: str, default: _Sentinel | _DT = _Sentinel.not_passed ) -> list[str] | _DT: """Returns a list of all the values for the named field. Returns an empty list if the key doesn't exist.""" try: vals = self._container[key.lower()] except KeyError: if default is _Sentinel.not_passed: # _DT is unbound; empty list is instance of List[str] return [] # _DT is bound; default is instance of _DT return default else: # _DT may or may not be bound; vals[1:] is instance of List[str], which # meets our external interface requirement of `Union[List[str], _DT]`. return vals[1:] def _prepare_for_method_change(self) -> Self: """ Remove content-specific header fields before changing the request method to GET or HEAD according to RFC 9110, Section 15.4. """ content_specific_headers = [ "Content-Encoding", "Content-Language", "Content-Location", "Content-Type", "Content-Length", "Digest", "Last-Modified", ] for header in content_specific_headers: self.discard(header) return self # Backwards compatibility for httplib getheaders = getlist getallmatchingheaders = getlist iget = getlist # Backwards compatibility for http.cookiejar get_all = getlist def __repr__(self) -> str: return f"{type(self).__name__}({dict(self.itermerged())})" def _copy_from(self, other: HTTPHeaderDict) -> None: for key in other: val = other.getlist(key) self._container[key.lower()] = [key, *val] def copy(self) -> HTTPHeaderDict: clone = type(self)() clone._copy_from(self) return clone def iteritems(self) -> typing.Iterator[tuple[str, str]]: """Iterate over all header lines, including duplicate ones.""" for key in self: vals = self._container[key.lower()] for val in vals[1:]: yield vals[0], val def itermerged(self) -> typing.Iterator[tuple[str, str]]: """Iterate over all headers, merging duplicate ones together.""" for key in self: val = self._container[key.lower()] yield val[0], ", ".join(val[1:]) def items(self) -> HTTPHeaderDictItemView: # type: ignore[override] return HTTPHeaderDictItemView(self) def _has_value_for_header(self, header_name: str, potential_value: str) -> bool: if header_name in self: return potential_value in self._container[header_name.lower()][1:] return False def __ior__(self, other: object) -> HTTPHeaderDict: # Supports extending a header dict in-place using operator |= # combining items with add instead of __setitem__ maybe_constructable = ensure_can_construct_http_header_dict(other) if maybe_constructable is None: return NotImplemented self.extend(maybe_constructable) return self def __or__(self, other: object) -> HTTPHeaderDict: # Supports merging header dicts using operator | # combining items with add instead of __setitem__ maybe_constructable = ensure_can_construct_http_header_dict(other) if maybe_constructable is None: return NotImplemented result = self.copy() result.extend(maybe_constructable) return result def __ror__(self, other: object) -> HTTPHeaderDict: # Supports merging header dicts using operator | when other is on left side # combining items with add instead of __setitem__ maybe_constructable = ensure_can_construct_http_header_dict(other) if maybe_constructable is None: return NotImplemented result = type(self)(maybe_constructable) result.extend(self) return result urllib3-2.0.7/src/urllib3/_request_methods.py0000644000000000000000000001711414513541732016122 0ustar00from __future__ import annotations import json as _json import typing from urllib.parse import urlencode from ._base_connection import _TYPE_BODY from ._collections import HTTPHeaderDict from .filepost import _TYPE_FIELDS, encode_multipart_formdata from .response import BaseHTTPResponse __all__ = ["RequestMethods"] _TYPE_ENCODE_URL_FIELDS = typing.Union[ typing.Sequence[typing.Tuple[str, typing.Union[str, bytes]]], typing.Mapping[str, typing.Union[str, bytes]], ] class RequestMethods: """ Convenience mixin for classes who implement a :meth:`urlopen` method, such as :class:`urllib3.HTTPConnectionPool` and :class:`urllib3.PoolManager`. Provides behavior for making common types of HTTP request methods and decides which type of request field encoding to use. Specifically, :meth:`.request_encode_url` is for sending requests whose fields are encoded in the URL (such as GET, HEAD, DELETE). :meth:`.request_encode_body` is for sending requests whose fields are encoded in the *body* of the request using multipart or www-form-urlencoded (such as for POST, PUT, PATCH). :meth:`.request` is for making any kind of request, it will look up the appropriate encoding format and use one of the above two methods to make the request. Initializer parameters: :param headers: Headers to include with all requests, unless other headers are given explicitly. """ _encode_url_methods = {"DELETE", "GET", "HEAD", "OPTIONS"} def __init__(self, headers: typing.Mapping[str, str] | None = None) -> None: self.headers = headers or {} def urlopen( self, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, encode_multipart: bool = True, multipart_boundary: str | None = None, **kw: typing.Any, ) -> BaseHTTPResponse: # Abstract raise NotImplementedError( "Classes extending RequestMethods must implement " "their own ``urlopen`` method." ) def request( self, method: str, url: str, body: _TYPE_BODY | None = None, fields: _TYPE_FIELDS | None = None, headers: typing.Mapping[str, str] | None = None, json: typing.Any | None = None, **urlopen_kw: typing.Any, ) -> BaseHTTPResponse: """ Make a request using :meth:`urlopen` with the appropriate encoding of ``fields`` based on the ``method`` used. This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as :meth:`request_encode_url`, :meth:`request_encode_body`, or even the lowest level :meth:`urlopen`. """ method = method.upper() if json is not None and body is not None: raise TypeError( "request got values for both 'body' and 'json' parameters which are mutually exclusive" ) if json is not None: if headers is None: headers = self.headers.copy() # type: ignore if not ("content-type" in map(str.lower, headers.keys())): headers["Content-Type"] = "application/json" # type: ignore body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode( "utf-8" ) if body is not None: urlopen_kw["body"] = body if method in self._encode_url_methods: return self.request_encode_url( method, url, fields=fields, # type: ignore[arg-type] headers=headers, **urlopen_kw, ) else: return self.request_encode_body( method, url, fields=fields, headers=headers, **urlopen_kw ) def request_encode_url( self, method: str, url: str, fields: _TYPE_ENCODE_URL_FIELDS | None = None, headers: typing.Mapping[str, str] | None = None, **urlopen_kw: str, ) -> BaseHTTPResponse: """ Make a request using :meth:`urlopen` with the ``fields`` encoded in the url. This is useful for request methods like GET, HEAD, DELETE, etc. """ if headers is None: headers = self.headers extra_kw: dict[str, typing.Any] = {"headers": headers} extra_kw.update(urlopen_kw) if fields: url += "?" + urlencode(fields) return self.urlopen(method, url, **extra_kw) def request_encode_body( self, method: str, url: str, fields: _TYPE_FIELDS | None = None, headers: typing.Mapping[str, str] | None = None, encode_multipart: bool = True, multipart_boundary: str | None = None, **urlopen_kw: str, ) -> BaseHTTPResponse: """ Make a request using :meth:`urlopen` with the ``fields`` encoded in the body. This is useful for request methods like POST, PUT, PATCH, etc. When ``encode_multipart=True`` (default), then :func:`urllib3.encode_multipart_formdata` is used to encode the payload with the appropriate content type. Otherwise :func:`urllib.parse.urlencode` is used with the 'application/x-www-form-urlencoded' content type. Multipart encoding must be used when posting files, and it's reasonably safe to use it in other times too. However, it may break request signing, such as with OAuth. Supports an optional ``fields`` parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:: fields = { 'foo': 'bar', 'fakefile': ('foofile.txt', 'contents of foofile'), 'realfile': ('barfile.txt', open('realfile').read()), 'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'), 'nonamefile': 'contents of nonamefile field', } When uploading a file, providing a filename (the first parameter of the tuple) is optional but recommended to best mimic behavior of browsers. Note that if ``headers`` are supplied, the 'Content-Type' header will be overwritten because it depends on the dynamic random boundary string which is used to compose the body of the request. The random boundary string can be explicitly set with the ``multipart_boundary`` parameter. """ if headers is None: headers = self.headers extra_kw: dict[str, typing.Any] = {"headers": HTTPHeaderDict(headers)} body: bytes | str if fields: if "body" in urlopen_kw: raise TypeError( "request got values for both 'fields' and 'body', can only specify one." ) if encode_multipart: body, content_type = encode_multipart_formdata( fields, boundary=multipart_boundary ) else: body, content_type = ( urlencode(fields), # type: ignore[arg-type] "application/x-www-form-urlencoded", ) extra_kw["body"] = body extra_kw["headers"].setdefault("Content-Type", content_type) extra_kw.update(urlopen_kw) return self.urlopen(method, url, **extra_kw) urllib3-2.0.7/src/urllib3/_version.py0000644000000000000000000000014214513541732014365 0ustar00# This file is protected via CODEOWNERS from __future__ import annotations __version__ = "2.0.7" urllib3-2.0.7/src/urllib3/connection.py0000644000000000000000000010205014513541732014701 0ustar00from __future__ import annotations import datetime import logging import os import re import socket import sys import typing import warnings from http.client import HTTPConnection as _HTTPConnection from http.client import HTTPException as HTTPException # noqa: F401 from http.client import ResponseNotReady from socket import timeout as SocketTimeout if typing.TYPE_CHECKING: from typing_extensions import Literal from .response import HTTPResponse from .util.ssl_ import _TYPE_PEER_CERT_RET_DICT from .util.ssltransport import SSLTransport from ._collections import HTTPHeaderDict from .util.response import assert_header_parsing from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT, Timeout from .util.util import to_str from .util.wait import wait_for_read try: # Compiled with SSL? import ssl BaseSSLError = ssl.SSLError except (ImportError, AttributeError): ssl = None # type: ignore[assignment] class BaseSSLError(BaseException): # type: ignore[no-redef] pass from ._base_connection import _TYPE_BODY from ._base_connection import ProxyConfig as ProxyConfig from ._base_connection import _ResponseOptions as _ResponseOptions from ._version import __version__ from .exceptions import ( ConnectTimeoutError, HeaderParsingError, NameResolutionError, NewConnectionError, ProxyError, SystemTimeWarning, ) from .util import SKIP_HEADER, SKIPPABLE_HEADERS, connection, ssl_ from .util.request import body_to_chunks from .util.ssl_ import assert_fingerprint as _assert_fingerprint from .util.ssl_ import ( create_urllib3_context, is_ipaddress, resolve_cert_reqs, resolve_ssl_version, ssl_wrap_socket, ) from .util.ssl_match_hostname import CertificateError, match_hostname from .util.url import Url # Not a no-op, we're adding this to the namespace so it can be imported. ConnectionError = ConnectionError BrokenPipeError = BrokenPipeError log = logging.getLogger(__name__) port_by_scheme = {"http": 80, "https": 443} # When it comes time to update this value as a part of regular maintenance # (ie test_recent_date is failing) update it to ~6 months before the current date. RECENT_DATE = datetime.date(2022, 1, 1) _CONTAINS_CONTROL_CHAR_RE = re.compile(r"[^-!#$%&'*+.^_`|~0-9a-zA-Z]") _HAS_SYS_AUDIT = hasattr(sys, "audit") class HTTPConnection(_HTTPConnection): """ Based on :class:`http.client.HTTPConnection` but provides an extra constructor backwards-compatibility layer between older and newer Pythons. Additional keyword parameters are used to configure attributes of the connection. Accepted parameters include: - ``source_address``: Set the source address for the current connection. - ``socket_options``: Set specific options on the underlying socket. If not specified, then defaults are loaded from ``HTTPConnection.default_socket_options`` which includes disabling Nagle's algorithm (sets TCP_NODELAY to 1) unless the connection is behind a proxy. For example, if you wish to enable TCP Keep Alive in addition to the defaults, you might pass: .. code-block:: python HTTPConnection.default_socket_options + [ (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1), ] Or you may want to disable the defaults by passing an empty list (e.g., ``[]``). """ default_port: typing.ClassVar[int] = port_by_scheme["http"] # type: ignore[misc] #: Disable Nagle's algorithm by default. #: ``[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]`` default_socket_options: typing.ClassVar[connection._TYPE_SOCKET_OPTIONS] = [ (socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) ] #: Whether this connection verifies the host's certificate. is_verified: bool = False #: Whether this proxy connection verified the proxy host's certificate. # If no proxy is currently connected to the value will be ``None``. proxy_is_verified: bool | None = None blocksize: int source_address: tuple[str, int] | None socket_options: connection._TYPE_SOCKET_OPTIONS | None _has_connected_to_proxy: bool _response_options: _ResponseOptions | None _tunnel_host: str | None _tunnel_port: int | None _tunnel_scheme: str | None def __init__( self, host: str, port: int | None = None, *, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, source_address: tuple[str, int] | None = None, blocksize: int = 16384, socket_options: None | (connection._TYPE_SOCKET_OPTIONS) = default_socket_options, proxy: Url | None = None, proxy_config: ProxyConfig | None = None, ) -> None: super().__init__( host=host, port=port, timeout=Timeout.resolve_default_timeout(timeout), source_address=source_address, blocksize=blocksize, ) self.socket_options = socket_options self.proxy = proxy self.proxy_config = proxy_config self._has_connected_to_proxy = False self._response_options = None self._tunnel_host: str | None = None self._tunnel_port: int | None = None self._tunnel_scheme: str | None = None # https://github.com/python/mypy/issues/4125 # Mypy treats this as LSP violation, which is considered a bug. # If `host` is made a property it violates LSP, because a writeable attribute is overridden with a read-only one. # However, there is also a `host` setter so LSP is not violated. # Potentially, a `@host.deleter` might be needed depending on how this issue will be fixed. @property def host(self) -> str: """ Getter method to remove any trailing dots that indicate the hostname is an FQDN. In general, SSL certificates don't include the trailing dot indicating a fully-qualified domain name, and thus, they don't validate properly when checked against a domain name that includes the dot. In addition, some servers may not expect to receive the trailing dot when provided. However, the hostname with trailing dot is critical to DNS resolution; doing a lookup with the trailing dot will properly only resolve the appropriate FQDN, whereas a lookup without a trailing dot will search the system's search domain list. Thus, it's important to keep the original host around for use only in those cases where it's appropriate (i.e., when doing DNS lookup to establish the actual TCP connection across which we're going to send HTTP requests). """ return self._dns_host.rstrip(".") @host.setter def host(self, value: str) -> None: """ Setter for the `host` property. We assume that only urllib3 uses the _dns_host attribute; httplib itself only uses `host`, and it seems reasonable that other libraries follow suit. """ self._dns_host = value def _new_conn(self) -> socket.socket: """Establish a socket connection and set nodelay settings on it. :return: New socket connection. """ try: sock = connection.create_connection( (self._dns_host, self.port), self.timeout, source_address=self.source_address, socket_options=self.socket_options, ) except socket.gaierror as e: raise NameResolutionError(self.host, self, e) from e except SocketTimeout as e: raise ConnectTimeoutError( self, f"Connection to {self.host} timed out. (connect timeout={self.timeout})", ) from e except OSError as e: raise NewConnectionError( self, f"Failed to establish a new connection: {e}" ) from e # Audit hooks are only available in Python 3.8+ if _HAS_SYS_AUDIT: sys.audit("http.client.connect", self, self.host, self.port) return sock def set_tunnel( self, host: str, port: int | None = None, headers: typing.Mapping[str, str] | None = None, scheme: str = "http", ) -> None: if scheme not in ("http", "https"): raise ValueError( f"Invalid proxy scheme for tunneling: {scheme!r}, must be either 'http' or 'https'" ) super().set_tunnel(host, port=port, headers=headers) self._tunnel_scheme = scheme def connect(self) -> None: self.sock = self._new_conn() if self._tunnel_host: # If we're tunneling it means we're connected to our proxy. self._has_connected_to_proxy = True # TODO: Fix tunnel so it doesn't depend on self.sock state. self._tunnel() # type: ignore[attr-defined] # If there's a proxy to be connected to we are fully connected. # This is set twice (once above and here) due to forwarding proxies # not using tunnelling. self._has_connected_to_proxy = bool(self.proxy) @property def is_closed(self) -> bool: return self.sock is None @property def is_connected(self) -> bool: if self.sock is None: return False return not wait_for_read(self.sock, timeout=0.0) @property def has_connected_to_proxy(self) -> bool: return self._has_connected_to_proxy def close(self) -> None: try: super().close() finally: # Reset all stateful properties so connection # can be re-used without leaking prior configs. self.sock = None self.is_verified = False self.proxy_is_verified = None self._has_connected_to_proxy = False self._response_options = None self._tunnel_host = None self._tunnel_port = None self._tunnel_scheme = None def putrequest( self, method: str, url: str, skip_host: bool = False, skip_accept_encoding: bool = False, ) -> None: """""" # Empty docstring because the indentation of CPython's implementation # is broken but we don't want this method in our documentation. match = _CONTAINS_CONTROL_CHAR_RE.search(method) if match: raise ValueError( f"Method cannot contain non-token characters {method!r} (found at least {match.group()!r})" ) return super().putrequest( method, url, skip_host=skip_host, skip_accept_encoding=skip_accept_encoding ) def putheader(self, header: str, *values: str) -> None: """""" if not any(isinstance(v, str) and v == SKIP_HEADER for v in values): super().putheader(header, *values) elif to_str(header.lower()) not in SKIPPABLE_HEADERS: skippable_headers = "', '".join( [str.title(header) for header in sorted(SKIPPABLE_HEADERS)] ) raise ValueError( f"urllib3.util.SKIP_HEADER only supports '{skippable_headers}'" ) # `request` method's signature intentionally violates LSP. # urllib3's API is different from `http.client.HTTPConnection` and the subclassing is only incidental. def request( # type: ignore[override] self, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, *, chunked: bool = False, preload_content: bool = True, decode_content: bool = True, enforce_content_length: bool = True, ) -> None: # Update the inner socket's timeout value to send the request. # This only triggers if the connection is re-used. if self.sock is not None: self.sock.settimeout(self.timeout) # Store these values to be fed into the HTTPResponse # object later. TODO: Remove this in favor of a real # HTTP lifecycle mechanism. # We have to store these before we call .request() # because sometimes we can still salvage a response # off the wire even if we aren't able to completely # send the request body. self._response_options = _ResponseOptions( request_method=method, request_url=url, preload_content=preload_content, decode_content=decode_content, enforce_content_length=enforce_content_length, ) if headers is None: headers = {} header_keys = frozenset(to_str(k.lower()) for k in headers) skip_accept_encoding = "accept-encoding" in header_keys skip_host = "host" in header_keys self.putrequest( method, url, skip_accept_encoding=skip_accept_encoding, skip_host=skip_host ) # Transform the body into an iterable of sendall()-able chunks # and detect if an explicit Content-Length is doable. chunks_and_cl = body_to_chunks(body, method=method, blocksize=self.blocksize) chunks = chunks_and_cl.chunks content_length = chunks_and_cl.content_length # When chunked is explicit set to 'True' we respect that. if chunked: if "transfer-encoding" not in header_keys: self.putheader("Transfer-Encoding", "chunked") else: # Detect whether a framing mechanism is already in use. If so # we respect that value, otherwise we pick chunked vs content-length # depending on the type of 'body'. if "content-length" in header_keys: chunked = False elif "transfer-encoding" in header_keys: chunked = True # Otherwise we go off the recommendation of 'body_to_chunks()'. else: chunked = False if content_length is None: if chunks is not None: chunked = True self.putheader("Transfer-Encoding", "chunked") else: self.putheader("Content-Length", str(content_length)) # Now that framing headers are out of the way we send all the other headers. if "user-agent" not in header_keys: self.putheader("User-Agent", _get_default_user_agent()) for header, value in headers.items(): self.putheader(header, value) self.endheaders() # If we're given a body we start sending that in chunks. if chunks is not None: for chunk in chunks: # Sending empty chunks isn't allowed for TE: chunked # as it indicates the end of the body. if not chunk: continue if isinstance(chunk, str): chunk = chunk.encode("utf-8") if chunked: self.send(b"%x\r\n%b\r\n" % (len(chunk), chunk)) else: self.send(chunk) # Regardless of whether we have a body or not, if we're in # chunked mode we want to send an explicit empty chunk. if chunked: self.send(b"0\r\n\r\n") def request_chunked( self, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, ) -> None: """ Alternative to the common request method, which sends the body with chunked encoding and not as one block """ warnings.warn( "HTTPConnection.request_chunked() is deprecated and will be removed " "in urllib3 v2.1.0. Instead use HTTPConnection.request(..., chunked=True).", category=DeprecationWarning, stacklevel=2, ) self.request(method, url, body=body, headers=headers, chunked=True) def getresponse( # type: ignore[override] self, ) -> HTTPResponse: """ Get the response from the server. If the HTTPConnection is in the correct state, returns an instance of HTTPResponse or of whatever object is returned by the response_class variable. If a request has not been sent or if a previous response has not be handled, ResponseNotReady is raised. If the HTTP response indicates that the connection should be closed, then it will be closed before the response is returned. When the connection is closed, the underlying socket is closed. """ # Raise the same error as http.client.HTTPConnection if self._response_options is None: raise ResponseNotReady() # Reset this attribute for being used again. resp_options = self._response_options self._response_options = None # Since the connection's timeout value may have been updated # we need to set the timeout on the socket. self.sock.settimeout(self.timeout) # This is needed here to avoid circular import errors from .response import HTTPResponse # Get the response from http.client.HTTPConnection httplib_response = super().getresponse() try: assert_header_parsing(httplib_response.msg) except (HeaderParsingError, TypeError) as hpe: log.warning( "Failed to parse headers (url=%s): %s", _url_from_connection(self, resp_options.request_url), hpe, exc_info=True, ) headers = HTTPHeaderDict(httplib_response.msg.items()) response = HTTPResponse( body=httplib_response, headers=headers, status=httplib_response.status, version=httplib_response.version, reason=httplib_response.reason, preload_content=resp_options.preload_content, decode_content=resp_options.decode_content, original_response=httplib_response, enforce_content_length=resp_options.enforce_content_length, request_method=resp_options.request_method, request_url=resp_options.request_url, ) return response class HTTPSConnection(HTTPConnection): """ Many of the parameters to this constructor are passed to the underlying SSL socket by means of :py:func:`urllib3.util.ssl_wrap_socket`. """ default_port = port_by_scheme["https"] # type: ignore[misc] cert_reqs: int | str | None = None ca_certs: str | None = None ca_cert_dir: str | None = None ca_cert_data: None | str | bytes = None ssl_version: int | str | None = None ssl_minimum_version: int | None = None ssl_maximum_version: int | None = None assert_fingerprint: str | None = None def __init__( self, host: str, port: int | None = None, *, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, source_address: tuple[str, int] | None = None, blocksize: int = 16384, socket_options: None | (connection._TYPE_SOCKET_OPTIONS) = HTTPConnection.default_socket_options, proxy: Url | None = None, proxy_config: ProxyConfig | None = None, cert_reqs: int | str | None = None, assert_hostname: None | str | Literal[False] = None, assert_fingerprint: str | None = None, server_hostname: str | None = None, ssl_context: ssl.SSLContext | None = None, ca_certs: str | None = None, ca_cert_dir: str | None = None, ca_cert_data: None | str | bytes = None, ssl_minimum_version: int | None = None, ssl_maximum_version: int | None = None, ssl_version: int | str | None = None, # Deprecated cert_file: str | None = None, key_file: str | None = None, key_password: str | None = None, ) -> None: super().__init__( host, port=port, timeout=timeout, source_address=source_address, blocksize=blocksize, socket_options=socket_options, proxy=proxy, proxy_config=proxy_config, ) self.key_file = key_file self.cert_file = cert_file self.key_password = key_password self.ssl_context = ssl_context self.server_hostname = server_hostname self.assert_hostname = assert_hostname self.assert_fingerprint = assert_fingerprint self.ssl_version = ssl_version self.ssl_minimum_version = ssl_minimum_version self.ssl_maximum_version = ssl_maximum_version self.ca_certs = ca_certs and os.path.expanduser(ca_certs) self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir) self.ca_cert_data = ca_cert_data # cert_reqs depends on ssl_context so calculate last. if cert_reqs is None: if self.ssl_context is not None: cert_reqs = self.ssl_context.verify_mode else: cert_reqs = resolve_cert_reqs(None) self.cert_reqs = cert_reqs def set_cert( self, key_file: str | None = None, cert_file: str | None = None, cert_reqs: int | str | None = None, key_password: str | None = None, ca_certs: str | None = None, assert_hostname: None | str | Literal[False] = None, assert_fingerprint: str | None = None, ca_cert_dir: str | None = None, ca_cert_data: None | str | bytes = None, ) -> None: """ This method should only be called once, before the connection is used. """ warnings.warn( "HTTPSConnection.set_cert() is deprecated and will be removed " "in urllib3 v2.1.0. Instead provide the parameters to the " "HTTPSConnection constructor.", category=DeprecationWarning, stacklevel=2, ) # If cert_reqs is not provided we'll assume CERT_REQUIRED unless we also # have an SSLContext object in which case we'll use its verify_mode. if cert_reqs is None: if self.ssl_context is not None: cert_reqs = self.ssl_context.verify_mode else: cert_reqs = resolve_cert_reqs(None) self.key_file = key_file self.cert_file = cert_file self.cert_reqs = cert_reqs self.key_password = key_password self.assert_hostname = assert_hostname self.assert_fingerprint = assert_fingerprint self.ca_certs = ca_certs and os.path.expanduser(ca_certs) self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir) self.ca_cert_data = ca_cert_data def connect(self) -> None: sock: socket.socket | ssl.SSLSocket self.sock = sock = self._new_conn() server_hostname: str = self.host tls_in_tls = False # Do we need to establish a tunnel? if self._tunnel_host is not None: # We're tunneling to an HTTPS origin so need to do TLS-in-TLS. if self._tunnel_scheme == "https": self.sock = sock = self._connect_tls_proxy(self.host, sock) tls_in_tls = True # If we're tunneling it means we're connected to our proxy. self._has_connected_to_proxy = True self._tunnel() # type: ignore[attr-defined] # Override the host with the one we're requesting data from. server_hostname = self._tunnel_host if self.server_hostname is not None: server_hostname = self.server_hostname is_time_off = datetime.date.today() < RECENT_DATE if is_time_off: warnings.warn( ( f"System time is way off (before {RECENT_DATE}). This will probably " "lead to SSL verification errors" ), SystemTimeWarning, ) sock_and_verified = _ssl_wrap_socket_and_match_hostname( sock=sock, cert_reqs=self.cert_reqs, ssl_version=self.ssl_version, ssl_minimum_version=self.ssl_minimum_version, ssl_maximum_version=self.ssl_maximum_version, ca_certs=self.ca_certs, ca_cert_dir=self.ca_cert_dir, ca_cert_data=self.ca_cert_data, cert_file=self.cert_file, key_file=self.key_file, key_password=self.key_password, server_hostname=server_hostname, ssl_context=self.ssl_context, tls_in_tls=tls_in_tls, assert_hostname=self.assert_hostname, assert_fingerprint=self.assert_fingerprint, ) self.sock = sock_and_verified.socket self.is_verified = sock_and_verified.is_verified # If there's a proxy to be connected to we are fully connected. # This is set twice (once above and here) due to forwarding proxies # not using tunnelling. self._has_connected_to_proxy = bool(self.proxy) def _connect_tls_proxy(self, hostname: str, sock: socket.socket) -> ssl.SSLSocket: """ Establish a TLS connection to the proxy using the provided SSL context. """ # `_connect_tls_proxy` is called when self._tunnel_host is truthy. proxy_config = typing.cast(ProxyConfig, self.proxy_config) ssl_context = proxy_config.ssl_context sock_and_verified = _ssl_wrap_socket_and_match_hostname( sock, cert_reqs=self.cert_reqs, ssl_version=self.ssl_version, ssl_minimum_version=self.ssl_minimum_version, ssl_maximum_version=self.ssl_maximum_version, ca_certs=self.ca_certs, ca_cert_dir=self.ca_cert_dir, ca_cert_data=self.ca_cert_data, server_hostname=hostname, ssl_context=ssl_context, assert_hostname=proxy_config.assert_hostname, assert_fingerprint=proxy_config.assert_fingerprint, # Features that aren't implemented for proxies yet: cert_file=None, key_file=None, key_password=None, tls_in_tls=False, ) self.proxy_is_verified = sock_and_verified.is_verified return sock_and_verified.socket # type: ignore[return-value] class _WrappedAndVerifiedSocket(typing.NamedTuple): """ Wrapped socket and whether the connection is verified after the TLS handshake """ socket: ssl.SSLSocket | SSLTransport is_verified: bool def _ssl_wrap_socket_and_match_hostname( sock: socket.socket, *, cert_reqs: None | str | int, ssl_version: None | str | int, ssl_minimum_version: int | None, ssl_maximum_version: int | None, cert_file: str | None, key_file: str | None, key_password: str | None, ca_certs: str | None, ca_cert_dir: str | None, ca_cert_data: None | str | bytes, assert_hostname: None | str | Literal[False], assert_fingerprint: str | None, server_hostname: str | None, ssl_context: ssl.SSLContext | None, tls_in_tls: bool = False, ) -> _WrappedAndVerifiedSocket: """Logic for constructing an SSLContext from all TLS parameters, passing that down into ssl_wrap_socket, and then doing certificate verification either via hostname or fingerprint. This function exists to guarantee that both proxies and targets have the same behavior when connecting via TLS. """ default_ssl_context = False if ssl_context is None: default_ssl_context = True context = create_urllib3_context( ssl_version=resolve_ssl_version(ssl_version), ssl_minimum_version=ssl_minimum_version, ssl_maximum_version=ssl_maximum_version, cert_reqs=resolve_cert_reqs(cert_reqs), ) else: context = ssl_context context.verify_mode = resolve_cert_reqs(cert_reqs) # In some cases, we want to verify hostnames ourselves if ( # `ssl` can't verify fingerprints or alternate hostnames assert_fingerprint or assert_hostname # assert_hostname can be set to False to disable hostname checking or assert_hostname is False # We still support OpenSSL 1.0.2, which prevents us from verifying # hostnames easily: https://github.com/pyca/pyopenssl/pull/933 or ssl_.IS_PYOPENSSL or not ssl_.HAS_NEVER_CHECK_COMMON_NAME ): context.check_hostname = False # Try to load OS default certs if none are given. # We need to do the hasattr() check for our custom # pyOpenSSL and SecureTransport SSLContext objects # because neither support load_default_certs(). if ( not ca_certs and not ca_cert_dir and not ca_cert_data and default_ssl_context and hasattr(context, "load_default_certs") ): context.load_default_certs() # Ensure that IPv6 addresses are in the proper format and don't have a # scope ID. Python's SSL module fails to recognize scoped IPv6 addresses # and interprets them as DNS hostnames. if server_hostname is not None: normalized = server_hostname.strip("[]") if "%" in normalized: normalized = normalized[: normalized.rfind("%")] if is_ipaddress(normalized): server_hostname = normalized ssl_sock = ssl_wrap_socket( sock=sock, keyfile=key_file, certfile=cert_file, key_password=key_password, ca_certs=ca_certs, ca_cert_dir=ca_cert_dir, ca_cert_data=ca_cert_data, server_hostname=server_hostname, ssl_context=context, tls_in_tls=tls_in_tls, ) try: if assert_fingerprint: _assert_fingerprint( ssl_sock.getpeercert(binary_form=True), assert_fingerprint ) elif ( context.verify_mode != ssl.CERT_NONE and not context.check_hostname and assert_hostname is not False ): cert: _TYPE_PEER_CERT_RET_DICT = ssl_sock.getpeercert() # type: ignore[assignment] # Need to signal to our match_hostname whether to use 'commonName' or not. # If we're using our own constructed SSLContext we explicitly set 'False' # because PyPy hard-codes 'True' from SSLContext.hostname_checks_common_name. if default_ssl_context: hostname_checks_common_name = False else: hostname_checks_common_name = ( getattr(context, "hostname_checks_common_name", False) or False ) _match_hostname( cert, assert_hostname or server_hostname, # type: ignore[arg-type] hostname_checks_common_name, ) return _WrappedAndVerifiedSocket( socket=ssl_sock, is_verified=context.verify_mode == ssl.CERT_REQUIRED or bool(assert_fingerprint), ) except BaseException: ssl_sock.close() raise def _match_hostname( cert: _TYPE_PEER_CERT_RET_DICT | None, asserted_hostname: str, hostname_checks_common_name: bool = False, ) -> None: # Our upstream implementation of ssl.match_hostname() # only applies this normalization to IP addresses so it doesn't # match DNS SANs so we do the same thing! stripped_hostname = asserted_hostname.strip("[]") if is_ipaddress(stripped_hostname): asserted_hostname = stripped_hostname try: match_hostname(cert, asserted_hostname, hostname_checks_common_name) except CertificateError as e: log.warning( "Certificate did not match expected hostname: %s. Certificate: %s", asserted_hostname, cert, ) # Add cert to exception and reraise so client code can inspect # the cert when catching the exception, if they want to e._peer_cert = cert # type: ignore[attr-defined] raise def _wrap_proxy_error(err: Exception, proxy_scheme: str | None) -> ProxyError: # Look for the phrase 'wrong version number', if found # then we should warn the user that we're very sure that # this proxy is HTTP-only and they have a configuration issue. error_normalized = " ".join(re.split("[^a-z]", str(err).lower())) is_likely_http_proxy = ( "wrong version number" in error_normalized or "unknown protocol" in error_normalized ) http_proxy_warning = ( ". Your proxy appears to only use HTTP and not HTTPS, " "try changing your proxy URL to be HTTP. See: " "https://urllib3.readthedocs.io/en/latest/advanced-usage.html" "#https-proxy-error-http-proxy" ) new_err = ProxyError( f"Unable to connect to proxy" f"{http_proxy_warning if is_likely_http_proxy and proxy_scheme == 'https' else ''}", err, ) new_err.__cause__ = err return new_err def _get_default_user_agent() -> str: return f"python-urllib3/{__version__}" class DummyConnection: """Used to detect a failed ConnectionCls import.""" if not ssl: HTTPSConnection = DummyConnection # type: ignore[misc, assignment] # noqa: F811 VerifiedHTTPSConnection = HTTPSConnection def _url_from_connection( conn: HTTPConnection | HTTPSConnection, path: str | None = None ) -> str: """Returns the URL from a given connection. This is mainly used for testing and logging.""" scheme = "https" if isinstance(conn, HTTPSConnection) else "http" return Url(scheme=scheme, host=conn.host, port=conn.port, path=path).url urllib3-2.0.7/src/urllib3/connectionpool.py0000644000000000000000000012437014513541732015604 0ustar00from __future__ import annotations import errno import logging import queue import sys import typing import warnings import weakref from socket import timeout as SocketTimeout from types import TracebackType from ._base_connection import _TYPE_BODY from ._collections import HTTPHeaderDict from ._request_methods import RequestMethods from .connection import ( BaseSSLError, BrokenPipeError, DummyConnection, HTTPConnection, HTTPException, HTTPSConnection, ProxyConfig, _wrap_proxy_error, ) from .connection import port_by_scheme as port_by_scheme from .exceptions import ( ClosedPoolError, EmptyPoolError, FullPoolError, HostChangedError, InsecureRequestWarning, LocationValueError, MaxRetryError, NewConnectionError, ProtocolError, ProxyError, ReadTimeoutError, SSLError, TimeoutError, ) from .response import BaseHTTPResponse from .util.connection import is_connection_dropped from .util.proxy import connection_requires_http_tunnel from .util.request import _TYPE_BODY_POSITION, set_file_position from .util.retry import Retry from .util.ssl_match_hostname import CertificateError from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_DEFAULT, Timeout from .util.url import Url, _encode_target from .util.url import _normalize_host as normalize_host from .util.url import parse_url from .util.util import to_str if typing.TYPE_CHECKING: import ssl from typing_extensions import Literal from ._base_connection import BaseHTTPConnection, BaseHTTPSConnection log = logging.getLogger(__name__) _TYPE_TIMEOUT = typing.Union[Timeout, float, _TYPE_DEFAULT, None] _SelfT = typing.TypeVar("_SelfT") # Pool objects class ConnectionPool: """ Base class for all connection pools, such as :class:`.HTTPConnectionPool` and :class:`.HTTPSConnectionPool`. .. note:: ConnectionPool.urlopen() does not normalize or percent-encode target URIs which is useful if your target server doesn't support percent-encoded target URIs. """ scheme: str | None = None QueueCls = queue.LifoQueue def __init__(self, host: str, port: int | None = None) -> None: if not host: raise LocationValueError("No host specified.") self.host = _normalize_host(host, scheme=self.scheme) self.port = port # This property uses 'normalize_host()' (not '_normalize_host()') # to avoid removing square braces around IPv6 addresses. # This value is sent to `HTTPConnection.set_tunnel()` if called # because square braces are required for HTTP CONNECT tunneling. self._tunnel_host = normalize_host(host, scheme=self.scheme).lower() def __str__(self) -> str: return f"{type(self).__name__}(host={self.host!r}, port={self.port!r})" def __enter__(self: _SelfT) -> _SelfT: return self def __exit__( self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None, ) -> Literal[False]: self.close() # Return False to re-raise any potential exceptions return False def close(self) -> None: """ Close all pooled connections and disable the pool. """ # This is taken from http://hg.python.org/cpython/file/7aaba721ebc0/Lib/socket.py#l252 _blocking_errnos = {errno.EAGAIN, errno.EWOULDBLOCK} class HTTPConnectionPool(ConnectionPool, RequestMethods): """ Thread-safe connection pool for one host. :param host: Host used for this HTTP Connection (e.g. "localhost"), passed into :class:`http.client.HTTPConnection`. :param port: Port used for this HTTP Connection (None is equivalent to 80), passed into :class:`http.client.HTTPConnection`. :param timeout: Socket timeout in seconds for each individual connection. This can be a float or integer, which sets the timeout for the HTTP request, or an instance of :class:`urllib3.util.Timeout` which gives you more fine-grained control over request timeouts. After the constructor has been parsed, this is always a `urllib3.util.Timeout` object. :param maxsize: Number of connections to save that can be reused. More than 1 is useful in multithreaded situations. If ``block`` is set to False, more connections will be created but they will not be saved once they've been used. :param block: If set to True, no more than ``maxsize`` connections will be used at a time. When no free connections are available, the call will block until a connection has been released. This is a useful side effect for particular multithreaded situations where one does not want to use more than maxsize connections per host to prevent flooding. :param headers: Headers to include with all requests, unless other headers are given explicitly. :param retries: Retry configuration to use by default with requests in this pool. :param _proxy: Parsed proxy URL, should not be used directly, instead, see :class:`urllib3.ProxyManager` :param _proxy_headers: A dictionary with proxy headers, should not be used directly, instead, see :class:`urllib3.ProxyManager` :param \\**conn_kw: Additional parameters are used to create fresh :class:`urllib3.connection.HTTPConnection`, :class:`urllib3.connection.HTTPSConnection` instances. """ scheme = "http" ConnectionCls: ( type[BaseHTTPConnection] | type[BaseHTTPSConnection] ) = HTTPConnection def __init__( self, host: str, port: int | None = None, timeout: _TYPE_TIMEOUT | None = _DEFAULT_TIMEOUT, maxsize: int = 1, block: bool = False, headers: typing.Mapping[str, str] | None = None, retries: Retry | bool | int | None = None, _proxy: Url | None = None, _proxy_headers: typing.Mapping[str, str] | None = None, _proxy_config: ProxyConfig | None = None, **conn_kw: typing.Any, ): ConnectionPool.__init__(self, host, port) RequestMethods.__init__(self, headers) if not isinstance(timeout, Timeout): timeout = Timeout.from_float(timeout) if retries is None: retries = Retry.DEFAULT self.timeout = timeout self.retries = retries self.pool: queue.LifoQueue[typing.Any] | None = self.QueueCls(maxsize) self.block = block self.proxy = _proxy self.proxy_headers = _proxy_headers or {} self.proxy_config = _proxy_config # Fill the queue up so that doing get() on it will block properly for _ in range(maxsize): self.pool.put(None) # These are mostly for testing and debugging purposes. self.num_connections = 0 self.num_requests = 0 self.conn_kw = conn_kw if self.proxy: # Enable Nagle's algorithm for proxies, to avoid packet fragmentation. # We cannot know if the user has added default socket options, so we cannot replace the # list. self.conn_kw.setdefault("socket_options", []) self.conn_kw["proxy"] = self.proxy self.conn_kw["proxy_config"] = self.proxy_config # Do not pass 'self' as callback to 'finalize'. # Then the 'finalize' would keep an endless living (leak) to self. # By just passing a reference to the pool allows the garbage collector # to free self if nobody else has a reference to it. pool = self.pool # Close all the HTTPConnections in the pool before the # HTTPConnectionPool object is garbage collected. weakref.finalize(self, _close_pool_connections, pool) def _new_conn(self) -> BaseHTTPConnection: """ Return a fresh :class:`HTTPConnection`. """ self.num_connections += 1 log.debug( "Starting new HTTP connection (%d): %s:%s", self.num_connections, self.host, self.port or "80", ) conn = self.ConnectionCls( host=self.host, port=self.port, timeout=self.timeout.connect_timeout, **self.conn_kw, ) return conn def _get_conn(self, timeout: float | None = None) -> BaseHTTPConnection: """ Get a connection. Will return a pooled connection if one is available. If no connections are available and :prop:`.block` is ``False``, then a fresh connection is returned. :param timeout: Seconds to wait before giving up and raising :class:`urllib3.exceptions.EmptyPoolError` if the pool is empty and :prop:`.block` is ``True``. """ conn = None if self.pool is None: raise ClosedPoolError(self, "Pool is closed.") try: conn = self.pool.get(block=self.block, timeout=timeout) except AttributeError: # self.pool is None raise ClosedPoolError(self, "Pool is closed.") from None # Defensive: except queue.Empty: if self.block: raise EmptyPoolError( self, "Pool is empty and a new connection can't be opened due to blocking mode.", ) from None pass # Oh well, we'll create a new connection then # If this is a persistent connection, check if it got disconnected if conn and is_connection_dropped(conn): log.debug("Resetting dropped connection: %s", self.host) conn.close() return conn or self._new_conn() def _put_conn(self, conn: BaseHTTPConnection | None) -> None: """ Put a connection back into the pool. :param conn: Connection object for the current host and port as returned by :meth:`._new_conn` or :meth:`._get_conn`. If the pool is already full, the connection is closed and discarded because we exceeded maxsize. If connections are discarded frequently, then maxsize should be increased. If the pool is closed, then the connection will be closed and discarded. """ if self.pool is not None: try: self.pool.put(conn, block=False) return # Everything is dandy, done. except AttributeError: # self.pool is None. pass except queue.Full: # Connection never got put back into the pool, close it. if conn: conn.close() if self.block: # This should never happen if you got the conn from self._get_conn raise FullPoolError( self, "Pool reached maximum size and no more connections are allowed.", ) from None log.warning( "Connection pool is full, discarding connection: %s. Connection pool size: %s", self.host, self.pool.qsize(), ) # Connection never got put back into the pool, close it. if conn: conn.close() def _validate_conn(self, conn: BaseHTTPConnection) -> None: """ Called right before a request is made, after the socket is created. """ def _prepare_proxy(self, conn: BaseHTTPConnection) -> None: # Nothing to do for HTTP connections. pass def _get_timeout(self, timeout: _TYPE_TIMEOUT) -> Timeout: """Helper that always returns a :class:`urllib3.util.Timeout`""" if timeout is _DEFAULT_TIMEOUT: return self.timeout.clone() if isinstance(timeout, Timeout): return timeout.clone() else: # User passed us an int/float. This is for backwards compatibility, # can be removed later return Timeout.from_float(timeout) def _raise_timeout( self, err: BaseSSLError | OSError | SocketTimeout, url: str, timeout_value: _TYPE_TIMEOUT | None, ) -> None: """Is the error actually a timeout? Will raise a ReadTimeout or pass""" if isinstance(err, SocketTimeout): raise ReadTimeoutError( self, url, f"Read timed out. (read timeout={timeout_value})" ) from err # See the above comment about EAGAIN in Python 3. if hasattr(err, "errno") and err.errno in _blocking_errnos: raise ReadTimeoutError( self, url, f"Read timed out. (read timeout={timeout_value})" ) from err def _make_request( self, conn: BaseHTTPConnection, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, retries: Retry | None = None, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, chunked: bool = False, response_conn: BaseHTTPConnection | None = None, preload_content: bool = True, decode_content: bool = True, enforce_content_length: bool = True, ) -> BaseHTTPResponse: """ Perform a request on a given urllib connection object taken from our pool. :param conn: a connection from one of our connection pools :param method: HTTP request method (such as GET, POST, PUT, etc.) :param url: The URL to perform the request on. :param body: Data to send in the request body, either :class:`str`, :class:`bytes`, an iterable of :class:`str`/:class:`bytes`, or a file-like object. :param headers: Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers. :param retries: Configure the number of retries to allow before raising a :class:`~urllib3.exceptions.MaxRetryError` exception. Pass ``None`` to retry until you receive a response. Pass a :class:`~urllib3.util.retry.Retry` object for fine-grained control over different types of retries. Pass an integer number to retry connection errors that many times, but no other types of errors. Pass zero to never retry. If ``False``, then retries are disabled and any exception is raised immediately. Also, instead of raising a MaxRetryError on redirects, the redirect response will be returned. :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. :param timeout: If specified, overrides the default timeout for this one request. It may be a float (in seconds) or an instance of :class:`urllib3.util.Timeout`. :param chunked: If True, urllib3 will send the body using chunked transfer encoding. Otherwise, urllib3 will send the body using the standard content-length form. Defaults to False. :param response_conn: Set this to ``None`` if you will handle releasing the connection or set the connection to have the response release it. :param preload_content: If True, the response's body will be preloaded during construction. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param enforce_content_length: Enforce content length checking. Body returned by server must match value of Content-Length header, if present. Otherwise, raise error. """ self.num_requests += 1 timeout_obj = self._get_timeout(timeout) timeout_obj.start_connect() conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) try: # Trigger any extra validation we need to do. try: self._validate_conn(conn) except (SocketTimeout, BaseSSLError) as e: self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) raise # _validate_conn() starts the connection to an HTTPS proxy # so we need to wrap errors with 'ProxyError' here too. except ( OSError, NewConnectionError, TimeoutError, BaseSSLError, CertificateError, SSLError, ) as e: new_e: Exception = e if isinstance(e, (BaseSSLError, CertificateError)): new_e = SSLError(e) # If the connection didn't successfully connect to it's proxy # then there if isinstance( new_e, (OSError, NewConnectionError, TimeoutError, SSLError) ) and (conn and conn.proxy and not conn.has_connected_to_proxy): new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) raise new_e # conn.request() calls http.client.*.request, not the method in # urllib3.request. It also calls makefile (recv) on the socket. try: conn.request( method, url, body=body, headers=headers, chunked=chunked, preload_content=preload_content, decode_content=decode_content, enforce_content_length=enforce_content_length, ) # We are swallowing BrokenPipeError (errno.EPIPE) since the server is # legitimately able to close the connection after sending a valid response. # With this behaviour, the received response is still readable. except BrokenPipeError: pass except OSError as e: # MacOS/Linux # EPROTOTYPE is needed on macOS # https://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ if e.errno != errno.EPROTOTYPE: raise # Reset the timeout for the recv() on the socket read_timeout = timeout_obj.read_timeout if not conn.is_closed: # In Python 3 socket.py will catch EAGAIN and return None when you # try and read into the file pointer created by http.client, which # instead raises a BadStatusLine exception. Instead of catching # the exception and assuming all BadStatusLine exceptions are read # timeouts, check for a zero timeout before making the request. if read_timeout == 0: raise ReadTimeoutError( self, url, f"Read timed out. (read timeout={read_timeout})" ) conn.timeout = read_timeout # Receive the response from the server try: response = conn.getresponse() except (BaseSSLError, OSError) as e: self._raise_timeout(err=e, url=url, timeout_value=read_timeout) raise # Set properties that are used by the pooling layer. response.retries = retries response._connection = response_conn # type: ignore[attr-defined] response._pool = self # type: ignore[attr-defined] log.debug( '%s://%s:%s "%s %s %s" %s %s', self.scheme, self.host, self.port, method, url, # HTTP version conn._http_vsn_str, # type: ignore[attr-defined] response.status, response.length_remaining, # type: ignore[attr-defined] ) return response def close(self) -> None: """ Close all pooled connections and disable the pool. """ if self.pool is None: return # Disable access to the pool old_pool, self.pool = self.pool, None # Close all the HTTPConnections in the pool. _close_pool_connections(old_pool) def is_same_host(self, url: str) -> bool: """ Check if the given ``url`` is a member of the same host as this connection pool. """ if url.startswith("/"): return True # TODO: Add optional support for socket.gethostbyname checking. scheme, _, host, port, *_ = parse_url(url) scheme = scheme or "http" if host is not None: host = _normalize_host(host, scheme=scheme) # Use explicit default port for comparison when none is given if self.port and not port: port = port_by_scheme.get(scheme) elif not self.port and port == port_by_scheme.get(scheme): port = None return (scheme, host, port) == (self.scheme, self.host, self.port) def urlopen( # type: ignore[override] self, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, retries: Retry | bool | int | None = None, redirect: bool = True, assert_same_host: bool = True, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, pool_timeout: int | None = None, release_conn: bool | None = None, chunked: bool = False, body_pos: _TYPE_BODY_POSITION | None = None, preload_content: bool = True, decode_content: bool = True, **response_kw: typing.Any, ) -> BaseHTTPResponse: """ Get a connection from the pool and perform an HTTP request. This is the lowest level call for making a request, so you'll need to specify all the raw details. .. note:: More commonly, it's appropriate to use a convenience method such as :meth:`request`. .. note:: `release_conn` will only behave as expected if `preload_content=False` because we want to make `preload_content=False` the default behaviour someday soon without breaking backwards compatibility. :param method: HTTP request method (such as GET, POST, PUT, etc.) :param url: The URL to perform the request on. :param body: Data to send in the request body, either :class:`str`, :class:`bytes`, an iterable of :class:`str`/:class:`bytes`, or a file-like object. :param headers: Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers. :param retries: Configure the number of retries to allow before raising a :class:`~urllib3.exceptions.MaxRetryError` exception. Pass ``None`` to retry until you receive a response. Pass a :class:`~urllib3.util.retry.Retry` object for fine-grained control over different types of retries. Pass an integer number to retry connection errors that many times, but no other types of errors. Pass zero to never retry. If ``False``, then retries are disabled and any exception is raised immediately. Also, instead of raising a MaxRetryError on redirects, the redirect response will be returned. :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. :param redirect: If True, automatically handle redirects (status codes 301, 302, 303, 307, 308). Each redirect counts as a retry. Disabling retries will disable redirect, too. :param assert_same_host: If ``True``, will make sure that the host of the pool requests is consistent else will raise HostChangedError. When ``False``, you can use the pool on an HTTP proxy and request foreign hosts. :param timeout: If specified, overrides the default timeout for this one request. It may be a float (in seconds) or an instance of :class:`urllib3.util.Timeout`. :param pool_timeout: If set and the pool is set to block=True, then this method will block for ``pool_timeout`` seconds and raise EmptyPoolError if no connection is available within the time period. :param bool preload_content: If True, the response's body will be preloaded into memory. :param bool decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param release_conn: If False, then the urlopen call will not release the connection back into the pool once a response is received (but will release if you read the entire contents of the response such as when `preload_content=True`). This is useful if you're not preloading the response's content immediately. You will need to call ``r.release_conn()`` on the response ``r`` to return the connection back into the pool. If None, it takes the value of ``preload_content`` which defaults to ``True``. :param bool chunked: If True, urllib3 will send the body using chunked transfer encoding. Otherwise, urllib3 will send the body using the standard content-length form. Defaults to False. :param int body_pos: Position to seek to in file-like body in the event of a retry or redirect. Typically this won't need to be set because urllib3 will auto-populate the value when needed. """ parsed_url = parse_url(url) destination_scheme = parsed_url.scheme if headers is None: headers = self.headers if not isinstance(retries, Retry): retries = Retry.from_int(retries, redirect=redirect, default=self.retries) if release_conn is None: release_conn = preload_content # Check host if assert_same_host and not self.is_same_host(url): raise HostChangedError(self, url, retries) # Ensure that the URL we're connecting to is properly encoded if url.startswith("/"): url = to_str(_encode_target(url)) else: url = to_str(parsed_url.url) conn = None # Track whether `conn` needs to be released before # returning/raising/recursing. Update this variable if necessary, and # leave `release_conn` constant throughout the function. That way, if # the function recurses, the original value of `release_conn` will be # passed down into the recursive call, and its value will be respected. # # See issue #651 [1] for details. # # [1] release_this_conn = release_conn http_tunnel_required = connection_requires_http_tunnel( self.proxy, self.proxy_config, destination_scheme ) # Merge the proxy headers. Only done when not using HTTP CONNECT. We # have to copy the headers dict so we can safely change it without those # changes being reflected in anyone else's copy. if not http_tunnel_required: headers = headers.copy() # type: ignore[attr-defined] headers.update(self.proxy_headers) # type: ignore[union-attr] # Must keep the exception bound to a separate variable or else Python 3 # complains about UnboundLocalError. err = None # Keep track of whether we cleanly exited the except block. This # ensures we do proper cleanup in finally. clean_exit = False # Rewind body position, if needed. Record current position # for future rewinds in the event of a redirect/retry. body_pos = set_file_position(body, body_pos) try: # Request a connection from the queue. timeout_obj = self._get_timeout(timeout) conn = self._get_conn(timeout=pool_timeout) conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] # Is this a closed/new connection that requires CONNECT tunnelling? if self.proxy is not None and http_tunnel_required and conn.is_closed: try: self._prepare_proxy(conn) except (BaseSSLError, OSError, SocketTimeout) as e: self._raise_timeout( err=e, url=self.proxy.url, timeout_value=conn.timeout ) raise # If we're going to release the connection in ``finally:``, then # the response doesn't need to know about the connection. Otherwise # it will also try to release it and we'll have a double-release # mess. response_conn = conn if not release_conn else None # Make the request on the HTTPConnection object response = self._make_request( conn, method, url, timeout=timeout_obj, body=body, headers=headers, chunked=chunked, retries=retries, response_conn=response_conn, preload_content=preload_content, decode_content=decode_content, **response_kw, ) # Everything went great! clean_exit = True except EmptyPoolError: # Didn't get a connection from the pool, no need to clean up clean_exit = True release_this_conn = False raise except ( TimeoutError, HTTPException, OSError, ProtocolError, BaseSSLError, SSLError, CertificateError, ProxyError, ) as e: # Discard the connection for these exceptions. It will be # replaced during the next _get_conn() call. clean_exit = False new_e: Exception = e if isinstance(e, (BaseSSLError, CertificateError)): new_e = SSLError(e) if isinstance( new_e, ( OSError, NewConnectionError, TimeoutError, SSLError, HTTPException, ), ) and (conn and conn.proxy and not conn.has_connected_to_proxy): new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) elif isinstance(new_e, (OSError, HTTPException)): new_e = ProtocolError("Connection aborted.", new_e) retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) retries.sleep() # Keep track of the error for the retry warning. err = e finally: if not clean_exit: # We hit some kind of exception, handled or otherwise. We need # to throw the connection away unless explicitly told not to. # Close the connection, set the variable to None, and make sure # we put the None back in the pool to avoid leaking it. if conn: conn.close() conn = None release_this_conn = True if release_this_conn: # Put the connection back to be reused. If the connection is # expired then it will be None, which will get replaced with a # fresh connection during _get_conn. self._put_conn(conn) if not conn: # Try again log.warning( "Retrying (%r) after connection broken by '%r': %s", retries, err, url ) return self.urlopen( method, url, body, headers, retries, redirect, assert_same_host, timeout=timeout, pool_timeout=pool_timeout, release_conn=release_conn, chunked=chunked, body_pos=body_pos, preload_content=preload_content, decode_content=decode_content, **response_kw, ) # Handle redirect? redirect_location = redirect and response.get_redirect_location() if redirect_location: if response.status == 303: # Change the method according to RFC 9110, Section 15.4.4. method = "GET" # And lose the body not to transfer anything sensitive. body = None headers = HTTPHeaderDict(headers)._prepare_for_method_change() try: retries = retries.increment(method, url, response=response, _pool=self) except MaxRetryError: if retries.raise_on_redirect: response.drain_conn() raise return response response.drain_conn() retries.sleep_for_retry(response) log.debug("Redirecting %s -> %s", url, redirect_location) return self.urlopen( method, redirect_location, body, headers, retries=retries, redirect=redirect, assert_same_host=assert_same_host, timeout=timeout, pool_timeout=pool_timeout, release_conn=release_conn, chunked=chunked, body_pos=body_pos, preload_content=preload_content, decode_content=decode_content, **response_kw, ) # Check if we should retry the HTTP response. has_retry_after = bool(response.headers.get("Retry-After")) if retries.is_retry(method, response.status, has_retry_after): try: retries = retries.increment(method, url, response=response, _pool=self) except MaxRetryError: if retries.raise_on_status: response.drain_conn() raise return response response.drain_conn() retries.sleep(response) log.debug("Retry: %s", url) return self.urlopen( method, url, body, headers, retries=retries, redirect=redirect, assert_same_host=assert_same_host, timeout=timeout, pool_timeout=pool_timeout, release_conn=release_conn, chunked=chunked, body_pos=body_pos, preload_content=preload_content, decode_content=decode_content, **response_kw, ) return response class HTTPSConnectionPool(HTTPConnectionPool): """ Same as :class:`.HTTPConnectionPool`, but HTTPS. :class:`.HTTPSConnection` uses one of ``assert_fingerprint``, ``assert_hostname`` and ``host`` in this order to verify connections. If ``assert_hostname`` is False, no verification is done. The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs``, ``ca_cert_dir``, ``ssl_version``, ``key_password`` are only used if :mod:`ssl` is available and are fed into :meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket into an SSL socket. """ scheme = "https" ConnectionCls: type[BaseHTTPSConnection] = HTTPSConnection def __init__( self, host: str, port: int | None = None, timeout: _TYPE_TIMEOUT | None = _DEFAULT_TIMEOUT, maxsize: int = 1, block: bool = False, headers: typing.Mapping[str, str] | None = None, retries: Retry | bool | int | None = None, _proxy: Url | None = None, _proxy_headers: typing.Mapping[str, str] | None = None, key_file: str | None = None, cert_file: str | None = None, cert_reqs: int | str | None = None, key_password: str | None = None, ca_certs: str | None = None, ssl_version: int | str | None = None, ssl_minimum_version: ssl.TLSVersion | None = None, ssl_maximum_version: ssl.TLSVersion | None = None, assert_hostname: str | Literal[False] | None = None, assert_fingerprint: str | None = None, ca_cert_dir: str | None = None, **conn_kw: typing.Any, ) -> None: super().__init__( host, port, timeout, maxsize, block, headers, retries, _proxy, _proxy_headers, **conn_kw, ) self.key_file = key_file self.cert_file = cert_file self.cert_reqs = cert_reqs self.key_password = key_password self.ca_certs = ca_certs self.ca_cert_dir = ca_cert_dir self.ssl_version = ssl_version self.ssl_minimum_version = ssl_minimum_version self.ssl_maximum_version = ssl_maximum_version self.assert_hostname = assert_hostname self.assert_fingerprint = assert_fingerprint def _prepare_proxy(self, conn: HTTPSConnection) -> None: # type: ignore[override] """Establishes a tunnel connection through HTTP CONNECT.""" if self.proxy and self.proxy.scheme == "https": tunnel_scheme = "https" else: tunnel_scheme = "http" conn.set_tunnel( scheme=tunnel_scheme, host=self._tunnel_host, port=self.port, headers=self.proxy_headers, ) conn.connect() def _new_conn(self) -> BaseHTTPSConnection: """ Return a fresh :class:`urllib3.connection.HTTPConnection`. """ self.num_connections += 1 log.debug( "Starting new HTTPS connection (%d): %s:%s", self.num_connections, self.host, self.port or "443", ) if not self.ConnectionCls or self.ConnectionCls is DummyConnection: # type: ignore[comparison-overlap] raise ImportError( "Can't connect to HTTPS URL because the SSL module is not available." ) actual_host: str = self.host actual_port = self.port if self.proxy is not None and self.proxy.host is not None: actual_host = self.proxy.host actual_port = self.proxy.port return self.ConnectionCls( host=actual_host, port=actual_port, timeout=self.timeout.connect_timeout, cert_file=self.cert_file, key_file=self.key_file, key_password=self.key_password, cert_reqs=self.cert_reqs, ca_certs=self.ca_certs, ca_cert_dir=self.ca_cert_dir, assert_hostname=self.assert_hostname, assert_fingerprint=self.assert_fingerprint, ssl_version=self.ssl_version, ssl_minimum_version=self.ssl_minimum_version, ssl_maximum_version=self.ssl_maximum_version, **self.conn_kw, ) def _validate_conn(self, conn: BaseHTTPConnection) -> None: """ Called right before a request is made, after the socket is created. """ super()._validate_conn(conn) # Force connect early to allow us to validate the connection. if conn.is_closed: conn.connect() if not conn.is_verified: warnings.warn( ( f"Unverified HTTPS request is being made to host '{conn.host}'. " "Adding certificate verification is strongly advised. See: " "https://urllib3.readthedocs.io/en/latest/advanced-usage.html" "#tls-warnings" ), InsecureRequestWarning, ) def connection_from_url(url: str, **kw: typing.Any) -> HTTPConnectionPool: """ Given a url, return an :class:`.ConnectionPool` instance of its host. This is a shortcut for not having to parse out the scheme, host, and port of the url before creating an :class:`.ConnectionPool` instance. :param url: Absolute URL string that must include the scheme. Port is optional. :param \\**kw: Passes additional parameters to the constructor of the appropriate :class:`.ConnectionPool`. Useful for specifying things like timeout, maxsize, headers, etc. Example:: >>> conn = connection_from_url('http://google.com/') >>> r = conn.request('GET', '/') """ scheme, _, host, port, *_ = parse_url(url) scheme = scheme or "http" port = port or port_by_scheme.get(scheme, 80) if scheme == "https": return HTTPSConnectionPool(host, port=port, **kw) # type: ignore[arg-type] else: return HTTPConnectionPool(host, port=port, **kw) # type: ignore[arg-type] @typing.overload def _normalize_host(host: None, scheme: str | None) -> None: ... @typing.overload def _normalize_host(host: str, scheme: str | None) -> str: ... def _normalize_host(host: str | None, scheme: str | None) -> str | None: """ Normalize hosts for comparisons and use with sockets. """ host = normalize_host(host, scheme) # httplib doesn't like it when we include brackets in IPv6 addresses # Specifically, if we include brackets but also pass the port then # httplib crazily doubles up the square brackets on the Host header. # Instead, we need to make sure we never pass ``None`` as the port. # However, for backward compatibility reasons we can't actually # *assert* that. See http://bugs.python.org/issue28539 if host and host.startswith("[") and host.endswith("]"): host = host[1:-1] return host def _url_from_pool( pool: HTTPConnectionPool | HTTPSConnectionPool, path: str | None = None ) -> str: """Returns the URL from a given connection pool. This is mainly used for testing and logging.""" return Url(scheme=pool.scheme, host=pool.host, port=pool.port, path=path).url def _close_pool_connections(pool: queue.LifoQueue[typing.Any]) -> None: """Drains a queue of connections and closes each one.""" try: while True: conn = pool.get(block=False) if conn: conn.close() except queue.Empty: pass # Done. urllib3-2.0.7/src/urllib3/exceptions.py0000644000000000000000000002225114513541732014727 0ustar00from __future__ import annotations import socket import typing import warnings from email.errors import MessageDefect from http.client import IncompleteRead as httplib_IncompleteRead if typing.TYPE_CHECKING: from .connection import HTTPConnection from .connectionpool import ConnectionPool from .response import HTTPResponse from .util.retry import Retry # Base Exceptions class HTTPError(Exception): """Base exception used by this module.""" class HTTPWarning(Warning): """Base warning used by this module.""" _TYPE_REDUCE_RESULT = typing.Tuple[ typing.Callable[..., object], typing.Tuple[object, ...] ] class PoolError(HTTPError): """Base exception for errors caused within a pool.""" def __init__(self, pool: ConnectionPool, message: str) -> None: self.pool = pool super().__init__(f"{pool}: {message}") def __reduce__(self) -> _TYPE_REDUCE_RESULT: # For pickling purposes. return self.__class__, (None, None) class RequestError(PoolError): """Base exception for PoolErrors that have associated URLs.""" def __init__(self, pool: ConnectionPool, url: str, message: str) -> None: self.url = url super().__init__(pool, message) def __reduce__(self) -> _TYPE_REDUCE_RESULT: # For pickling purposes. return self.__class__, (None, self.url, None) class SSLError(HTTPError): """Raised when SSL certificate fails in an HTTPS connection.""" class ProxyError(HTTPError): """Raised when the connection to a proxy fails.""" # The original error is also available as __cause__. original_error: Exception def __init__(self, message: str, error: Exception) -> None: super().__init__(message, error) self.original_error = error class DecodeError(HTTPError): """Raised when automatic decoding based on Content-Type fails.""" class ProtocolError(HTTPError): """Raised when something unexpected happens mid-request/response.""" #: Renamed to ProtocolError but aliased for backwards compatibility. ConnectionError = ProtocolError # Leaf Exceptions class MaxRetryError(RequestError): """Raised when the maximum number of retries is exceeded. :param pool: The connection pool :type pool: :class:`~urllib3.connectionpool.HTTPConnectionPool` :param str url: The requested Url :param reason: The underlying error :type reason: :class:`Exception` """ def __init__( self, pool: ConnectionPool, url: str, reason: Exception | None = None ) -> None: self.reason = reason message = f"Max retries exceeded with url: {url} (Caused by {reason!r})" super().__init__(pool, url, message) class HostChangedError(RequestError): """Raised when an existing pool gets a request for a foreign host.""" def __init__( self, pool: ConnectionPool, url: str, retries: Retry | int = 3 ) -> None: message = f"Tried to open a foreign host with url: {url}" super().__init__(pool, url, message) self.retries = retries class TimeoutStateError(HTTPError): """Raised when passing an invalid state to a timeout""" class TimeoutError(HTTPError): """Raised when a socket timeout error occurs. Catching this error will catch both :exc:`ReadTimeoutErrors ` and :exc:`ConnectTimeoutErrors `. """ class ReadTimeoutError(TimeoutError, RequestError): """Raised when a socket timeout occurs while receiving data from a server""" # This timeout error does not have a URL attached and needs to inherit from the # base HTTPError class ConnectTimeoutError(TimeoutError): """Raised when a socket timeout occurs while connecting to a server""" class NewConnectionError(ConnectTimeoutError, HTTPError): """Raised when we fail to establish a new connection. Usually ECONNREFUSED.""" def __init__(self, conn: HTTPConnection, message: str) -> None: self.conn = conn super().__init__(f"{conn}: {message}") @property def pool(self) -> HTTPConnection: warnings.warn( "The 'pool' property is deprecated and will be removed " "in urllib3 v2.1.0. Use 'conn' instead.", DeprecationWarning, stacklevel=2, ) return self.conn class NameResolutionError(NewConnectionError): """Raised when host name resolution fails.""" def __init__(self, host: str, conn: HTTPConnection, reason: socket.gaierror): message = f"Failed to resolve '{host}' ({reason})" super().__init__(conn, message) class EmptyPoolError(PoolError): """Raised when a pool runs out of connections and no more are allowed.""" class FullPoolError(PoolError): """Raised when we try to add a connection to a full pool in blocking mode.""" class ClosedPoolError(PoolError): """Raised when a request enters a pool after the pool has been closed.""" class LocationValueError(ValueError, HTTPError): """Raised when there is something wrong with a given URL input.""" class LocationParseError(LocationValueError): """Raised when get_host or similar fails to parse the URL input.""" def __init__(self, location: str) -> None: message = f"Failed to parse: {location}" super().__init__(message) self.location = location class URLSchemeUnknown(LocationValueError): """Raised when a URL input has an unsupported scheme.""" def __init__(self, scheme: str): message = f"Not supported URL scheme {scheme}" super().__init__(message) self.scheme = scheme class ResponseError(HTTPError): """Used as a container for an error reason supplied in a MaxRetryError.""" GENERIC_ERROR = "too many error responses" SPECIFIC_ERROR = "too many {status_code} error responses" class SecurityWarning(HTTPWarning): """Warned when performing security reducing actions""" class InsecureRequestWarning(SecurityWarning): """Warned when making an unverified HTTPS request.""" class NotOpenSSLWarning(SecurityWarning): """Warned when using unsupported SSL library""" class SystemTimeWarning(SecurityWarning): """Warned when system time is suspected to be wrong""" class InsecurePlatformWarning(SecurityWarning): """Warned when certain TLS/SSL configuration is not available on a platform.""" class DependencyWarning(HTTPWarning): """ Warned when an attempt is made to import a module with missing optional dependencies. """ class ResponseNotChunked(ProtocolError, ValueError): """Response needs to be chunked in order to read it as chunks.""" class BodyNotHttplibCompatible(HTTPError): """ Body should be :class:`http.client.HTTPResponse` like (have an fp attribute which returns raw chunks) for read_chunked(). """ class IncompleteRead(HTTPError, httplib_IncompleteRead): """ Response length doesn't match expected Content-Length Subclass of :class:`http.client.IncompleteRead` to allow int value for ``partial`` to avoid creating large objects on streamed reads. """ def __init__(self, partial: int, expected: int) -> None: self.partial = partial # type: ignore[assignment] self.expected = expected def __repr__(self) -> str: return "IncompleteRead(%i bytes read, %i more expected)" % ( self.partial, # type: ignore[str-format] self.expected, ) class InvalidChunkLength(HTTPError, httplib_IncompleteRead): """Invalid chunk length in a chunked response.""" def __init__(self, response: HTTPResponse, length: bytes) -> None: self.partial: int = response.tell() # type: ignore[assignment] self.expected: int | None = response.length_remaining self.response = response self.length = length def __repr__(self) -> str: return "InvalidChunkLength(got length %r, %i bytes read)" % ( self.length, self.partial, ) class InvalidHeader(HTTPError): """The header provided was somehow invalid.""" class ProxySchemeUnknown(AssertionError, URLSchemeUnknown): """ProxyManager does not support the supplied scheme""" # TODO(t-8ch): Stop inheriting from AssertionError in v2.0. def __init__(self, scheme: str | None) -> None: # 'localhost' is here because our URL parser parses # localhost:8080 -> scheme=localhost, remove if we fix this. if scheme == "localhost": scheme = None if scheme is None: message = "Proxy URL had no scheme, should start with http:// or https://" else: message = f"Proxy URL had unsupported scheme {scheme}, should use http:// or https://" super().__init__(message) class ProxySchemeUnsupported(ValueError): """Fetching HTTPS resources through HTTPS proxies is unsupported""" class HeaderParsingError(HTTPError): """Raised by assert_header_parsing, but we convert it to a log.warning statement.""" def __init__( self, defects: list[MessageDefect], unparsed_data: bytes | str | None ) -> None: message = f"{defects or 'Unknown'}, unparsed data: {unparsed_data!r}" super().__init__(message) class UnrewindableBodyError(HTTPError): """urllib3 encountered an error when trying to rewind a body""" urllib3-2.0.7/src/urllib3/fields.py0000644000000000000000000002542214513541732014017 0ustar00from __future__ import annotations import email.utils import mimetypes import typing _TYPE_FIELD_VALUE = typing.Union[str, bytes] _TYPE_FIELD_VALUE_TUPLE = typing.Union[ _TYPE_FIELD_VALUE, typing.Tuple[str, _TYPE_FIELD_VALUE], typing.Tuple[str, _TYPE_FIELD_VALUE, str], ] def guess_content_type( filename: str | None, default: str = "application/octet-stream" ) -> str: """ Guess the "Content-Type" of a file. :param filename: The filename to guess the "Content-Type" of using :mod:`mimetypes`. :param default: If no "Content-Type" can be guessed, default to `default`. """ if filename: return mimetypes.guess_type(filename)[0] or default return default def format_header_param_rfc2231(name: str, value: _TYPE_FIELD_VALUE) -> str: """ Helper function to format and quote a single header parameter using the strategy defined in RFC 2231. Particularly useful for header parameters which might contain non-ASCII values, like file names. This follows `RFC 2388 Section 4.4 `_. :param name: The name of the parameter, a string expected to be ASCII only. :param value: The value of the parameter, provided as ``bytes`` or `str``. :returns: An RFC-2231-formatted unicode string. .. deprecated:: 2.0.0 Will be removed in urllib3 v2.1.0. This is not valid for ``multipart/form-data`` header parameters. """ import warnings warnings.warn( "'format_header_param_rfc2231' is deprecated and will be " "removed in urllib3 v2.1.0. This is not valid for " "multipart/form-data header parameters.", DeprecationWarning, stacklevel=2, ) if isinstance(value, bytes): value = value.decode("utf-8") if not any(ch in value for ch in '"\\\r\n'): result = f'{name}="{value}"' try: result.encode("ascii") except (UnicodeEncodeError, UnicodeDecodeError): pass else: return result value = email.utils.encode_rfc2231(value, "utf-8") value = f"{name}*={value}" return value def format_multipart_header_param(name: str, value: _TYPE_FIELD_VALUE) -> str: """ Format and quote a single multipart header parameter. This follows the `WHATWG HTML Standard`_ as of 2021/06/10, matching the behavior of current browser and curl versions. Values are assumed to be UTF-8. The ``\\n``, ``\\r``, and ``"`` characters are percent encoded. .. _WHATWG HTML Standard: https://html.spec.whatwg.org/multipage/ form-control-infrastructure.html#multipart-form-data :param name: The name of the parameter, an ASCII-only ``str``. :param value: The value of the parameter, a ``str`` or UTF-8 encoded ``bytes``. :returns: A string ``name="value"`` with the escaped value. .. versionchanged:: 2.0.0 Matches the WHATWG HTML Standard as of 2021/06/10. Control characters are no longer percent encoded. .. versionchanged:: 2.0.0 Renamed from ``format_header_param_html5`` and ``format_header_param``. The old names will be removed in urllib3 v2.1.0. """ if isinstance(value, bytes): value = value.decode("utf-8") # percent encode \n \r " value = value.translate({10: "%0A", 13: "%0D", 34: "%22"}) return f'{name}="{value}"' def format_header_param_html5(name: str, value: _TYPE_FIELD_VALUE) -> str: """ .. deprecated:: 2.0.0 Renamed to :func:`format_multipart_header_param`. Will be removed in urllib3 v2.1.0. """ import warnings warnings.warn( "'format_header_param_html5' has been renamed to " "'format_multipart_header_param'. The old name will be " "removed in urllib3 v2.1.0.", DeprecationWarning, stacklevel=2, ) return format_multipart_header_param(name, value) def format_header_param(name: str, value: _TYPE_FIELD_VALUE) -> str: """ .. deprecated:: 2.0.0 Renamed to :func:`format_multipart_header_param`. Will be removed in urllib3 v2.1.0. """ import warnings warnings.warn( "'format_header_param' has been renamed to " "'format_multipart_header_param'. The old name will be " "removed in urllib3 v2.1.0.", DeprecationWarning, stacklevel=2, ) return format_multipart_header_param(name, value) class RequestField: """ A data container for request body parameters. :param name: The name of this request field. Must be unicode. :param data: The data/value body. :param filename: An optional filename of the request field. Must be unicode. :param headers: An optional dict-like object of headers to initially use for the field. .. versionchanged:: 2.0.0 The ``header_formatter`` parameter is deprecated and will be removed in urllib3 v2.1.0. """ def __init__( self, name: str, data: _TYPE_FIELD_VALUE, filename: str | None = None, headers: typing.Mapping[str, str] | None = None, header_formatter: typing.Callable[[str, _TYPE_FIELD_VALUE], str] | None = None, ): self._name = name self._filename = filename self.data = data self.headers: dict[str, str | None] = {} if headers: self.headers = dict(headers) if header_formatter is not None: import warnings warnings.warn( "The 'header_formatter' parameter is deprecated and " "will be removed in urllib3 v2.1.0.", DeprecationWarning, stacklevel=2, ) self.header_formatter = header_formatter else: self.header_formatter = format_multipart_header_param @classmethod def from_tuples( cls, fieldname: str, value: _TYPE_FIELD_VALUE_TUPLE, header_formatter: typing.Callable[[str, _TYPE_FIELD_VALUE], str] | None = None, ) -> RequestField: """ A :class:`~urllib3.fields.RequestField` factory from old-style tuple parameters. Supports constructing :class:`~urllib3.fields.RequestField` from parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:: 'foo': 'bar', 'fakefile': ('foofile.txt', 'contents of foofile'), 'realfile': ('barfile.txt', open('realfile').read()), 'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'), 'nonamefile': 'contents of nonamefile field', Field names and filenames must be unicode. """ filename: str | None content_type: str | None data: _TYPE_FIELD_VALUE if isinstance(value, tuple): if len(value) == 3: filename, data, content_type = typing.cast( typing.Tuple[str, _TYPE_FIELD_VALUE, str], value ) else: filename, data = typing.cast( typing.Tuple[str, _TYPE_FIELD_VALUE], value ) content_type = guess_content_type(filename) else: filename = None content_type = None data = value request_param = cls( fieldname, data, filename=filename, header_formatter=header_formatter ) request_param.make_multipart(content_type=content_type) return request_param def _render_part(self, name: str, value: _TYPE_FIELD_VALUE) -> str: """ Override this method to change how each multipart header parameter is formatted. By default, this calls :func:`format_multipart_header_param`. :param name: The name of the parameter, an ASCII-only ``str``. :param value: The value of the parameter, a ``str`` or UTF-8 encoded ``bytes``. :meta public: """ return self.header_formatter(name, value) def _render_parts( self, header_parts: ( dict[str, _TYPE_FIELD_VALUE | None] | typing.Sequence[tuple[str, _TYPE_FIELD_VALUE | None]] ), ) -> str: """ Helper function to format and quote a single header. Useful for single headers that are composed of multiple items. E.g., 'Content-Disposition' fields. :param header_parts: A sequence of (k, v) tuples or a :class:`dict` of (k, v) to format as `k1="v1"; k2="v2"; ...`. """ iterable: typing.Iterable[tuple[str, _TYPE_FIELD_VALUE | None]] parts = [] if isinstance(header_parts, dict): iterable = header_parts.items() else: iterable = header_parts for name, value in iterable: if value is not None: parts.append(self._render_part(name, value)) return "; ".join(parts) def render_headers(self) -> str: """ Renders the headers for this request field. """ lines = [] sort_keys = ["Content-Disposition", "Content-Type", "Content-Location"] for sort_key in sort_keys: if self.headers.get(sort_key, False): lines.append(f"{sort_key}: {self.headers[sort_key]}") for header_name, header_value in self.headers.items(): if header_name not in sort_keys: if header_value: lines.append(f"{header_name}: {header_value}") lines.append("\r\n") return "\r\n".join(lines) def make_multipart( self, content_disposition: str | None = None, content_type: str | None = None, content_location: str | None = None, ) -> None: """ Makes this request field into a multipart request field. This method overrides "Content-Disposition", "Content-Type" and "Content-Location" headers to the request parameter. :param content_disposition: The 'Content-Disposition' of the request body. Defaults to 'form-data' :param content_type: The 'Content-Type' of the request body. :param content_location: The 'Content-Location' of the request body. """ content_disposition = (content_disposition or "form-data") + "; ".join( [ "", self._render_parts( (("name", self._name), ("filename", self._filename)) ), ] ) self.headers["Content-Disposition"] = content_disposition self.headers["Content-Type"] = content_type self.headers["Content-Location"] = content_location urllib3-2.0.7/src/urllib3/filepost.py0000644000000000000000000000453314513541732014376 0ustar00from __future__ import annotations import binascii import codecs import os import typing from io import BytesIO from .fields import _TYPE_FIELD_VALUE_TUPLE, RequestField writer = codecs.lookup("utf-8")[3] _TYPE_FIELDS_SEQUENCE = typing.Sequence[ typing.Union[typing.Tuple[str, _TYPE_FIELD_VALUE_TUPLE], RequestField] ] _TYPE_FIELDS = typing.Union[ _TYPE_FIELDS_SEQUENCE, typing.Mapping[str, _TYPE_FIELD_VALUE_TUPLE], ] def choose_boundary() -> str: """ Our embarrassingly-simple replacement for mimetools.choose_boundary. """ return binascii.hexlify(os.urandom(16)).decode() def iter_field_objects(fields: _TYPE_FIELDS) -> typing.Iterable[RequestField]: """ Iterate over fields. Supports list of (k, v) tuples and dicts, and lists of :class:`~urllib3.fields.RequestField`. """ iterable: typing.Iterable[RequestField | tuple[str, _TYPE_FIELD_VALUE_TUPLE]] if isinstance(fields, typing.Mapping): iterable = fields.items() else: iterable = fields for field in iterable: if isinstance(field, RequestField): yield field else: yield RequestField.from_tuples(*field) def encode_multipart_formdata( fields: _TYPE_FIELDS, boundary: str | None = None ) -> tuple[bytes, str]: """ Encode a dictionary of ``fields`` using the multipart/form-data MIME format. :param fields: Dictionary of fields or list of (key, :class:`~urllib3.fields.RequestField`). Values are processed by :func:`urllib3.fields.RequestField.from_tuples`. :param boundary: If not specified, then a random boundary will be generated using :func:`urllib3.filepost.choose_boundary`. """ body = BytesIO() if boundary is None: boundary = choose_boundary() for field in iter_field_objects(fields): body.write(f"--{boundary}\r\n".encode("latin-1")) writer(body).write(field.render_headers()) data = field.data if isinstance(data, int): data = str(data) # Backwards compatibility if isinstance(data, str): writer(body).write(data) else: body.write(data) body.write(b"\r\n") body.write(f"--{boundary}--\r\n".encode("latin-1")) content_type = f"multipart/form-data; boundary={boundary}" return body.getvalue(), content_type urllib3-2.0.7/src/urllib3/poolmanager.py0000644000000000000000000005454614513541732015066 0ustar00from __future__ import annotations import functools import logging import typing import warnings from types import TracebackType from urllib.parse import urljoin from ._collections import HTTPHeaderDict, RecentlyUsedContainer from ._request_methods import RequestMethods from .connection import ProxyConfig from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, port_by_scheme from .exceptions import ( LocationValueError, MaxRetryError, ProxySchemeUnknown, URLSchemeUnknown, ) from .response import BaseHTTPResponse from .util.connection import _TYPE_SOCKET_OPTIONS from .util.proxy import connection_requires_http_tunnel from .util.retry import Retry from .util.timeout import Timeout from .util.url import Url, parse_url if typing.TYPE_CHECKING: import ssl from typing_extensions import Literal __all__ = ["PoolManager", "ProxyManager", "proxy_from_url"] log = logging.getLogger(__name__) SSL_KEYWORDS = ( "key_file", "cert_file", "cert_reqs", "ca_certs", "ssl_version", "ssl_minimum_version", "ssl_maximum_version", "ca_cert_dir", "ssl_context", "key_password", "server_hostname", ) # Default value for `blocksize` - a new parameter introduced to # http.client.HTTPConnection & http.client.HTTPSConnection in Python 3.7 _DEFAULT_BLOCKSIZE = 16384 _SelfT = typing.TypeVar("_SelfT") class PoolKey(typing.NamedTuple): """ All known keyword arguments that could be provided to the pool manager, its pools, or the underlying connections. All custom key schemes should include the fields in this key at a minimum. """ key_scheme: str key_host: str key_port: int | None key_timeout: Timeout | float | int | None key_retries: Retry | bool | int | None key_block: bool | None key_source_address: tuple[str, int] | None key_key_file: str | None key_key_password: str | None key_cert_file: str | None key_cert_reqs: str | None key_ca_certs: str | None key_ssl_version: int | str | None key_ssl_minimum_version: ssl.TLSVersion | None key_ssl_maximum_version: ssl.TLSVersion | None key_ca_cert_dir: str | None key_ssl_context: ssl.SSLContext | None key_maxsize: int | None key_headers: frozenset[tuple[str, str]] | None key__proxy: Url | None key__proxy_headers: frozenset[tuple[str, str]] | None key__proxy_config: ProxyConfig | None key_socket_options: _TYPE_SOCKET_OPTIONS | None key__socks_options: frozenset[tuple[str, str]] | None key_assert_hostname: bool | str | None key_assert_fingerprint: str | None key_server_hostname: str | None key_blocksize: int | None def _default_key_normalizer( key_class: type[PoolKey], request_context: dict[str, typing.Any] ) -> PoolKey: """ Create a pool key out of a request context dictionary. According to RFC 3986, both the scheme and host are case-insensitive. Therefore, this function normalizes both before constructing the pool key for an HTTPS request. If you wish to change this behaviour, provide alternate callables to ``key_fn_by_scheme``. :param key_class: The class to use when constructing the key. This should be a namedtuple with the ``scheme`` and ``host`` keys at a minimum. :type key_class: namedtuple :param request_context: A dictionary-like object that contain the context for a request. :type request_context: dict :return: A namedtuple that can be used as a connection pool key. :rtype: PoolKey """ # Since we mutate the dictionary, make a copy first context = request_context.copy() context["scheme"] = context["scheme"].lower() context["host"] = context["host"].lower() # These are both dictionaries and need to be transformed into frozensets for key in ("headers", "_proxy_headers", "_socks_options"): if key in context and context[key] is not None: context[key] = frozenset(context[key].items()) # The socket_options key may be a list and needs to be transformed into a # tuple. socket_opts = context.get("socket_options") if socket_opts is not None: context["socket_options"] = tuple(socket_opts) # Map the kwargs to the names in the namedtuple - this is necessary since # namedtuples can't have fields starting with '_'. for key in list(context.keys()): context["key_" + key] = context.pop(key) # Default to ``None`` for keys missing from the context for field in key_class._fields: if field not in context: context[field] = None # Default key_blocksize to _DEFAULT_BLOCKSIZE if missing from the context if context.get("key_blocksize") is None: context["key_blocksize"] = _DEFAULT_BLOCKSIZE return key_class(**context) #: A dictionary that maps a scheme to a callable that creates a pool key. #: This can be used to alter the way pool keys are constructed, if desired. #: Each PoolManager makes a copy of this dictionary so they can be configured #: globally here, or individually on the instance. key_fn_by_scheme = { "http": functools.partial(_default_key_normalizer, PoolKey), "https": functools.partial(_default_key_normalizer, PoolKey), } pool_classes_by_scheme = {"http": HTTPConnectionPool, "https": HTTPSConnectionPool} class PoolManager(RequestMethods): """ Allows for arbitrary requests while transparently keeping track of necessary connection pools for you. :param num_pools: Number of connection pools to cache before discarding the least recently used pool. :param headers: Headers to include with all requests, unless other headers are given explicitly. :param \\**connection_pool_kw: Additional parameters are used to create fresh :class:`urllib3.connectionpool.ConnectionPool` instances. Example: .. code-block:: python import urllib3 http = urllib3.PoolManager(num_pools=2) resp1 = http.request("GET", "https://google.com/") resp2 = http.request("GET", "https://google.com/mail") resp3 = http.request("GET", "https://yahoo.com/") print(len(http.pools)) # 2 """ proxy: Url | None = None proxy_config: ProxyConfig | None = None def __init__( self, num_pools: int = 10, headers: typing.Mapping[str, str] | None = None, **connection_pool_kw: typing.Any, ) -> None: super().__init__(headers) self.connection_pool_kw = connection_pool_kw self.pools: RecentlyUsedContainer[PoolKey, HTTPConnectionPool] self.pools = RecentlyUsedContainer(num_pools) # Locally set the pool classes and keys so other PoolManagers can # override them. self.pool_classes_by_scheme = pool_classes_by_scheme self.key_fn_by_scheme = key_fn_by_scheme.copy() def __enter__(self: _SelfT) -> _SelfT: return self def __exit__( self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None, ) -> Literal[False]: self.clear() # Return False to re-raise any potential exceptions return False def _new_pool( self, scheme: str, host: str, port: int, request_context: dict[str, typing.Any] | None = None, ) -> HTTPConnectionPool: """ Create a new :class:`urllib3.connectionpool.ConnectionPool` based on host, port, scheme, and any additional pool keyword arguments. If ``request_context`` is provided, it is provided as keyword arguments to the pool class used. This method is used to actually create the connection pools handed out by :meth:`connection_from_url` and companion methods. It is intended to be overridden for customization. """ pool_cls: type[HTTPConnectionPool] = self.pool_classes_by_scheme[scheme] if request_context is None: request_context = self.connection_pool_kw.copy() # Default blocksize to _DEFAULT_BLOCKSIZE if missing or explicitly # set to 'None' in the request_context. if request_context.get("blocksize") is None: request_context["blocksize"] = _DEFAULT_BLOCKSIZE # Although the context has everything necessary to create the pool, # this function has historically only used the scheme, host, and port # in the positional args. When an API change is acceptable these can # be removed. for key in ("scheme", "host", "port"): request_context.pop(key, None) if scheme == "http": for kw in SSL_KEYWORDS: request_context.pop(kw, None) return pool_cls(host, port, **request_context) def clear(self) -> None: """ Empty our store of pools and direct them all to close. This will not affect in-flight connections, but they will not be re-used after completion. """ self.pools.clear() def connection_from_host( self, host: str | None, port: int | None = None, scheme: str | None = "http", pool_kwargs: dict[str, typing.Any] | None = None, ) -> HTTPConnectionPool: """ Get a :class:`urllib3.connectionpool.ConnectionPool` based on the host, port, and scheme. If ``port`` isn't given, it will be derived from the ``scheme`` using ``urllib3.connectionpool.port_by_scheme``. If ``pool_kwargs`` is provided, it is merged with the instance's ``connection_pool_kw`` variable and used to create the new connection pool, if one is needed. """ if not host: raise LocationValueError("No host specified.") request_context = self._merge_pool_kwargs(pool_kwargs) request_context["scheme"] = scheme or "http" if not port: port = port_by_scheme.get(request_context["scheme"].lower(), 80) request_context["port"] = port request_context["host"] = host return self.connection_from_context(request_context) def connection_from_context( self, request_context: dict[str, typing.Any] ) -> HTTPConnectionPool: """ Get a :class:`urllib3.connectionpool.ConnectionPool` based on the request context. ``request_context`` must at least contain the ``scheme`` key and its value must be a key in ``key_fn_by_scheme`` instance variable. """ if "strict" in request_context: warnings.warn( "The 'strict' parameter is no longer needed on Python 3+. " "This will raise an error in urllib3 v2.1.0.", DeprecationWarning, ) request_context.pop("strict") scheme = request_context["scheme"].lower() pool_key_constructor = self.key_fn_by_scheme.get(scheme) if not pool_key_constructor: raise URLSchemeUnknown(scheme) pool_key = pool_key_constructor(request_context) return self.connection_from_pool_key(pool_key, request_context=request_context) def connection_from_pool_key( self, pool_key: PoolKey, request_context: dict[str, typing.Any] ) -> HTTPConnectionPool: """ Get a :class:`urllib3.connectionpool.ConnectionPool` based on the provided pool key. ``pool_key`` should be a namedtuple that only contains immutable objects. At a minimum it must have the ``scheme``, ``host``, and ``port`` fields. """ with self.pools.lock: # If the scheme, host, or port doesn't match existing open # connections, open a new ConnectionPool. pool = self.pools.get(pool_key) if pool: return pool # Make a fresh ConnectionPool of the desired type scheme = request_context["scheme"] host = request_context["host"] port = request_context["port"] pool = self._new_pool(scheme, host, port, request_context=request_context) self.pools[pool_key] = pool return pool def connection_from_url( self, url: str, pool_kwargs: dict[str, typing.Any] | None = None ) -> HTTPConnectionPool: """ Similar to :func:`urllib3.connectionpool.connection_from_url`. If ``pool_kwargs`` is not provided and a new pool needs to be constructed, ``self.connection_pool_kw`` is used to initialize the :class:`urllib3.connectionpool.ConnectionPool`. If ``pool_kwargs`` is provided, it is used instead. Note that if a new pool does not need to be created for the request, the provided ``pool_kwargs`` are not used. """ u = parse_url(url) return self.connection_from_host( u.host, port=u.port, scheme=u.scheme, pool_kwargs=pool_kwargs ) def _merge_pool_kwargs( self, override: dict[str, typing.Any] | None ) -> dict[str, typing.Any]: """ Merge a dictionary of override values for self.connection_pool_kw. This does not modify self.connection_pool_kw and returns a new dict. Any keys in the override dictionary with a value of ``None`` are removed from the merged dictionary. """ base_pool_kwargs = self.connection_pool_kw.copy() if override: for key, value in override.items(): if value is None: try: del base_pool_kwargs[key] except KeyError: pass else: base_pool_kwargs[key] = value return base_pool_kwargs def _proxy_requires_url_absolute_form(self, parsed_url: Url) -> bool: """ Indicates if the proxy requires the complete destination URL in the request. Normally this is only needed when not using an HTTP CONNECT tunnel. """ if self.proxy is None: return False return not connection_requires_http_tunnel( self.proxy, self.proxy_config, parsed_url.scheme ) def urlopen( # type: ignore[override] self, method: str, url: str, redirect: bool = True, **kw: typing.Any ) -> BaseHTTPResponse: """ Same as :meth:`urllib3.HTTPConnectionPool.urlopen` with custom cross-host redirect logic and only sends the request-uri portion of the ``url``. The given ``url`` parameter must be absolute, such that an appropriate :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it. """ u = parse_url(url) if u.scheme is None: warnings.warn( "URLs without a scheme (ie 'https://') are deprecated and will raise an error " "in a future version of urllib3. To avoid this DeprecationWarning ensure all URLs " "start with 'https://' or 'http://'. Read more in this issue: " "https://github.com/urllib3/urllib3/issues/2920", category=DeprecationWarning, stacklevel=2, ) conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) kw["assert_same_host"] = False kw["redirect"] = False if "headers" not in kw: kw["headers"] = self.headers if self._proxy_requires_url_absolute_form(u): response = conn.urlopen(method, url, **kw) else: response = conn.urlopen(method, u.request_uri, **kw) redirect_location = redirect and response.get_redirect_location() if not redirect_location: return response # Support relative URLs for redirecting. redirect_location = urljoin(url, redirect_location) if response.status == 303: # Change the method according to RFC 9110, Section 15.4.4. method = "GET" # And lose the body not to transfer anything sensitive. kw["body"] = None kw["headers"] = HTTPHeaderDict(kw["headers"])._prepare_for_method_change() retries = kw.get("retries") if not isinstance(retries, Retry): retries = Retry.from_int(retries, redirect=redirect) # Strip headers marked as unsafe to forward to the redirected location. # Check remove_headers_on_redirect to avoid a potential network call within # conn.is_same_host() which may use socket.gethostbyname() in the future. if retries.remove_headers_on_redirect and not conn.is_same_host( redirect_location ): new_headers = kw["headers"].copy() for header in kw["headers"]: if header.lower() in retries.remove_headers_on_redirect: new_headers.pop(header, None) kw["headers"] = new_headers try: retries = retries.increment(method, url, response=response, _pool=conn) except MaxRetryError: if retries.raise_on_redirect: response.drain_conn() raise return response kw["retries"] = retries kw["redirect"] = redirect log.info("Redirecting %s -> %s", url, redirect_location) response.drain_conn() return self.urlopen(method, redirect_location, **kw) class ProxyManager(PoolManager): """ Behaves just like :class:`PoolManager`, but sends all requests through the defined proxy, using the CONNECT method for HTTPS URLs. :param proxy_url: The URL of the proxy to be used. :param proxy_headers: A dictionary containing headers that will be sent to the proxy. In case of HTTP they are being sent with each request, while in the HTTPS/CONNECT case they are sent only once. Could be used for proxy authentication. :param proxy_ssl_context: The proxy SSL context is used to establish the TLS connection to the proxy when using HTTPS proxies. :param use_forwarding_for_https: (Defaults to False) If set to True will forward requests to the HTTPS proxy to be made on behalf of the client instead of creating a TLS tunnel via the CONNECT method. **Enabling this flag means that request and response headers and content will be visible from the HTTPS proxy** whereas tunneling keeps request and response headers and content private. IP address, target hostname, SNI, and port are always visible to an HTTPS proxy even when this flag is disabled. :param proxy_assert_hostname: The hostname of the certificate to verify against. :param proxy_assert_fingerprint: The fingerprint of the certificate to verify against. Example: .. code-block:: python import urllib3 proxy = urllib3.ProxyManager("https://localhost:3128/") resp1 = proxy.request("GET", "https://google.com/") resp2 = proxy.request("GET", "https://httpbin.org/") print(len(proxy.pools)) # 1 resp3 = proxy.request("GET", "https://httpbin.org/") resp4 = proxy.request("GET", "https://twitter.com/") print(len(proxy.pools)) # 3 """ def __init__( self, proxy_url: str, num_pools: int = 10, headers: typing.Mapping[str, str] | None = None, proxy_headers: typing.Mapping[str, str] | None = None, proxy_ssl_context: ssl.SSLContext | None = None, use_forwarding_for_https: bool = False, proxy_assert_hostname: None | str | Literal[False] = None, proxy_assert_fingerprint: str | None = None, **connection_pool_kw: typing.Any, ) -> None: if isinstance(proxy_url, HTTPConnectionPool): str_proxy_url = f"{proxy_url.scheme}://{proxy_url.host}:{proxy_url.port}" else: str_proxy_url = proxy_url proxy = parse_url(str_proxy_url) if proxy.scheme not in ("http", "https"): raise ProxySchemeUnknown(proxy.scheme) if not proxy.port: port = port_by_scheme.get(proxy.scheme, 80) proxy = proxy._replace(port=port) self.proxy = proxy self.proxy_headers = proxy_headers or {} self.proxy_ssl_context = proxy_ssl_context self.proxy_config = ProxyConfig( proxy_ssl_context, use_forwarding_for_https, proxy_assert_hostname, proxy_assert_fingerprint, ) connection_pool_kw["_proxy"] = self.proxy connection_pool_kw["_proxy_headers"] = self.proxy_headers connection_pool_kw["_proxy_config"] = self.proxy_config super().__init__(num_pools, headers, **connection_pool_kw) def connection_from_host( self, host: str | None, port: int | None = None, scheme: str | None = "http", pool_kwargs: dict[str, typing.Any] | None = None, ) -> HTTPConnectionPool: if scheme == "https": return super().connection_from_host( host, port, scheme, pool_kwargs=pool_kwargs ) return super().connection_from_host( self.proxy.host, self.proxy.port, self.proxy.scheme, pool_kwargs=pool_kwargs # type: ignore[union-attr] ) def _set_proxy_headers( self, url: str, headers: typing.Mapping[str, str] | None = None ) -> typing.Mapping[str, str]: """ Sets headers needed by proxies: specifically, the Accept and Host headers. Only sets headers not provided by the user. """ headers_ = {"Accept": "*/*"} netloc = parse_url(url).netloc if netloc: headers_["Host"] = netloc if headers: headers_.update(headers) return headers_ def urlopen( # type: ignore[override] self, method: str, url: str, redirect: bool = True, **kw: typing.Any ) -> BaseHTTPResponse: "Same as HTTP(S)ConnectionPool.urlopen, ``url`` must be absolute." u = parse_url(url) if not connection_requires_http_tunnel(self.proxy, self.proxy_config, u.scheme): # For connections using HTTP CONNECT, httplib sets the necessary # headers on the CONNECT to the proxy. If we're not using CONNECT, # we'll definitely need to set 'Host' at the very least. headers = kw.get("headers", self.headers) kw["headers"] = self._set_proxy_headers(url, headers) return super().urlopen(method, url, redirect=redirect, **kw) def proxy_from_url(url: str, **kw: typing.Any) -> ProxyManager: return ProxyManager(proxy_url=url, **kw) urllib3-2.0.7/src/urllib3/py.typed0000644000000000000000000000013514513541732013670 0ustar00# Instruct type checkers to look for inline type annotations in this package. # See PEP 561. urllib3-2.0.7/src/urllib3/response.py0000644000000000000000000011607714513541732014416 0ustar00from __future__ import annotations import collections import io import json as _json import logging import re import sys import typing import warnings import zlib from contextlib import contextmanager from http.client import HTTPMessage as _HttplibHTTPMessage from http.client import HTTPResponse as _HttplibHTTPResponse from socket import timeout as SocketTimeout try: try: import brotlicffi as brotli # type: ignore[import] except ImportError: import brotli # type: ignore[import] except ImportError: brotli = None try: import zstandard as zstd # type: ignore[import] # The package 'zstandard' added the 'eof' property starting # in v0.18.0 which we require to ensure a complete and # valid zstd stream was fed into the ZstdDecoder. # See: https://github.com/urllib3/urllib3/pull/2624 _zstd_version = _zstd_version = tuple( map(int, re.search(r"^([0-9]+)\.([0-9]+)", zstd.__version__).groups()) # type: ignore[union-attr] ) if _zstd_version < (0, 18): # Defensive: zstd = None except (AttributeError, ImportError, ValueError): # Defensive: zstd = None from . import util from ._base_connection import _TYPE_BODY from ._collections import HTTPHeaderDict from .connection import BaseSSLError, HTTPConnection, HTTPException from .exceptions import ( BodyNotHttplibCompatible, DecodeError, HTTPError, IncompleteRead, InvalidChunkLength, InvalidHeader, ProtocolError, ReadTimeoutError, ResponseNotChunked, SSLError, ) from .util.response import is_fp_closed, is_response_to_head from .util.retry import Retry if typing.TYPE_CHECKING: from typing_extensions import Literal from .connectionpool import HTTPConnectionPool log = logging.getLogger(__name__) class ContentDecoder: def decompress(self, data: bytes) -> bytes: raise NotImplementedError() def flush(self) -> bytes: raise NotImplementedError() class DeflateDecoder(ContentDecoder): def __init__(self) -> None: self._first_try = True self._data = b"" self._obj = zlib.decompressobj() def decompress(self, data: bytes) -> bytes: if not data: return data if not self._first_try: return self._obj.decompress(data) self._data += data try: decompressed = self._obj.decompress(data) if decompressed: self._first_try = False self._data = None # type: ignore[assignment] return decompressed except zlib.error: self._first_try = False self._obj = zlib.decompressobj(-zlib.MAX_WBITS) try: return self.decompress(self._data) finally: self._data = None # type: ignore[assignment] def flush(self) -> bytes: return self._obj.flush() class GzipDecoderState: FIRST_MEMBER = 0 OTHER_MEMBERS = 1 SWALLOW_DATA = 2 class GzipDecoder(ContentDecoder): def __init__(self) -> None: self._obj = zlib.decompressobj(16 + zlib.MAX_WBITS) self._state = GzipDecoderState.FIRST_MEMBER def decompress(self, data: bytes) -> bytes: ret = bytearray() if self._state == GzipDecoderState.SWALLOW_DATA or not data: return bytes(ret) while True: try: ret += self._obj.decompress(data) except zlib.error: previous_state = self._state # Ignore data after the first error self._state = GzipDecoderState.SWALLOW_DATA if previous_state == GzipDecoderState.OTHER_MEMBERS: # Allow trailing garbage acceptable in other gzip clients return bytes(ret) raise data = self._obj.unused_data if not data: return bytes(ret) self._state = GzipDecoderState.OTHER_MEMBERS self._obj = zlib.decompressobj(16 + zlib.MAX_WBITS) def flush(self) -> bytes: return self._obj.flush() if brotli is not None: class BrotliDecoder(ContentDecoder): # Supports both 'brotlipy' and 'Brotli' packages # since they share an import name. The top branches # are for 'brotlipy' and bottom branches for 'Brotli' def __init__(self) -> None: self._obj = brotli.Decompressor() if hasattr(self._obj, "decompress"): setattr(self, "decompress", self._obj.decompress) else: setattr(self, "decompress", self._obj.process) def flush(self) -> bytes: if hasattr(self._obj, "flush"): return self._obj.flush() # type: ignore[no-any-return] return b"" if zstd is not None: class ZstdDecoder(ContentDecoder): def __init__(self) -> None: self._obj = zstd.ZstdDecompressor().decompressobj() def decompress(self, data: bytes) -> bytes: if not data: return b"" data_parts = [self._obj.decompress(data)] while self._obj.eof and self._obj.unused_data: unused_data = self._obj.unused_data self._obj = zstd.ZstdDecompressor().decompressobj() data_parts.append(self._obj.decompress(unused_data)) return b"".join(data_parts) def flush(self) -> bytes: ret = self._obj.flush() # note: this is a no-op if not self._obj.eof: raise DecodeError("Zstandard data is incomplete") return ret # type: ignore[no-any-return] class MultiDecoder(ContentDecoder): """ From RFC7231: If one or more encodings have been applied to a representation, the sender that applied the encodings MUST generate a Content-Encoding header field that lists the content codings in the order in which they were applied. """ def __init__(self, modes: str) -> None: self._decoders = [_get_decoder(m.strip()) for m in modes.split(",")] def flush(self) -> bytes: return self._decoders[0].flush() def decompress(self, data: bytes) -> bytes: for d in reversed(self._decoders): data = d.decompress(data) return data def _get_decoder(mode: str) -> ContentDecoder: if "," in mode: return MultiDecoder(mode) if mode == "gzip": return GzipDecoder() if brotli is not None and mode == "br": return BrotliDecoder() if zstd is not None and mode == "zstd": return ZstdDecoder() return DeflateDecoder() class BytesQueueBuffer: """Memory-efficient bytes buffer To return decoded data in read() and still follow the BufferedIOBase API, we need a buffer to always return the correct amount of bytes. This buffer should be filled using calls to put() Our maximum memory usage is determined by the sum of the size of: * self.buffer, which contains the full data * the largest chunk that we will copy in get() The worst case scenario is a single chunk, in which case we'll make a full copy of the data inside get(). """ def __init__(self) -> None: self.buffer: typing.Deque[bytes] = collections.deque() self._size: int = 0 def __len__(self) -> int: return self._size def put(self, data: bytes) -> None: self.buffer.append(data) self._size += len(data) def get(self, n: int) -> bytes: if n == 0: return b"" elif not self.buffer: raise RuntimeError("buffer is empty") elif n < 0: raise ValueError("n should be > 0") fetched = 0 ret = io.BytesIO() while fetched < n: remaining = n - fetched chunk = self.buffer.popleft() chunk_length = len(chunk) if remaining < chunk_length: left_chunk, right_chunk = chunk[:remaining], chunk[remaining:] ret.write(left_chunk) self.buffer.appendleft(right_chunk) self._size -= remaining break else: ret.write(chunk) self._size -= chunk_length fetched += chunk_length if not self.buffer: break return ret.getvalue() class BaseHTTPResponse(io.IOBase): CONTENT_DECODERS = ["gzip", "deflate"] if brotli is not None: CONTENT_DECODERS += ["br"] if zstd is not None: CONTENT_DECODERS += ["zstd"] REDIRECT_STATUSES = [301, 302, 303, 307, 308] DECODER_ERROR_CLASSES: tuple[type[Exception], ...] = (IOError, zlib.error) if brotli is not None: DECODER_ERROR_CLASSES += (brotli.error,) if zstd is not None: DECODER_ERROR_CLASSES += (zstd.ZstdError,) def __init__( self, *, headers: typing.Mapping[str, str] | typing.Mapping[bytes, bytes] | None = None, status: int, version: int, reason: str | None, decode_content: bool, request_url: str | None, retries: Retry | None = None, ) -> None: if isinstance(headers, HTTPHeaderDict): self.headers = headers else: self.headers = HTTPHeaderDict(headers) # type: ignore[arg-type] self.status = status self.version = version self.reason = reason self.decode_content = decode_content self._has_decoded_content = False self._request_url: str | None = request_url self.retries = retries self.chunked = False tr_enc = self.headers.get("transfer-encoding", "").lower() # Don't incur the penalty of creating a list and then discarding it encodings = (enc.strip() for enc in tr_enc.split(",")) if "chunked" in encodings: self.chunked = True self._decoder: ContentDecoder | None = None def get_redirect_location(self) -> str | None | Literal[False]: """ Should we redirect and where to? :returns: Truthy redirect location string if we got a redirect status code and valid location. ``None`` if redirect status and no location. ``False`` if not a redirect status code. """ if self.status in self.REDIRECT_STATUSES: return self.headers.get("location") return False @property def data(self) -> bytes: raise NotImplementedError() def json(self) -> typing.Any: """ Parses the body of the HTTP response as JSON. To use a custom JSON decoder pass the result of :attr:`HTTPResponse.data` to the decoder. This method can raise either `UnicodeDecodeError` or `json.JSONDecodeError`. Read more :ref:`here `. """ data = self.data.decode("utf-8") return _json.loads(data) @property def url(self) -> str | None: raise NotImplementedError() @url.setter def url(self, url: str | None) -> None: raise NotImplementedError() @property def connection(self) -> HTTPConnection | None: raise NotImplementedError() @property def retries(self) -> Retry | None: return self._retries @retries.setter def retries(self, retries: Retry | None) -> None: # Override the request_url if retries has a redirect location. if retries is not None and retries.history: self.url = retries.history[-1].redirect_location self._retries = retries def stream( self, amt: int | None = 2**16, decode_content: bool | None = None ) -> typing.Iterator[bytes]: raise NotImplementedError() def read( self, amt: int | None = None, decode_content: bool | None = None, cache_content: bool = False, ) -> bytes: raise NotImplementedError() def read_chunked( self, amt: int | None = None, decode_content: bool | None = None, ) -> typing.Iterator[bytes]: raise NotImplementedError() def release_conn(self) -> None: raise NotImplementedError() def drain_conn(self) -> None: raise NotImplementedError() def close(self) -> None: raise NotImplementedError() def _init_decoder(self) -> None: """ Set-up the _decoder attribute if necessary. """ # Note: content-encoding value should be case-insensitive, per RFC 7230 # Section 3.2 content_encoding = self.headers.get("content-encoding", "").lower() if self._decoder is None: if content_encoding in self.CONTENT_DECODERS: self._decoder = _get_decoder(content_encoding) elif "," in content_encoding: encodings = [ e.strip() for e in content_encoding.split(",") if e.strip() in self.CONTENT_DECODERS ] if encodings: self._decoder = _get_decoder(content_encoding) def _decode( self, data: bytes, decode_content: bool | None, flush_decoder: bool ) -> bytes: """ Decode the data passed in and potentially flush the decoder. """ if not decode_content: if self._has_decoded_content: raise RuntimeError( "Calling read(decode_content=False) is not supported after " "read(decode_content=True) was called." ) return data try: if self._decoder: data = self._decoder.decompress(data) self._has_decoded_content = True except self.DECODER_ERROR_CLASSES as e: content_encoding = self.headers.get("content-encoding", "").lower() raise DecodeError( "Received response with content-encoding: %s, but " "failed to decode it." % content_encoding, e, ) from e if flush_decoder: data += self._flush_decoder() return data def _flush_decoder(self) -> bytes: """ Flushes the decoder. Should only be called if the decoder is actually being used. """ if self._decoder: return self._decoder.decompress(b"") + self._decoder.flush() return b"" # Compatibility methods for `io` module def readinto(self, b: bytearray) -> int: temp = self.read(len(b)) if len(temp) == 0: return 0 else: b[: len(temp)] = temp return len(temp) # Compatibility methods for http.client.HTTPResponse def getheaders(self) -> HTTPHeaderDict: warnings.warn( "HTTPResponse.getheaders() is deprecated and will be removed " "in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.", category=DeprecationWarning, stacklevel=2, ) return self.headers def getheader(self, name: str, default: str | None = None) -> str | None: warnings.warn( "HTTPResponse.getheader() is deprecated and will be removed " "in urllib3 v2.1.0. Instead use HTTPResponse.headers.get(name, default).", category=DeprecationWarning, stacklevel=2, ) return self.headers.get(name, default) # Compatibility method for http.cookiejar def info(self) -> HTTPHeaderDict: return self.headers def geturl(self) -> str | None: return self.url class HTTPResponse(BaseHTTPResponse): """ HTTP Response container. Backwards-compatible with :class:`http.client.HTTPResponse` but the response ``body`` is loaded and decoded on-demand when the ``data`` property is accessed. This class is also compatible with the Python standard library's :mod:`io` module, and can hence be treated as a readable object in the context of that framework. Extra parameters for behaviour not present in :class:`http.client.HTTPResponse`: :param preload_content: If True, the response's body will be preloaded during construction. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param original_response: When this HTTPResponse wrapper is generated from an :class:`http.client.HTTPResponse` object, it's convenient to include the original for debug purposes. It's otherwise unused. :param retries: The retries contains the last :class:`~urllib3.util.retry.Retry` that was used during the request. :param enforce_content_length: Enforce content length checking. Body returned by server must match value of Content-Length header, if present. Otherwise, raise error. """ def __init__( self, body: _TYPE_BODY = "", headers: typing.Mapping[str, str] | typing.Mapping[bytes, bytes] | None = None, status: int = 0, version: int = 0, reason: str | None = None, preload_content: bool = True, decode_content: bool = True, original_response: _HttplibHTTPResponse | None = None, pool: HTTPConnectionPool | None = None, connection: HTTPConnection | None = None, msg: _HttplibHTTPMessage | None = None, retries: Retry | None = None, enforce_content_length: bool = True, request_method: str | None = None, request_url: str | None = None, auto_close: bool = True, ) -> None: super().__init__( headers=headers, status=status, version=version, reason=reason, decode_content=decode_content, request_url=request_url, retries=retries, ) self.enforce_content_length = enforce_content_length self.auto_close = auto_close self._body = None self._fp: _HttplibHTTPResponse | None = None self._original_response = original_response self._fp_bytes_read = 0 self.msg = msg if body and isinstance(body, (str, bytes)): self._body = body self._pool = pool self._connection = connection if hasattr(body, "read"): self._fp = body # type: ignore[assignment] # Are we using the chunked-style of transfer encoding? self.chunk_left: int | None = None # Determine length of response self.length_remaining = self._init_length(request_method) # Used to return the correct amount of bytes for partial read()s self._decoded_buffer = BytesQueueBuffer() # If requested, preload the body. if preload_content and not self._body: self._body = self.read(decode_content=decode_content) def release_conn(self) -> None: if not self._pool or not self._connection: return None self._pool._put_conn(self._connection) self._connection = None def drain_conn(self) -> None: """ Read and discard any remaining HTTP response data in the response connection. Unread data in the HTTPResponse connection blocks the connection from being released back to the pool. """ try: self.read() except (HTTPError, OSError, BaseSSLError, HTTPException): pass @property def data(self) -> bytes: # For backwards-compat with earlier urllib3 0.4 and earlier. if self._body: return self._body # type: ignore[return-value] if self._fp: return self.read(cache_content=True) return None # type: ignore[return-value] @property def connection(self) -> HTTPConnection | None: return self._connection def isclosed(self) -> bool: return is_fp_closed(self._fp) def tell(self) -> int: """ Obtain the number of bytes pulled over the wire so far. May differ from the amount of content returned by :meth:``urllib3.response.HTTPResponse.read`` if bytes are encoded on the wire (e.g, compressed). """ return self._fp_bytes_read def _init_length(self, request_method: str | None) -> int | None: """ Set initial length value for Response content if available. """ length: int | None content_length: str | None = self.headers.get("content-length") if content_length is not None: if self.chunked: # This Response will fail with an IncompleteRead if it can't be # received as chunked. This method falls back to attempt reading # the response before raising an exception. log.warning( "Received response with both Content-Length and " "Transfer-Encoding set. This is expressly forbidden " "by RFC 7230 sec 3.3.2. Ignoring Content-Length and " "attempting to process response as Transfer-Encoding: " "chunked." ) return None try: # RFC 7230 section 3.3.2 specifies multiple content lengths can # be sent in a single Content-Length header # (e.g. Content-Length: 42, 42). This line ensures the values # are all valid ints and that as long as the `set` length is 1, # all values are the same. Otherwise, the header is invalid. lengths = {int(val) for val in content_length.split(",")} if len(lengths) > 1: raise InvalidHeader( "Content-Length contained multiple " "unmatching values (%s)" % content_length ) length = lengths.pop() except ValueError: length = None else: if length < 0: length = None else: # if content_length is None length = None # Convert status to int for comparison # In some cases, httplib returns a status of "_UNKNOWN" try: status = int(self.status) except ValueError: status = 0 # Check for responses that shouldn't include a body if status in (204, 304) or 100 <= status < 200 or request_method == "HEAD": length = 0 return length @contextmanager def _error_catcher(self) -> typing.Generator[None, None, None]: """ Catch low-level python exceptions, instead re-raising urllib3 variants, so that low-level exceptions are not leaked in the high-level api. On exit, release the connection back to the pool. """ clean_exit = False try: try: yield except SocketTimeout as e: # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but # there is yet no clean way to get at it from this context. raise ReadTimeoutError(self._pool, None, "Read timed out.") from e # type: ignore[arg-type] except BaseSSLError as e: # FIXME: Is there a better way to differentiate between SSLErrors? if "read operation timed out" not in str(e): # SSL errors related to framing/MAC get wrapped and reraised here raise SSLError(e) from e raise ReadTimeoutError(self._pool, None, "Read timed out.") from e # type: ignore[arg-type] except (HTTPException, OSError) as e: # This includes IncompleteRead. raise ProtocolError(f"Connection broken: {e!r}", e) from e # If no exception is thrown, we should avoid cleaning up # unnecessarily. clean_exit = True finally: # If we didn't terminate cleanly, we need to throw away our # connection. if not clean_exit: # The response may not be closed but we're not going to use it # anymore so close it now to ensure that the connection is # released back to the pool. if self._original_response: self._original_response.close() # Closing the response may not actually be sufficient to close # everything, so if we have a hold of the connection close that # too. if self._connection: self._connection.close() # If we hold the original response but it's closed now, we should # return the connection back to the pool. if self._original_response and self._original_response.isclosed(): self.release_conn() def _fp_read(self, amt: int | None = None) -> bytes: """ Read a response with the thought that reading the number of bytes larger than can fit in a 32-bit int at a time via SSL in some known cases leads to an overflow error that has to be prevented if `amt` or `self.length_remaining` indicate that a problem may happen. The known cases: * 3.8 <= CPython < 3.9.7 because of a bug https://github.com/urllib3/urllib3/issues/2513#issuecomment-1152559900. * urllib3 injected with pyOpenSSL-backed SSL-support. * CPython < 3.10 only when `amt` does not fit 32-bit int. """ assert self._fp c_int_max = 2**31 - 1 if ( ( (amt and amt > c_int_max) or (self.length_remaining and self.length_remaining > c_int_max) ) and not util.IS_SECURETRANSPORT and (util.IS_PYOPENSSL or sys.version_info < (3, 10)) ): buffer = io.BytesIO() # Besides `max_chunk_amt` being a maximum chunk size, it # affects memory overhead of reading a response by this # method in CPython. # `c_int_max` equal to 2 GiB - 1 byte is the actual maximum # chunk size that does not lead to an overflow error, but # 256 MiB is a compromise. max_chunk_amt = 2**28 while amt is None or amt != 0: if amt is not None: chunk_amt = min(amt, max_chunk_amt) amt -= chunk_amt else: chunk_amt = max_chunk_amt data = self._fp.read(chunk_amt) if not data: break buffer.write(data) del data # to reduce peak memory usage by `max_chunk_amt`. return buffer.getvalue() else: # StringIO doesn't like amt=None return self._fp.read(amt) if amt is not None else self._fp.read() def _raw_read( self, amt: int | None = None, ) -> bytes: """ Reads `amt` of bytes from the socket. """ if self._fp is None: return None # type: ignore[return-value] fp_closed = getattr(self._fp, "closed", False) with self._error_catcher(): data = self._fp_read(amt) if not fp_closed else b"" if amt is not None and amt != 0 and not data: # Platform-specific: Buggy versions of Python. # Close the connection when no data is returned # # This is redundant to what httplib/http.client _should_ # already do. However, versions of python released before # December 15, 2012 (http://bugs.python.org/issue16298) do # not properly close the connection in all cases. There is # no harm in redundantly calling close. self._fp.close() if ( self.enforce_content_length and self.length_remaining is not None and self.length_remaining != 0 ): # This is an edge case that httplib failed to cover due # to concerns of backward compatibility. We're # addressing it here to make sure IncompleteRead is # raised during streaming, so all calls with incorrect # Content-Length are caught. raise IncompleteRead(self._fp_bytes_read, self.length_remaining) if data: self._fp_bytes_read += len(data) if self.length_remaining is not None: self.length_remaining -= len(data) return data def read( self, amt: int | None = None, decode_content: bool | None = None, cache_content: bool = False, ) -> bytes: """ Similar to :meth:`http.client.HTTPResponse.read`, but with two additional parameters: ``decode_content`` and ``cache_content``. :param amt: How much of the content to read. If specified, caching is skipped because it doesn't make sense to cache partial content as the full response. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param cache_content: If True, will save the returned data such that the same result is returned despite of the state of the underlying file object. This is useful if you want the ``.data`` property to continue working after having ``.read()`` the file object. (Overridden if ``amt`` is set.) """ self._init_decoder() if decode_content is None: decode_content = self.decode_content if amt is not None: cache_content = False if len(self._decoded_buffer) >= amt: return self._decoded_buffer.get(amt) data = self._raw_read(amt) flush_decoder = amt is None or (amt != 0 and not data) if not data and len(self._decoded_buffer) == 0: return data if amt is None: data = self._decode(data, decode_content, flush_decoder) if cache_content: self._body = data else: # do not waste memory on buffer when not decoding if not decode_content: if self._has_decoded_content: raise RuntimeError( "Calling read(decode_content=False) is not supported after " "read(decode_content=True) was called." ) return data decoded_data = self._decode(data, decode_content, flush_decoder) self._decoded_buffer.put(decoded_data) while len(self._decoded_buffer) < amt and data: # TODO make sure to initially read enough data to get past the headers # For example, the GZ file header takes 10 bytes, we don't want to read # it one byte at a time data = self._raw_read(amt) decoded_data = self._decode(data, decode_content, flush_decoder) self._decoded_buffer.put(decoded_data) data = self._decoded_buffer.get(amt) return data def stream( self, amt: int | None = 2**16, decode_content: bool | None = None ) -> typing.Generator[bytes, None, None]: """ A generator wrapper for the read() method. A call will block until ``amt`` bytes have been read from the connection or until the connection is closed. :param amt: How much of the content to read. The generator will return up to much data per iteration, but may return less. This is particularly likely when using compressed data. However, the empty string will never be returned. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. """ if self.chunked and self.supports_chunked_reads(): yield from self.read_chunked(amt, decode_content=decode_content) else: while not is_fp_closed(self._fp) or len(self._decoded_buffer) > 0: data = self.read(amt=amt, decode_content=decode_content) if data: yield data # Overrides from io.IOBase def readable(self) -> bool: return True def close(self) -> None: if not self.closed and self._fp: self._fp.close() if self._connection: self._connection.close() if not self.auto_close: io.IOBase.close(self) @property def closed(self) -> bool: if not self.auto_close: return io.IOBase.closed.__get__(self) # type: ignore[no-any-return] elif self._fp is None: return True elif hasattr(self._fp, "isclosed"): return self._fp.isclosed() elif hasattr(self._fp, "closed"): return self._fp.closed else: return True def fileno(self) -> int: if self._fp is None: raise OSError("HTTPResponse has no file to get a fileno from") elif hasattr(self._fp, "fileno"): return self._fp.fileno() else: raise OSError( "The file-like object this HTTPResponse is wrapped " "around has no file descriptor" ) def flush(self) -> None: if ( self._fp is not None and hasattr(self._fp, "flush") and not getattr(self._fp, "closed", False) ): return self._fp.flush() def supports_chunked_reads(self) -> bool: """ Checks if the underlying file-like object looks like a :class:`http.client.HTTPResponse` object. We do this by testing for the fp attribute. If it is present we assume it returns raw chunks as processed by read_chunked(). """ return hasattr(self._fp, "fp") def _update_chunk_length(self) -> None: # First, we'll figure out length of a chunk and then # we'll try to read it from socket. if self.chunk_left is not None: return None line = self._fp.fp.readline() # type: ignore[union-attr] line = line.split(b";", 1)[0] try: self.chunk_left = int(line, 16) except ValueError: # Invalid chunked protocol response, abort. self.close() raise InvalidChunkLength(self, line) from None def _handle_chunk(self, amt: int | None) -> bytes: returned_chunk = None if amt is None: chunk = self._fp._safe_read(self.chunk_left) # type: ignore[union-attr] returned_chunk = chunk self._fp._safe_read(2) # type: ignore[union-attr] # Toss the CRLF at the end of the chunk. self.chunk_left = None elif self.chunk_left is not None and amt < self.chunk_left: value = self._fp._safe_read(amt) # type: ignore[union-attr] self.chunk_left = self.chunk_left - amt returned_chunk = value elif amt == self.chunk_left: value = self._fp._safe_read(amt) # type: ignore[union-attr] self._fp._safe_read(2) # type: ignore[union-attr] # Toss the CRLF at the end of the chunk. self.chunk_left = None returned_chunk = value else: # amt > self.chunk_left returned_chunk = self._fp._safe_read(self.chunk_left) # type: ignore[union-attr] self._fp._safe_read(2) # type: ignore[union-attr] # Toss the CRLF at the end of the chunk. self.chunk_left = None return returned_chunk # type: ignore[no-any-return] def read_chunked( self, amt: int | None = None, decode_content: bool | None = None ) -> typing.Generator[bytes, None, None]: """ Similar to :meth:`HTTPResponse.read`, but with an additional parameter: ``decode_content``. :param amt: How much of the content to read. If specified, caching is skipped because it doesn't make sense to cache partial content as the full response. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. """ self._init_decoder() # FIXME: Rewrite this method and make it a class with a better structured logic. if not self.chunked: raise ResponseNotChunked( "Response is not chunked. " "Header 'transfer-encoding: chunked' is missing." ) if not self.supports_chunked_reads(): raise BodyNotHttplibCompatible( "Body should be http.client.HTTPResponse like. " "It should have have an fp attribute which returns raw chunks." ) with self._error_catcher(): # Don't bother reading the body of a HEAD request. if self._original_response and is_response_to_head(self._original_response): self._original_response.close() return None # If a response is already read and closed # then return immediately. if self._fp.fp is None: # type: ignore[union-attr] return None while True: self._update_chunk_length() if self.chunk_left == 0: break chunk = self._handle_chunk(amt) decoded = self._decode( chunk, decode_content=decode_content, flush_decoder=False ) if decoded: yield decoded if decode_content: # On CPython and PyPy, we should never need to flush the # decoder. However, on Jython we *might* need to, so # lets defensively do it anyway. decoded = self._flush_decoder() if decoded: # Platform-specific: Jython. yield decoded # Chunk content ends with \r\n: discard it. while self._fp is not None: line = self._fp.fp.readline() if not line: # Some sites may not end with '\r\n'. break if line == b"\r\n": break # We read everything; close the "file". if self._original_response: self._original_response.close() @property def url(self) -> str | None: """ Returns the URL that was the source of this response. If the request that generated this response redirected, this method will return the final redirect location. """ return self._request_url @url.setter def url(self, url: str) -> None: self._request_url = url def __iter__(self) -> typing.Iterator[bytes]: buffer: list[bytes] = [] for chunk in self.stream(decode_content=True): if b"\n" in chunk: chunks = chunk.split(b"\n") yield b"".join(buffer) + chunks[0] + b"\n" for x in chunks[1:-1]: yield x + b"\n" if chunks[-1]: buffer = [chunks[-1]] else: buffer = [] else: buffer.append(chunk) if buffer: yield b"".join(buffer) urllib3-2.0.7/src/urllib3/contrib/__init__.py0000644000000000000000000000000014513541732015731 0ustar00urllib3-2.0.7/src/urllib3/contrib/pyopenssl.py0000644000000000000000000004534314513541732016251 0ustar00""" Module for using pyOpenSSL as a TLS backend. This module was relevant before the standard library ``ssl`` module supported SNI, but now that we've dropped support for Python 2.7 all relevant Python versions support SNI so **this module is no longer recommended**. This needs the following packages installed: * `pyOpenSSL`_ (tested with 16.0.0) * `cryptography`_ (minimum 1.3.4, from pyopenssl) * `idna`_ (minimum 2.0, from cryptography) However, pyOpenSSL depends on cryptography, which depends on idna, so while we use all three directly here we end up having relatively few packages required. You can install them with the following command: .. code-block:: bash $ python -m pip install pyopenssl cryptography idna To activate certificate checking, call :func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code before you begin making HTTP requests. This can be done in a ``sitecustomize`` module, or at any other time before your application begins using ``urllib3``, like this: .. code-block:: python try: import urllib3.contrib.pyopenssl urllib3.contrib.pyopenssl.inject_into_urllib3() except ImportError: pass .. _pyopenssl: https://www.pyopenssl.org .. _cryptography: https://cryptography.io .. _idna: https://github.com/kjd/idna """ from __future__ import annotations import OpenSSL.SSL # type: ignore[import] from cryptography import x509 try: from cryptography.x509 import UnsupportedExtension # type: ignore[attr-defined] except ImportError: # UnsupportedExtension is gone in cryptography >= 2.1.0 class UnsupportedExtension(Exception): # type: ignore[no-redef] pass import logging import ssl import typing from io import BytesIO from socket import socket as socket_cls from socket import timeout from .. import util if typing.TYPE_CHECKING: from OpenSSL.crypto import X509 # type: ignore[import] __all__ = ["inject_into_urllib3", "extract_from_urllib3"] # Map from urllib3 to PyOpenSSL compatible parameter-values. _openssl_versions = { util.ssl_.PROTOCOL_TLS: OpenSSL.SSL.SSLv23_METHOD, # type: ignore[attr-defined] util.ssl_.PROTOCOL_TLS_CLIENT: OpenSSL.SSL.SSLv23_METHOD, # type: ignore[attr-defined] ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD, } if hasattr(ssl, "PROTOCOL_TLSv1_1") and hasattr(OpenSSL.SSL, "TLSv1_1_METHOD"): _openssl_versions[ssl.PROTOCOL_TLSv1_1] = OpenSSL.SSL.TLSv1_1_METHOD if hasattr(ssl, "PROTOCOL_TLSv1_2") and hasattr(OpenSSL.SSL, "TLSv1_2_METHOD"): _openssl_versions[ssl.PROTOCOL_TLSv1_2] = OpenSSL.SSL.TLSv1_2_METHOD _stdlib_to_openssl_verify = { ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE, ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER, ssl.CERT_REQUIRED: OpenSSL.SSL.VERIFY_PEER + OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT, } _openssl_to_stdlib_verify = {v: k for k, v in _stdlib_to_openssl_verify.items()} # The SSLvX values are the most likely to be missing in the future # but we check them all just to be sure. _OP_NO_SSLv2_OR_SSLv3: int = getattr(OpenSSL.SSL, "OP_NO_SSLv2", 0) | getattr( OpenSSL.SSL, "OP_NO_SSLv3", 0 ) _OP_NO_TLSv1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1", 0) _OP_NO_TLSv1_1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_1", 0) _OP_NO_TLSv1_2: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_2", 0) _OP_NO_TLSv1_3: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_3", 0) _openssl_to_ssl_minimum_version: dict[int, int] = { ssl.TLSVersion.MINIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3, ssl.TLSVersion.TLSv1: _OP_NO_SSLv2_OR_SSLv3, ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1, ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1, ssl.TLSVersion.TLSv1_3: ( _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 ), ssl.TLSVersion.MAXIMUM_SUPPORTED: ( _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 ), } _openssl_to_ssl_maximum_version: dict[int, int] = { ssl.TLSVersion.MINIMUM_SUPPORTED: ( _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3 ), ssl.TLSVersion.TLSv1: ( _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3 ), ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3, ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_3, ssl.TLSVersion.TLSv1_3: _OP_NO_SSLv2_OR_SSLv3, ssl.TLSVersion.MAXIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3, } # OpenSSL will only write 16K at a time SSL_WRITE_BLOCKSIZE = 16384 orig_util_SSLContext = util.ssl_.SSLContext log = logging.getLogger(__name__) def inject_into_urllib3() -> None: "Monkey-patch urllib3 with PyOpenSSL-backed SSL-support." _validate_dependencies_met() util.SSLContext = PyOpenSSLContext # type: ignore[assignment] util.ssl_.SSLContext = PyOpenSSLContext # type: ignore[assignment] util.IS_PYOPENSSL = True util.ssl_.IS_PYOPENSSL = True def extract_from_urllib3() -> None: "Undo monkey-patching by :func:`inject_into_urllib3`." util.SSLContext = orig_util_SSLContext util.ssl_.SSLContext = orig_util_SSLContext util.IS_PYOPENSSL = False util.ssl_.IS_PYOPENSSL = False def _validate_dependencies_met() -> None: """ Verifies that PyOpenSSL's package-level dependencies have been met. Throws `ImportError` if they are not met. """ # Method added in `cryptography==1.1`; not available in older versions from cryptography.x509.extensions import Extensions if getattr(Extensions, "get_extension_for_class", None) is None: raise ImportError( "'cryptography' module missing required functionality. " "Try upgrading to v1.3.4 or newer." ) # pyOpenSSL 0.14 and above use cryptography for OpenSSL bindings. The _x509 # attribute is only present on those versions. from OpenSSL.crypto import X509 x509 = X509() if getattr(x509, "_x509", None) is None: raise ImportError( "'pyOpenSSL' module missing required functionality. " "Try upgrading to v0.14 or newer." ) def _dnsname_to_stdlib(name: str) -> str | None: """ Converts a dNSName SubjectAlternativeName field to the form used by the standard library on the given Python version. Cryptography produces a dNSName as a unicode string that was idna-decoded from ASCII bytes. We need to idna-encode that string to get it back, and then on Python 3 we also need to convert to unicode via UTF-8 (the stdlib uses PyUnicode_FromStringAndSize on it, which decodes via UTF-8). If the name cannot be idna-encoded then we return None signalling that the name given should be skipped. """ def idna_encode(name: str) -> bytes | None: """ Borrowed wholesale from the Python Cryptography Project. It turns out that we can't just safely call `idna.encode`: it can explode for wildcard names. This avoids that problem. """ import idna try: for prefix in ["*.", "."]: if name.startswith(prefix): name = name[len(prefix) :] return prefix.encode("ascii") + idna.encode(name) return idna.encode(name) except idna.core.IDNAError: return None # Don't send IPv6 addresses through the IDNA encoder. if ":" in name: return name encoded_name = idna_encode(name) if encoded_name is None: return None return encoded_name.decode("utf-8") def get_subj_alt_name(peer_cert: X509) -> list[tuple[str, str]]: """ Given an PyOpenSSL certificate, provides all the subject alternative names. """ cert = peer_cert.to_cryptography() # We want to find the SAN extension. Ask Cryptography to locate it (it's # faster than looping in Python) try: ext = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName).value except x509.ExtensionNotFound: # No such extension, return the empty list. return [] except ( x509.DuplicateExtension, UnsupportedExtension, x509.UnsupportedGeneralNameType, UnicodeError, ) as e: # A problem has been found with the quality of the certificate. Assume # no SAN field is present. log.warning( "A problem was encountered with the certificate that prevented " "urllib3 from finding the SubjectAlternativeName field. This can " "affect certificate validation. The error was %s", e, ) return [] # We want to return dNSName and iPAddress fields. We need to cast the IPs # back to strings because the match_hostname function wants them as # strings. # Sadly the DNS names need to be idna encoded and then, on Python 3, UTF-8 # decoded. This is pretty frustrating, but that's what the standard library # does with certificates, and so we need to attempt to do the same. # We also want to skip over names which cannot be idna encoded. names = [ ("DNS", name) for name in map(_dnsname_to_stdlib, ext.get_values_for_type(x509.DNSName)) if name is not None ] names.extend( ("IP Address", str(name)) for name in ext.get_values_for_type(x509.IPAddress) ) return names class WrappedSocket: """API-compatibility wrapper for Python OpenSSL's Connection-class.""" def __init__( self, connection: OpenSSL.SSL.Connection, socket: socket_cls, suppress_ragged_eofs: bool = True, ) -> None: self.connection = connection self.socket = socket self.suppress_ragged_eofs = suppress_ragged_eofs self._io_refs = 0 self._closed = False def fileno(self) -> int: return self.socket.fileno() # Copy-pasted from Python 3.5 source code def _decref_socketios(self) -> None: if self._io_refs > 0: self._io_refs -= 1 if self._closed: self.close() def recv(self, *args: typing.Any, **kwargs: typing.Any) -> bytes: try: data = self.connection.recv(*args, **kwargs) except OpenSSL.SSL.SysCallError as e: if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"): return b"" else: raise OSError(e.args[0], str(e)) from e except OpenSSL.SSL.ZeroReturnError: if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN: return b"" else: raise except OpenSSL.SSL.WantReadError as e: if not util.wait_for_read(self.socket, self.socket.gettimeout()): raise timeout("The read operation timed out") from e else: return self.recv(*args, **kwargs) # TLS 1.3 post-handshake authentication except OpenSSL.SSL.Error as e: raise ssl.SSLError(f"read error: {e!r}") from e else: return data # type: ignore[no-any-return] def recv_into(self, *args: typing.Any, **kwargs: typing.Any) -> int: try: return self.connection.recv_into(*args, **kwargs) # type: ignore[no-any-return] except OpenSSL.SSL.SysCallError as e: if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"): return 0 else: raise OSError(e.args[0], str(e)) from e except OpenSSL.SSL.ZeroReturnError: if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN: return 0 else: raise except OpenSSL.SSL.WantReadError as e: if not util.wait_for_read(self.socket, self.socket.gettimeout()): raise timeout("The read operation timed out") from e else: return self.recv_into(*args, **kwargs) # TLS 1.3 post-handshake authentication except OpenSSL.SSL.Error as e: raise ssl.SSLError(f"read error: {e!r}") from e def settimeout(self, timeout: float) -> None: return self.socket.settimeout(timeout) def _send_until_done(self, data: bytes) -> int: while True: try: return self.connection.send(data) # type: ignore[no-any-return] except OpenSSL.SSL.WantWriteError as e: if not util.wait_for_write(self.socket, self.socket.gettimeout()): raise timeout() from e continue except OpenSSL.SSL.SysCallError as e: raise OSError(e.args[0], str(e)) from e def sendall(self, data: bytes) -> None: total_sent = 0 while total_sent < len(data): sent = self._send_until_done( data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE] ) total_sent += sent def shutdown(self) -> None: # FIXME rethrow compatible exceptions should we ever use this self.connection.shutdown() def close(self) -> None: self._closed = True if self._io_refs <= 0: self._real_close() def _real_close(self) -> None: try: return self.connection.close() # type: ignore[no-any-return] except OpenSSL.SSL.Error: return def getpeercert( self, binary_form: bool = False ) -> dict[str, list[typing.Any]] | None: x509 = self.connection.get_peer_certificate() if not x509: return x509 # type: ignore[no-any-return] if binary_form: return OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_ASN1, x509) # type: ignore[no-any-return] return { "subject": ((("commonName", x509.get_subject().CN),),), # type: ignore[dict-item] "subjectAltName": get_subj_alt_name(x509), } def version(self) -> str: return self.connection.get_protocol_version_name() # type: ignore[no-any-return] WrappedSocket.makefile = socket_cls.makefile # type: ignore[attr-defined] class PyOpenSSLContext: """ I am a wrapper class for the PyOpenSSL ``Context`` object. I am responsible for translating the interface of the standard library ``SSLContext`` object to calls into PyOpenSSL. """ def __init__(self, protocol: int) -> None: self.protocol = _openssl_versions[protocol] self._ctx = OpenSSL.SSL.Context(self.protocol) self._options = 0 self.check_hostname = False self._minimum_version: int = ssl.TLSVersion.MINIMUM_SUPPORTED self._maximum_version: int = ssl.TLSVersion.MAXIMUM_SUPPORTED @property def options(self) -> int: return self._options @options.setter def options(self, value: int) -> None: self._options = value self._set_ctx_options() @property def verify_mode(self) -> int: return _openssl_to_stdlib_verify[self._ctx.get_verify_mode()] @verify_mode.setter def verify_mode(self, value: ssl.VerifyMode) -> None: self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback) def set_default_verify_paths(self) -> None: self._ctx.set_default_verify_paths() def set_ciphers(self, ciphers: bytes | str) -> None: if isinstance(ciphers, str): ciphers = ciphers.encode("utf-8") self._ctx.set_cipher_list(ciphers) def load_verify_locations( self, cafile: str | None = None, capath: str | None = None, cadata: bytes | None = None, ) -> None: if cafile is not None: cafile = cafile.encode("utf-8") # type: ignore[assignment] if capath is not None: capath = capath.encode("utf-8") # type: ignore[assignment] try: self._ctx.load_verify_locations(cafile, capath) if cadata is not None: self._ctx.load_verify_locations(BytesIO(cadata)) except OpenSSL.SSL.Error as e: raise ssl.SSLError(f"unable to load trusted certificates: {e!r}") from e def load_cert_chain( self, certfile: str, keyfile: str | None = None, password: str | None = None, ) -> None: try: self._ctx.use_certificate_chain_file(certfile) if password is not None: if not isinstance(password, bytes): password = password.encode("utf-8") # type: ignore[assignment] self._ctx.set_passwd_cb(lambda *_: password) self._ctx.use_privatekey_file(keyfile or certfile) except OpenSSL.SSL.Error as e: raise ssl.SSLError(f"Unable to load certificate chain: {e!r}") from e def set_alpn_protocols(self, protocols: list[bytes | str]) -> None: protocols = [util.util.to_bytes(p, "ascii") for p in protocols] return self._ctx.set_alpn_protos(protocols) # type: ignore[no-any-return] def wrap_socket( self, sock: socket_cls, server_side: bool = False, do_handshake_on_connect: bool = True, suppress_ragged_eofs: bool = True, server_hostname: bytes | str | None = None, ) -> WrappedSocket: cnx = OpenSSL.SSL.Connection(self._ctx, sock) # If server_hostname is an IP, don't use it for SNI, per RFC6066 Section 3 if server_hostname and not util.ssl_.is_ipaddress(server_hostname): if isinstance(server_hostname, str): server_hostname = server_hostname.encode("utf-8") cnx.set_tlsext_host_name(server_hostname) cnx.set_connect_state() while True: try: cnx.do_handshake() except OpenSSL.SSL.WantReadError as e: if not util.wait_for_read(sock, sock.gettimeout()): raise timeout("select timed out") from e continue except OpenSSL.SSL.Error as e: raise ssl.SSLError(f"bad handshake: {e!r}") from e break return WrappedSocket(cnx, sock) def _set_ctx_options(self) -> None: self._ctx.set_options( self._options | _openssl_to_ssl_minimum_version[self._minimum_version] | _openssl_to_ssl_maximum_version[self._maximum_version] ) @property def minimum_version(self) -> int: return self._minimum_version @minimum_version.setter def minimum_version(self, minimum_version: int) -> None: self._minimum_version = minimum_version self._set_ctx_options() @property def maximum_version(self) -> int: return self._maximum_version @maximum_version.setter def maximum_version(self, maximum_version: int) -> None: self._maximum_version = maximum_version self._set_ctx_options() def _verify_callback( cnx: OpenSSL.SSL.Connection, x509: X509, err_no: int, err_depth: int, return_code: int, ) -> bool: return err_no == 0 urllib3-2.0.7/src/urllib3/contrib/securetransport.py0000644000000000000000000010251114513541732017447 0ustar00""" SecureTranport support for urllib3 via ctypes. This makes platform-native TLS available to urllib3 users on macOS without the use of a compiler. This is an important feature because the Python Package Index is moving to become a TLSv1.2-or-higher server, and the default OpenSSL that ships with macOS is not capable of doing TLSv1.2. The only way to resolve this is to give macOS users an alternative solution to the problem, and that solution is to use SecureTransport. We use ctypes here because this solution must not require a compiler. That's because pip is not allowed to require a compiler either. This is not intended to be a seriously long-term solution to this problem. The hope is that PEP 543 will eventually solve this issue for us, at which point we can retire this contrib module. But in the short term, we need to solve the impending tire fire that is Python on Mac without this kind of contrib module. So...here we are. To use this module, simply import and inject it:: import urllib3.contrib.securetransport urllib3.contrib.securetransport.inject_into_urllib3() Happy TLSing! This code is a bastardised version of the code found in Will Bond's oscrypto library. An enormous debt is owed to him for blazing this trail for us. For that reason, this code should be considered to be covered both by urllib3's license and by oscrypto's: .. code-block:: Copyright (c) 2015-2016 Will Bond Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ from __future__ import annotations import contextlib import ctypes import errno import os.path import shutil import socket import ssl import struct import threading import typing import warnings import weakref from socket import socket as socket_cls from .. import util from ._securetransport.bindings import ( # type: ignore[attr-defined] CoreFoundation, Security, ) from ._securetransport.low_level import ( SecurityConst, _assert_no_error, _build_tls_unknown_ca_alert, _cert_array_from_pem, _create_cfstring_array, _load_client_cert_chain, _temporary_keychain, ) warnings.warn( "'urllib3.contrib.securetransport' module is deprecated and will be removed " "in urllib3 v2.1.0. Read more in this issue: " "https://github.com/urllib3/urllib3/issues/2681", category=DeprecationWarning, stacklevel=2, ) if typing.TYPE_CHECKING: from typing_extensions import Literal __all__ = ["inject_into_urllib3", "extract_from_urllib3"] orig_util_SSLContext = util.ssl_.SSLContext # This dictionary is used by the read callback to obtain a handle to the # calling wrapped socket. This is a pretty silly approach, but for now it'll # do. I feel like I should be able to smuggle a handle to the wrapped socket # directly in the SSLConnectionRef, but for now this approach will work I # guess. # # We need to lock around this structure for inserts, but we don't do it for # reads/writes in the callbacks. The reasoning here goes as follows: # # 1. It is not possible to call into the callbacks before the dictionary is # populated, so once in the callback the id must be in the dictionary. # 2. The callbacks don't mutate the dictionary, they only read from it, and # so cannot conflict with any of the insertions. # # This is good: if we had to lock in the callbacks we'd drastically slow down # the performance of this code. _connection_refs: weakref.WeakValueDictionary[ int, WrappedSocket ] = weakref.WeakValueDictionary() _connection_ref_lock = threading.Lock() # Limit writes to 16kB. This is OpenSSL's limit, but we'll cargo-cult it over # for no better reason than we need *a* limit, and this one is right there. SSL_WRITE_BLOCKSIZE = 16384 # Basically this is simple: for PROTOCOL_SSLv23 we turn it into a low of # TLSv1 and a high of TLSv1.2. For everything else, we pin to that version. # TLSv1 to 1.2 are supported on macOS 10.8+ _protocol_to_min_max = { util.ssl_.PROTOCOL_TLS: (SecurityConst.kTLSProtocol1, SecurityConst.kTLSProtocol12), # type: ignore[attr-defined] util.ssl_.PROTOCOL_TLS_CLIENT: ( # type: ignore[attr-defined] SecurityConst.kTLSProtocol1, SecurityConst.kTLSProtocol12, ), } if hasattr(ssl, "PROTOCOL_SSLv2"): _protocol_to_min_max[ssl.PROTOCOL_SSLv2] = ( SecurityConst.kSSLProtocol2, SecurityConst.kSSLProtocol2, ) if hasattr(ssl, "PROTOCOL_SSLv3"): _protocol_to_min_max[ssl.PROTOCOL_SSLv3] = ( SecurityConst.kSSLProtocol3, SecurityConst.kSSLProtocol3, ) if hasattr(ssl, "PROTOCOL_TLSv1"): _protocol_to_min_max[ssl.PROTOCOL_TLSv1] = ( SecurityConst.kTLSProtocol1, SecurityConst.kTLSProtocol1, ) if hasattr(ssl, "PROTOCOL_TLSv1_1"): _protocol_to_min_max[ssl.PROTOCOL_TLSv1_1] = ( SecurityConst.kTLSProtocol11, SecurityConst.kTLSProtocol11, ) if hasattr(ssl, "PROTOCOL_TLSv1_2"): _protocol_to_min_max[ssl.PROTOCOL_TLSv1_2] = ( SecurityConst.kTLSProtocol12, SecurityConst.kTLSProtocol12, ) _tls_version_to_st: dict[int, int] = { ssl.TLSVersion.MINIMUM_SUPPORTED: SecurityConst.kTLSProtocol1, ssl.TLSVersion.TLSv1: SecurityConst.kTLSProtocol1, ssl.TLSVersion.TLSv1_1: SecurityConst.kTLSProtocol11, ssl.TLSVersion.TLSv1_2: SecurityConst.kTLSProtocol12, ssl.TLSVersion.MAXIMUM_SUPPORTED: SecurityConst.kTLSProtocol12, } def inject_into_urllib3() -> None: """ Monkey-patch urllib3 with SecureTransport-backed SSL-support. """ util.SSLContext = SecureTransportContext # type: ignore[assignment] util.ssl_.SSLContext = SecureTransportContext # type: ignore[assignment] util.IS_SECURETRANSPORT = True util.ssl_.IS_SECURETRANSPORT = True def extract_from_urllib3() -> None: """ Undo monkey-patching by :func:`inject_into_urllib3`. """ util.SSLContext = orig_util_SSLContext util.ssl_.SSLContext = orig_util_SSLContext util.IS_SECURETRANSPORT = False util.ssl_.IS_SECURETRANSPORT = False def _read_callback( connection_id: int, data_buffer: int, data_length_pointer: bytearray ) -> int: """ SecureTransport read callback. This is called by ST to request that data be returned from the socket. """ wrapped_socket = None try: wrapped_socket = _connection_refs.get(connection_id) if wrapped_socket is None: return SecurityConst.errSSLInternal base_socket = wrapped_socket.socket requested_length = data_length_pointer[0] timeout = wrapped_socket.gettimeout() error = None read_count = 0 try: while read_count < requested_length: if timeout is None or timeout >= 0: if not util.wait_for_read(base_socket, timeout): raise OSError(errno.EAGAIN, "timed out") remaining = requested_length - read_count buffer = (ctypes.c_char * remaining).from_address( data_buffer + read_count ) chunk_size = base_socket.recv_into(buffer, remaining) read_count += chunk_size if not chunk_size: if not read_count: return SecurityConst.errSSLClosedGraceful break except OSError as e: error = e.errno if error is not None and error != errno.EAGAIN: data_length_pointer[0] = read_count if error == errno.ECONNRESET or error == errno.EPIPE: return SecurityConst.errSSLClosedAbort raise data_length_pointer[0] = read_count if read_count != requested_length: return SecurityConst.errSSLWouldBlock return 0 except Exception as e: if wrapped_socket is not None: wrapped_socket._exception = e return SecurityConst.errSSLInternal def _write_callback( connection_id: int, data_buffer: int, data_length_pointer: bytearray ) -> int: """ SecureTransport write callback. This is called by ST to request that data actually be sent on the network. """ wrapped_socket = None try: wrapped_socket = _connection_refs.get(connection_id) if wrapped_socket is None: return SecurityConst.errSSLInternal base_socket = wrapped_socket.socket bytes_to_write = data_length_pointer[0] data = ctypes.string_at(data_buffer, bytes_to_write) timeout = wrapped_socket.gettimeout() error = None sent = 0 try: while sent < bytes_to_write: if timeout is None or timeout >= 0: if not util.wait_for_write(base_socket, timeout): raise OSError(errno.EAGAIN, "timed out") chunk_sent = base_socket.send(data) sent += chunk_sent # This has some needless copying here, but I'm not sure there's # much value in optimising this data path. data = data[chunk_sent:] except OSError as e: error = e.errno if error is not None and error != errno.EAGAIN: data_length_pointer[0] = sent if error == errno.ECONNRESET or error == errno.EPIPE: return SecurityConst.errSSLClosedAbort raise data_length_pointer[0] = sent if sent != bytes_to_write: return SecurityConst.errSSLWouldBlock return 0 except Exception as e: if wrapped_socket is not None: wrapped_socket._exception = e return SecurityConst.errSSLInternal # We need to keep these two objects references alive: if they get GC'd while # in use then SecureTransport could attempt to call a function that is in freed # memory. That would be...uh...bad. Yeah, that's the word. Bad. _read_callback_pointer = Security.SSLReadFunc(_read_callback) _write_callback_pointer = Security.SSLWriteFunc(_write_callback) class WrappedSocket: """ API-compatibility wrapper for Python's OpenSSL wrapped socket object. """ def __init__(self, socket: socket_cls) -> None: self.socket = socket self.context = None self._io_refs = 0 self._closed = False self._real_closed = False self._exception: Exception | None = None self._keychain = None self._keychain_dir: str | None = None self._client_cert_chain = None # We save off the previously-configured timeout and then set it to # zero. This is done because we use select and friends to handle the # timeouts, but if we leave the timeout set on the lower socket then # Python will "kindly" call select on that socket again for us. Avoid # that by forcing the timeout to zero. self._timeout = self.socket.gettimeout() self.socket.settimeout(0) @contextlib.contextmanager def _raise_on_error(self) -> typing.Generator[None, None, None]: """ A context manager that can be used to wrap calls that do I/O from SecureTransport. If any of the I/O callbacks hit an exception, this context manager will correctly propagate the exception after the fact. This avoids silently swallowing those exceptions. It also correctly forces the socket closed. """ self._exception = None # We explicitly don't catch around this yield because in the unlikely # event that an exception was hit in the block we don't want to swallow # it. yield if self._exception is not None: exception, self._exception = self._exception, None self._real_close() raise exception def _set_alpn_protocols(self, protocols: list[bytes] | None) -> None: """ Sets up the ALPN protocols on the context. """ if not protocols: return protocols_arr = _create_cfstring_array(protocols) try: result = Security.SSLSetALPNProtocols(self.context, protocols_arr) _assert_no_error(result) finally: CoreFoundation.CFRelease(protocols_arr) def _custom_validate(self, verify: bool, trust_bundle: bytes | None) -> None: """ Called when we have set custom validation. We do this in two cases: first, when cert validation is entirely disabled; and second, when using a custom trust DB. Raises an SSLError if the connection is not trusted. """ # If we disabled cert validation, just say: cool. if not verify or trust_bundle is None: return successes = ( SecurityConst.kSecTrustResultUnspecified, SecurityConst.kSecTrustResultProceed, ) try: trust_result = self._evaluate_trust(trust_bundle) if trust_result in successes: return reason = f"error code: {int(trust_result)}" exc = None except Exception as e: # Do not trust on error reason = f"exception: {e!r}" exc = e # SecureTransport does not send an alert nor shuts down the connection. rec = _build_tls_unknown_ca_alert(self.version()) self.socket.sendall(rec) # close the connection immediately # l_onoff = 1, activate linger # l_linger = 0, linger for 0 seoncds opts = struct.pack("ii", 1, 0) self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, opts) self._real_close() raise ssl.SSLError(f"certificate verify failed, {reason}") from exc def _evaluate_trust(self, trust_bundle: bytes) -> int: # We want data in memory, so load it up. if os.path.isfile(trust_bundle): with open(trust_bundle, "rb") as f: trust_bundle = f.read() cert_array = None trust = Security.SecTrustRef() try: # Get a CFArray that contains the certs we want. cert_array = _cert_array_from_pem(trust_bundle) # Ok, now the hard part. We want to get the SecTrustRef that ST has # created for this connection, shove our CAs into it, tell ST to # ignore everything else it knows, and then ask if it can build a # chain. This is a buuuunch of code. result = Security.SSLCopyPeerTrust(self.context, ctypes.byref(trust)) _assert_no_error(result) if not trust: raise ssl.SSLError("Failed to copy trust reference") result = Security.SecTrustSetAnchorCertificates(trust, cert_array) _assert_no_error(result) result = Security.SecTrustSetAnchorCertificatesOnly(trust, True) _assert_no_error(result) trust_result = Security.SecTrustResultType() result = Security.SecTrustEvaluate(trust, ctypes.byref(trust_result)) _assert_no_error(result) finally: if trust: CoreFoundation.CFRelease(trust) if cert_array is not None: CoreFoundation.CFRelease(cert_array) return trust_result.value # type: ignore[no-any-return] def handshake( self, server_hostname: bytes | str | None, verify: bool, trust_bundle: bytes | None, min_version: int, max_version: int, client_cert: str | None, client_key: str | None, client_key_passphrase: typing.Any, alpn_protocols: list[bytes] | None, ) -> None: """ Actually performs the TLS handshake. This is run automatically by wrapped socket, and shouldn't be needed in user code. """ # First, we do the initial bits of connection setup. We need to create # a context, set its I/O funcs, and set the connection reference. self.context = Security.SSLCreateContext( None, SecurityConst.kSSLClientSide, SecurityConst.kSSLStreamType ) result = Security.SSLSetIOFuncs( self.context, _read_callback_pointer, _write_callback_pointer ) _assert_no_error(result) # Here we need to compute the handle to use. We do this by taking the # id of self modulo 2**31 - 1. If this is already in the dictionary, we # just keep incrementing by one until we find a free space. with _connection_ref_lock: handle = id(self) % 2147483647 while handle in _connection_refs: handle = (handle + 1) % 2147483647 _connection_refs[handle] = self result = Security.SSLSetConnection(self.context, handle) _assert_no_error(result) # If we have a server hostname, we should set that too. # RFC6066 Section 3 tells us not to use SNI when the host is an IP, but we have # to do it anyway to match server_hostname against the server certificate if server_hostname: if not isinstance(server_hostname, bytes): server_hostname = server_hostname.encode("utf-8") result = Security.SSLSetPeerDomainName( self.context, server_hostname, len(server_hostname) ) _assert_no_error(result) # Setup the ALPN protocols. self._set_alpn_protocols(alpn_protocols) # Set the minimum and maximum TLS versions. result = Security.SSLSetProtocolVersionMin(self.context, min_version) _assert_no_error(result) result = Security.SSLSetProtocolVersionMax(self.context, max_version) _assert_no_error(result) # If there's a trust DB, we need to use it. We do that by telling # SecureTransport to break on server auth. We also do that if we don't # want to validate the certs at all: we just won't actually do any # authing in that case. if not verify or trust_bundle is not None: result = Security.SSLSetSessionOption( self.context, SecurityConst.kSSLSessionOptionBreakOnServerAuth, True ) _assert_no_error(result) # If there's a client cert, we need to use it. if client_cert: self._keychain, self._keychain_dir = _temporary_keychain() self._client_cert_chain = _load_client_cert_chain( self._keychain, client_cert, client_key ) result = Security.SSLSetCertificate(self.context, self._client_cert_chain) _assert_no_error(result) while True: with self._raise_on_error(): result = Security.SSLHandshake(self.context) if result == SecurityConst.errSSLWouldBlock: raise socket.timeout("handshake timed out") elif result == SecurityConst.errSSLServerAuthCompleted: self._custom_validate(verify, trust_bundle) continue else: _assert_no_error(result) break def fileno(self) -> int: return self.socket.fileno() # Copy-pasted from Python 3.5 source code def _decref_socketios(self) -> None: if self._io_refs > 0: self._io_refs -= 1 if self._closed: self.close() def recv(self, bufsiz: int) -> bytes: buffer = ctypes.create_string_buffer(bufsiz) bytes_read = self.recv_into(buffer, bufsiz) data = buffer[:bytes_read] return typing.cast(bytes, data) def recv_into( self, buffer: ctypes.Array[ctypes.c_char], nbytes: int | None = None ) -> int: # Read short on EOF. if self._real_closed: return 0 if nbytes is None: nbytes = len(buffer) buffer = (ctypes.c_char * nbytes).from_buffer(buffer) processed_bytes = ctypes.c_size_t(0) with self._raise_on_error(): result = Security.SSLRead( self.context, buffer, nbytes, ctypes.byref(processed_bytes) ) # There are some result codes that we want to treat as "not always # errors". Specifically, those are errSSLWouldBlock, # errSSLClosedGraceful, and errSSLClosedNoNotify. if result == SecurityConst.errSSLWouldBlock: # If we didn't process any bytes, then this was just a time out. # However, we can get errSSLWouldBlock in situations when we *did* # read some data, and in those cases we should just read "short" # and return. if processed_bytes.value == 0: # Timed out, no data read. raise socket.timeout("recv timed out") elif result in ( SecurityConst.errSSLClosedGraceful, SecurityConst.errSSLClosedNoNotify, ): # The remote peer has closed this connection. We should do so as # well. Note that we don't actually return here because in # principle this could actually be fired along with return data. # It's unlikely though. self._real_close() else: _assert_no_error(result) # Ok, we read and probably succeeded. We should return whatever data # was actually read. return processed_bytes.value def settimeout(self, timeout: float) -> None: self._timeout = timeout def gettimeout(self) -> float | None: return self._timeout def send(self, data: bytes) -> int: processed_bytes = ctypes.c_size_t(0) with self._raise_on_error(): result = Security.SSLWrite( self.context, data, len(data), ctypes.byref(processed_bytes) ) if result == SecurityConst.errSSLWouldBlock and processed_bytes.value == 0: # Timed out raise socket.timeout("send timed out") else: _assert_no_error(result) # We sent, and probably succeeded. Tell them how much we sent. return processed_bytes.value def sendall(self, data: bytes) -> None: total_sent = 0 while total_sent < len(data): sent = self.send(data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE]) total_sent += sent def shutdown(self) -> None: with self._raise_on_error(): Security.SSLClose(self.context) def close(self) -> None: self._closed = True # TODO: should I do clean shutdown here? Do I have to? if self._io_refs <= 0: self._real_close() def _real_close(self) -> None: self._real_closed = True if self.context: CoreFoundation.CFRelease(self.context) self.context = None if self._client_cert_chain: CoreFoundation.CFRelease(self._client_cert_chain) self._client_cert_chain = None if self._keychain: Security.SecKeychainDelete(self._keychain) CoreFoundation.CFRelease(self._keychain) shutil.rmtree(self._keychain_dir) self._keychain = self._keychain_dir = None return self.socket.close() def getpeercert(self, binary_form: bool = False) -> bytes | None: # Urgh, annoying. # # Here's how we do this: # # 1. Call SSLCopyPeerTrust to get hold of the trust object for this # connection. # 2. Call SecTrustGetCertificateAtIndex for index 0 to get the leaf. # 3. To get the CN, call SecCertificateCopyCommonName and process that # string so that it's of the appropriate type. # 4. To get the SAN, we need to do something a bit more complex: # a. Call SecCertificateCopyValues to get the data, requesting # kSecOIDSubjectAltName. # b. Mess about with this dictionary to try to get the SANs out. # # This is gross. Really gross. It's going to be a few hundred LoC extra # just to repeat something that SecureTransport can *already do*. So my # operating assumption at this time is that what we want to do is # instead to just flag to urllib3 that it shouldn't do its own hostname # validation when using SecureTransport. if not binary_form: raise ValueError("SecureTransport only supports dumping binary certs") trust = Security.SecTrustRef() certdata = None der_bytes = None try: # Grab the trust store. result = Security.SSLCopyPeerTrust(self.context, ctypes.byref(trust)) _assert_no_error(result) if not trust: # Probably we haven't done the handshake yet. No biggie. return None cert_count = Security.SecTrustGetCertificateCount(trust) if not cert_count: # Also a case that might happen if we haven't handshaked. # Handshook? Handshaken? return None leaf = Security.SecTrustGetCertificateAtIndex(trust, 0) assert leaf # Ok, now we want the DER bytes. certdata = Security.SecCertificateCopyData(leaf) assert certdata data_length = CoreFoundation.CFDataGetLength(certdata) data_buffer = CoreFoundation.CFDataGetBytePtr(certdata) der_bytes = ctypes.string_at(data_buffer, data_length) finally: if certdata: CoreFoundation.CFRelease(certdata) if trust: CoreFoundation.CFRelease(trust) return der_bytes def version(self) -> str: protocol = Security.SSLProtocol() result = Security.SSLGetNegotiatedProtocolVersion( self.context, ctypes.byref(protocol) ) _assert_no_error(result) if protocol.value == SecurityConst.kTLSProtocol13: raise ssl.SSLError("SecureTransport does not support TLS 1.3") elif protocol.value == SecurityConst.kTLSProtocol12: return "TLSv1.2" elif protocol.value == SecurityConst.kTLSProtocol11: return "TLSv1.1" elif protocol.value == SecurityConst.kTLSProtocol1: return "TLSv1" elif protocol.value == SecurityConst.kSSLProtocol3: return "SSLv3" elif protocol.value == SecurityConst.kSSLProtocol2: return "SSLv2" else: raise ssl.SSLError(f"Unknown TLS version: {protocol!r}") def makefile( self: socket_cls, mode: ( Literal["r"] | Literal["w"] | Literal["rw"] | Literal["wr"] | Literal[""] ) = "r", buffering: int | None = None, *args: typing.Any, **kwargs: typing.Any, ) -> typing.BinaryIO | typing.TextIO: # We disable buffering with SecureTransport because it conflicts with # the buffering that ST does internally (see issue #1153 for more). buffering = 0 return socket_cls.makefile(self, mode, buffering, *args, **kwargs) WrappedSocket.makefile = makefile # type: ignore[attr-defined] class SecureTransportContext: """ I am a wrapper class for the SecureTransport library, to translate the interface of the standard library ``SSLContext`` object to calls into SecureTransport. """ def __init__(self, protocol: int) -> None: self._minimum_version: int = ssl.TLSVersion.MINIMUM_SUPPORTED self._maximum_version: int = ssl.TLSVersion.MAXIMUM_SUPPORTED if protocol not in (None, ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS_CLIENT): self._min_version, self._max_version = _protocol_to_min_max[protocol] self._options = 0 self._verify = False self._trust_bundle: bytes | None = None self._client_cert: str | None = None self._client_key: str | None = None self._client_key_passphrase = None self._alpn_protocols: list[bytes] | None = None @property def check_hostname(self) -> Literal[True]: """ SecureTransport cannot have its hostname checking disabled. For more, see the comment on getpeercert() in this file. """ return True @check_hostname.setter def check_hostname(self, value: typing.Any) -> None: """ SecureTransport cannot have its hostname checking disabled. For more, see the comment on getpeercert() in this file. """ @property def options(self) -> int: # TODO: Well, crap. # # So this is the bit of the code that is the most likely to cause us # trouble. Essentially we need to enumerate all of the SSL options that # users might want to use and try to see if we can sensibly translate # them, or whether we should just ignore them. return self._options @options.setter def options(self, value: int) -> None: # TODO: Update in line with above. self._options = value @property def verify_mode(self) -> int: return ssl.CERT_REQUIRED if self._verify else ssl.CERT_NONE @verify_mode.setter def verify_mode(self, value: int) -> None: self._verify = value == ssl.CERT_REQUIRED def set_default_verify_paths(self) -> None: # So, this has to do something a bit weird. Specifically, what it does # is nothing. # # This means that, if we had previously had load_verify_locations # called, this does not undo that. We need to do that because it turns # out that the rest of the urllib3 code will attempt to load the # default verify paths if it hasn't been told about any paths, even if # the context itself was sometime earlier. We resolve that by just # ignoring it. pass def load_default_certs(self) -> None: return self.set_default_verify_paths() def set_ciphers(self, ciphers: typing.Any) -> None: raise ValueError("SecureTransport doesn't support custom cipher strings") def load_verify_locations( self, cafile: str | None = None, capath: str | None = None, cadata: bytes | None = None, ) -> None: # OK, we only really support cadata and cafile. if capath is not None: raise ValueError("SecureTransport does not support cert directories") # Raise if cafile does not exist. if cafile is not None: with open(cafile): pass self._trust_bundle = cafile or cadata # type: ignore[assignment] def load_cert_chain( self, certfile: str, keyfile: str | None = None, password: str | None = None, ) -> None: self._client_cert = certfile self._client_key = keyfile self._client_cert_passphrase = password def set_alpn_protocols(self, protocols: list[str | bytes]) -> None: """ Sets the ALPN protocols that will later be set on the context. Raises a NotImplementedError if ALPN is not supported. """ if not hasattr(Security, "SSLSetALPNProtocols"): raise NotImplementedError( "SecureTransport supports ALPN only in macOS 10.12+" ) self._alpn_protocols = [util.util.to_bytes(p, "ascii") for p in protocols] def wrap_socket( self, sock: socket_cls, server_side: bool = False, do_handshake_on_connect: bool = True, suppress_ragged_eofs: bool = True, server_hostname: bytes | str | None = None, ) -> WrappedSocket: # So, what do we do here? Firstly, we assert some properties. This is a # stripped down shim, so there is some functionality we don't support. # See PEP 543 for the real deal. assert not server_side assert do_handshake_on_connect assert suppress_ragged_eofs # Ok, we're good to go. Now we want to create the wrapped socket object # and store it in the appropriate place. wrapped_socket = WrappedSocket(sock) # Now we can handshake wrapped_socket.handshake( server_hostname, self._verify, self._trust_bundle, _tls_version_to_st[self._minimum_version], _tls_version_to_st[self._maximum_version], self._client_cert, self._client_key, self._client_key_passphrase, self._alpn_protocols, ) return wrapped_socket @property def minimum_version(self) -> int: return self._minimum_version @minimum_version.setter def minimum_version(self, minimum_version: int) -> None: self._minimum_version = minimum_version @property def maximum_version(self) -> int: return self._maximum_version @maximum_version.setter def maximum_version(self, maximum_version: int) -> None: self._maximum_version = maximum_version urllib3-2.0.7/src/urllib3/contrib/socks.py0000644000000000000000000001704314513541732015333 0ustar00""" This module contains provisional support for SOCKS proxies from within urllib3. This module supports SOCKS4, SOCKS4A (an extension of SOCKS4), and SOCKS5. To enable its functionality, either install PySocks or install this module with the ``socks`` extra. The SOCKS implementation supports the full range of urllib3 features. It also supports the following SOCKS features: - SOCKS4A (``proxy_url='socks4a://...``) - SOCKS4 (``proxy_url='socks4://...``) - SOCKS5 with remote DNS (``proxy_url='socks5h://...``) - SOCKS5 with local DNS (``proxy_url='socks5://...``) - Usernames and passwords for the SOCKS proxy .. note:: It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in your ``proxy_url`` to ensure that DNS resolution is done from the remote server instead of client-side when connecting to a domain name. SOCKS4 supports IPv4 and domain names with the SOCKS4A extension. SOCKS5 supports IPv4, IPv6, and domain names. When connecting to a SOCKS4 proxy the ``username`` portion of the ``proxy_url`` will be sent as the ``userid`` section of the SOCKS request: .. code-block:: python proxy_url="socks4a://@proxy-host" When connecting to a SOCKS5 proxy the ``username`` and ``password`` portion of the ``proxy_url`` will be sent as the username/password to authenticate with the proxy: .. code-block:: python proxy_url="socks5h://:@proxy-host" """ from __future__ import annotations try: import socks # type: ignore[import] except ImportError: import warnings from ..exceptions import DependencyWarning warnings.warn( ( "SOCKS support in urllib3 requires the installation of optional " "dependencies: specifically, PySocks. For more information, see " "https://urllib3.readthedocs.io/en/latest/contrib.html#socks-proxies" ), DependencyWarning, ) raise import typing from socket import timeout as SocketTimeout from ..connection import HTTPConnection, HTTPSConnection from ..connectionpool import HTTPConnectionPool, HTTPSConnectionPool from ..exceptions import ConnectTimeoutError, NewConnectionError from ..poolmanager import PoolManager from ..util.url import parse_url try: import ssl except ImportError: ssl = None # type: ignore[assignment] try: from typing import TypedDict class _TYPE_SOCKS_OPTIONS(TypedDict): socks_version: int proxy_host: str | None proxy_port: str | None username: str | None password: str | None rdns: bool except ImportError: # Python 3.7 _TYPE_SOCKS_OPTIONS = typing.Dict[str, typing.Any] # type: ignore[misc, assignment] class SOCKSConnection(HTTPConnection): """ A plain-text HTTP connection that connects via a SOCKS proxy. """ def __init__( self, _socks_options: _TYPE_SOCKS_OPTIONS, *args: typing.Any, **kwargs: typing.Any, ) -> None: self._socks_options = _socks_options super().__init__(*args, **kwargs) def _new_conn(self) -> socks.socksocket: """ Establish a new connection via the SOCKS proxy. """ extra_kw: dict[str, typing.Any] = {} if self.source_address: extra_kw["source_address"] = self.source_address if self.socket_options: extra_kw["socket_options"] = self.socket_options try: conn = socks.create_connection( (self.host, self.port), proxy_type=self._socks_options["socks_version"], proxy_addr=self._socks_options["proxy_host"], proxy_port=self._socks_options["proxy_port"], proxy_username=self._socks_options["username"], proxy_password=self._socks_options["password"], proxy_rdns=self._socks_options["rdns"], timeout=self.timeout, **extra_kw, ) except SocketTimeout as e: raise ConnectTimeoutError( self, f"Connection to {self.host} timed out. (connect timeout={self.timeout})", ) from e except socks.ProxyError as e: # This is fragile as hell, but it seems to be the only way to raise # useful errors here. if e.socket_err: error = e.socket_err if isinstance(error, SocketTimeout): raise ConnectTimeoutError( self, f"Connection to {self.host} timed out. (connect timeout={self.timeout})", ) from e else: # Adding `from e` messes with coverage somehow, so it's omitted. # See #2386. raise NewConnectionError( self, f"Failed to establish a new connection: {error}" ) else: raise NewConnectionError( self, f"Failed to establish a new connection: {e}" ) from e except OSError as e: # Defensive: PySocks should catch all these. raise NewConnectionError( self, f"Failed to establish a new connection: {e}" ) from e return conn # We don't need to duplicate the Verified/Unverified distinction from # urllib3/connection.py here because the HTTPSConnection will already have been # correctly set to either the Verified or Unverified form by that module. This # means the SOCKSHTTPSConnection will automatically be the correct type. class SOCKSHTTPSConnection(SOCKSConnection, HTTPSConnection): pass class SOCKSHTTPConnectionPool(HTTPConnectionPool): ConnectionCls = SOCKSConnection class SOCKSHTTPSConnectionPool(HTTPSConnectionPool): ConnectionCls = SOCKSHTTPSConnection class SOCKSProxyManager(PoolManager): """ A version of the urllib3 ProxyManager that routes connections via the defined SOCKS proxy. """ pool_classes_by_scheme = { "http": SOCKSHTTPConnectionPool, "https": SOCKSHTTPSConnectionPool, } def __init__( self, proxy_url: str, username: str | None = None, password: str | None = None, num_pools: int = 10, headers: typing.Mapping[str, str] | None = None, **connection_pool_kw: typing.Any, ): parsed = parse_url(proxy_url) if username is None and password is None and parsed.auth is not None: split = parsed.auth.split(":") if len(split) == 2: username, password = split if parsed.scheme == "socks5": socks_version = socks.PROXY_TYPE_SOCKS5 rdns = False elif parsed.scheme == "socks5h": socks_version = socks.PROXY_TYPE_SOCKS5 rdns = True elif parsed.scheme == "socks4": socks_version = socks.PROXY_TYPE_SOCKS4 rdns = False elif parsed.scheme == "socks4a": socks_version = socks.PROXY_TYPE_SOCKS4 rdns = True else: raise ValueError(f"Unable to determine SOCKS version from {proxy_url}") self.proxy_url = proxy_url socks_options = { "socks_version": socks_version, "proxy_host": parsed.host, "proxy_port": parsed.port, "username": username, "password": password, "rdns": rdns, } connection_pool_kw["_socks_options"] = socks_options super().__init__(num_pools, headers, **connection_pool_kw) self.pool_classes_by_scheme = SOCKSProxyManager.pool_classes_by_scheme urllib3-2.0.7/src/urllib3/contrib/_securetransport/__init__.py0000644000000000000000000000000014513541732021333 0ustar00urllib3-2.0.7/src/urllib3/contrib/_securetransport/bindings.py0000644000000000000000000003416414513541732021413 0ustar00# type: ignore """ This module uses ctypes to bind a whole bunch of functions and constants from SecureTransport. The goal here is to provide the low-level API to SecureTransport. These are essentially the C-level functions and constants, and they're pretty gross to work with. This code is a bastardised version of the code found in Will Bond's oscrypto library. An enormous debt is owed to him for blazing this trail for us. For that reason, this code should be considered to be covered both by urllib3's license and by oscrypto's: Copyright (c) 2015-2016 Will Bond Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ from __future__ import annotations import platform from ctypes import ( CDLL, CFUNCTYPE, POINTER, c_bool, c_byte, c_char_p, c_int32, c_long, c_size_t, c_uint32, c_ulong, c_void_p, ) from ctypes.util import find_library if platform.system() != "Darwin": raise ImportError("Only macOS is supported") version = platform.mac_ver()[0] version_info = tuple(map(int, version.split("."))) if version_info < (10, 8): raise OSError( f"Only OS X 10.8 and newer are supported, not {version_info[0]}.{version_info[1]}" ) def load_cdll(name: str, macos10_16_path: str) -> CDLL: """Loads a CDLL by name, falling back to known path on 10.16+""" try: # Big Sur is technically 11 but we use 10.16 due to the Big Sur # beta being labeled as 10.16. path: str | None if version_info >= (10, 16): path = macos10_16_path else: path = find_library(name) if not path: raise OSError # Caught and reraised as 'ImportError' return CDLL(path, use_errno=True) except OSError: raise ImportError(f"The library {name} failed to load") from None Security = load_cdll( "Security", "/System/Library/Frameworks/Security.framework/Security" ) CoreFoundation = load_cdll( "CoreFoundation", "/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation", ) Boolean = c_bool CFIndex = c_long CFStringEncoding = c_uint32 CFData = c_void_p CFString = c_void_p CFArray = c_void_p CFMutableArray = c_void_p CFDictionary = c_void_p CFError = c_void_p CFType = c_void_p CFTypeID = c_ulong CFTypeRef = POINTER(CFType) CFAllocatorRef = c_void_p OSStatus = c_int32 CFDataRef = POINTER(CFData) CFStringRef = POINTER(CFString) CFArrayRef = POINTER(CFArray) CFMutableArrayRef = POINTER(CFMutableArray) CFDictionaryRef = POINTER(CFDictionary) CFArrayCallBacks = c_void_p CFDictionaryKeyCallBacks = c_void_p CFDictionaryValueCallBacks = c_void_p SecCertificateRef = POINTER(c_void_p) SecExternalFormat = c_uint32 SecExternalItemType = c_uint32 SecIdentityRef = POINTER(c_void_p) SecItemImportExportFlags = c_uint32 SecItemImportExportKeyParameters = c_void_p SecKeychainRef = POINTER(c_void_p) SSLProtocol = c_uint32 SSLCipherSuite = c_uint32 SSLContextRef = POINTER(c_void_p) SecTrustRef = POINTER(c_void_p) SSLConnectionRef = c_uint32 SecTrustResultType = c_uint32 SecTrustOptionFlags = c_uint32 SSLProtocolSide = c_uint32 SSLConnectionType = c_uint32 SSLSessionOption = c_uint32 try: Security.SecItemImport.argtypes = [ CFDataRef, CFStringRef, POINTER(SecExternalFormat), POINTER(SecExternalItemType), SecItemImportExportFlags, POINTER(SecItemImportExportKeyParameters), SecKeychainRef, POINTER(CFArrayRef), ] Security.SecItemImport.restype = OSStatus Security.SecCertificateGetTypeID.argtypes = [] Security.SecCertificateGetTypeID.restype = CFTypeID Security.SecIdentityGetTypeID.argtypes = [] Security.SecIdentityGetTypeID.restype = CFTypeID Security.SecKeyGetTypeID.argtypes = [] Security.SecKeyGetTypeID.restype = CFTypeID Security.SecCertificateCreateWithData.argtypes = [CFAllocatorRef, CFDataRef] Security.SecCertificateCreateWithData.restype = SecCertificateRef Security.SecCertificateCopyData.argtypes = [SecCertificateRef] Security.SecCertificateCopyData.restype = CFDataRef Security.SecCopyErrorMessageString.argtypes = [OSStatus, c_void_p] Security.SecCopyErrorMessageString.restype = CFStringRef Security.SecIdentityCreateWithCertificate.argtypes = [ CFTypeRef, SecCertificateRef, POINTER(SecIdentityRef), ] Security.SecIdentityCreateWithCertificate.restype = OSStatus Security.SecKeychainCreate.argtypes = [ c_char_p, c_uint32, c_void_p, Boolean, c_void_p, POINTER(SecKeychainRef), ] Security.SecKeychainCreate.restype = OSStatus Security.SecKeychainDelete.argtypes = [SecKeychainRef] Security.SecKeychainDelete.restype = OSStatus Security.SecPKCS12Import.argtypes = [ CFDataRef, CFDictionaryRef, POINTER(CFArrayRef), ] Security.SecPKCS12Import.restype = OSStatus SSLReadFunc = CFUNCTYPE(OSStatus, SSLConnectionRef, c_void_p, POINTER(c_size_t)) SSLWriteFunc = CFUNCTYPE( OSStatus, SSLConnectionRef, POINTER(c_byte), POINTER(c_size_t) ) Security.SSLSetIOFuncs.argtypes = [SSLContextRef, SSLReadFunc, SSLWriteFunc] Security.SSLSetIOFuncs.restype = OSStatus Security.SSLSetPeerID.argtypes = [SSLContextRef, c_char_p, c_size_t] Security.SSLSetPeerID.restype = OSStatus Security.SSLSetCertificate.argtypes = [SSLContextRef, CFArrayRef] Security.SSLSetCertificate.restype = OSStatus Security.SSLSetCertificateAuthorities.argtypes = [SSLContextRef, CFTypeRef, Boolean] Security.SSLSetCertificateAuthorities.restype = OSStatus Security.SSLSetConnection.argtypes = [SSLContextRef, SSLConnectionRef] Security.SSLSetConnection.restype = OSStatus Security.SSLSetPeerDomainName.argtypes = [SSLContextRef, c_char_p, c_size_t] Security.SSLSetPeerDomainName.restype = OSStatus Security.SSLHandshake.argtypes = [SSLContextRef] Security.SSLHandshake.restype = OSStatus Security.SSLRead.argtypes = [SSLContextRef, c_char_p, c_size_t, POINTER(c_size_t)] Security.SSLRead.restype = OSStatus Security.SSLWrite.argtypes = [SSLContextRef, c_char_p, c_size_t, POINTER(c_size_t)] Security.SSLWrite.restype = OSStatus Security.SSLClose.argtypes = [SSLContextRef] Security.SSLClose.restype = OSStatus Security.SSLGetNumberSupportedCiphers.argtypes = [SSLContextRef, POINTER(c_size_t)] Security.SSLGetNumberSupportedCiphers.restype = OSStatus Security.SSLGetSupportedCiphers.argtypes = [ SSLContextRef, POINTER(SSLCipherSuite), POINTER(c_size_t), ] Security.SSLGetSupportedCiphers.restype = OSStatus Security.SSLSetEnabledCiphers.argtypes = [ SSLContextRef, POINTER(SSLCipherSuite), c_size_t, ] Security.SSLSetEnabledCiphers.restype = OSStatus Security.SSLGetNumberEnabledCiphers.argtype = [SSLContextRef, POINTER(c_size_t)] Security.SSLGetNumberEnabledCiphers.restype = OSStatus Security.SSLGetEnabledCiphers.argtypes = [ SSLContextRef, POINTER(SSLCipherSuite), POINTER(c_size_t), ] Security.SSLGetEnabledCiphers.restype = OSStatus Security.SSLGetNegotiatedCipher.argtypes = [SSLContextRef, POINTER(SSLCipherSuite)] Security.SSLGetNegotiatedCipher.restype = OSStatus Security.SSLGetNegotiatedProtocolVersion.argtypes = [ SSLContextRef, POINTER(SSLProtocol), ] Security.SSLGetNegotiatedProtocolVersion.restype = OSStatus Security.SSLCopyPeerTrust.argtypes = [SSLContextRef, POINTER(SecTrustRef)] Security.SSLCopyPeerTrust.restype = OSStatus Security.SecTrustSetAnchorCertificates.argtypes = [SecTrustRef, CFArrayRef] Security.SecTrustSetAnchorCertificates.restype = OSStatus Security.SecTrustSetAnchorCertificatesOnly.argstypes = [SecTrustRef, Boolean] Security.SecTrustSetAnchorCertificatesOnly.restype = OSStatus Security.SecTrustEvaluate.argtypes = [SecTrustRef, POINTER(SecTrustResultType)] Security.SecTrustEvaluate.restype = OSStatus Security.SecTrustGetCertificateCount.argtypes = [SecTrustRef] Security.SecTrustGetCertificateCount.restype = CFIndex Security.SecTrustGetCertificateAtIndex.argtypes = [SecTrustRef, CFIndex] Security.SecTrustGetCertificateAtIndex.restype = SecCertificateRef Security.SSLCreateContext.argtypes = [ CFAllocatorRef, SSLProtocolSide, SSLConnectionType, ] Security.SSLCreateContext.restype = SSLContextRef Security.SSLSetSessionOption.argtypes = [SSLContextRef, SSLSessionOption, Boolean] Security.SSLSetSessionOption.restype = OSStatus Security.SSLSetProtocolVersionMin.argtypes = [SSLContextRef, SSLProtocol] Security.SSLSetProtocolVersionMin.restype = OSStatus Security.SSLSetProtocolVersionMax.argtypes = [SSLContextRef, SSLProtocol] Security.SSLSetProtocolVersionMax.restype = OSStatus try: Security.SSLSetALPNProtocols.argtypes = [SSLContextRef, CFArrayRef] Security.SSLSetALPNProtocols.restype = OSStatus except AttributeError: # Supported only in 10.12+ pass Security.SecCopyErrorMessageString.argtypes = [OSStatus, c_void_p] Security.SecCopyErrorMessageString.restype = CFStringRef Security.SSLReadFunc = SSLReadFunc Security.SSLWriteFunc = SSLWriteFunc Security.SSLContextRef = SSLContextRef Security.SSLProtocol = SSLProtocol Security.SSLCipherSuite = SSLCipherSuite Security.SecIdentityRef = SecIdentityRef Security.SecKeychainRef = SecKeychainRef Security.SecTrustRef = SecTrustRef Security.SecTrustResultType = SecTrustResultType Security.SecExternalFormat = SecExternalFormat Security.OSStatus = OSStatus Security.kSecImportExportPassphrase = CFStringRef.in_dll( Security, "kSecImportExportPassphrase" ) Security.kSecImportItemIdentity = CFStringRef.in_dll( Security, "kSecImportItemIdentity" ) # CoreFoundation time! CoreFoundation.CFRetain.argtypes = [CFTypeRef] CoreFoundation.CFRetain.restype = CFTypeRef CoreFoundation.CFRelease.argtypes = [CFTypeRef] CoreFoundation.CFRelease.restype = None CoreFoundation.CFGetTypeID.argtypes = [CFTypeRef] CoreFoundation.CFGetTypeID.restype = CFTypeID CoreFoundation.CFStringCreateWithCString.argtypes = [ CFAllocatorRef, c_char_p, CFStringEncoding, ] CoreFoundation.CFStringCreateWithCString.restype = CFStringRef CoreFoundation.CFStringGetCStringPtr.argtypes = [CFStringRef, CFStringEncoding] CoreFoundation.CFStringGetCStringPtr.restype = c_char_p CoreFoundation.CFStringGetCString.argtypes = [ CFStringRef, c_char_p, CFIndex, CFStringEncoding, ] CoreFoundation.CFStringGetCString.restype = c_bool CoreFoundation.CFDataCreate.argtypes = [CFAllocatorRef, c_char_p, CFIndex] CoreFoundation.CFDataCreate.restype = CFDataRef CoreFoundation.CFDataGetLength.argtypes = [CFDataRef] CoreFoundation.CFDataGetLength.restype = CFIndex CoreFoundation.CFDataGetBytePtr.argtypes = [CFDataRef] CoreFoundation.CFDataGetBytePtr.restype = c_void_p CoreFoundation.CFDictionaryCreate.argtypes = [ CFAllocatorRef, POINTER(CFTypeRef), POINTER(CFTypeRef), CFIndex, CFDictionaryKeyCallBacks, CFDictionaryValueCallBacks, ] CoreFoundation.CFDictionaryCreate.restype = CFDictionaryRef CoreFoundation.CFDictionaryGetValue.argtypes = [CFDictionaryRef, CFTypeRef] CoreFoundation.CFDictionaryGetValue.restype = CFTypeRef CoreFoundation.CFArrayCreate.argtypes = [ CFAllocatorRef, POINTER(CFTypeRef), CFIndex, CFArrayCallBacks, ] CoreFoundation.CFArrayCreate.restype = CFArrayRef CoreFoundation.CFArrayCreateMutable.argtypes = [ CFAllocatorRef, CFIndex, CFArrayCallBacks, ] CoreFoundation.CFArrayCreateMutable.restype = CFMutableArrayRef CoreFoundation.CFArrayAppendValue.argtypes = [CFMutableArrayRef, c_void_p] CoreFoundation.CFArrayAppendValue.restype = None CoreFoundation.CFArrayGetCount.argtypes = [CFArrayRef] CoreFoundation.CFArrayGetCount.restype = CFIndex CoreFoundation.CFArrayGetValueAtIndex.argtypes = [CFArrayRef, CFIndex] CoreFoundation.CFArrayGetValueAtIndex.restype = c_void_p CoreFoundation.kCFAllocatorDefault = CFAllocatorRef.in_dll( CoreFoundation, "kCFAllocatorDefault" ) CoreFoundation.kCFTypeArrayCallBacks = c_void_p.in_dll( CoreFoundation, "kCFTypeArrayCallBacks" ) CoreFoundation.kCFTypeDictionaryKeyCallBacks = c_void_p.in_dll( CoreFoundation, "kCFTypeDictionaryKeyCallBacks" ) CoreFoundation.kCFTypeDictionaryValueCallBacks = c_void_p.in_dll( CoreFoundation, "kCFTypeDictionaryValueCallBacks" ) CoreFoundation.CFTypeRef = CFTypeRef CoreFoundation.CFArrayRef = CFArrayRef CoreFoundation.CFStringRef = CFStringRef CoreFoundation.CFDictionaryRef = CFDictionaryRef except AttributeError: raise ImportError("Error initializing ctypes") from None class CFConst: """ A class object that acts as essentially a namespace for CoreFoundation constants. """ kCFStringEncodingUTF8 = CFStringEncoding(0x08000100) urllib3-2.0.7/src/urllib3/contrib/_securetransport/low_level.py0000644000000000000000000003753414513541732021612 0ustar00""" Low-level helpers for the SecureTransport bindings. These are Python functions that are not directly related to the high-level APIs but are necessary to get them to work. They include a whole bunch of low-level CoreFoundation messing about and memory management. The concerns in this module are almost entirely about trying to avoid memory leaks and providing appropriate and useful assistance to the higher-level code. """ from __future__ import annotations import base64 import ctypes import itertools import os import re import ssl import struct import tempfile import typing from .bindings import ( # type: ignore[attr-defined] CFArray, CFConst, CFData, CFDictionary, CFMutableArray, CFString, CFTypeRef, CoreFoundation, SecKeychainRef, Security, ) # This regular expression is used to grab PEM data out of a PEM bundle. _PEM_CERTS_RE = re.compile( b"-----BEGIN CERTIFICATE-----\n(.*?)\n-----END CERTIFICATE-----", re.DOTALL ) def _cf_data_from_bytes(bytestring: bytes) -> CFData: """ Given a bytestring, create a CFData object from it. This CFData object must be CFReleased by the caller. """ return CoreFoundation.CFDataCreate( CoreFoundation.kCFAllocatorDefault, bytestring, len(bytestring) ) def _cf_dictionary_from_tuples( tuples: list[tuple[typing.Any, typing.Any]] ) -> CFDictionary: """ Given a list of Python tuples, create an associated CFDictionary. """ dictionary_size = len(tuples) # We need to get the dictionary keys and values out in the same order. keys = (t[0] for t in tuples) values = (t[1] for t in tuples) cf_keys = (CoreFoundation.CFTypeRef * dictionary_size)(*keys) cf_values = (CoreFoundation.CFTypeRef * dictionary_size)(*values) return CoreFoundation.CFDictionaryCreate( CoreFoundation.kCFAllocatorDefault, cf_keys, cf_values, dictionary_size, CoreFoundation.kCFTypeDictionaryKeyCallBacks, CoreFoundation.kCFTypeDictionaryValueCallBacks, ) def _cfstr(py_bstr: bytes) -> CFString: """ Given a Python binary data, create a CFString. The string must be CFReleased by the caller. """ c_str = ctypes.c_char_p(py_bstr) cf_str = CoreFoundation.CFStringCreateWithCString( CoreFoundation.kCFAllocatorDefault, c_str, CFConst.kCFStringEncodingUTF8, ) return cf_str def _create_cfstring_array(lst: list[bytes]) -> CFMutableArray: """ Given a list of Python binary data, create an associated CFMutableArray. The array must be CFReleased by the caller. Raises an ssl.SSLError on failure. """ cf_arr = None try: cf_arr = CoreFoundation.CFArrayCreateMutable( CoreFoundation.kCFAllocatorDefault, 0, ctypes.byref(CoreFoundation.kCFTypeArrayCallBacks), ) if not cf_arr: raise MemoryError("Unable to allocate memory!") for item in lst: cf_str = _cfstr(item) if not cf_str: raise MemoryError("Unable to allocate memory!") try: CoreFoundation.CFArrayAppendValue(cf_arr, cf_str) finally: CoreFoundation.CFRelease(cf_str) except BaseException as e: if cf_arr: CoreFoundation.CFRelease(cf_arr) raise ssl.SSLError(f"Unable to allocate array: {e}") from None return cf_arr def _cf_string_to_unicode(value: CFString) -> str | None: """ Creates a Unicode string from a CFString object. Used entirely for error reporting. Yes, it annoys me quite a lot that this function is this complex. """ value_as_void_p = ctypes.cast(value, ctypes.POINTER(ctypes.c_void_p)) string = CoreFoundation.CFStringGetCStringPtr( value_as_void_p, CFConst.kCFStringEncodingUTF8 ) if string is None: buffer = ctypes.create_string_buffer(1024) result = CoreFoundation.CFStringGetCString( value_as_void_p, buffer, 1024, CFConst.kCFStringEncodingUTF8 ) if not result: raise OSError("Error copying C string from CFStringRef") string = buffer.value if string is not None: string = string.decode("utf-8") return string # type: ignore[no-any-return] def _assert_no_error( error: int, exception_class: type[BaseException] | None = None ) -> None: """ Checks the return code and throws an exception if there is an error to report """ if error == 0: return cf_error_string = Security.SecCopyErrorMessageString(error, None) output = _cf_string_to_unicode(cf_error_string) CoreFoundation.CFRelease(cf_error_string) if output is None or output == "": output = f"OSStatus {error}" if exception_class is None: exception_class = ssl.SSLError raise exception_class(output) def _cert_array_from_pem(pem_bundle: bytes) -> CFArray: """ Given a bundle of certs in PEM format, turns them into a CFArray of certs that can be used to validate a cert chain. """ # Normalize the PEM bundle's line endings. pem_bundle = pem_bundle.replace(b"\r\n", b"\n") der_certs = [ base64.b64decode(match.group(1)) for match in _PEM_CERTS_RE.finditer(pem_bundle) ] if not der_certs: raise ssl.SSLError("No root certificates specified") cert_array = CoreFoundation.CFArrayCreateMutable( CoreFoundation.kCFAllocatorDefault, 0, ctypes.byref(CoreFoundation.kCFTypeArrayCallBacks), ) if not cert_array: raise ssl.SSLError("Unable to allocate memory!") try: for der_bytes in der_certs: certdata = _cf_data_from_bytes(der_bytes) if not certdata: raise ssl.SSLError("Unable to allocate memory!") cert = Security.SecCertificateCreateWithData( CoreFoundation.kCFAllocatorDefault, certdata ) CoreFoundation.CFRelease(certdata) if not cert: raise ssl.SSLError("Unable to build cert object!") CoreFoundation.CFArrayAppendValue(cert_array, cert) CoreFoundation.CFRelease(cert) except Exception: # We need to free the array before the exception bubbles further. # We only want to do that if an error occurs: otherwise, the caller # should free. CoreFoundation.CFRelease(cert_array) raise return cert_array def _is_cert(item: CFTypeRef) -> bool: """ Returns True if a given CFTypeRef is a certificate. """ expected = Security.SecCertificateGetTypeID() return CoreFoundation.CFGetTypeID(item) == expected # type: ignore[no-any-return] def _is_identity(item: CFTypeRef) -> bool: """ Returns True if a given CFTypeRef is an identity. """ expected = Security.SecIdentityGetTypeID() return CoreFoundation.CFGetTypeID(item) == expected # type: ignore[no-any-return] def _temporary_keychain() -> tuple[SecKeychainRef, str]: """ This function creates a temporary Mac keychain that we can use to work with credentials. This keychain uses a one-time password and a temporary file to store the data. We expect to have one keychain per socket. The returned SecKeychainRef must be freed by the caller, including calling SecKeychainDelete. Returns a tuple of the SecKeychainRef and the path to the temporary directory that contains it. """ # Unfortunately, SecKeychainCreate requires a path to a keychain. This # means we cannot use mkstemp to use a generic temporary file. Instead, # we're going to create a temporary directory and a filename to use there. # This filename will be 8 random bytes expanded into base64. We also need # some random bytes to password-protect the keychain we're creating, so we # ask for 40 random bytes. random_bytes = os.urandom(40) filename = base64.b16encode(random_bytes[:8]).decode("utf-8") password = base64.b16encode(random_bytes[8:]) # Must be valid UTF-8 tempdirectory = tempfile.mkdtemp() keychain_path = os.path.join(tempdirectory, filename).encode("utf-8") # We now want to create the keychain itself. keychain = Security.SecKeychainRef() status = Security.SecKeychainCreate( keychain_path, len(password), password, False, None, ctypes.byref(keychain) ) _assert_no_error(status) # Having created the keychain, we want to pass it off to the caller. return keychain, tempdirectory def _load_items_from_file( keychain: SecKeychainRef, path: str ) -> tuple[list[CFTypeRef], list[CFTypeRef]]: """ Given a single file, loads all the trust objects from it into arrays and the keychain. Returns a tuple of lists: the first list is a list of identities, the second a list of certs. """ certificates = [] identities = [] result_array = None with open(path, "rb") as f: raw_filedata = f.read() try: filedata = CoreFoundation.CFDataCreate( CoreFoundation.kCFAllocatorDefault, raw_filedata, len(raw_filedata) ) result_array = CoreFoundation.CFArrayRef() result = Security.SecItemImport( filedata, # cert data None, # Filename, leaving it out for now None, # What the type of the file is, we don't care None, # what's in the file, we don't care 0, # import flags None, # key params, can include passphrase in the future keychain, # The keychain to insert into ctypes.byref(result_array), # Results ) _assert_no_error(result) # A CFArray is not very useful to us as an intermediary # representation, so we are going to extract the objects we want # and then free the array. We don't need to keep hold of keys: the # keychain already has them! result_count = CoreFoundation.CFArrayGetCount(result_array) for index in range(result_count): item = CoreFoundation.CFArrayGetValueAtIndex(result_array, index) item = ctypes.cast(item, CoreFoundation.CFTypeRef) if _is_cert(item): CoreFoundation.CFRetain(item) certificates.append(item) elif _is_identity(item): CoreFoundation.CFRetain(item) identities.append(item) finally: if result_array: CoreFoundation.CFRelease(result_array) CoreFoundation.CFRelease(filedata) return (identities, certificates) def _load_client_cert_chain(keychain: SecKeychainRef, *paths: str | None) -> CFArray: """ Load certificates and maybe keys from a number of files. Has the end goal of returning a CFArray containing one SecIdentityRef, and then zero or more SecCertificateRef objects, suitable for use as a client certificate trust chain. """ # Ok, the strategy. # # This relies on knowing that macOS will not give you a SecIdentityRef # unless you have imported a key into a keychain. This is a somewhat # artificial limitation of macOS (for example, it doesn't necessarily # affect iOS), but there is nothing inside Security.framework that lets you # get a SecIdentityRef without having a key in a keychain. # # So the policy here is we take all the files and iterate them in order. # Each one will use SecItemImport to have one or more objects loaded from # it. We will also point at a keychain that macOS can use to work with the # private key. # # Once we have all the objects, we'll check what we actually have. If we # already have a SecIdentityRef in hand, fab: we'll use that. Otherwise, # we'll take the first certificate (which we assume to be our leaf) and # ask the keychain to give us a SecIdentityRef with that cert's associated # key. # # We'll then return a CFArray containing the trust chain: one # SecIdentityRef and then zero-or-more SecCertificateRef objects. The # responsibility for freeing this CFArray will be with the caller. This # CFArray must remain alive for the entire connection, so in practice it # will be stored with a single SSLSocket, along with the reference to the # keychain. certificates = [] identities = [] # Filter out bad paths. filtered_paths = (path for path in paths if path) try: for file_path in filtered_paths: new_identities, new_certs = _load_items_from_file(keychain, file_path) identities.extend(new_identities) certificates.extend(new_certs) # Ok, we have everything. The question is: do we have an identity? If # not, we want to grab one from the first cert we have. if not identities: new_identity = Security.SecIdentityRef() status = Security.SecIdentityCreateWithCertificate( keychain, certificates[0], ctypes.byref(new_identity) ) _assert_no_error(status) identities.append(new_identity) # We now want to release the original certificate, as we no longer # need it. CoreFoundation.CFRelease(certificates.pop(0)) # We now need to build a new CFArray that holds the trust chain. trust_chain = CoreFoundation.CFArrayCreateMutable( CoreFoundation.kCFAllocatorDefault, 0, ctypes.byref(CoreFoundation.kCFTypeArrayCallBacks), ) for item in itertools.chain(identities, certificates): # ArrayAppendValue does a CFRetain on the item. That's fine, # because the finally block will release our other refs to them. CoreFoundation.CFArrayAppendValue(trust_chain, item) return trust_chain finally: for obj in itertools.chain(identities, certificates): CoreFoundation.CFRelease(obj) TLS_PROTOCOL_VERSIONS = { "SSLv2": (0, 2), "SSLv3": (3, 0), "TLSv1": (3, 1), "TLSv1.1": (3, 2), "TLSv1.2": (3, 3), } def _build_tls_unknown_ca_alert(version: str) -> bytes: """ Builds a TLS alert record for an unknown CA. """ ver_maj, ver_min = TLS_PROTOCOL_VERSIONS[version] severity_fatal = 0x02 description_unknown_ca = 0x30 msg = struct.pack(">BB", severity_fatal, description_unknown_ca) msg_len = len(msg) record_type_alert = 0x15 record = struct.pack(">BBBH", record_type_alert, ver_maj, ver_min, msg_len) + msg return record class SecurityConst: """ A class object that acts as essentially a namespace for Security constants. """ kSSLSessionOptionBreakOnServerAuth = 0 kSSLProtocol2 = 1 kSSLProtocol3 = 2 kTLSProtocol1 = 4 kTLSProtocol11 = 7 kTLSProtocol12 = 8 # SecureTransport does not support TLS 1.3 even if there's a constant for it kTLSProtocol13 = 10 kTLSProtocolMaxSupported = 999 kSSLClientSide = 1 kSSLStreamType = 0 kSecFormatPEMSequence = 10 kSecTrustResultInvalid = 0 kSecTrustResultProceed = 1 # This gap is present on purpose: this was kSecTrustResultConfirm, which # is deprecated. kSecTrustResultDeny = 3 kSecTrustResultUnspecified = 4 kSecTrustResultRecoverableTrustFailure = 5 kSecTrustResultFatalTrustFailure = 6 kSecTrustResultOtherError = 7 errSSLProtocol = -9800 errSSLWouldBlock = -9803 errSSLClosedGraceful = -9805 errSSLClosedNoNotify = -9816 errSSLClosedAbort = -9806 errSSLXCertChainInvalid = -9807 errSSLCrypto = -9809 errSSLInternal = -9810 errSSLCertExpired = -9814 errSSLCertNotYetValid = -9815 errSSLUnknownRootCert = -9812 errSSLNoRootCert = -9813 errSSLHostNameMismatch = -9843 errSSLPeerHandshakeFail = -9824 errSSLPeerUserCancelled = -9839 errSSLWeakPeerEphemeralDHKey = -9850 errSSLServerAuthCompleted = -9841 errSSLRecordOverflow = -9847 errSecVerifyFailed = -67808 errSecNoTrustSettings = -25263 errSecItemNotFound = -25300 errSecInvalidTrustSettings = -25262 urllib3-2.0.7/src/urllib3/util/__init__.py0000644000000000000000000000203314513541732015256 0ustar00# For backwards compatibility, provide imports that used to be here. from __future__ import annotations from .connection import is_connection_dropped from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers from .response import is_fp_closed from .retry import Retry from .ssl_ import ( ALPN_PROTOCOLS, IS_PYOPENSSL, IS_SECURETRANSPORT, SSLContext, assert_fingerprint, create_urllib3_context, resolve_cert_reqs, resolve_ssl_version, ssl_wrap_socket, ) from .timeout import Timeout from .url import Url, parse_url from .wait import wait_for_read, wait_for_write __all__ = ( "IS_PYOPENSSL", "IS_SECURETRANSPORT", "SSLContext", "ALPN_PROTOCOLS", "Retry", "Timeout", "Url", "assert_fingerprint", "create_urllib3_context", "is_connection_dropped", "is_fp_closed", "parse_url", "make_headers", "resolve_cert_reqs", "resolve_ssl_version", "ssl_wrap_socket", "wait_for_read", "wait_for_write", "SKIP_HEADER", "SKIPPABLE_HEADERS", ) urllib3-2.0.7/src/urllib3/util/connection.py0000644000000000000000000001055614513541732015667 0ustar00from __future__ import annotations import socket import typing from ..exceptions import LocationParseError from .timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT _TYPE_SOCKET_OPTIONS = typing.Sequence[typing.Tuple[int, int, typing.Union[int, bytes]]] if typing.TYPE_CHECKING: from .._base_connection import BaseHTTPConnection def is_connection_dropped(conn: BaseHTTPConnection) -> bool: # Platform-specific """ Returns True if the connection is dropped and should be closed. :param conn: :class:`urllib3.connection.HTTPConnection` object. """ return not conn.is_connected # This function is copied from socket.py in the Python 2.7 standard # library test suite. Added to its signature is only `socket_options`. # One additional modification is that we avoid binding to IPv6 servers # discovered in DNS if the system doesn't have IPv6 functionality. def create_connection( address: tuple[str, int], timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, source_address: tuple[str, int] | None = None, socket_options: _TYPE_SOCKET_OPTIONS | None = None, ) -> socket.socket: """Connect to *address* and return the socket object. Convenience function. Connect to *address* (a 2-tuple ``(host, port)``) and return the socket object. Passing the optional *timeout* parameter will set the timeout on the socket instance before attempting to connect. If no *timeout* is supplied, the global default timeout setting returned by :func:`socket.getdefaulttimeout` is used. If *source_address* is set it must be a tuple of (host, port) for the socket to bind as a source address before making the connection. An host of '' or port 0 tells the OS to use the default. """ host, port = address if host.startswith("["): host = host.strip("[]") err = None # Using the value from allowed_gai_family() in the context of getaddrinfo lets # us select whether to work with IPv4 DNS records, IPv6 records, or both. # The original create_connection function always returns all records. family = allowed_gai_family() try: host.encode("idna") except UnicodeError: raise LocationParseError(f"'{host}', label empty or too long") from None for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): af, socktype, proto, canonname, sa = res sock = None try: sock = socket.socket(af, socktype, proto) # If provided, set socket level options before connecting. _set_socket_options(sock, socket_options) if timeout is not _DEFAULT_TIMEOUT: sock.settimeout(timeout) if source_address: sock.bind(source_address) sock.connect(sa) # Break explicitly a reference cycle err = None return sock except OSError as _: err = _ if sock is not None: sock.close() if err is not None: try: raise err finally: # Break explicitly a reference cycle err = None else: raise OSError("getaddrinfo returns an empty list") def _set_socket_options( sock: socket.socket, options: _TYPE_SOCKET_OPTIONS | None ) -> None: if options is None: return for opt in options: sock.setsockopt(*opt) def allowed_gai_family() -> socket.AddressFamily: """This function is designed to work in the context of getaddrinfo, where family=socket.AF_UNSPEC is the default and will perform a DNS search for both IPv6 and IPv4 records.""" family = socket.AF_INET if HAS_IPV6: family = socket.AF_UNSPEC return family def _has_ipv6(host: str) -> bool: """Returns True if the system can bind an IPv6 address.""" sock = None has_ipv6 = False if socket.has_ipv6: # has_ipv6 returns true if cPython was compiled with IPv6 support. # It does not tell us if the system has IPv6 support enabled. To # determine that we must bind to an IPv6 address. # https://github.com/urllib3/urllib3/pull/611 # https://bugs.python.org/issue658327 try: sock = socket.socket(socket.AF_INET6) sock.bind((host, 0)) has_ipv6 = True except Exception: pass if sock: sock.close() return has_ipv6 HAS_IPV6 = _has_ipv6("::1") urllib3-2.0.7/src/urllib3/util/proxy.py0000644000000000000000000000217414513541732014706 0ustar00from __future__ import annotations import typing from .url import Url if typing.TYPE_CHECKING: from ..connection import ProxyConfig def connection_requires_http_tunnel( proxy_url: Url | None = None, proxy_config: ProxyConfig | None = None, destination_scheme: str | None = None, ) -> bool: """ Returns True if the connection requires an HTTP CONNECT through the proxy. :param URL proxy_url: URL of the proxy. :param ProxyConfig proxy_config: Proxy configuration from poolmanager.py :param str destination_scheme: The scheme of the destination. (i.e https, http, etc) """ # If we're not using a proxy, no way to use a tunnel. if proxy_url is None: return False # HTTP destinations never require tunneling, we always forward. if destination_scheme == "http": return False # Support for forwarding with HTTPS proxies and HTTPS destinations. if ( proxy_url.scheme == "https" and proxy_config and proxy_config.use_forwarding_for_https ): return False # Otherwise always use a tunnel. return True urllib3-2.0.7/src/urllib3/util/request.py0000644000000000000000000001762314513541732015222 0ustar00from __future__ import annotations import io import typing from base64 import b64encode from enum import Enum from ..exceptions import UnrewindableBodyError from .util import to_bytes if typing.TYPE_CHECKING: from typing_extensions import Final # Pass as a value within ``headers`` to skip # emitting some HTTP headers that are added automatically. # The only headers that are supported are ``Accept-Encoding``, # ``Host``, and ``User-Agent``. SKIP_HEADER = "@@@SKIP_HEADER@@@" SKIPPABLE_HEADERS = frozenset(["accept-encoding", "host", "user-agent"]) ACCEPT_ENCODING = "gzip,deflate" try: try: import brotlicffi as _unused_module_brotli # type: ignore[import] # noqa: F401 except ImportError: import brotli as _unused_module_brotli # type: ignore[import] # noqa: F401 except ImportError: pass else: ACCEPT_ENCODING += ",br" try: import zstandard as _unused_module_zstd # type: ignore[import] # noqa: F401 except ImportError: pass else: ACCEPT_ENCODING += ",zstd" class _TYPE_FAILEDTELL(Enum): token = 0 _FAILEDTELL: Final[_TYPE_FAILEDTELL] = _TYPE_FAILEDTELL.token _TYPE_BODY_POSITION = typing.Union[int, _TYPE_FAILEDTELL] # When sending a request with these methods we aren't expecting # a body so don't need to set an explicit 'Content-Length: 0' # The reason we do this in the negative instead of tracking methods # which 'should' have a body is because unknown methods should be # treated as if they were 'POST' which *does* expect a body. _METHODS_NOT_EXPECTING_BODY = {"GET", "HEAD", "DELETE", "TRACE", "OPTIONS", "CONNECT"} def make_headers( keep_alive: bool | None = None, accept_encoding: bool | list[str] | str | None = None, user_agent: str | None = None, basic_auth: str | None = None, proxy_basic_auth: str | None = None, disable_cache: bool | None = None, ) -> dict[str, str]: """ Shortcuts for generating request headers. :param keep_alive: If ``True``, adds 'connection: keep-alive' header. :param accept_encoding: Can be a boolean, list, or string. ``True`` translates to 'gzip,deflate'. If either the ``brotli`` or ``brotlicffi`` package is installed 'gzip,deflate,br' is used instead. List will get joined by comma. String will be used as provided. :param user_agent: String representing the user-agent you want, such as "python-urllib3/0.6" :param basic_auth: Colon-separated username:password string for 'authorization: basic ...' auth header. :param proxy_basic_auth: Colon-separated username:password string for 'proxy-authorization: basic ...' auth header. :param disable_cache: If ``True``, adds 'cache-control: no-cache' header. Example: .. code-block:: python import urllib3 print(urllib3.util.make_headers(keep_alive=True, user_agent="Batman/1.0")) # {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'} print(urllib3.util.make_headers(accept_encoding=True)) # {'accept-encoding': 'gzip,deflate'} """ headers: dict[str, str] = {} if accept_encoding: if isinstance(accept_encoding, str): pass elif isinstance(accept_encoding, list): accept_encoding = ",".join(accept_encoding) else: accept_encoding = ACCEPT_ENCODING headers["accept-encoding"] = accept_encoding if user_agent: headers["user-agent"] = user_agent if keep_alive: headers["connection"] = "keep-alive" if basic_auth: headers[ "authorization" ] = f"Basic {b64encode(basic_auth.encode('latin-1')).decode()}" if proxy_basic_auth: headers[ "proxy-authorization" ] = f"Basic {b64encode(proxy_basic_auth.encode('latin-1')).decode()}" if disable_cache: headers["cache-control"] = "no-cache" return headers def set_file_position( body: typing.Any, pos: _TYPE_BODY_POSITION | None ) -> _TYPE_BODY_POSITION | None: """ If a position is provided, move file to that point. Otherwise, we'll attempt to record a position for future use. """ if pos is not None: rewind_body(body, pos) elif getattr(body, "tell", None) is not None: try: pos = body.tell() except OSError: # This differentiates from None, allowing us to catch # a failed `tell()` later when trying to rewind the body. pos = _FAILEDTELL return pos def rewind_body(body: typing.IO[typing.AnyStr], body_pos: _TYPE_BODY_POSITION) -> None: """ Attempt to rewind body to a certain position. Primarily used for request redirects and retries. :param body: File-like object that supports seek. :param int pos: Position to seek to in file. """ body_seek = getattr(body, "seek", None) if body_seek is not None and isinstance(body_pos, int): try: body_seek(body_pos) except OSError as e: raise UnrewindableBodyError( "An error occurred when rewinding request body for redirect/retry." ) from e elif body_pos is _FAILEDTELL: raise UnrewindableBodyError( "Unable to record file position for rewinding " "request body during a redirect/retry." ) else: raise ValueError( f"body_pos must be of type integer, instead it was {type(body_pos)}." ) class ChunksAndContentLength(typing.NamedTuple): chunks: typing.Iterable[bytes] | None content_length: int | None def body_to_chunks( body: typing.Any | None, method: str, blocksize: int ) -> ChunksAndContentLength: """Takes the HTTP request method, body, and blocksize and transforms them into an iterable of chunks to pass to socket.sendall() and an optional 'Content-Length' header. A 'Content-Length' of 'None' indicates the length of the body can't be determined so should use 'Transfer-Encoding: chunked' for framing instead. """ chunks: typing.Iterable[bytes] | None content_length: int | None # No body, we need to make a recommendation on 'Content-Length' # based on whether that request method is expected to have # a body or not. if body is None: chunks = None if method.upper() not in _METHODS_NOT_EXPECTING_BODY: content_length = 0 else: content_length = None # Bytes or strings become bytes elif isinstance(body, (str, bytes)): chunks = (to_bytes(body),) content_length = len(chunks[0]) # File-like object, TODO: use seek() and tell() for length? elif hasattr(body, "read"): def chunk_readable() -> typing.Iterable[bytes]: nonlocal body, blocksize encode = isinstance(body, io.TextIOBase) while True: datablock = body.read(blocksize) if not datablock: break if encode: datablock = datablock.encode("iso-8859-1") yield datablock chunks = chunk_readable() content_length = None # Otherwise we need to start checking via duck-typing. else: try: # Check if the body implements the buffer API. mv = memoryview(body) except TypeError: try: # Check if the body is an iterable chunks = iter(body) content_length = None except TypeError: raise TypeError( f"'body' must be a bytes-like object, file-like " f"object, or iterable. Instead was {body!r}" ) from None else: # Since it implements the buffer API can be passed directly to socket.sendall() chunks = (body,) content_length = mv.nbytes return ChunksAndContentLength(chunks=chunks, content_length=content_length) urllib3-2.0.7/src/urllib3/util/response.py0000644000000000000000000000645614513541732015372 0ustar00from __future__ import annotations import http.client as httplib from email.errors import MultipartInvariantViolationDefect, StartBoundaryNotFoundDefect from ..exceptions import HeaderParsingError def is_fp_closed(obj: object) -> bool: """ Checks whether a given file-like object is closed. :param obj: The file-like object to check. """ try: # Check `isclosed()` first, in case Python3 doesn't set `closed`. # GH Issue #928 return obj.isclosed() # type: ignore[no-any-return, attr-defined] except AttributeError: pass try: # Check via the official file-like-object way. return obj.closed # type: ignore[no-any-return, attr-defined] except AttributeError: pass try: # Check if the object is a container for another file-like object that # gets released on exhaustion (e.g. HTTPResponse). return obj.fp is None # type: ignore[attr-defined] except AttributeError: pass raise ValueError("Unable to determine whether fp is closed.") def assert_header_parsing(headers: httplib.HTTPMessage) -> None: """ Asserts whether all headers have been successfully parsed. Extracts encountered errors from the result of parsing headers. Only works on Python 3. :param http.client.HTTPMessage headers: Headers to verify. :raises urllib3.exceptions.HeaderParsingError: If parsing errors are found. """ # This will fail silently if we pass in the wrong kind of parameter. # To make debugging easier add an explicit check. if not isinstance(headers, httplib.HTTPMessage): raise TypeError(f"expected httplib.Message, got {type(headers)}.") unparsed_data = None # get_payload is actually email.message.Message.get_payload; # we're only interested in the result if it's not a multipart message if not headers.is_multipart(): payload = headers.get_payload() if isinstance(payload, (bytes, str)): unparsed_data = payload # httplib is assuming a response body is available # when parsing headers even when httplib only sends # header data to parse_headers() This results in # defects on multipart responses in particular. # See: https://github.com/urllib3/urllib3/issues/800 # So we ignore the following defects: # - StartBoundaryNotFoundDefect: # The claimed start boundary was never found. # - MultipartInvariantViolationDefect: # A message claimed to be a multipart but no subparts were found. defects = [ defect for defect in headers.defects if not isinstance( defect, (StartBoundaryNotFoundDefect, MultipartInvariantViolationDefect) ) ] if defects or unparsed_data: raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data) def is_response_to_head(response: httplib.HTTPResponse) -> bool: """ Checks whether the request of a response has been a HEAD-request. :param http.client.HTTPResponse response: Response to check if the originating request used 'HEAD' as a method. """ # FIXME: Can we do this somehow without accessing private httplib _method? method_str = response._method # type: str # type: ignore[attr-defined] return method_str.upper() == "HEAD" urllib3-2.0.7/src/urllib3/util/retry.py0000644000000000000000000004372014513541732014674 0ustar00from __future__ import annotations import email import logging import random import re import time import typing from itertools import takewhile from types import TracebackType from ..exceptions import ( ConnectTimeoutError, InvalidHeader, MaxRetryError, ProtocolError, ProxyError, ReadTimeoutError, ResponseError, ) from .util import reraise if typing.TYPE_CHECKING: from ..connectionpool import ConnectionPool from ..response import BaseHTTPResponse log = logging.getLogger(__name__) # Data structure for representing the metadata of requests that result in a retry. class RequestHistory(typing.NamedTuple): method: str | None url: str | None error: Exception | None status: int | None redirect_location: str | None class Retry: """Retry configuration. Each retry attempt will create a new Retry object with updated values, so they can be safely reused. Retries can be defined as a default for a pool: .. code-block:: python retries = Retry(connect=5, read=2, redirect=5) http = PoolManager(retries=retries) response = http.request("GET", "https://example.com/") Or per-request (which overrides the default for the pool): .. code-block:: python response = http.request("GET", "https://example.com/", retries=Retry(10)) Retries can be disabled by passing ``False``: .. code-block:: python response = http.request("GET", "https://example.com/", retries=False) Errors will be wrapped in :class:`~urllib3.exceptions.MaxRetryError` unless retries are disabled, in which case the causing exception will be raised. :param int total: Total number of retries to allow. Takes precedence over other counts. Set to ``None`` to remove this constraint and fall back on other counts. Set to ``0`` to fail on the first retry. Set to ``False`` to disable and imply ``raise_on_redirect=False``. :param int connect: How many connection-related errors to retry on. These are errors raised before the request is sent to the remote server, which we assume has not triggered the server to process the request. Set to ``0`` to fail on the first retry of this type. :param int read: How many times to retry on read errors. These errors are raised after the request was sent to the server, so the request may have side-effects. Set to ``0`` to fail on the first retry of this type. :param int redirect: How many redirects to perform. Limit this to avoid infinite redirect loops. A redirect is a HTTP response with a status code 301, 302, 303, 307 or 308. Set to ``0`` to fail on the first retry of this type. Set to ``False`` to disable and imply ``raise_on_redirect=False``. :param int status: How many times to retry on bad status codes. These are retries made on responses, where status code matches ``status_forcelist``. Set to ``0`` to fail on the first retry of this type. :param int other: How many times to retry on other errors. Other errors are errors that are not connect, read, redirect or status errors. These errors might be raised after the request was sent to the server, so the request might have side-effects. Set to ``0`` to fail on the first retry of this type. If ``total`` is not set, it's a good idea to set this to 0 to account for unexpected edge cases and avoid infinite retry loops. :param Collection allowed_methods: Set of uppercased HTTP method verbs that we should retry on. By default, we only retry on methods which are considered to be idempotent (multiple requests with the same parameters end with the same state). See :attr:`Retry.DEFAULT_ALLOWED_METHODS`. Set to a ``None`` value to retry on any verb. :param Collection status_forcelist: A set of integer HTTP status codes that we should force a retry on. A retry is initiated if the request method is in ``allowed_methods`` and the response status code is in ``status_forcelist``. By default, this is disabled with ``None``. :param float backoff_factor: A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for:: {backoff factor} * (2 ** ({number of previous retries})) seconds. If `backoff_jitter` is non-zero, this sleep is extended by:: random.uniform(0, {backoff jitter}) seconds. For example, if the backoff_factor is 0.1, then :func:`Retry.sleep` will sleep for [0.0s, 0.2s, 0.4s, 0.8s, ...] between retries. No backoff will ever be longer than `backoff_max`. By default, backoff is disabled (factor set to 0). :param bool raise_on_redirect: Whether, if the number of redirects is exhausted, to raise a MaxRetryError, or to return a response with a response code in the 3xx range. :param bool raise_on_status: Similar meaning to ``raise_on_redirect``: whether we should raise an exception, or return a response, if status falls in ``status_forcelist`` range and retries have been exhausted. :param tuple history: The history of the request encountered during each call to :meth:`~Retry.increment`. The list is in the order the requests occurred. Each list item is of class :class:`RequestHistory`. :param bool respect_retry_after_header: Whether to respect Retry-After header on status codes defined as :attr:`Retry.RETRY_AFTER_STATUS_CODES` or not. :param Collection remove_headers_on_redirect: Sequence of headers to remove from the request when a response indicating a redirect is returned before firing off the redirected request. """ #: Default methods to be used for ``allowed_methods`` DEFAULT_ALLOWED_METHODS = frozenset( ["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"] ) #: Default status codes to be used for ``status_forcelist`` RETRY_AFTER_STATUS_CODES = frozenset([413, 429, 503]) #: Default headers to be used for ``remove_headers_on_redirect`` DEFAULT_REMOVE_HEADERS_ON_REDIRECT = frozenset(["Cookie", "Authorization"]) #: Default maximum backoff time. DEFAULT_BACKOFF_MAX = 120 # Backward compatibility; assigned outside of the class. DEFAULT: typing.ClassVar[Retry] def __init__( self, total: bool | int | None = 10, connect: int | None = None, read: int | None = None, redirect: bool | int | None = None, status: int | None = None, other: int | None = None, allowed_methods: typing.Collection[str] | None = DEFAULT_ALLOWED_METHODS, status_forcelist: typing.Collection[int] | None = None, backoff_factor: float = 0, backoff_max: float = DEFAULT_BACKOFF_MAX, raise_on_redirect: bool = True, raise_on_status: bool = True, history: tuple[RequestHistory, ...] | None = None, respect_retry_after_header: bool = True, remove_headers_on_redirect: typing.Collection[ str ] = DEFAULT_REMOVE_HEADERS_ON_REDIRECT, backoff_jitter: float = 0.0, ) -> None: self.total = total self.connect = connect self.read = read self.status = status self.other = other if redirect is False or total is False: redirect = 0 raise_on_redirect = False self.redirect = redirect self.status_forcelist = status_forcelist or set() self.allowed_methods = allowed_methods self.backoff_factor = backoff_factor self.backoff_max = backoff_max self.raise_on_redirect = raise_on_redirect self.raise_on_status = raise_on_status self.history = history or () self.respect_retry_after_header = respect_retry_after_header self.remove_headers_on_redirect = frozenset( h.lower() for h in remove_headers_on_redirect ) self.backoff_jitter = backoff_jitter def new(self, **kw: typing.Any) -> Retry: params = dict( total=self.total, connect=self.connect, read=self.read, redirect=self.redirect, status=self.status, other=self.other, allowed_methods=self.allowed_methods, status_forcelist=self.status_forcelist, backoff_factor=self.backoff_factor, backoff_max=self.backoff_max, raise_on_redirect=self.raise_on_redirect, raise_on_status=self.raise_on_status, history=self.history, remove_headers_on_redirect=self.remove_headers_on_redirect, respect_retry_after_header=self.respect_retry_after_header, backoff_jitter=self.backoff_jitter, ) params.update(kw) return type(self)(**params) # type: ignore[arg-type] @classmethod def from_int( cls, retries: Retry | bool | int | None, redirect: bool | int | None = True, default: Retry | bool | int | None = None, ) -> Retry: """Backwards-compatibility for the old retries format.""" if retries is None: retries = default if default is not None else cls.DEFAULT if isinstance(retries, Retry): return retries redirect = bool(redirect) and None new_retries = cls(retries, redirect=redirect) log.debug("Converted retries value: %r -> %r", retries, new_retries) return new_retries def get_backoff_time(self) -> float: """Formula for computing the current backoff :rtype: float """ # We want to consider only the last consecutive errors sequence (Ignore redirects). consecutive_errors_len = len( list( takewhile(lambda x: x.redirect_location is None, reversed(self.history)) ) ) if consecutive_errors_len <= 1: return 0 backoff_value = self.backoff_factor * (2 ** (consecutive_errors_len - 1)) if self.backoff_jitter != 0.0: backoff_value += random.random() * self.backoff_jitter return float(max(0, min(self.backoff_max, backoff_value))) def parse_retry_after(self, retry_after: str) -> float: seconds: float # Whitespace: https://tools.ietf.org/html/rfc7230#section-3.2.4 if re.match(r"^\s*[0-9]+\s*$", retry_after): seconds = int(retry_after) else: retry_date_tuple = email.utils.parsedate_tz(retry_after) if retry_date_tuple is None: raise InvalidHeader(f"Invalid Retry-After header: {retry_after}") retry_date = email.utils.mktime_tz(retry_date_tuple) seconds = retry_date - time.time() seconds = max(seconds, 0) return seconds def get_retry_after(self, response: BaseHTTPResponse) -> float | None: """Get the value of Retry-After in seconds.""" retry_after = response.headers.get("Retry-After") if retry_after is None: return None return self.parse_retry_after(retry_after) def sleep_for_retry(self, response: BaseHTTPResponse) -> bool: retry_after = self.get_retry_after(response) if retry_after: time.sleep(retry_after) return True return False def _sleep_backoff(self) -> None: backoff = self.get_backoff_time() if backoff <= 0: return time.sleep(backoff) def sleep(self, response: BaseHTTPResponse | None = None) -> None: """Sleep between retry attempts. This method will respect a server's ``Retry-After`` response header and sleep the duration of the time requested. If that is not present, it will use an exponential backoff. By default, the backoff factor is 0 and this method will return immediately. """ if self.respect_retry_after_header and response: slept = self.sleep_for_retry(response) if slept: return self._sleep_backoff() def _is_connection_error(self, err: Exception) -> bool: """Errors when we're fairly sure that the server did not receive the request, so it should be safe to retry. """ if isinstance(err, ProxyError): err = err.original_error return isinstance(err, ConnectTimeoutError) def _is_read_error(self, err: Exception) -> bool: """Errors that occur after the request has been started, so we should assume that the server began processing it. """ return isinstance(err, (ReadTimeoutError, ProtocolError)) def _is_method_retryable(self, method: str) -> bool: """Checks if a given HTTP method should be retried upon, depending if it is included in the allowed_methods """ if self.allowed_methods and method.upper() not in self.allowed_methods: return False return True def is_retry( self, method: str, status_code: int, has_retry_after: bool = False ) -> bool: """Is this method/status code retryable? (Based on allowlists and control variables such as the number of total retries to allow, whether to respect the Retry-After header, whether this header is present, and whether the returned status code is on the list of status codes to be retried upon on the presence of the aforementioned header) """ if not self._is_method_retryable(method): return False if self.status_forcelist and status_code in self.status_forcelist: return True return bool( self.total and self.respect_retry_after_header and has_retry_after and (status_code in self.RETRY_AFTER_STATUS_CODES) ) def is_exhausted(self) -> bool: """Are we out of retries?""" retry_counts = [ x for x in ( self.total, self.connect, self.read, self.redirect, self.status, self.other, ) if x ] if not retry_counts: return False return min(retry_counts) < 0 def increment( self, method: str | None = None, url: str | None = None, response: BaseHTTPResponse | None = None, error: Exception | None = None, _pool: ConnectionPool | None = None, _stacktrace: TracebackType | None = None, ) -> Retry: """Return a new Retry object with incremented retry counters. :param response: A response object, or None, if the server did not return a response. :type response: :class:`~urllib3.response.BaseHTTPResponse` :param Exception error: An error encountered during the request, or None if the response was received successfully. :return: A new ``Retry`` object. """ if self.total is False and error: # Disabled, indicate to re-raise the error. raise reraise(type(error), error, _stacktrace) total = self.total if total is not None: total -= 1 connect = self.connect read = self.read redirect = self.redirect status_count = self.status other = self.other cause = "unknown" status = None redirect_location = None if error and self._is_connection_error(error): # Connect retry? if connect is False: raise reraise(type(error), error, _stacktrace) elif connect is not None: connect -= 1 elif error and self._is_read_error(error): # Read retry? if read is False or method is None or not self._is_method_retryable(method): raise reraise(type(error), error, _stacktrace) elif read is not None: read -= 1 elif error: # Other retry? if other is not None: other -= 1 elif response and response.get_redirect_location(): # Redirect retry? if redirect is not None: redirect -= 1 cause = "too many redirects" response_redirect_location = response.get_redirect_location() if response_redirect_location: redirect_location = response_redirect_location status = response.status else: # Incrementing because of a server error like a 500 in # status_forcelist and the given method is in the allowed_methods cause = ResponseError.GENERIC_ERROR if response and response.status: if status_count is not None: status_count -= 1 cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) status = response.status history = self.history + ( RequestHistory(method, url, error, status, redirect_location), ) new_retry = self.new( total=total, connect=connect, read=read, redirect=redirect, status=status_count, other=other, history=history, ) if new_retry.is_exhausted(): reason = error or ResponseError(cause) raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] log.debug("Incremented Retry for (url='%s'): %r", url, new_retry) return new_retry def __repr__(self) -> str: return ( f"{type(self).__name__}(total={self.total}, connect={self.connect}, " f"read={self.read}, redirect={self.redirect}, status={self.status})" ) # For backwards compatibility (equivalent to pre-v1.9): Retry.DEFAULT = Retry(3) urllib3-2.0.7/src/urllib3/util/ssl_.py0000644000000000000000000004571314513541732014473 0ustar00from __future__ import annotations import hmac import os import socket import sys import typing import warnings from binascii import unhexlify from hashlib import md5, sha1, sha256 from ..exceptions import ProxySchemeUnsupported, SSLError from .url import _BRACELESS_IPV6_ADDRZ_RE, _IPV4_RE SSLContext = None SSLTransport = None HAS_NEVER_CHECK_COMMON_NAME = False IS_PYOPENSSL = False IS_SECURETRANSPORT = False ALPN_PROTOCOLS = ["http/1.1"] _TYPE_VERSION_INFO = typing.Tuple[int, int, int, str, int] # Maps the length of a digest to a possible hash function producing this digest HASHFUNC_MAP = {32: md5, 40: sha1, 64: sha256} def _is_bpo_43522_fixed( implementation_name: str, version_info: _TYPE_VERSION_INFO, pypy_version_info: _TYPE_VERSION_INFO | None, ) -> bool: """Return True for CPython 3.8.9+, 3.9.3+ or 3.10+ and PyPy 7.3.8+ where setting SSLContext.hostname_checks_common_name to False works. Outside of CPython and PyPy we don't know which implementations work or not so we conservatively use our hostname matching as we know that works on all implementations. https://github.com/urllib3/urllib3/issues/2192#issuecomment-821832963 https://foss.heptapod.net/pypy/pypy/-/issues/3539 """ if implementation_name == "pypy": # https://foss.heptapod.net/pypy/pypy/-/issues/3129 return pypy_version_info >= (7, 3, 8) and version_info >= (3, 8) # type: ignore[operator] elif implementation_name == "cpython": major_minor = version_info[:2] micro = version_info[2] return ( (major_minor == (3, 8) and micro >= 9) or (major_minor == (3, 9) and micro >= 3) or major_minor >= (3, 10) ) else: # Defensive: return False def _is_has_never_check_common_name_reliable( openssl_version: str, openssl_version_number: int, implementation_name: str, version_info: _TYPE_VERSION_INFO, pypy_version_info: _TYPE_VERSION_INFO | None, ) -> bool: # As of May 2023, all released versions of LibreSSL fail to reject certificates with # only common names, see https://github.com/urllib3/urllib3/pull/3024 is_openssl = openssl_version.startswith("OpenSSL ") # Before fixing OpenSSL issue #14579, the SSL_new() API was not copying hostflags # like X509_CHECK_FLAG_NEVER_CHECK_SUBJECT, which tripped up CPython. # https://github.com/openssl/openssl/issues/14579 # This was released in OpenSSL 1.1.1l+ (>=0x101010cf) is_openssl_issue_14579_fixed = openssl_version_number >= 0x101010CF return is_openssl and ( is_openssl_issue_14579_fixed or _is_bpo_43522_fixed(implementation_name, version_info, pypy_version_info) ) if typing.TYPE_CHECKING: from ssl import VerifyMode from typing_extensions import Literal, TypedDict from .ssltransport import SSLTransport as SSLTransportType class _TYPE_PEER_CERT_RET_DICT(TypedDict, total=False): subjectAltName: tuple[tuple[str, str], ...] subject: tuple[tuple[tuple[str, str], ...], ...] serialNumber: str # Mapping from 'ssl.PROTOCOL_TLSX' to 'TLSVersion.X' _SSL_VERSION_TO_TLS_VERSION: dict[int, int] = {} try: # Do we have ssl at all? import ssl from ssl import ( # type: ignore[assignment] CERT_REQUIRED, HAS_NEVER_CHECK_COMMON_NAME, OP_NO_COMPRESSION, OP_NO_TICKET, OPENSSL_VERSION, OPENSSL_VERSION_NUMBER, PROTOCOL_TLS, PROTOCOL_TLS_CLIENT, OP_NO_SSLv2, OP_NO_SSLv3, SSLContext, TLSVersion, ) PROTOCOL_SSLv23 = PROTOCOL_TLS # Setting SSLContext.hostname_checks_common_name = False didn't work before CPython # 3.8.9, 3.9.3, and 3.10 (but OK on PyPy) or OpenSSL 1.1.1l+ if HAS_NEVER_CHECK_COMMON_NAME and not _is_has_never_check_common_name_reliable( OPENSSL_VERSION, OPENSSL_VERSION_NUMBER, sys.implementation.name, sys.version_info, sys.pypy_version_info if sys.implementation.name == "pypy" else None, # type: ignore[attr-defined] ): HAS_NEVER_CHECK_COMMON_NAME = False # Need to be careful here in case old TLS versions get # removed in future 'ssl' module implementations. for attr in ("TLSv1", "TLSv1_1", "TLSv1_2"): try: _SSL_VERSION_TO_TLS_VERSION[getattr(ssl, f"PROTOCOL_{attr}")] = getattr( TLSVersion, attr ) except AttributeError: # Defensive: continue from .ssltransport import SSLTransport # type: ignore[assignment] except ImportError: OP_NO_COMPRESSION = 0x20000 # type: ignore[assignment] OP_NO_TICKET = 0x4000 # type: ignore[assignment] OP_NO_SSLv2 = 0x1000000 # type: ignore[assignment] OP_NO_SSLv3 = 0x2000000 # type: ignore[assignment] PROTOCOL_SSLv23 = PROTOCOL_TLS = 2 # type: ignore[assignment] PROTOCOL_TLS_CLIENT = 16 # type: ignore[assignment] _TYPE_PEER_CERT_RET = typing.Union["_TYPE_PEER_CERT_RET_DICT", bytes, None] def assert_fingerprint(cert: bytes | None, fingerprint: str) -> None: """ Checks if given fingerprint matches the supplied certificate. :param cert: Certificate as bytes object. :param fingerprint: Fingerprint as string of hexdigits, can be interspersed by colons. """ if cert is None: raise SSLError("No certificate for the peer.") fingerprint = fingerprint.replace(":", "").lower() digest_length = len(fingerprint) hashfunc = HASHFUNC_MAP.get(digest_length) if not hashfunc: raise SSLError(f"Fingerprint of invalid length: {fingerprint}") # We need encode() here for py32; works on py2 and p33. fingerprint_bytes = unhexlify(fingerprint.encode()) cert_digest = hashfunc(cert).digest() if not hmac.compare_digest(cert_digest, fingerprint_bytes): raise SSLError( f'Fingerprints did not match. Expected "{fingerprint}", got "{cert_digest.hex()}"' ) def resolve_cert_reqs(candidate: None | int | str) -> VerifyMode: """ Resolves the argument to a numeric constant, which can be passed to the wrap_socket function/method from the ssl module. Defaults to :data:`ssl.CERT_REQUIRED`. If given a string it is assumed to be the name of the constant in the :mod:`ssl` module or its abbreviation. (So you can specify `REQUIRED` instead of `CERT_REQUIRED`. If it's neither `None` nor a string we assume it is already the numeric constant which can directly be passed to wrap_socket. """ if candidate is None: return CERT_REQUIRED if isinstance(candidate, str): res = getattr(ssl, candidate, None) if res is None: res = getattr(ssl, "CERT_" + candidate) return res # type: ignore[no-any-return] return candidate # type: ignore[return-value] def resolve_ssl_version(candidate: None | int | str) -> int: """ like resolve_cert_reqs """ if candidate is None: return PROTOCOL_TLS if isinstance(candidate, str): res = getattr(ssl, candidate, None) if res is None: res = getattr(ssl, "PROTOCOL_" + candidate) return typing.cast(int, res) return candidate def create_urllib3_context( ssl_version: int | None = None, cert_reqs: int | None = None, options: int | None = None, ciphers: str | None = None, ssl_minimum_version: int | None = None, ssl_maximum_version: int | None = None, ) -> ssl.SSLContext: """Creates and configures an :class:`ssl.SSLContext` instance for use with urllib3. :param ssl_version: The desired protocol version to use. This will default to PROTOCOL_SSLv23 which will negotiate the highest protocol that both the server and your installation of OpenSSL support. This parameter is deprecated instead use 'ssl_minimum_version'. :param ssl_minimum_version: The minimum version of TLS to be used. Use the 'ssl.TLSVersion' enum for specifying the value. :param ssl_maximum_version: The maximum version of TLS to be used. Use the 'ssl.TLSVersion' enum for specifying the value. Not recommended to set to anything other than 'ssl.TLSVersion.MAXIMUM_SUPPORTED' which is the default value. :param cert_reqs: Whether to require the certificate verification. This defaults to ``ssl.CERT_REQUIRED``. :param options: Specific OpenSSL options. These default to ``ssl.OP_NO_SSLv2``, ``ssl.OP_NO_SSLv3``, ``ssl.OP_NO_COMPRESSION``, and ``ssl.OP_NO_TICKET``. :param ciphers: Which cipher suites to allow the server to select. Defaults to either system configured ciphers if OpenSSL 1.1.1+, otherwise uses a secure default set of ciphers. :returns: Constructed SSLContext object with specified options :rtype: SSLContext """ if SSLContext is None: raise TypeError("Can't create an SSLContext object without an ssl module") # This means 'ssl_version' was specified as an exact value. if ssl_version not in (None, PROTOCOL_TLS, PROTOCOL_TLS_CLIENT): # Disallow setting 'ssl_version' and 'ssl_minimum|maximum_version' # to avoid conflicts. if ssl_minimum_version is not None or ssl_maximum_version is not None: raise ValueError( "Can't specify both 'ssl_version' and either " "'ssl_minimum_version' or 'ssl_maximum_version'" ) # 'ssl_version' is deprecated and will be removed in the future. else: # Use 'ssl_minimum_version' and 'ssl_maximum_version' instead. ssl_minimum_version = _SSL_VERSION_TO_TLS_VERSION.get( ssl_version, TLSVersion.MINIMUM_SUPPORTED ) ssl_maximum_version = _SSL_VERSION_TO_TLS_VERSION.get( ssl_version, TLSVersion.MAXIMUM_SUPPORTED ) # This warning message is pushing users to use 'ssl_minimum_version' # instead of both min/max. Best practice is to only set the minimum version and # keep the maximum version to be it's default value: 'TLSVersion.MAXIMUM_SUPPORTED' warnings.warn( "'ssl_version' option is deprecated and will be " "removed in urllib3 v2.1.0. Instead use 'ssl_minimum_version'", category=DeprecationWarning, stacklevel=2, ) # PROTOCOL_TLS is deprecated in Python 3.10 so we always use PROTOCOL_TLS_CLIENT context = SSLContext(PROTOCOL_TLS_CLIENT) if ssl_minimum_version is not None: context.minimum_version = ssl_minimum_version else: # Python <3.10 defaults to 'MINIMUM_SUPPORTED' so explicitly set TLSv1.2 here context.minimum_version = TLSVersion.TLSv1_2 if ssl_maximum_version is not None: context.maximum_version = ssl_maximum_version # Unless we're given ciphers defer to either system ciphers in # the case of OpenSSL 1.1.1+ or use our own secure default ciphers. if ciphers: context.set_ciphers(ciphers) # Setting the default here, as we may have no ssl module on import cert_reqs = ssl.CERT_REQUIRED if cert_reqs is None else cert_reqs if options is None: options = 0 # SSLv2 is easily broken and is considered harmful and dangerous options |= OP_NO_SSLv2 # SSLv3 has several problems and is now dangerous options |= OP_NO_SSLv3 # Disable compression to prevent CRIME attacks for OpenSSL 1.0+ # (issue #309) options |= OP_NO_COMPRESSION # TLSv1.2 only. Unless set explicitly, do not request tickets. # This may save some bandwidth on wire, and although the ticket is encrypted, # there is a risk associated with it being on wire, # if the server is not rotating its ticketing keys properly. options |= OP_NO_TICKET context.options |= options # Enable post-handshake authentication for TLS 1.3, see GH #1634. PHA is # necessary for conditional client cert authentication with TLS 1.3. # The attribute is None for OpenSSL <= 1.1.0 or does not exist in older # versions of Python. We only enable on Python 3.7.4+ or if certificate # verification is enabled to work around Python issue #37428 # See: https://bugs.python.org/issue37428 if (cert_reqs == ssl.CERT_REQUIRED or sys.version_info >= (3, 7, 4)) and getattr( context, "post_handshake_auth", None ) is not None: context.post_handshake_auth = True # The order of the below lines setting verify_mode and check_hostname # matter due to safe-guards SSLContext has to prevent an SSLContext with # check_hostname=True, verify_mode=NONE/OPTIONAL. # We always set 'check_hostname=False' for pyOpenSSL so we rely on our own # 'ssl.match_hostname()' implementation. if cert_reqs == ssl.CERT_REQUIRED and not IS_PYOPENSSL: context.verify_mode = cert_reqs context.check_hostname = True else: context.check_hostname = False context.verify_mode = cert_reqs try: context.hostname_checks_common_name = False except AttributeError: # Defensive: for CPython < 3.8.9 and 3.9.3; for PyPy < 7.3.8 pass # Enable logging of TLS session keys via defacto standard environment variable # 'SSLKEYLOGFILE', if the feature is available (Python 3.8+). Skip empty values. if hasattr(context, "keylog_filename"): sslkeylogfile = os.environ.get("SSLKEYLOGFILE") if sslkeylogfile: context.keylog_filename = sslkeylogfile return context @typing.overload def ssl_wrap_socket( sock: socket.socket, keyfile: str | None = ..., certfile: str | None = ..., cert_reqs: int | None = ..., ca_certs: str | None = ..., server_hostname: str | None = ..., ssl_version: int | None = ..., ciphers: str | None = ..., ssl_context: ssl.SSLContext | None = ..., ca_cert_dir: str | None = ..., key_password: str | None = ..., ca_cert_data: None | str | bytes = ..., tls_in_tls: Literal[False] = ..., ) -> ssl.SSLSocket: ... @typing.overload def ssl_wrap_socket( sock: socket.socket, keyfile: str | None = ..., certfile: str | None = ..., cert_reqs: int | None = ..., ca_certs: str | None = ..., server_hostname: str | None = ..., ssl_version: int | None = ..., ciphers: str | None = ..., ssl_context: ssl.SSLContext | None = ..., ca_cert_dir: str | None = ..., key_password: str | None = ..., ca_cert_data: None | str | bytes = ..., tls_in_tls: bool = ..., ) -> ssl.SSLSocket | SSLTransportType: ... def ssl_wrap_socket( sock: socket.socket, keyfile: str | None = None, certfile: str | None = None, cert_reqs: int | None = None, ca_certs: str | None = None, server_hostname: str | None = None, ssl_version: int | None = None, ciphers: str | None = None, ssl_context: ssl.SSLContext | None = None, ca_cert_dir: str | None = None, key_password: str | None = None, ca_cert_data: None | str | bytes = None, tls_in_tls: bool = False, ) -> ssl.SSLSocket | SSLTransportType: """ All arguments except for server_hostname, ssl_context, tls_in_tls, ca_cert_data and ca_cert_dir have the same meaning as they do when using :func:`ssl.create_default_context`, :meth:`ssl.SSLContext.load_cert_chain`, :meth:`ssl.SSLContext.set_ciphers` and :meth:`ssl.SSLContext.wrap_socket`. :param server_hostname: When SNI is supported, the expected hostname of the certificate :param ssl_context: A pre-made :class:`SSLContext` object. If none is provided, one will be created using :func:`create_urllib3_context`. :param ciphers: A string of ciphers we wish the client to support. :param ca_cert_dir: A directory containing CA certificates in multiple separate files, as supported by OpenSSL's -CApath flag or the capath argument to SSLContext.load_verify_locations(). :param key_password: Optional password if the keyfile is encrypted. :param ca_cert_data: Optional string containing CA certificates in PEM format suitable for passing as the cadata parameter to SSLContext.load_verify_locations() :param tls_in_tls: Use SSLTransport to wrap the existing socket. """ context = ssl_context if context is None: # Note: This branch of code and all the variables in it are only used in tests. # We should consider deprecating and removing this code. context = create_urllib3_context(ssl_version, cert_reqs, ciphers=ciphers) if ca_certs or ca_cert_dir or ca_cert_data: try: context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data) except OSError as e: raise SSLError(e) from e elif ssl_context is None and hasattr(context, "load_default_certs"): # try to load OS default certs; works well on Windows. context.load_default_certs() # Attempt to detect if we get the goofy behavior of the # keyfile being encrypted and OpenSSL asking for the # passphrase via the terminal and instead error out. if keyfile and key_password is None and _is_key_file_encrypted(keyfile): raise SSLError("Client private key is encrypted, password is required") if certfile: if key_password is None: context.load_cert_chain(certfile, keyfile) else: context.load_cert_chain(certfile, keyfile, key_password) try: context.set_alpn_protocols(ALPN_PROTOCOLS) except NotImplementedError: # Defensive: in CI, we always have set_alpn_protocols pass ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname) return ssl_sock def is_ipaddress(hostname: str | bytes) -> bool: """Detects whether the hostname given is an IPv4 or IPv6 address. Also detects IPv6 addresses with Zone IDs. :param str hostname: Hostname to examine. :return: True if the hostname is an IP address, False otherwise. """ if isinstance(hostname, bytes): # IDN A-label bytes are ASCII compatible. hostname = hostname.decode("ascii") return bool(_IPV4_RE.match(hostname) or _BRACELESS_IPV6_ADDRZ_RE.match(hostname)) def _is_key_file_encrypted(key_file: str) -> bool: """Detects if a key file is encrypted or not.""" with open(key_file) as f: for line in f: # Look for Proc-Type: 4,ENCRYPTED if "ENCRYPTED" in line: return True return False def _ssl_wrap_socket_impl( sock: socket.socket, ssl_context: ssl.SSLContext, tls_in_tls: bool, server_hostname: str | None = None, ) -> ssl.SSLSocket | SSLTransportType: if tls_in_tls: if not SSLTransport: # Import error, ssl is not available. raise ProxySchemeUnsupported( "TLS in TLS requires support for the 'ssl' module" ) SSLTransport._validate_ssl_context_for_tls_in_tls(ssl_context) return SSLTransport(sock, ssl_context, server_hostname) return ssl_context.wrap_socket(sock, server_hostname=server_hostname) urllib3-2.0.7/src/urllib3/util/ssl_match_hostname.py0000644000000000000000000001326414513541732017402 0ustar00"""The match_hostname() function from Python 3.5, essential when using SSL.""" # Note: This file is under the PSF license as the code comes from the python # stdlib. http://docs.python.org/3/license.html # It is modified to remove commonName support. from __future__ import annotations import ipaddress import re import typing from ipaddress import IPv4Address, IPv6Address if typing.TYPE_CHECKING: from .ssl_ import _TYPE_PEER_CERT_RET_DICT __version__ = "3.5.0.1" class CertificateError(ValueError): pass def _dnsname_match( dn: typing.Any, hostname: str, max_wildcards: int = 1 ) -> typing.Match[str] | None | bool: """Matching according to RFC 6125, section 6.4.3 http://tools.ietf.org/html/rfc6125#section-6.4.3 """ pats = [] if not dn: return False # Ported from python3-syntax: # leftmost, *remainder = dn.split(r'.') parts = dn.split(r".") leftmost = parts[0] remainder = parts[1:] wildcards = leftmost.count("*") if wildcards > max_wildcards: # Issue #17980: avoid denials of service by refusing more # than one wildcard per fragment. A survey of established # policy among SSL implementations showed it to be a # reasonable choice. raise CertificateError( "too many wildcards in certificate DNS name: " + repr(dn) ) # speed up common case w/o wildcards if not wildcards: return bool(dn.lower() == hostname.lower()) # RFC 6125, section 6.4.3, subitem 1. # The client SHOULD NOT attempt to match a presented identifier in which # the wildcard character comprises a label other than the left-most label. if leftmost == "*": # When '*' is a fragment by itself, it matches a non-empty dotless # fragment. pats.append("[^.]+") elif leftmost.startswith("xn--") or hostname.startswith("xn--"): # RFC 6125, section 6.4.3, subitem 3. # The client SHOULD NOT attempt to match a presented identifier # where the wildcard character is embedded within an A-label or # U-label of an internationalized domain name. pats.append(re.escape(leftmost)) else: # Otherwise, '*' matches any dotless string, e.g. www* pats.append(re.escape(leftmost).replace(r"\*", "[^.]*")) # add the remaining fragments, ignore any wildcards for frag in remainder: pats.append(re.escape(frag)) pat = re.compile(r"\A" + r"\.".join(pats) + r"\Z", re.IGNORECASE) return pat.match(hostname) def _ipaddress_match(ipname: str, host_ip: IPv4Address | IPv6Address) -> bool: """Exact matching of IP addresses. RFC 9110 section 4.3.5: "A reference identity of IP-ID contains the decoded bytes of the IP address. An IP version 4 address is 4 octets, and an IP version 6 address is 16 octets. [...] A reference identity of type IP-ID matches if the address is identical to an iPAddress value of the subjectAltName extension of the certificate." """ # OpenSSL may add a trailing newline to a subjectAltName's IP address # Divergence from upstream: ipaddress can't handle byte str ip = ipaddress.ip_address(ipname.rstrip()) return bool(ip.packed == host_ip.packed) def match_hostname( cert: _TYPE_PEER_CERT_RET_DICT | None, hostname: str, hostname_checks_common_name: bool = False, ) -> None: """Verify that *cert* (in decoded format as returned by SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 and RFC 6125 rules are followed, but IP addresses are not accepted for *hostname*. CertificateError is raised on failure. On success, the function returns nothing. """ if not cert: raise ValueError( "empty or no certificate, match_hostname needs a " "SSL socket or SSL context with either " "CERT_OPTIONAL or CERT_REQUIRED" ) try: # Divergence from upstream: ipaddress can't handle byte str # # The ipaddress module shipped with Python < 3.9 does not support # scoped IPv6 addresses so we unconditionally strip the Zone IDs for # now. Once we drop support for Python 3.9 we can remove this branch. if "%" in hostname: host_ip = ipaddress.ip_address(hostname[: hostname.rfind("%")]) else: host_ip = ipaddress.ip_address(hostname) except ValueError: # Not an IP address (common case) host_ip = None dnsnames = [] san: tuple[tuple[str, str], ...] = cert.get("subjectAltName", ()) key: str value: str for key, value in san: if key == "DNS": if host_ip is None and _dnsname_match(value, hostname): return dnsnames.append(value) elif key == "IP Address": if host_ip is not None and _ipaddress_match(value, host_ip): return dnsnames.append(value) # We only check 'commonName' if it's enabled and we're not verifying # an IP address. IP addresses aren't valid within 'commonName'. if hostname_checks_common_name and host_ip is None and not dnsnames: for sub in cert.get("subject", ()): for key, value in sub: if key == "commonName": if _dnsname_match(value, hostname): return dnsnames.append(value) if len(dnsnames) > 1: raise CertificateError( "hostname %r " "doesn't match either of %s" % (hostname, ", ".join(map(repr, dnsnames))) ) elif len(dnsnames) == 1: raise CertificateError(f"hostname {hostname!r} doesn't match {dnsnames[0]!r}") else: raise CertificateError("no appropriate subjectAltName fields were found") urllib3-2.0.7/src/urllib3/util/ssltransport.py0000644000000000000000000002152514513541732016304 0ustar00from __future__ import annotations import io import socket import ssl import typing from ..exceptions import ProxySchemeUnsupported if typing.TYPE_CHECKING: from typing_extensions import Literal from .ssl_ import _TYPE_PEER_CERT_RET, _TYPE_PEER_CERT_RET_DICT _SelfT = typing.TypeVar("_SelfT", bound="SSLTransport") _WriteBuffer = typing.Union[bytearray, memoryview] _ReturnValue = typing.TypeVar("_ReturnValue") SSL_BLOCKSIZE = 16384 class SSLTransport: """ The SSLTransport wraps an existing socket and establishes an SSL connection. Contrary to Python's implementation of SSLSocket, it allows you to chain multiple TLS connections together. It's particularly useful if you need to implement TLS within TLS. The class supports most of the socket API operations. """ @staticmethod def _validate_ssl_context_for_tls_in_tls(ssl_context: ssl.SSLContext) -> None: """ Raises a ProxySchemeUnsupported if the provided ssl_context can't be used for TLS in TLS. The only requirement is that the ssl_context provides the 'wrap_bio' methods. """ if not hasattr(ssl_context, "wrap_bio"): raise ProxySchemeUnsupported( "TLS in TLS requires SSLContext.wrap_bio() which isn't " "available on non-native SSLContext" ) def __init__( self, socket: socket.socket, ssl_context: ssl.SSLContext, server_hostname: str | None = None, suppress_ragged_eofs: bool = True, ) -> None: """ Create an SSLTransport around socket using the provided ssl_context. """ self.incoming = ssl.MemoryBIO() self.outgoing = ssl.MemoryBIO() self.suppress_ragged_eofs = suppress_ragged_eofs self.socket = socket self.sslobj = ssl_context.wrap_bio( self.incoming, self.outgoing, server_hostname=server_hostname ) # Perform initial handshake. self._ssl_io_loop(self.sslobj.do_handshake) def __enter__(self: _SelfT) -> _SelfT: return self def __exit__(self, *_: typing.Any) -> None: self.close() def fileno(self) -> int: return self.socket.fileno() def read(self, len: int = 1024, buffer: typing.Any | None = None) -> int | bytes: return self._wrap_ssl_read(len, buffer) def recv(self, buflen: int = 1024, flags: int = 0) -> int | bytes: if flags != 0: raise ValueError("non-zero flags not allowed in calls to recv") return self._wrap_ssl_read(buflen) def recv_into( self, buffer: _WriteBuffer, nbytes: int | None = None, flags: int = 0, ) -> None | int | bytes: if flags != 0: raise ValueError("non-zero flags not allowed in calls to recv_into") if nbytes is None: nbytes = len(buffer) return self.read(nbytes, buffer) def sendall(self, data: bytes, flags: int = 0) -> None: if flags != 0: raise ValueError("non-zero flags not allowed in calls to sendall") count = 0 with memoryview(data) as view, view.cast("B") as byte_view: amount = len(byte_view) while count < amount: v = self.send(byte_view[count:]) count += v def send(self, data: bytes, flags: int = 0) -> int: if flags != 0: raise ValueError("non-zero flags not allowed in calls to send") return self._ssl_io_loop(self.sslobj.write, data) def makefile( self, mode: str, buffering: int | None = None, *, encoding: str | None = None, errors: str | None = None, newline: str | None = None, ) -> typing.BinaryIO | typing.TextIO | socket.SocketIO: """ Python's httpclient uses makefile and buffered io when reading HTTP messages and we need to support it. This is unfortunately a copy and paste of socket.py makefile with small changes to point to the socket directly. """ if not set(mode) <= {"r", "w", "b"}: raise ValueError(f"invalid mode {mode!r} (only r, w, b allowed)") writing = "w" in mode reading = "r" in mode or not writing assert reading or writing binary = "b" in mode rawmode = "" if reading: rawmode += "r" if writing: rawmode += "w" raw = socket.SocketIO(self, rawmode) # type: ignore[arg-type] self.socket._io_refs += 1 # type: ignore[attr-defined] if buffering is None: buffering = -1 if buffering < 0: buffering = io.DEFAULT_BUFFER_SIZE if buffering == 0: if not binary: raise ValueError("unbuffered streams must be binary") return raw buffer: typing.BinaryIO if reading and writing: buffer = io.BufferedRWPair(raw, raw, buffering) # type: ignore[assignment] elif reading: buffer = io.BufferedReader(raw, buffering) else: assert writing buffer = io.BufferedWriter(raw, buffering) if binary: return buffer text = io.TextIOWrapper(buffer, encoding, errors, newline) text.mode = mode # type: ignore[misc] return text def unwrap(self) -> None: self._ssl_io_loop(self.sslobj.unwrap) def close(self) -> None: self.socket.close() @typing.overload def getpeercert( self, binary_form: Literal[False] = ... ) -> _TYPE_PEER_CERT_RET_DICT | None: ... @typing.overload def getpeercert(self, binary_form: Literal[True]) -> bytes | None: ... def getpeercert(self, binary_form: bool = False) -> _TYPE_PEER_CERT_RET: return self.sslobj.getpeercert(binary_form) # type: ignore[return-value] def version(self) -> str | None: return self.sslobj.version() def cipher(self) -> tuple[str, str, int] | None: return self.sslobj.cipher() def selected_alpn_protocol(self) -> str | None: return self.sslobj.selected_alpn_protocol() def selected_npn_protocol(self) -> str | None: return self.sslobj.selected_npn_protocol() def shared_ciphers(self) -> list[tuple[str, str, int]] | None: return self.sslobj.shared_ciphers() def compression(self) -> str | None: return self.sslobj.compression() def settimeout(self, value: float | None) -> None: self.socket.settimeout(value) def gettimeout(self) -> float | None: return self.socket.gettimeout() def _decref_socketios(self) -> None: self.socket._decref_socketios() # type: ignore[attr-defined] def _wrap_ssl_read(self, len: int, buffer: bytearray | None = None) -> int | bytes: try: return self._ssl_io_loop(self.sslobj.read, len, buffer) except ssl.SSLError as e: if e.errno == ssl.SSL_ERROR_EOF and self.suppress_ragged_eofs: return 0 # eof, return 0. else: raise # func is sslobj.do_handshake or sslobj.unwrap @typing.overload def _ssl_io_loop(self, func: typing.Callable[[], None]) -> None: ... # func is sslobj.write, arg1 is data @typing.overload def _ssl_io_loop(self, func: typing.Callable[[bytes], int], arg1: bytes) -> int: ... # func is sslobj.read, arg1 is len, arg2 is buffer @typing.overload def _ssl_io_loop( self, func: typing.Callable[[int, bytearray | None], bytes], arg1: int, arg2: bytearray | None, ) -> bytes: ... def _ssl_io_loop( self, func: typing.Callable[..., _ReturnValue], arg1: None | bytes | int = None, arg2: bytearray | None = None, ) -> _ReturnValue: """Performs an I/O loop between incoming/outgoing and the socket.""" should_loop = True ret = None while should_loop: errno = None try: if arg1 is None and arg2 is None: ret = func() elif arg2 is None: ret = func(arg1) else: ret = func(arg1, arg2) except ssl.SSLError as e: if e.errno not in (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE): # WANT_READ, and WANT_WRITE are expected, others are not. raise e errno = e.errno buf = self.outgoing.read() self.socket.sendall(buf) if errno is None: should_loop = False elif errno == ssl.SSL_ERROR_WANT_READ: buf = self.socket.recv(SSL_BLOCKSIZE) if buf: self.incoming.write(buf) else: self.incoming.write_eof() return typing.cast(_ReturnValue, ret) urllib3-2.0.7/src/urllib3/util/timeout.py0000644000000000000000000002444114513541732015214 0ustar00from __future__ import annotations import time import typing from enum import Enum from socket import getdefaulttimeout from ..exceptions import TimeoutStateError if typing.TYPE_CHECKING: from typing_extensions import Final class _TYPE_DEFAULT(Enum): # This value should never be passed to socket.settimeout() so for safety we use a -1. # socket.settimout() raises a ValueError for negative values. token = -1 _DEFAULT_TIMEOUT: Final[_TYPE_DEFAULT] = _TYPE_DEFAULT.token _TYPE_TIMEOUT = typing.Optional[typing.Union[float, _TYPE_DEFAULT]] class Timeout: """Timeout configuration. Timeouts can be defined as a default for a pool: .. code-block:: python import urllib3 timeout = urllib3.util.Timeout(connect=2.0, read=7.0) http = urllib3.PoolManager(timeout=timeout) resp = http.request("GET", "https://example.com/") print(resp.status) Or per-request (which overrides the default for the pool): .. code-block:: python response = http.request("GET", "https://example.com/", timeout=Timeout(10)) Timeouts can be disabled by setting all the parameters to ``None``: .. code-block:: python no_timeout = Timeout(connect=None, read=None) response = http.request("GET", "https://example.com/", timeout=no_timeout) :param total: This combines the connect and read timeouts into one; the read timeout will be set to the time leftover from the connect attempt. In the event that both a connect timeout and a total are specified, or a read timeout and a total are specified, the shorter timeout will be applied. Defaults to None. :type total: int, float, or None :param connect: The maximum amount of time (in seconds) to wait for a connection attempt to a server to succeed. Omitting the parameter will default the connect timeout to the system default, probably `the global default timeout in socket.py `_. None will set an infinite timeout for connection attempts. :type connect: int, float, or None :param read: The maximum amount of time (in seconds) to wait between consecutive read operations for a response from the server. Omitting the parameter will default the read timeout to the system default, probably `the global default timeout in socket.py `_. None will set an infinite timeout. :type read: int, float, or None .. note:: Many factors can affect the total amount of time for urllib3 to return an HTTP response. For example, Python's DNS resolver does not obey the timeout specified on the socket. Other factors that can affect total request time include high CPU load, high swap, the program running at a low priority level, or other behaviors. In addition, the read and total timeouts only measure the time between read operations on the socket connecting the client and the server, not the total amount of time for the request to return a complete response. For most requests, the timeout is raised because the server has not sent the first byte in the specified time. This is not always the case; if a server streams one byte every fifteen seconds, a timeout of 20 seconds will not trigger, even though the request will take several minutes to complete. If your goal is to cut off any request after a set amount of wall clock time, consider having a second "watcher" thread to cut off a slow request. """ #: A sentinel object representing the default timeout value DEFAULT_TIMEOUT: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT def __init__( self, total: _TYPE_TIMEOUT = None, connect: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, read: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, ) -> None: self._connect = self._validate_timeout(connect, "connect") self._read = self._validate_timeout(read, "read") self.total = self._validate_timeout(total, "total") self._start_connect: float | None = None def __repr__(self) -> str: return f"{type(self).__name__}(connect={self._connect!r}, read={self._read!r}, total={self.total!r})" # __str__ provided for backwards compatibility __str__ = __repr__ @staticmethod def resolve_default_timeout(timeout: _TYPE_TIMEOUT) -> float | None: return getdefaulttimeout() if timeout is _DEFAULT_TIMEOUT else timeout @classmethod def _validate_timeout(cls, value: _TYPE_TIMEOUT, name: str) -> _TYPE_TIMEOUT: """Check that a timeout attribute is valid. :param value: The timeout value to validate :param name: The name of the timeout attribute to validate. This is used to specify in error messages. :return: The validated and casted version of the given value. :raises ValueError: If it is a numeric value less than or equal to zero, or the type is not an integer, float, or None. """ if value is None or value is _DEFAULT_TIMEOUT: return value if isinstance(value, bool): raise ValueError( "Timeout cannot be a boolean value. It must " "be an int, float or None." ) try: float(value) except (TypeError, ValueError): raise ValueError( "Timeout value %s was %s, but it must be an " "int, float or None." % (name, value) ) from None try: if value <= 0: raise ValueError( "Attempted to set %s timeout to %s, but the " "timeout cannot be set to a value less " "than or equal to 0." % (name, value) ) except TypeError: raise ValueError( "Timeout value %s was %s, but it must be an " "int, float or None." % (name, value) ) from None return value @classmethod def from_float(cls, timeout: _TYPE_TIMEOUT) -> Timeout: """Create a new Timeout from a legacy timeout value. The timeout value used by httplib.py sets the same timeout on the connect(), and recv() socket requests. This creates a :class:`Timeout` object that sets the individual timeouts to the ``timeout`` value passed to this function. :param timeout: The legacy timeout value. :type timeout: integer, float, :attr:`urllib3.util.Timeout.DEFAULT_TIMEOUT`, or None :return: Timeout object :rtype: :class:`Timeout` """ return Timeout(read=timeout, connect=timeout) def clone(self) -> Timeout: """Create a copy of the timeout object Timeout properties are stored per-pool but each request needs a fresh Timeout object to ensure each one has its own start/stop configured. :return: a copy of the timeout object :rtype: :class:`Timeout` """ # We can't use copy.deepcopy because that will also create a new object # for _GLOBAL_DEFAULT_TIMEOUT, which socket.py uses as a sentinel to # detect the user default. return Timeout(connect=self._connect, read=self._read, total=self.total) def start_connect(self) -> float: """Start the timeout clock, used during a connect() attempt :raises urllib3.exceptions.TimeoutStateError: if you attempt to start a timer that has been started already. """ if self._start_connect is not None: raise TimeoutStateError("Timeout timer has already been started.") self._start_connect = time.monotonic() return self._start_connect def get_connect_duration(self) -> float: """Gets the time elapsed since the call to :meth:`start_connect`. :return: Elapsed time in seconds. :rtype: float :raises urllib3.exceptions.TimeoutStateError: if you attempt to get duration for a timer that hasn't been started. """ if self._start_connect is None: raise TimeoutStateError( "Can't get connect duration for timer that has not started." ) return time.monotonic() - self._start_connect @property def connect_timeout(self) -> _TYPE_TIMEOUT: """Get the value to use when setting a connection timeout. This will be a positive float or integer, the value None (never timeout), or the default system timeout. :return: Connect timeout. :rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None """ if self.total is None: return self._connect if self._connect is None or self._connect is _DEFAULT_TIMEOUT: return self.total return min(self._connect, self.total) # type: ignore[type-var] @property def read_timeout(self) -> float | None: """Get the value for the read timeout. This assumes some time has elapsed in the connection timeout and computes the read timeout appropriately. If self.total is set, the read timeout is dependent on the amount of time taken by the connect timeout. If the connection time has not been established, a :exc:`~urllib3.exceptions.TimeoutStateError` will be raised. :return: Value to use for the read timeout. :rtype: int, float or None :raises urllib3.exceptions.TimeoutStateError: If :meth:`start_connect` has not yet been called on this object. """ if ( self.total is not None and self.total is not _DEFAULT_TIMEOUT and self._read is not None and self._read is not _DEFAULT_TIMEOUT ): # In case the connect timeout has not yet been established. if self._start_connect is None: return self._read return max(0, min(self.total - self.get_connect_duration(), self._read)) elif self.total is not None and self.total is not _DEFAULT_TIMEOUT: return max(0, self.total - self.get_connect_duration()) else: return self.resolve_default_timeout(self._read) urllib3-2.0.7/src/urllib3/util/url.py0000644000000000000000000003555514513541732014340 0ustar00from __future__ import annotations import re import typing from ..exceptions import LocationParseError from .util import to_str # We only want to normalize urls with an HTTP(S) scheme. # urllib3 infers URLs without a scheme (None) to be http. _NORMALIZABLE_SCHEMES = ("http", "https", None) # Almost all of these patterns were derived from the # 'rfc3986' module: https://github.com/python-hyper/rfc3986 _PERCENT_RE = re.compile(r"%[a-fA-F0-9]{2}") _SCHEME_RE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+-]*:|/)") _URI_RE = re.compile( r"^(?:([a-zA-Z][a-zA-Z0-9+.-]*):)?" r"(?://([^\\/?#]*))?" r"([^?#]*)" r"(?:\?([^#]*))?" r"(?:#(.*))?$", re.UNICODE | re.DOTALL, ) _IPV4_PAT = r"(?:[0-9]{1,3}\.){3}[0-9]{1,3}" _HEX_PAT = "[0-9A-Fa-f]{1,4}" _LS32_PAT = "(?:{hex}:{hex}|{ipv4})".format(hex=_HEX_PAT, ipv4=_IPV4_PAT) _subs = {"hex": _HEX_PAT, "ls32": _LS32_PAT} _variations = [ # 6( h16 ":" ) ls32 "(?:%(hex)s:){6}%(ls32)s", # "::" 5( h16 ":" ) ls32 "::(?:%(hex)s:){5}%(ls32)s", # [ h16 ] "::" 4( h16 ":" ) ls32 "(?:%(hex)s)?::(?:%(hex)s:){4}%(ls32)s", # [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32 "(?:(?:%(hex)s:)?%(hex)s)?::(?:%(hex)s:){3}%(ls32)s", # [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32 "(?:(?:%(hex)s:){0,2}%(hex)s)?::(?:%(hex)s:){2}%(ls32)s", # [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32 "(?:(?:%(hex)s:){0,3}%(hex)s)?::%(hex)s:%(ls32)s", # [ *4( h16 ":" ) h16 ] "::" ls32 "(?:(?:%(hex)s:){0,4}%(hex)s)?::%(ls32)s", # [ *5( h16 ":" ) h16 ] "::" h16 "(?:(?:%(hex)s:){0,5}%(hex)s)?::%(hex)s", # [ *6( h16 ":" ) h16 ] "::" "(?:(?:%(hex)s:){0,6}%(hex)s)?::", ] _UNRESERVED_PAT = r"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._\-~" _IPV6_PAT = "(?:" + "|".join([x % _subs for x in _variations]) + ")" _ZONE_ID_PAT = "(?:%25|%)(?:[" + _UNRESERVED_PAT + "]|%[a-fA-F0-9]{2})+" _IPV6_ADDRZ_PAT = r"\[" + _IPV6_PAT + r"(?:" + _ZONE_ID_PAT + r")?\]" _REG_NAME_PAT = r"(?:[^\[\]%:/?#]|%[a-fA-F0-9]{2})*" _TARGET_RE = re.compile(r"^(/[^?#]*)(?:\?([^#]*))?(?:#.*)?$") _IPV4_RE = re.compile("^" + _IPV4_PAT + "$") _IPV6_RE = re.compile("^" + _IPV6_PAT + "$") _IPV6_ADDRZ_RE = re.compile("^" + _IPV6_ADDRZ_PAT + "$") _BRACELESS_IPV6_ADDRZ_RE = re.compile("^" + _IPV6_ADDRZ_PAT[2:-2] + "$") _ZONE_ID_RE = re.compile("(" + _ZONE_ID_PAT + r")\]$") _HOST_PORT_PAT = ("^(%s|%s|%s)(?::0*?(|0|[1-9][0-9]{0,4}))?$") % ( _REG_NAME_PAT, _IPV4_PAT, _IPV6_ADDRZ_PAT, ) _HOST_PORT_RE = re.compile(_HOST_PORT_PAT, re.UNICODE | re.DOTALL) _UNRESERVED_CHARS = set( "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-~" ) _SUB_DELIM_CHARS = set("!$&'()*+,;=") _USERINFO_CHARS = _UNRESERVED_CHARS | _SUB_DELIM_CHARS | {":"} _PATH_CHARS = _USERINFO_CHARS | {"@", "/"} _QUERY_CHARS = _FRAGMENT_CHARS = _PATH_CHARS | {"?"} class Url( typing.NamedTuple( "Url", [ ("scheme", typing.Optional[str]), ("auth", typing.Optional[str]), ("host", typing.Optional[str]), ("port", typing.Optional[int]), ("path", typing.Optional[str]), ("query", typing.Optional[str]), ("fragment", typing.Optional[str]), ], ) ): """ Data structure for representing an HTTP URL. Used as a return value for :func:`parse_url`. Both the scheme and host are normalized as they are both case-insensitive according to RFC 3986. """ def __new__( # type: ignore[no-untyped-def] cls, scheme: str | None = None, auth: str | None = None, host: str | None = None, port: int | None = None, path: str | None = None, query: str | None = None, fragment: str | None = None, ): if path and not path.startswith("/"): path = "/" + path if scheme is not None: scheme = scheme.lower() return super().__new__(cls, scheme, auth, host, port, path, query, fragment) @property def hostname(self) -> str | None: """For backwards-compatibility with urlparse. We're nice like that.""" return self.host @property def request_uri(self) -> str: """Absolute path including the query string.""" uri = self.path or "/" if self.query is not None: uri += "?" + self.query return uri @property def authority(self) -> str | None: """ Authority component as defined in RFC 3986 3.2. This includes userinfo (auth), host and port. i.e. userinfo@host:port """ userinfo = self.auth netloc = self.netloc if netloc is None or userinfo is None: return netloc else: return f"{userinfo}@{netloc}" @property def netloc(self) -> str | None: """ Network location including host and port. If you need the equivalent of urllib.parse's ``netloc``, use the ``authority`` property instead. """ if self.host is None: return None if self.port: return f"{self.host}:{self.port}" return self.host @property def url(self) -> str: """ Convert self into a url This function should more or less round-trip with :func:`.parse_url`. The returned url may not be exactly the same as the url inputted to :func:`.parse_url`, but it should be equivalent by the RFC (e.g., urls with a blank port will have : removed). Example: .. code-block:: python import urllib3 U = urllib3.util.parse_url("https://google.com/mail/") print(U.url) # "https://google.com/mail/" print( urllib3.util.Url("https", "username:password", "host.com", 80, "/path", "query", "fragment" ).url ) # "https://username:password@host.com:80/path?query#fragment" """ scheme, auth, host, port, path, query, fragment = self url = "" # We use "is not None" we want things to happen with empty strings (or 0 port) if scheme is not None: url += scheme + "://" if auth is not None: url += auth + "@" if host is not None: url += host if port is not None: url += ":" + str(port) if path is not None: url += path if query is not None: url += "?" + query if fragment is not None: url += "#" + fragment return url def __str__(self) -> str: return self.url @typing.overload def _encode_invalid_chars( component: str, allowed_chars: typing.Container[str] ) -> str: # Abstract ... @typing.overload def _encode_invalid_chars( component: None, allowed_chars: typing.Container[str] ) -> None: # Abstract ... def _encode_invalid_chars( component: str | None, allowed_chars: typing.Container[str] ) -> str | None: """Percent-encodes a URI component without reapplying onto an already percent-encoded component. """ if component is None: return component component = to_str(component) # Normalize existing percent-encoded bytes. # Try to see if the component we're encoding is already percent-encoded # so we can skip all '%' characters but still encode all others. component, percent_encodings = _PERCENT_RE.subn( lambda match: match.group(0).upper(), component ) uri_bytes = component.encode("utf-8", "surrogatepass") is_percent_encoded = percent_encodings == uri_bytes.count(b"%") encoded_component = bytearray() for i in range(0, len(uri_bytes)): # Will return a single character bytestring byte = uri_bytes[i : i + 1] byte_ord = ord(byte) if (is_percent_encoded and byte == b"%") or ( byte_ord < 128 and byte.decode() in allowed_chars ): encoded_component += byte continue encoded_component.extend(b"%" + (hex(byte_ord)[2:].encode().zfill(2).upper())) return encoded_component.decode() def _remove_path_dot_segments(path: str) -> str: # See http://tools.ietf.org/html/rfc3986#section-5.2.4 for pseudo-code segments = path.split("/") # Turn the path into a list of segments output = [] # Initialize the variable to use to store output for segment in segments: # '.' is the current directory, so ignore it, it is superfluous if segment == ".": continue # Anything other than '..', should be appended to the output if segment != "..": output.append(segment) # In this case segment == '..', if we can, we should pop the last # element elif output: output.pop() # If the path starts with '/' and the output is empty or the first string # is non-empty if path.startswith("/") and (not output or output[0]): output.insert(0, "") # If the path starts with '/.' or '/..' ensure we add one more empty # string to add a trailing '/' if path.endswith(("/.", "/..")): output.append("") return "/".join(output) @typing.overload def _normalize_host(host: None, scheme: str | None) -> None: ... @typing.overload def _normalize_host(host: str, scheme: str | None) -> str: ... def _normalize_host(host: str | None, scheme: str | None) -> str | None: if host: if scheme in _NORMALIZABLE_SCHEMES: is_ipv6 = _IPV6_ADDRZ_RE.match(host) if is_ipv6: # IPv6 hosts of the form 'a::b%zone' are encoded in a URL as # such per RFC 6874: 'a::b%25zone'. Unquote the ZoneID # separator as necessary to return a valid RFC 4007 scoped IP. match = _ZONE_ID_RE.search(host) if match: start, end = match.span(1) zone_id = host[start:end] if zone_id.startswith("%25") and zone_id != "%25": zone_id = zone_id[3:] else: zone_id = zone_id[1:] zone_id = _encode_invalid_chars(zone_id, _UNRESERVED_CHARS) return f"{host[:start].lower()}%{zone_id}{host[end:]}" else: return host.lower() elif not _IPV4_RE.match(host): return to_str( b".".join([_idna_encode(label) for label in host.split(".")]), "ascii", ) return host def _idna_encode(name: str) -> bytes: if not name.isascii(): try: import idna except ImportError: raise LocationParseError( "Unable to parse URL without the 'idna' module" ) from None try: return idna.encode(name.lower(), strict=True, std3_rules=True) except idna.IDNAError: raise LocationParseError( f"Name '{name}' is not a valid IDNA label" ) from None return name.lower().encode("ascii") def _encode_target(target: str) -> str: """Percent-encodes a request target so that there are no invalid characters Pre-condition for this function is that 'target' must start with '/'. If that is the case then _TARGET_RE will always produce a match. """ match = _TARGET_RE.match(target) if not match: # Defensive: raise LocationParseError(f"{target!r} is not a valid request URI") path, query = match.groups() encoded_target = _encode_invalid_chars(path, _PATH_CHARS) if query is not None: query = _encode_invalid_chars(query, _QUERY_CHARS) encoded_target += "?" + query return encoded_target def parse_url(url: str) -> Url: """ Given a url, return a parsed :class:`.Url` namedtuple. Best-effort is performed to parse incomplete urls. Fields not provided will be None. This parser is RFC 3986 and RFC 6874 compliant. The parser logic and helper functions are based heavily on work done in the ``rfc3986`` module. :param str url: URL to parse into a :class:`.Url` namedtuple. Partly backwards-compatible with :mod:`urllib.parse`. Example: .. code-block:: python import urllib3 print( urllib3.util.parse_url('http://google.com/mail/')) # Url(scheme='http', host='google.com', port=None, path='/mail/', ...) print( urllib3.util.parse_url('google.com:80')) # Url(scheme=None, host='google.com', port=80, path=None, ...) print( urllib3.util.parse_url('/foo?bar')) # Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...) """ if not url: # Empty return Url() source_url = url if not _SCHEME_RE.search(url): url = "//" + url scheme: str | None authority: str | None auth: str | None host: str | None port: str | None port_int: int | None path: str | None query: str | None fragment: str | None try: scheme, authority, path, query, fragment = _URI_RE.match(url).groups() # type: ignore[union-attr] normalize_uri = scheme is None or scheme.lower() in _NORMALIZABLE_SCHEMES if scheme: scheme = scheme.lower() if authority: auth, _, host_port = authority.rpartition("@") auth = auth or None host, port = _HOST_PORT_RE.match(host_port).groups() # type: ignore[union-attr] if auth and normalize_uri: auth = _encode_invalid_chars(auth, _USERINFO_CHARS) if port == "": port = None else: auth, host, port = None, None, None if port is not None: port_int = int(port) if not (0 <= port_int <= 65535): raise LocationParseError(url) else: port_int = None host = _normalize_host(host, scheme) if normalize_uri and path: path = _remove_path_dot_segments(path) path = _encode_invalid_chars(path, _PATH_CHARS) if normalize_uri and query: query = _encode_invalid_chars(query, _QUERY_CHARS) if normalize_uri and fragment: fragment = _encode_invalid_chars(fragment, _FRAGMENT_CHARS) except (ValueError, AttributeError) as e: raise LocationParseError(source_url) from e # For the sake of backwards compatibility we put empty # string values for path if there are any defined values # beyond the path in the URL. # TODO: Remove this when we break backwards compatibility. if not path: if query is not None or fragment is not None: path = "" else: path = None return Url( scheme=scheme, auth=auth, host=host, port=port_int, path=path, query=query, fragment=fragment, ) urllib3-2.0.7/src/urllib3/util/util.py0000644000000000000000000000217214513541732014500 0ustar00from __future__ import annotations import typing from types import TracebackType def to_bytes( x: str | bytes, encoding: str | None = None, errors: str | None = None ) -> bytes: if isinstance(x, bytes): return x elif not isinstance(x, str): raise TypeError(f"not expecting type {type(x).__name__}") if encoding or errors: return x.encode(encoding or "utf-8", errors=errors or "strict") return x.encode() def to_str( x: str | bytes, encoding: str | None = None, errors: str | None = None ) -> str: if isinstance(x, str): return x elif not isinstance(x, bytes): raise TypeError(f"not expecting type {type(x).__name__}") if encoding or errors: return x.decode(encoding or "utf-8", errors=errors or "strict") return x.decode() def reraise( tp: type[BaseException] | None, value: BaseException, tb: TracebackType | None = None, ) -> typing.NoReturn: try: if value.__traceback__ is not tb: raise value.with_traceback(tb) raise value finally: value = None # type: ignore[assignment] tb = None urllib3-2.0.7/src/urllib3/util/wait.py0000644000000000000000000001050714513541732014470 0ustar00from __future__ import annotations import select import socket from functools import partial __all__ = ["wait_for_read", "wait_for_write"] # How should we wait on sockets? # # There are two types of APIs you can use for waiting on sockets: the fancy # modern stateful APIs like epoll/kqueue, and the older stateless APIs like # select/poll. The stateful APIs are more efficient when you have a lots of # sockets to keep track of, because you can set them up once and then use them # lots of times. But we only ever want to wait on a single socket at a time # and don't want to keep track of state, so the stateless APIs are actually # more efficient. So we want to use select() or poll(). # # Now, how do we choose between select() and poll()? On traditional Unixes, # select() has a strange calling convention that makes it slow, or fail # altogether, for high-numbered file descriptors. The point of poll() is to fix # that, so on Unixes, we prefer poll(). # # On Windows, there is no poll() (or at least Python doesn't provide a wrapper # for it), but that's OK, because on Windows, select() doesn't have this # strange calling convention; plain select() works fine. # # So: on Windows we use select(), and everywhere else we use poll(). We also # fall back to select() in case poll() is somehow broken or missing. def select_wait_for_socket( sock: socket.socket, read: bool = False, write: bool = False, timeout: float | None = None, ) -> bool: if not read and not write: raise RuntimeError("must specify at least one of read=True, write=True") rcheck = [] wcheck = [] if read: rcheck.append(sock) if write: wcheck.append(sock) # When doing a non-blocking connect, most systems signal success by # marking the socket writable. Windows, though, signals success by marked # it as "exceptional". We paper over the difference by checking the write # sockets for both conditions. (The stdlib selectors module does the same # thing.) fn = partial(select.select, rcheck, wcheck, wcheck) rready, wready, xready = fn(timeout) return bool(rready or wready or xready) def poll_wait_for_socket( sock: socket.socket, read: bool = False, write: bool = False, timeout: float | None = None, ) -> bool: if not read and not write: raise RuntimeError("must specify at least one of read=True, write=True") mask = 0 if read: mask |= select.POLLIN if write: mask |= select.POLLOUT poll_obj = select.poll() poll_obj.register(sock, mask) # For some reason, poll() takes timeout in milliseconds def do_poll(t: float | None) -> list[tuple[int, int]]: if t is not None: t *= 1000 return poll_obj.poll(t) return bool(do_poll(timeout)) def _have_working_poll() -> bool: # Apparently some systems have a select.poll that fails as soon as you try # to use it, either due to strange configuration or broken monkeypatching # from libraries like eventlet/greenlet. try: poll_obj = select.poll() poll_obj.poll(0) except (AttributeError, OSError): return False else: return True def wait_for_socket( sock: socket.socket, read: bool = False, write: bool = False, timeout: float | None = None, ) -> bool: # We delay choosing which implementation to use until the first time we're # called. We could do it at import time, but then we might make the wrong # decision if someone goes wild with monkeypatching select.poll after # we're imported. global wait_for_socket if _have_working_poll(): wait_for_socket = poll_wait_for_socket elif hasattr(select, "select"): wait_for_socket = select_wait_for_socket return wait_for_socket(sock, read, write, timeout) def wait_for_read(sock: socket.socket, timeout: float | None = None) -> bool: """Waits for reading to be available on a given socket. Returns True if the socket is readable, or False if the timeout expired. """ return wait_for_socket(sock, read=True, timeout=timeout) def wait_for_write(sock: socket.socket, timeout: float | None = None) -> bool: """Waits for writing to be available on a given socket. Returns True if the socket is readable, or False if the timeout expired. """ return wait_for_socket(sock, write=True, timeout=timeout) urllib3-2.0.7/test/__init__.py0000644000000000000000000002501314513541732013120 0ustar00from __future__ import annotations import errno import importlib.util import logging import os import platform import socket import sys import typing import warnings from collections.abc import Sequence from importlib.abc import Loader, MetaPathFinder from importlib.machinery import ModuleSpec from types import ModuleType, TracebackType import pytest try: try: import brotlicffi as brotli # type: ignore[import] except ImportError: import brotli # type: ignore[import] except ImportError: brotli = None try: import zstandard as zstd # type: ignore[import] except ImportError: zstd = None import functools from urllib3 import util from urllib3.connectionpool import ConnectionPool from urllib3.exceptions import HTTPWarning from urllib3.util import ssl_ try: import urllib3.contrib.pyopenssl as pyopenssl except ImportError: pyopenssl = None # type: ignore[assignment] if typing.TYPE_CHECKING: import ssl from typing_extensions import Literal _RT = typing.TypeVar("_RT") # return type _TestFuncT = typing.TypeVar("_TestFuncT", bound=typing.Callable[..., typing.Any]) # We need a host that will not immediately close the connection with a TCP # Reset. if platform.system() == "Windows": # Reserved loopback subnet address TARPIT_HOST = "127.0.0.0" else: # Reserved internet scoped address # https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml TARPIT_HOST = "240.0.0.0" # (Arguments for socket, is it IPv6 address?) VALID_SOURCE_ADDRESSES = [(("::1", 0), True), (("127.0.0.1", 0), False)] # RFC 5737: 192.0.2.0/24 is for testing only. # RFC 3849: 2001:db8::/32 is for documentation only. INVALID_SOURCE_ADDRESSES = [(("192.0.2.255", 0), False), (("2001:db8::1", 0), True)] # We use timeouts in three different ways in our tests # # 1. To make sure that the operation timeouts, we can use a short timeout. # 2. To make sure that the test does not hang even if the operation should succeed, we # want to use a long timeout, even more so on CI where tests can be really slow # 3. To test our timeout logic by using two different values, eg. by using different # values at the pool level and at the request level. SHORT_TIMEOUT = 0.001 LONG_TIMEOUT = 0.01 if os.environ.get("CI") or os.environ.get("GITHUB_ACTIONS") == "true": LONG_TIMEOUT = 0.5 DUMMY_POOL = ConnectionPool("dummy") def _can_resolve(host: str) -> bool: """Returns True if the system can resolve host to an address.""" try: socket.getaddrinfo(host, None, socket.AF_UNSPEC) return True except socket.gaierror: return False def has_alpn(ctx_cls: type[ssl.SSLContext] | None = None) -> bool: """Detect if ALPN support is enabled.""" ctx_cls = ctx_cls or util.SSLContext ctx = ctx_cls(protocol=ssl_.PROTOCOL_TLS) # type: ignore[misc, attr-defined] try: if hasattr(ctx, "set_alpn_protocols"): ctx.set_alpn_protocols(ssl_.ALPN_PROTOCOLS) return True except NotImplementedError: pass return False # Some systems might not resolve "localhost." correctly. # See https://github.com/urllib3/urllib3/issues/1809 and # https://github.com/urllib3/urllib3/pull/1475#issuecomment-440788064. RESOLVES_LOCALHOST_FQDN = _can_resolve("localhost.") def clear_warnings(cls: type[Warning] = HTTPWarning) -> None: new_filters = [] for f in warnings.filters: if issubclass(f[2], cls): continue new_filters.append(f) warnings.filters[:] = new_filters # type: ignore[index] def setUp() -> None: clear_warnings() warnings.simplefilter("ignore", HTTPWarning) def notWindows() -> typing.Callable[[_TestFuncT], _TestFuncT]: """Skips this test on Windows""" return pytest.mark.skipif( platform.system() == "Windows", reason="Test does not run on Windows", ) def onlyBrotli() -> typing.Callable[[_TestFuncT], _TestFuncT]: return pytest.mark.skipif( brotli is None, reason="only run if brotli library is present" ) def notBrotli() -> typing.Callable[[_TestFuncT], _TestFuncT]: return pytest.mark.skipif( brotli is not None, reason="only run if a brotli library is absent" ) def onlyZstd() -> typing.Callable[[_TestFuncT], _TestFuncT]: return pytest.mark.skipif( zstd is None, reason="only run if a python-zstandard library is installed" ) def notZstd() -> typing.Callable[[_TestFuncT], _TestFuncT]: return pytest.mark.skipif( zstd is not None, reason="only run if a python-zstandard library is not installed", ) # Hack to make pytest evaluate a condition at test runtime instead of collection time. def lazy_condition(condition: typing.Callable[[], bool]) -> bool: class LazyCondition: def __bool__(self) -> bool: return condition() return typing.cast(bool, LazyCondition()) def onlySecureTransport() -> typing.Callable[[_TestFuncT], _TestFuncT]: """Runs this test when SecureTransport is in use.""" return pytest.mark.skipif( lazy_condition(lambda: not ssl_.IS_SECURETRANSPORT), reason="Test only runs with SecureTransport", ) def notSecureTransport() -> typing.Callable[[_TestFuncT], _TestFuncT]: """Skips this test when SecureTransport is in use.""" return pytest.mark.skipif( lazy_condition(lambda: ssl_.IS_SECURETRANSPORT), reason="Test does not run with SecureTransport", ) _requires_network_has_route = None def requires_network() -> typing.Callable[[_TestFuncT], _TestFuncT]: """Helps you skip tests that require the network""" def _is_unreachable_err(err: Exception) -> bool: return getattr(err, "errno", None) in ( errno.ENETUNREACH, errno.EHOSTUNREACH, # For OSX ) def _has_route() -> bool: try: sock = socket.create_connection((TARPIT_HOST, 80), 0.0001) sock.close() return True except socket.timeout: return True except OSError as e: if _is_unreachable_err(e): return False else: raise global _requires_network_has_route if _requires_network_has_route is None: _requires_network_has_route = _has_route() return pytest.mark.skipif( not _requires_network_has_route, reason="Can't run the test because the network is unreachable", ) def requires_ssl_context_keyfile_password() -> ( typing.Callable[[_TestFuncT], _TestFuncT] ): return pytest.mark.skipif( lazy_condition(lambda: ssl_.IS_SECURETRANSPORT), reason="Test requires password parameter for SSLContext.load_cert_chain()", ) def resolvesLocalhostFQDN() -> typing.Callable[[_TestFuncT], _TestFuncT]: """Test requires successful resolving of 'localhost.'""" return pytest.mark.skipif( not RESOLVES_LOCALHOST_FQDN, reason="Can't resolve localhost.", ) def withPyOpenSSL(test: typing.Callable[..., _RT]) -> typing.Callable[..., _RT]: @functools.wraps(test) def wrapper(*args: typing.Any, **kwargs: typing.Any) -> _RT: if not pyopenssl: pytest.skip("pyopenssl not available, skipping test.") return test(*args, **kwargs) pyopenssl.inject_into_urllib3() result = test(*args, **kwargs) pyopenssl.extract_from_urllib3() return result return wrapper class _ListHandler(logging.Handler): def __init__(self) -> None: super().__init__() self.records: list[logging.LogRecord] = [] def emit(self, record: logging.LogRecord) -> None: self.records.append(record) class LogRecorder: def __init__(self, target: logging.Logger = logging.root) -> None: super().__init__() self._target = target self._handler = _ListHandler() @property def records(self) -> list[logging.LogRecord]: return self._handler.records def install(self) -> None: self._target.addHandler(self._handler) def uninstall(self) -> None: self._target.removeHandler(self._handler) def __enter__(self) -> list[logging.LogRecord]: self.install() return self.records def __exit__( self, exc_type: type[BaseException] | None, exc_value: BaseException | None, traceback: TracebackType | None, ) -> Literal[False]: self.uninstall() return False class ImportBlockerLoader(Loader): def __init__(self, fullname: str) -> None: self._fullname = fullname def load_module(self, fullname: str) -> ModuleType: raise ImportError(f"import of {fullname} is blocked") def exec_module(self, module: ModuleType) -> None: raise ImportError(f"import of {self._fullname} is blocked") class ImportBlocker(MetaPathFinder): """ Block Imports To be placed on ``sys.meta_path``. This ensures that the modules specified cannot be imported, even if they are a builtin. """ def __init__(self, *namestoblock: str) -> None: self.namestoblock = namestoblock def find_module( self, fullname: str, path: typing.Sequence[bytes | str] | None = None ) -> Loader | None: if fullname in self.namestoblock: return ImportBlockerLoader(fullname) return None def find_spec( self, fullname: str, path: Sequence[bytes | str] | None, target: ModuleType | None = None, ) -> ModuleSpec | None: loader = self.find_module(fullname, path) if loader is None: return None return importlib.util.spec_from_loader(fullname, loader) class ModuleStash(MetaPathFinder): """ Stashes away previously imported modules If we reimport a module the data from coverage is lost, so we reuse the old modules """ def __init__( self, namespace: str, modules: dict[str, ModuleType] = sys.modules ) -> None: self.namespace = namespace self.modules = modules self._data: dict[str, ModuleType] = {} def stash(self) -> None: if self.namespace in self.modules: self._data[self.namespace] = self.modules.pop(self.namespace) for module in list(self.modules.keys()): if module.startswith(self.namespace + "."): self._data[module] = self.modules.pop(module) def pop(self) -> None: self.modules.pop(self.namespace, None) for module in list(self.modules.keys()): if module.startswith(self.namespace + "."): self.modules.pop(module) self.modules.update(self._data) urllib3-2.0.7/test/conftest.py0000644000000000000000000002571314513541732013215 0ustar00from __future__ import annotations import asyncio import contextlib import socket import ssl import typing from pathlib import Path import pytest import trustme from tornado import web from dummyserver.handlers import TestingApp from dummyserver.proxy import ProxyHandler from dummyserver.server import HAS_IPV6, run_loop_in_thread, run_tornado_app from dummyserver.testcase import HTTPSDummyServerTestCase from urllib3.util import ssl_ from .tz_stub import stub_timezone_ctx class ServerConfig(typing.NamedTuple): scheme: str host: str port: int ca_certs: str @property def base_url(self) -> str: host = self.host if ":" in host: host = f"[{host}]" return f"{self.scheme}://{host}:{self.port}" def _write_cert_to_dir( cert: trustme.LeafCert, tmpdir: Path, file_prefix: str = "server" ) -> dict[str, str]: cert_path = str(tmpdir / ("%s.pem" % file_prefix)) key_path = str(tmpdir / ("%s.key" % file_prefix)) cert.private_key_pem.write_to_path(key_path) cert.cert_chain_pems[0].write_to_path(cert_path) certs = {"keyfile": key_path, "certfile": cert_path} return certs @contextlib.contextmanager def run_server_in_thread( scheme: str, host: str, tmpdir: Path, ca: trustme.CA, server_cert: trustme.LeafCert ) -> typing.Generator[ServerConfig, None, None]: ca_cert_path = str(tmpdir / "ca.pem") ca.cert_pem.write_to_path(ca_cert_path) server_certs = _write_cert_to_dir(server_cert, tmpdir) with run_loop_in_thread() as io_loop: async def run_app() -> int: app = web.Application([(r".*", TestingApp)]) server, port = run_tornado_app(app, server_certs, scheme, host) return port port = asyncio.run_coroutine_threadsafe( run_app(), io_loop.asyncio_loop # type: ignore[attr-defined] ).result() yield ServerConfig("https", host, port, ca_cert_path) @contextlib.contextmanager def run_server_and_proxy_in_thread( proxy_scheme: str, proxy_host: str, tmpdir: Path, ca: trustme.CA, proxy_cert: trustme.LeafCert, server_cert: trustme.LeafCert, ) -> typing.Generator[tuple[ServerConfig, ServerConfig], None, None]: ca_cert_path = str(tmpdir / "ca.pem") ca.cert_pem.write_to_path(ca_cert_path) server_certs = _write_cert_to_dir(server_cert, tmpdir) proxy_certs = _write_cert_to_dir(proxy_cert, tmpdir, "proxy") with run_loop_in_thread() as io_loop: async def run_app() -> tuple[ServerConfig, ServerConfig]: app = web.Application([(r".*", TestingApp)]) server_app, port = run_tornado_app(app, server_certs, "https", "localhost") server_config = ServerConfig("https", "localhost", port, ca_cert_path) proxy = web.Application([(r".*", ProxyHandler)]) proxy_app, proxy_port = run_tornado_app( proxy, proxy_certs, proxy_scheme, proxy_host ) proxy_config = ServerConfig( proxy_scheme, proxy_host, proxy_port, ca_cert_path ) return proxy_config, server_config proxy_config, server_config = asyncio.run_coroutine_threadsafe( run_app(), io_loop.asyncio_loop # type: ignore[attr-defined] ).result() yield (proxy_config, server_config) @pytest.fixture(params=["localhost", "127.0.0.1", "::1"]) def loopback_host(request: typing.Any) -> typing.Generator[str, None, None]: host = request.param if host == "::1" and not HAS_IPV6: pytest.skip("Test requires IPv6 on loopback") yield host @pytest.fixture() def san_server( loopback_host: str, tmp_path_factory: pytest.TempPathFactory ) -> typing.Generator[ServerConfig, None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() server_cert = ca.issue_cert(loopback_host) with run_server_in_thread("https", loopback_host, tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture() def no_san_server( loopback_host: str, tmp_path_factory: pytest.TempPathFactory ) -> typing.Generator[ServerConfig, None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() server_cert = ca.issue_cert(common_name=loopback_host) with run_server_in_thread("https", loopback_host, tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture() def no_san_server_with_different_commmon_name( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[ServerConfig, None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() server_cert = ca.issue_cert(common_name="example.com") with run_server_in_thread("https", "localhost", tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture def san_proxy_with_server( loopback_host: str, tmp_path_factory: pytest.TempPathFactory ) -> typing.Generator[tuple[ServerConfig, ServerConfig], None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() proxy_cert = ca.issue_cert(loopback_host) server_cert = ca.issue_cert("localhost") with run_server_and_proxy_in_thread( "https", loopback_host, tmpdir, ca, proxy_cert, server_cert ) as cfg: yield cfg @pytest.fixture def no_san_proxy_with_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[tuple[ServerConfig, ServerConfig], None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # only common name, no subject alternative names proxy_cert = ca.issue_cert(common_name="localhost") server_cert = ca.issue_cert("localhost") with run_server_and_proxy_in_thread( "https", "localhost", tmpdir, ca, proxy_cert, server_cert ) as cfg: yield cfg @pytest.fixture def no_localhost_san_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[ServerConfig, None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # non localhost common name server_cert = ca.issue_cert("example.com") with run_server_in_thread("https", "localhost", tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture def ipv4_san_proxy_with_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[tuple[ServerConfig, ServerConfig], None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # IP address in Subject Alternative Name proxy_cert = ca.issue_cert("127.0.0.1") server_cert = ca.issue_cert("localhost") with run_server_and_proxy_in_thread( "https", "127.0.0.1", tmpdir, ca, proxy_cert, server_cert ) as cfg: yield cfg @pytest.fixture def ipv6_san_proxy_with_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[tuple[ServerConfig, ServerConfig], None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # IP addresses in Subject Alternative Name proxy_cert = ca.issue_cert("::1") server_cert = ca.issue_cert("localhost") with run_server_and_proxy_in_thread( "https", "::1", tmpdir, ca, proxy_cert, server_cert ) as cfg: yield cfg @pytest.fixture def ipv4_san_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[ServerConfig, None, None]: tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # IP address in Subject Alternative Name server_cert = ca.issue_cert("127.0.0.1") with run_server_in_thread("https", "127.0.0.1", tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture def ipv6_san_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[ServerConfig, None, None]: if not HAS_IPV6: pytest.skip("Only runs on IPv6 systems") tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # IP address in Subject Alternative Name server_cert = ca.issue_cert("::1") with run_server_in_thread("https", "::1", tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture def ipv6_no_san_server( tmp_path_factory: pytest.TempPathFactory, ) -> typing.Generator[ServerConfig, None, None]: if not HAS_IPV6: pytest.skip("Only runs on IPv6 systems") tmpdir = tmp_path_factory.mktemp("certs") ca = trustme.CA() # IP address in Common Name server_cert = ca.issue_cert(common_name="::1") with run_server_in_thread("https", "::1", tmpdir, ca, server_cert) as cfg: yield cfg @pytest.fixture def stub_timezone(request: pytest.FixtureRequest) -> typing.Generator[None, None, None]: """ A pytest fixture that runs the test with a stub timezone. """ with stub_timezone_ctx(request.param): yield @pytest.fixture(scope="session") def supported_tls_versions() -> typing.AbstractSet[str | None]: # We have to create an actual TLS connection # to test if the TLS version is not disabled by # OpenSSL config. Ubuntu 20.04 specifically # disables TLSv1 and TLSv1.1. tls_versions = set() _server = HTTPSDummyServerTestCase() _server._start_server() for _ssl_version_name, min_max_version in ( ("PROTOCOL_TLSv1", ssl.TLSVersion.TLSv1), ("PROTOCOL_TLSv1_1", ssl.TLSVersion.TLSv1_1), ("PROTOCOL_TLSv1_2", ssl.TLSVersion.TLSv1_2), ("PROTOCOL_TLS", None), ): _ssl_version = getattr(ssl, _ssl_version_name, 0) if _ssl_version == 0: continue _sock = socket.create_connection((_server.host, _server.port)) try: _sock = ssl_.ssl_wrap_socket( _sock, ssl_context=ssl_.create_urllib3_context( cert_reqs=ssl.CERT_NONE, ssl_minimum_version=min_max_version, ssl_maximum_version=min_max_version, ), ) except ssl.SSLError: pass else: tls_versions.add(_sock.version()) _sock.close() _server._stop_server() return tls_versions @pytest.fixture(scope="function") def requires_tlsv1(supported_tls_versions: typing.AbstractSet[str]) -> None: """Test requires TLSv1 available""" if not hasattr(ssl, "PROTOCOL_TLSv1") or "TLSv1" not in supported_tls_versions: pytest.skip("Test requires TLSv1") @pytest.fixture(scope="function") def requires_tlsv1_1(supported_tls_versions: typing.AbstractSet[str]) -> None: """Test requires TLSv1.1 available""" if not hasattr(ssl, "PROTOCOL_TLSv1_1") or "TLSv1.1" not in supported_tls_versions: pytest.skip("Test requires TLSv1.1") @pytest.fixture(scope="function") def requires_tlsv1_2(supported_tls_versions: typing.AbstractSet[str]) -> None: """Test requires TLSv1.2 available""" if not hasattr(ssl, "PROTOCOL_TLSv1_2") or "TLSv1.2" not in supported_tls_versions: pytest.skip("Test requires TLSv1.2") @pytest.fixture(scope="function") def requires_tlsv1_3(supported_tls_versions: typing.AbstractSet[str]) -> None: """Test requires TLSv1.3 available""" if ( not getattr(ssl, "HAS_TLSv1_3", False) or "TLSv1.3" not in supported_tls_versions ): pytest.skip("Test requires TLSv1.3") urllib3-2.0.7/test/port_helpers.py0000644000000000000000000001411614513541732014071 0ustar00# These helpers are copied from test/support/socket_helper.py in the Python 3.9 standard # library test suite. from __future__ import annotations import socket # Don't use "localhost", since resolving it uses the DNS under recent # Windows versions (see issue #18792). HOST = "127.0.0.1" HOSTv6 = "::1" def find_unused_port( family: socket.AddressFamily = socket.AF_INET, socktype: socket.SocketKind = socket.SOCK_STREAM, ) -> int: """Returns an unused port that should be suitable for binding. This is achieved by creating a temporary socket with the same family and type as the 'sock' parameter (default is AF_INET, SOCK_STREAM), and binding it to the specified host address (defaults to 0.0.0.0) with the port set to 0, eliciting an unused ephemeral port from the OS. The temporary socket is then closed and deleted, and the ephemeral port is returned. Either this method or bind_port() should be used for any tests where a server socket needs to be bound to a particular port for the duration of the test. Which one to use depends on whether the calling code is creating a python socket, or if an unused port needs to be provided in a constructor or passed to an external program (i.e. the -accept argument to openssl's s_server mode). Always prefer bind_port() over find_unused_port() where possible. Hard coded ports should *NEVER* be used. As soon as a server socket is bound to a hard coded port, the ability to run multiple instances of the test simultaneously on the same host is compromised, which makes the test a ticking time bomb in a buildbot environment. On Unix buildbots, this may simply manifest as a failed test, which can be recovered from without intervention in most cases, but on Windows, the entire python process can completely and utterly wedge, requiring someone to log in to the buildbot and manually kill the affected process. (This is easy to reproduce on Windows, unfortunately, and can be traced to the SO_REUSEADDR socket option having different semantics on Windows versus Unix/Linux. On Unix, you can't have two AF_INET SOCK_STREAM sockets bind, listen and then accept connections on identical host/ports. An EADDRINUSE OSError will be raised at some point (depending on the platform and the order bind and listen were called on each socket). However, on Windows, if SO_REUSEADDR is set on the sockets, no EADDRINUSE will ever be raised when attempting to bind two identical host/ports. When accept() is called on each socket, the second caller's process will steal the port from the first caller, leaving them both in an awkwardly wedged state where they'll no longer respond to any signals or graceful kills, and must be forcibly killed via OpenProcess()/TerminateProcess(). The solution on Windows is to use the SO_EXCLUSIVEADDRUSE socket option instead of SO_REUSEADDR, which effectively affords the same semantics as SO_REUSEADDR on Unix. Given the propensity of Unix developers in the Open Source world compared to Windows ones, this is a common mistake. A quick look over OpenSSL's 0.9.8g source shows that they use SO_REUSEADDR when openssl.exe is called with the 's_server' option, for example. See http://bugs.python.org/issue2550 for more info. The following site also has a very thorough description about the implications of both REUSEADDR and EXCLUSIVEADDRUSE on Windows: http://msdn2.microsoft.com/en-us/library/ms740621(VS.85).aspx) XXX: although this approach is a vast improvement on previous attempts to elicit unused ports, it rests heavily on the assumption that the ephemeral port returned to us by the OS won't immediately be dished back out to some other process when we close and delete our temporary socket but before our calling code has a chance to bind the returned port. We can deal with this issue if/when we come across it.""" with socket.socket(family, socktype) as tempsock: port = bind_port(tempsock) del tempsock return port def bind_port(sock: socket.socket, host: str = HOST) -> int: """Bind the socket to a free port and return the port number. Relies on ephemeral ports in order to ensure we are using an unbound port. This is important as many tests may be running simultaneously, especially in a buildbot environment. This method raises an exception if the sock.family is AF_INET and sock.type is SOCK_STREAM, *and* the socket has SO_REUSEADDR or SO_REUSEPORT set on it. Tests should *never* set these socket options for TCP/IP sockets. The only case for setting these options is testing multicasting via multiple UDP sockets. Additionally, if the SO_EXCLUSIVEADDRUSE socket option is available (i.e. on Windows), it will be set on the socket. This will prevent anyone else from bind()'ing to our host/port for the duration of the test. """ if sock.family == socket.AF_INET and sock.type == socket.SOCK_STREAM: if hasattr(socket, "SO_REUSEADDR"): if sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR) == 1: raise ValueError( "tests should never set the SO_REUSEADDR " "socket option on TCP/IP sockets!" ) if hasattr(socket, "SO_REUSEPORT"): try: if sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT) == 1: raise ValueError( "tests should never set the SO_REUSEPORT " "socket option on TCP/IP sockets!" ) except OSError: # Python's socket module was compiled using modern headers # thus defining SO_REUSEPORT but this process is running # under an older kernel that does not support SO_REUSEPORT. pass if hasattr(socket, "SO_EXCLUSIVEADDRUSE"): sock.setsockopt(socket.SOL_SOCKET, socket.SO_EXCLUSIVEADDRUSE, 1) sock.bind((host, 0)) port = sock.getsockname()[1] assert isinstance(port, int) return port urllib3-2.0.7/test/test_collections.py0000644000000000000000000003375114513541732014746 0ustar00from __future__ import annotations import typing import pytest from urllib3._collections import HTTPHeaderDict from urllib3._collections import RecentlyUsedContainer as Container class TestLRUContainer: def test_maxsize(self) -> None: d: Container[int, str] = Container(5) for i in range(5): d[i] = str(i) assert len(d) == 5 for i in range(5): assert d[i] == str(i) d[i + 1] = str(i + 1) assert len(d) == 5 assert 0 not in d assert (i + 1) in d def test_maxsize_0(self) -> None: d: Container[int, int] = Container(0) d[1] = 1 assert len(d) == 0 def test_expire(self) -> None: d: Container[int, str] = Container(5) for i in range(5): d[i] = str(i) for i in range(5): d.get(0) # Add one more entry d[5] = "5" # Check state assert list(d._container.keys()) == [2, 3, 4, 0, 5] def test_same_key(self) -> None: d: Container[str, int] = Container(5) for i in range(10): d["foo"] = i assert list(d._container.keys()) == ["foo"] assert len(d) == 1 def test_access_ordering(self) -> None: d: Container[int, bool] = Container(5) for i in range(10): d[i] = True # Keys should be ordered by access time assert list(d._container.keys()) == [5, 6, 7, 8, 9] new_order = [7, 8, 6, 9, 5] for k in new_order: d[k] assert list(d._container.keys()) == new_order def test_delete(self) -> None: d: Container[int, bool] = Container(5) for i in range(5): d[i] = True del d[0] assert 0 not in d d.pop(1) assert 1 not in d d.pop(1, None) def test_get(self) -> None: d: Container[int, bool | int] = Container(5) for i in range(5): d[i] = True r = d.get(4) assert r is True r = d.get(5) assert r is None r = d.get(5, 42) assert r == 42 with pytest.raises(KeyError): d[5] def test_disposal(self) -> None: evicted_items: list[int] = [] def dispose_func(arg: int) -> None: # Save the evicted datum for inspection evicted_items.append(arg) d: Container[int, int] = Container(5, dispose_func=dispose_func) for i in range(5): d[i] = i assert list(d._container.keys()) == list(range(5)) assert evicted_items == [] # Nothing disposed d[5] = 5 assert list(d._container.keys()) == list(range(1, 6)) assert evicted_items == [0] del d[1] assert evicted_items == [0, 1] d.clear() assert evicted_items == [0, 1, 2, 3, 4, 5] def test_iter(self) -> None: d: Container[str, str] = Container() with pytest.raises(NotImplementedError): d.__iter__() class NonMappingHeaderContainer: def __init__(self, **kwargs: str) -> None: self._data = {} self._data.update(kwargs) def keys(self) -> typing.Iterator[str]: return iter(self._data) def __getitem__(self, key: str) -> str: return self._data[key] @pytest.fixture() def d() -> HTTPHeaderDict: header_dict = HTTPHeaderDict(Cookie="foo") header_dict.add("cookie", "bar") return header_dict class TestHTTPHeaderDict: def test_create_from_kwargs(self) -> None: h = HTTPHeaderDict(ab="1", cd="2", ef="3", gh="4") assert len(h) == 4 assert "ab" in h def test_setdefault(self) -> None: h = HTTPHeaderDict(a="1") assert h.setdefault("A", "3") == "1" assert h.setdefault("b", "2") == "2" assert h.setdefault("c") == "" assert h["c"] == "" assert h["b"] == "2" def test_create_from_dict(self) -> None: h = HTTPHeaderDict(dict(ab="1", cd="2", ef="3", gh="4")) assert len(h) == 4 assert "ab" in h def test_create_from_iterator(self) -> None: teststr = "urllib3ontherocks" h = HTTPHeaderDict((c, c * 5) for c in teststr) assert len(h) == len(set(teststr)) def test_create_from_list(self) -> None: headers = [ ("ab", "A"), ("cd", "B"), ("cookie", "C"), ("cookie", "D"), ("cookie", "E"), ] h = HTTPHeaderDict(headers) assert len(h) == 3 assert "ab" in h clist = h.getlist("cookie") assert len(clist) == 3 assert clist[0] == "C" assert clist[-1] == "E" def test_create_from_headerdict(self) -> None: headers = [ ("ab", "A"), ("cd", "B"), ("cookie", "C"), ("cookie", "D"), ("cookie", "E"), ] org = HTTPHeaderDict(headers) h = HTTPHeaderDict(org) assert len(h) == 3 assert "ab" in h clist = h.getlist("cookie") assert len(clist) == 3 assert clist[0] == "C" assert clist[-1] == "E" assert h is not org assert h == org def test_setitem(self, d: HTTPHeaderDict) -> None: d["Cookie"] = "foo" # The bytes value gets converted to str. The API is typed for str only, # but the implementation continues supports bytes. d[b"Cookie"] = "bar" # type: ignore[index] assert d["cookie"] == "bar" d["cookie"] = "with, comma" assert d.getlist("cookie") == ["with, comma"] def test_update(self, d: HTTPHeaderDict) -> None: d.update(dict(Cookie="foo")) assert d["cookie"] == "foo" d.update(dict(cookie="with, comma")) assert d.getlist("cookie") == ["with, comma"] def test_delitem(self, d: HTTPHeaderDict) -> None: del d["cookie"] assert "cookie" not in d assert "COOKIE" not in d def test_add_well_known_multiheader(self, d: HTTPHeaderDict) -> None: d.add("COOKIE", "asdf") assert d.getlist("cookie") == ["foo", "bar", "asdf"] assert d["cookie"] == "foo, bar, asdf" def test_add_comma_separated_multiheader(self, d: HTTPHeaderDict) -> None: d.add("bar", "foo") # The bytes value gets converted to str. The API is typed for str only, # but the implementation continues supports bytes. d.add(b"BAR", "bar") # type: ignore[arg-type] d.add("Bar", "asdf") assert d.getlist("bar") == ["foo", "bar", "asdf"] assert d["bar"] == "foo, bar, asdf" def test_extend_from_list(self, d: HTTPHeaderDict) -> None: d.extend([("set-cookie", "100"), ("set-cookie", "200"), ("set-cookie", "300")]) assert d["set-cookie"] == "100, 200, 300" def test_extend_from_dict(self, d: HTTPHeaderDict) -> None: d.extend(dict(cookie="asdf"), b="100") assert d["cookie"] == "foo, bar, asdf" assert d["b"] == "100" d.add("cookie", "with, comma") assert d.getlist("cookie") == ["foo", "bar", "asdf", "with, comma"] def test_extend_from_container(self, d: HTTPHeaderDict) -> None: h = NonMappingHeaderContainer(Cookie="foo", e="foofoo") d.extend(h) assert d["cookie"] == "foo, bar, foo" assert d["e"] == "foofoo" assert len(d) == 2 def test_header_repeat(self, d: HTTPHeaderDict) -> None: d["other-header"] = "hello" d.add("other-header", "world") assert list(d.items()) == [ ("Cookie", "foo"), ("Cookie", "bar"), ("other-header", "hello"), ("other-header", "world"), ] d.add("other-header", "!", combine=True) expected_results = [ ("Cookie", "foo"), ("Cookie", "bar"), ("other-header", "hello"), ("other-header", "world, !"), ] assert list(d.items()) == expected_results # make sure the values persist over copys assert list(d.copy().items()) == expected_results other_dict = HTTPHeaderDict() # we also need for extensions to properly maintain results other_dict.extend(d) assert list(other_dict.items()) == expected_results def test_extend_from_headerdict(self, d: HTTPHeaderDict) -> None: h = HTTPHeaderDict(Cookie="foo", e="foofoo") d.extend(h) assert d["cookie"] == "foo, bar, foo" assert d["e"] == "foofoo" assert len(d) == 2 @pytest.mark.parametrize("args", [(1, 2), (1, 2, 3, 4, 5)]) def test_extend_with_wrong_number_of_args_is_typeerror( self, d: HTTPHeaderDict, args: tuple[int, ...] ) -> None: with pytest.raises( TypeError, match=r"extend\(\) takes at most 1 positional arguments" ): d.extend(*args) # type: ignore[arg-type] def test_copy(self, d: HTTPHeaderDict) -> None: h = d.copy() assert d is not h assert d == h def test_getlist(self, d: HTTPHeaderDict) -> None: assert d.getlist("cookie") == ["foo", "bar"] assert d.getlist("Cookie") == ["foo", "bar"] assert d.getlist("b") == [] d.add("b", "asdf") assert d.getlist("b") == ["asdf"] def test_getlist_after_copy(self, d: HTTPHeaderDict) -> None: assert d.getlist("cookie") == HTTPHeaderDict(d).getlist("cookie") def test_equal(self, d: HTTPHeaderDict) -> None: b = HTTPHeaderDict(cookie="foo, bar") c = NonMappingHeaderContainer(cookie="foo, bar") e = [("cookie", "foo, bar")] assert d == b assert d == c assert d == e assert d != 2 def test_not_equal(self, d: HTTPHeaderDict) -> None: b = HTTPHeaderDict(cookie="foo, bar") c = NonMappingHeaderContainer(cookie="foo, bar") e = [("cookie", "foo, bar")] assert not (d != b) assert not (d != c) assert not (d != e) assert d != 2 def test_pop(self, d: HTTPHeaderDict) -> None: key = "Cookie" a = d[key] b = d.pop(key) assert a == b assert key not in d with pytest.raises(KeyError): d.pop(key) dummy = object() assert dummy is d.pop(key, dummy) def test_discard(self, d: HTTPHeaderDict) -> None: d.discard("cookie") assert "cookie" not in d d.discard("cookie") def test_len(self, d: HTTPHeaderDict) -> None: assert len(d) == 1 d.add("cookie", "bla") d.add("asdf", "foo") # len determined by unique fieldnames assert len(d) == 2 def test_repr(self, d: HTTPHeaderDict) -> None: rep = "HTTPHeaderDict({'Cookie': 'foo, bar'})" assert repr(d) == rep def test_items(self, d: HTTPHeaderDict) -> None: items = d.items() assert len(items) == 2 assert list(items) == [ ("Cookie", "foo"), ("Cookie", "bar"), ] assert ("Cookie", "foo") in items assert ("Cookie", "bar") in items assert ("X-Some-Header", "foo") not in items assert ("Cookie", "not_present") not in items assert ("Cookie", 1) not in items # type: ignore[comparison-overlap] assert "Cookie" not in items # type: ignore[comparison-overlap] def test_dict_conversion(self, d: HTTPHeaderDict) -> None: # Also tested in connectionpool, needs to preserve case hdict = { "Content-Length": "0", "Content-type": "text/plain", "Server": "TornadoServer/1.2.3", } h = dict(HTTPHeaderDict(hdict).items()) assert hdict == h assert hdict == dict(HTTPHeaderDict(hdict)) def test_string_enforcement(self, d: HTTPHeaderDict) -> None: # This currently throws AttributeError on key.lower(), should # probably be something nicer with pytest.raises(Exception): d[3] = "5" # type: ignore[index] with pytest.raises(Exception): d.add(3, "4") # type: ignore[arg-type] with pytest.raises(Exception): del d[3] # type: ignore[arg-type] with pytest.raises(Exception): HTTPHeaderDict({3: 3}) # type: ignore[arg-type] def test_dunder_contains(self, d: HTTPHeaderDict) -> None: """ Test: HTTPHeaderDict.__contains__ returns True - for matched string objects - for case-similar string objects HTTPHeaderDict.__contains__ returns False - for non-similar strings - for non-strings, even if they are keys in the underlying datastructure """ assert "cookie" in d assert "CoOkIe" in d assert "Not a cookie" not in d marker = object() d._container[marker] = ["some", "strings"] # type: ignore[index] assert marker not in d assert marker in d._container def test_union(self, d: HTTPHeaderDict) -> None: to_merge = {"Cookie": "tim-tam"} result = d | to_merge assert result == HTTPHeaderDict({"Cookie": "foo, bar, tim-tam"}) assert to_merge == {"Cookie": "tim-tam"} assert d == HTTPHeaderDict({"Cookie": "foo, bar"}) def test_union_rhs(self, d: HTTPHeaderDict) -> None: to_merge = {"Cookie": "tim-tam"} result = to_merge | d assert result == HTTPHeaderDict({"Cookie": "tim-tam, foo, bar"}) assert to_merge == {"Cookie": "tim-tam"} assert d == HTTPHeaderDict({"Cookie": "foo, bar"}) def test_inplace_union(self, d: HTTPHeaderDict) -> None: to_merge = {"Cookie": "tim-tam"} d |= to_merge assert d == HTTPHeaderDict({"Cookie": "foo, bar, tim-tam"}) def test_union_with_unsupported_type(self, d: HTTPHeaderDict) -> None: with pytest.raises(TypeError, match="unsupported operand type.*'int'"): d | 42 with pytest.raises(TypeError, match="unsupported operand type.*'float'"): 3.14 | d def test_inplace_union_with_unsupported_type(self, d: HTTPHeaderDict) -> None: with pytest.raises(TypeError, match="unsupported operand type.*'NoneType'"): d |= None urllib3-2.0.7/test/test_compatibility.py0000644000000000000000000000126414513541732015273 0ustar00from __future__ import annotations import http.cookiejar import urllib from urllib3.response import HTTPResponse class TestCookiejar: def test_extract(self) -> None: request = urllib.request.Request("http://google.com") cookiejar = http.cookiejar.CookieJar() response = HTTPResponse() cookies = [ "sessionhash=abcabcabcabcab; path=/; HttpOnly", "lastvisit=1348253375; expires=Sat, 21-Sep-2050 18:49:35 GMT; path=/", ] for c in cookies: response.headers.add("set-cookie", c) cookiejar.extract_cookies(response, request) # type: ignore[arg-type] assert len(cookiejar) == len(cookies) urllib3-2.0.7/test/test_connection.py0000644000000000000000000002476114513541732014570 0ustar00from __future__ import annotations import datetime import socket import typing from http.client import ResponseNotReady from unittest import mock import pytest from urllib3.connection import ( # type: ignore[attr-defined] RECENT_DATE, CertificateError, HTTPConnection, HTTPSConnection, _match_hostname, _url_from_connection, _wrap_proxy_error, ) from urllib3.exceptions import HTTPError, ProxyError, SSLError from urllib3.util import ssl_ from urllib3.util.ssl_match_hostname import ( CertificateError as ImplementationCertificateError, ) from urllib3.util.ssl_match_hostname import _dnsname_match, match_hostname if typing.TYPE_CHECKING: from urllib3.util.ssl_ import _TYPE_PEER_CERT_RET_DICT class TestConnection: """ Tests in this suite should not make any network requests or connections. """ def test_match_hostname_no_cert(self) -> None: cert = None asserted_hostname = "foo" with pytest.raises(ValueError): _match_hostname(cert, asserted_hostname) def test_match_hostname_empty_cert(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {} asserted_hostname = "foo" with pytest.raises(ValueError): _match_hostname(cert, asserted_hostname) def test_match_hostname_match(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subjectAltName": (("DNS", "foo"),)} asserted_hostname = "foo" _match_hostname(cert, asserted_hostname) def test_match_hostname_mismatch(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subjectAltName": (("DNS", "foo"),)} asserted_hostname = "bar" try: with mock.patch("urllib3.connection.log.warning") as mock_log: _match_hostname(cert, asserted_hostname) except CertificateError as e: assert "hostname 'bar' doesn't match 'foo'" in str(e) mock_log.assert_called_once_with( "Certificate did not match expected hostname: %s. Certificate: %s", "bar", {"subjectAltName": (("DNS", "foo"),)}, ) assert e._peer_cert == cert # type: ignore[attr-defined] def test_match_hostname_no_dns(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subjectAltName": (("DNS", ""),)} asserted_hostname = "bar" try: with mock.patch("urllib3.connection.log.warning") as mock_log: _match_hostname(cert, asserted_hostname) except CertificateError as e: assert "hostname 'bar' doesn't match ''" in str(e) mock_log.assert_called_once_with( "Certificate did not match expected hostname: %s. Certificate: %s", "bar", {"subjectAltName": (("DNS", ""),)}, ) assert e._peer_cert == cert # type: ignore[attr-defined] def test_match_hostname_startwith_wildcard(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subjectAltName": (("DNS", "*"),)} asserted_hostname = "foo" _match_hostname(cert, asserted_hostname) def test_match_hostname_dnsname(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = { "subjectAltName": (("DNS", "xn--p1b6ci4b4b3a*.xn--11b5bs8d"),) } asserted_hostname = "xn--p1b6ci4b4b3a*.xn--11b5bs8d" _match_hostname(cert, asserted_hostname) def test_match_hostname_include_wildcard(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subjectAltName": (("DNS", "foo*"),)} asserted_hostname = "foobar" _match_hostname(cert, asserted_hostname) def test_match_hostname_more_than_one_dnsname_error(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = { "subjectAltName": (("DNS", "foo*"), ("DNS", "fo*")) } asserted_hostname = "bar" with pytest.raises(CertificateError, match="doesn't match either of"): _match_hostname(cert, asserted_hostname) def test_dnsname_match_include_more_than_one_wildcard_error(self) -> None: with pytest.raises(CertificateError, match="too many wildcards in certificate"): _dnsname_match("foo**", "foobar") def test_match_hostname_ignore_common_name(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subject": ((("commonName", "foo"),),)} asserted_hostname = "foo" with pytest.raises( ImplementationCertificateError, match="no appropriate subjectAltName fields were found", ): match_hostname(cert, asserted_hostname) def test_match_hostname_check_common_name(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = {"subject": ((("commonName", "foo"),),)} asserted_hostname = "foo" match_hostname(cert, asserted_hostname, True) def test_match_hostname_ip_address(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = { "subjectAltName": (("IP Address", "1.1.1.1"),) } asserted_hostname = "1.1.1.2" try: with mock.patch("urllib3.connection.log.warning") as mock_log: _match_hostname(cert, asserted_hostname) except CertificateError as e: assert "hostname '1.1.1.2' doesn't match '1.1.1.1'" in str(e) mock_log.assert_called_once_with( "Certificate did not match expected hostname: %s. Certificate: %s", "1.1.1.2", {"subjectAltName": (("IP Address", "1.1.1.1"),)}, ) assert e._peer_cert == cert # type: ignore[attr-defined] @pytest.mark.parametrize( ["asserted_hostname", "san_ip"], [ ("1:2::3:4", "1:2:0:0:0:0:3:4"), ("1:2:0:0::3:4", "1:2:0:0:0:0:3:4"), ("::0.1.0.2", "0:0:0:0:0:0:1:2"), ("::1%42", "0:0:0:0:0:0:0:1"), ("::2%iface", "0:0:0:0:0:0:0:2"), ], ) def test_match_hostname_ip_address_ipv6( self, asserted_hostname: str, san_ip: str ) -> None: """Check that hostname matches follow RFC 9110 rules for IPv6.""" cert: _TYPE_PEER_CERT_RET_DICT = {"subjectAltName": (("IP Address", san_ip),)} match_hostname(cert, asserted_hostname) def test_match_hostname_ip_address_ipv6_doesnt_match(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = { "subjectAltName": (("IP Address", "1:2::2:1"),) } asserted_hostname = "1:2::2:2" try: with mock.patch("urllib3.connection.log.warning") as mock_log: _match_hostname(cert, asserted_hostname) except CertificateError as e: assert "hostname '1:2::2:2' doesn't match '1:2::2:1'" in str(e) mock_log.assert_called_once_with( "Certificate did not match expected hostname: %s. Certificate: %s", "1:2::2:2", {"subjectAltName": (("IP Address", "1:2::2:1"),)}, ) assert e._peer_cert == cert # type: ignore[attr-defined] def test_match_hostname_dns_with_brackets_doesnt_match(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = { "subjectAltName": ( ("DNS", "localhost"), ("IP Address", "localhost"), ) } asserted_hostname = "[localhost]" with pytest.raises(CertificateError) as e: _match_hostname(cert, asserted_hostname) assert ( "hostname '[localhost]' doesn't match either of 'localhost', 'localhost'" in str(e.value) ) def test_match_hostname_ip_address_ipv6_brackets(self) -> None: cert: _TYPE_PEER_CERT_RET_DICT = { "subjectAltName": (("IP Address", "1:2::2:1"),) } asserted_hostname = "[1:2::2:1]" # Assert no error is raised _match_hostname(cert, asserted_hostname) def test_recent_date(self) -> None: # This test is to make sure that the RECENT_DATE value # doesn't get too far behind what the current date is. # When this test fails update urllib3.connection.RECENT_DATE # according to the rules defined in that file. two_years = datetime.timedelta(days=365 * 2) assert RECENT_DATE > (datetime.datetime.today() - two_years).date() def test_HTTPSConnection_default_socket_options(self) -> None: conn = HTTPSConnection("not.a.real.host", port=443) assert conn.socket_options == [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)] @pytest.mark.parametrize( "proxy_scheme, err_part", [ ("http", "Unable to connect to proxy"), ( "https", "Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS", ), ], ) def test_wrap_proxy_error(self, proxy_scheme: str, err_part: str) -> None: new_err = _wrap_proxy_error(HTTPError("unknown protocol"), proxy_scheme) assert isinstance(new_err, ProxyError) is True assert err_part in new_err.args[0] def test_url_from_pool(self) -> None: conn = HTTPConnection("google.com", port=80) path = "path?query=foo" assert f"http://google.com:80/{path}" == _url_from_connection(conn, path) def test_getresponse_requires_reponseoptions(self) -> None: conn = HTTPConnection("google.com", port=80) # Should error if a request has not been sent with pytest.raises(ResponseNotReady): conn.getresponse() def test_assert_fingerprint_closes_socket(self) -> None: context = mock.create_autospec(ssl_.SSLContext) context.wrap_socket.return_value.getpeercert.return_value = b"fake cert" conn = HTTPSConnection( "google.com", port=443, assert_fingerprint="AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA", ssl_context=context, ) with mock.patch.object(conn, "_new_conn"): with pytest.raises(SSLError): conn.connect() context.wrap_socket.return_value.close.assert_called_once_with() def test_assert_hostname_closes_socket(self) -> None: context = mock.create_autospec(ssl_.SSLContext) context.wrap_socket.return_value.getpeercert.return_value = { "subjectAltName": (("DNS", "google.com"),) } conn = HTTPSConnection( "google.com", port=443, assert_hostname="example.com", ssl_context=context ) with mock.patch.object(conn, "_new_conn"): with pytest.raises(ImplementationCertificateError): conn.connect() context.wrap_socket.return_value.close.assert_called_once_with() urllib3-2.0.7/test/test_connectionpool.py0000644000000000000000000005436414513541732015464 0ustar00from __future__ import annotations import http.client as httplib import ssl import typing from http.client import HTTPException from queue import Empty from socket import error as SocketError from ssl import SSLError as BaseSSLError from test import SHORT_TIMEOUT from unittest.mock import Mock, patch import pytest from dummyserver.server import DEFAULT_CA from urllib3 import Retry from urllib3.connection import HTTPConnection from urllib3.connectionpool import ( HTTPConnectionPool, HTTPSConnectionPool, _url_from_pool, connection_from_url, ) from urllib3.exceptions import ( ClosedPoolError, EmptyPoolError, FullPoolError, HostChangedError, LocationValueError, MaxRetryError, ProtocolError, ReadTimeoutError, SSLError, TimeoutError, ) from urllib3.response import HTTPResponse from urllib3.util.ssl_match_hostname import CertificateError from urllib3.util.timeout import _DEFAULT_TIMEOUT, Timeout from .test_response import MockChunkedEncodingResponse, MockSock class HTTPUnixConnection(HTTPConnection): def __init__(self, host: str, timeout: int = 60, **kwargs: typing.Any) -> None: super().__init__("localhost") self.unix_socket = host self.timeout = timeout self.sock = None class HTTPUnixConnectionPool(HTTPConnectionPool): scheme = "http+unix" ConnectionCls = HTTPUnixConnection class TestConnectionPool: """ Tests in this suite should exercise the ConnectionPool functionality without actually making any network requests or connections. """ @pytest.mark.parametrize( "a, b", [ ("http://google.com/", "/"), ("http://google.com/", "http://google.com/"), ("http://google.com/", "http://google.com"), ("http://google.com/", "http://google.com/abra/cadabra"), ("http://google.com:42/", "http://google.com:42/abracadabra"), # Test comparison using default ports ("http://google.com:80/", "http://google.com/abracadabra"), ("http://google.com/", "http://google.com:80/abracadabra"), ("https://google.com:443/", "https://google.com/abracadabra"), ("https://google.com/", "https://google.com:443/abracadabra"), ( "http://[2607:f8b0:4005:805::200e%25eth0]/", "http://[2607:f8b0:4005:805::200e%eth0]/", ), ( "https://[2607:f8b0:4005:805::200e%25eth0]:443/", "https://[2607:f8b0:4005:805::200e%eth0]:443/", ), ("http://[::1]/", "http://[::1]"), ( "http://[2001:558:fc00:200:f816:3eff:fef9:b954%lo]/", "http://[2001:558:fc00:200:f816:3eff:fef9:b954%25lo]", ), ], ) def test_same_host(self, a: str, b: str) -> None: with connection_from_url(a) as c: assert c.is_same_host(b) @pytest.mark.parametrize( "a, b", [ ("https://google.com/", "http://google.com/"), ("http://google.com/", "https://google.com/"), ("http://yahoo.com/", "http://google.com/"), ("http://google.com:42", "https://google.com/abracadabra"), ("http://google.com", "https://google.net/"), # Test comparison with default ports ("http://google.com:42", "http://google.com"), ("https://google.com:42", "https://google.com"), ("http://google.com:443", "http://google.com"), ("https://google.com:80", "https://google.com"), ("http://google.com:443", "https://google.com"), ("https://google.com:80", "http://google.com"), ("https://google.com:443", "http://google.com"), ("http://google.com:80", "https://google.com"), # Zone identifiers are unique connection end points and should # never be equivalent. ("http://[dead::beef]", "https://[dead::beef%en5]/"), ], ) def test_not_same_host(self, a: str, b: str) -> None: with connection_from_url(a) as c: assert not c.is_same_host(b) with connection_from_url(b) as c: assert not c.is_same_host(a) @pytest.mark.parametrize( "a, b", [ ("google.com", "/"), ("google.com", "http://google.com/"), ("google.com", "http://google.com"), ("google.com", "http://google.com/abra/cadabra"), # Test comparison using default ports ("google.com", "http://google.com:80/abracadabra"), ], ) def test_same_host_no_port_http(self, a: str, b: str) -> None: # This test was introduced in #801 to deal with the fact that urllib3 # never initializes ConnectionPool objects with port=None. with HTTPConnectionPool(a) as c: assert c.is_same_host(b) @pytest.mark.parametrize( "a, b", [ ("google.com", "/"), ("google.com", "https://google.com/"), ("google.com", "https://google.com"), ("google.com", "https://google.com/abra/cadabra"), # Test comparison using default ports ("google.com", "https://google.com:443/abracadabra"), ], ) def test_same_host_no_port_https(self, a: str, b: str) -> None: # This test was introduced in #801 to deal with the fact that urllib3 # never initializes ConnectionPool objects with port=None. with HTTPSConnectionPool(a) as c: assert c.is_same_host(b) @pytest.mark.parametrize( "a, b", [ ("google.com", "https://google.com/"), ("yahoo.com", "http://google.com/"), ("google.com", "https://google.net/"), ("google.com", "http://google.com./"), ], ) def test_not_same_host_no_port_http(self, a: str, b: str) -> None: with HTTPConnectionPool(a) as c: assert not c.is_same_host(b) with HTTPConnectionPool(b) as c: assert not c.is_same_host(a) @pytest.mark.parametrize( "a, b", [ ("google.com", "http://google.com/"), ("yahoo.com", "https://google.com/"), ("google.com", "https://google.net/"), ("google.com", "https://google.com./"), ], ) def test_not_same_host_no_port_https(self, a: str, b: str) -> None: with HTTPSConnectionPool(a) as c: assert not c.is_same_host(b) with HTTPSConnectionPool(b) as c: assert not c.is_same_host(a) @pytest.mark.parametrize( "a, b", [ ("%2Fvar%2Frun%2Fdocker.sock", "http+unix://%2Fvar%2Frun%2Fdocker.sock"), ("%2Fvar%2Frun%2Fdocker.sock", "http+unix://%2Fvar%2Frun%2Fdocker.sock/"), ( "%2Fvar%2Frun%2Fdocker.sock", "http+unix://%2Fvar%2Frun%2Fdocker.sock/abracadabra", ), ("%2Ftmp%2FTEST.sock", "http+unix://%2Ftmp%2FTEST.sock"), ("%2Ftmp%2FTEST.sock", "http+unix://%2Ftmp%2FTEST.sock/"), ("%2Ftmp%2FTEST.sock", "http+unix://%2Ftmp%2FTEST.sock/abracadabra"), ], ) def test_same_host_custom_protocol(self, a: str, b: str) -> None: with HTTPUnixConnectionPool(a) as c: assert c.is_same_host(b) @pytest.mark.parametrize( "a, b", [ ("%2Ftmp%2Ftest.sock", "http+unix://%2Ftmp%2FTEST.sock"), ("%2Ftmp%2Ftest.sock", "http+unix://%2Ftmp%2FTEST.sock/"), ("%2Ftmp%2Ftest.sock", "http+unix://%2Ftmp%2FTEST.sock/abracadabra"), ("%2Fvar%2Frun%2Fdocker.sock", "http+unix://%2Ftmp%2FTEST.sock"), ], ) def test_not_same_host_custom_protocol(self, a: str, b: str) -> None: with HTTPUnixConnectionPool(a) as c: assert not c.is_same_host(b) def test_max_connections(self) -> None: with HTTPConnectionPool(host="localhost", maxsize=1, block=True) as pool: pool._get_conn(timeout=SHORT_TIMEOUT) with pytest.raises(EmptyPoolError): pool._get_conn(timeout=SHORT_TIMEOUT) with pytest.raises(EmptyPoolError): pool.request("GET", "/", pool_timeout=SHORT_TIMEOUT) assert pool.num_connections == 1 def test_put_conn_when_pool_is_full_nonblocking( self, caplog: pytest.LogCaptureFixture ) -> None: """ If maxsize = n and we _put_conn n + 1 conns, the n + 1th conn will get closed and will not get added to the pool. """ with HTTPConnectionPool(host="localhost", maxsize=1, block=False) as pool: conn1 = pool._get_conn() # pool.pool is empty because we popped the one None that pool.pool was initialized with # but this pool._get_conn call will not raise EmptyPoolError because block is False conn2 = pool._get_conn() with patch.object(conn1, "close") as conn1_close: with patch.object(conn2, "close") as conn2_close: pool._put_conn(conn1) pool._put_conn(conn2) assert conn1_close.called is False assert conn2_close.called is True assert conn1 == pool._get_conn() assert conn2 != pool._get_conn() assert pool.num_connections == 3 assert "Connection pool is full, discarding connection" in caplog.text assert "Connection pool size: 1" in caplog.text def test_put_conn_when_pool_is_full_blocking(self) -> None: """ If maxsize = n and we _put_conn n + 1 conns, the n + 1th conn will cause a FullPoolError. """ with HTTPConnectionPool(host="localhost", maxsize=1, block=True) as pool: conn1 = pool._get_conn() conn2 = pool._new_conn() with patch.object(conn1, "close") as conn1_close: with patch.object(conn2, "close") as conn2_close: pool._put_conn(conn1) with pytest.raises(FullPoolError): pool._put_conn(conn2) assert conn1_close.called is False assert conn2_close.called is True assert conn1 == pool._get_conn() def test_put_conn_closed_pool(self) -> None: with HTTPConnectionPool(host="localhost", maxsize=1, block=True) as pool: conn1 = pool._get_conn() with patch.object(conn1, "close") as conn1_close: pool.close() assert pool.pool is None # Accessing pool.pool will raise AttributeError, which will get # caught and will close conn1 pool._put_conn(conn1) assert conn1_close.called is True def test_exception_str(self) -> None: assert ( str(EmptyPoolError(HTTPConnectionPool(host="localhost"), "Test.")) == "HTTPConnectionPool(host='localhost', port=None): Test." ) def test_retry_exception_str(self) -> None: assert ( str(MaxRetryError(HTTPConnectionPool(host="localhost"), "Test.", None)) == "HTTPConnectionPool(host='localhost', port=None): " "Max retries exceeded with url: Test. (Caused by None)" ) err = SocketError("Test") # using err.__class__ here, as socket.error is an alias for OSError # since Py3.3 and gets printed as this assert ( str(MaxRetryError(HTTPConnectionPool(host="localhost"), "Test.", err)) == "HTTPConnectionPool(host='localhost', port=None): " "Max retries exceeded with url: Test. " "(Caused by %r)" % err ) def test_pool_size(self) -> None: POOL_SIZE = 1 with HTTPConnectionPool( host="localhost", maxsize=POOL_SIZE, block=True ) as pool: def _test( exception: type[BaseException], expect: type[BaseException], reason: type[BaseException] | None = None, ) -> None: with patch.object(pool, "_make_request", side_effect=exception()): with pytest.raises(expect) as excinfo: pool.request("GET", "/") if reason is not None: assert isinstance(excinfo.value.reason, reason) # type: ignore[attr-defined] assert pool.pool is not None assert pool.pool.qsize() == POOL_SIZE # Make sure that all of the exceptions return the connection # to the pool _test(BaseSSLError, MaxRetryError, SSLError) _test(CertificateError, MaxRetryError, SSLError) # The pool should never be empty, and with these two exceptions # being raised, a retry will be triggered, but that retry will # fail, eventually raising MaxRetryError, not EmptyPoolError # See: https://github.com/urllib3/urllib3/issues/76 with patch.object(pool, "_make_request", side_effect=HTTPException()): with pytest.raises(MaxRetryError): pool.request("GET", "/", retries=1, pool_timeout=SHORT_TIMEOUT) assert pool.pool is not None assert pool.pool.qsize() == POOL_SIZE def test_empty_does_not_put_conn(self) -> None: """Do not put None back in the pool if the pool was empty""" with HTTPConnectionPool(host="localhost", maxsize=1, block=True) as pool: with patch.object( pool, "_get_conn", side_effect=EmptyPoolError(pool, "Pool is empty") ): with patch.object( pool, "_put_conn", side_effect=AssertionError("Unexpected _put_conn"), ): with pytest.raises(EmptyPoolError): pool.request("GET", "/") def test_assert_same_host(self) -> None: with connection_from_url("http://google.com:80") as c: with pytest.raises(HostChangedError): c.request("GET", "http://yahoo.com:80", assert_same_host=True) def test_pool_close(self) -> None: pool = connection_from_url("http://google.com:80") # Populate with some connections conn1 = pool._get_conn() conn2 = pool._get_conn() conn3 = pool._get_conn() pool._put_conn(conn1) pool._put_conn(conn2) old_pool_queue = pool.pool pool.close() assert pool.pool is None with pytest.raises(ClosedPoolError): pool._get_conn() pool._put_conn(conn3) with pytest.raises(ClosedPoolError): pool._get_conn() with pytest.raises(Empty): assert old_pool_queue is not None old_pool_queue.get(block=False) def test_pool_close_twice(self) -> None: pool = connection_from_url("http://google.com:80") # Populate with some connections conn1 = pool._get_conn() conn2 = pool._get_conn() pool._put_conn(conn1) pool._put_conn(conn2) pool.close() assert pool.pool is None try: pool.close() except AttributeError: pytest.fail("Pool of the ConnectionPool is None and has no attribute get.") def test_pool_timeouts(self) -> None: with HTTPConnectionPool(host="localhost") as pool: conn = pool._new_conn() assert conn.__class__ == HTTPConnection assert pool.timeout.__class__ == Timeout assert pool.timeout._read == _DEFAULT_TIMEOUT assert pool.timeout._connect == _DEFAULT_TIMEOUT assert pool.timeout.total is None pool = HTTPConnectionPool(host="localhost", timeout=SHORT_TIMEOUT) assert pool.timeout._read == SHORT_TIMEOUT assert pool.timeout._connect == SHORT_TIMEOUT assert pool.timeout.total is None def test_no_host(self) -> None: with pytest.raises(LocationValueError): HTTPConnectionPool(None) # type: ignore[arg-type] def test_contextmanager(self) -> None: with connection_from_url("http://google.com:80") as pool: # Populate with some connections conn1 = pool._get_conn() conn2 = pool._get_conn() conn3 = pool._get_conn() pool._put_conn(conn1) pool._put_conn(conn2) old_pool_queue = pool.pool assert pool.pool is None with pytest.raises(ClosedPoolError): pool._get_conn() pool._put_conn(conn3) with pytest.raises(ClosedPoolError): pool._get_conn() with pytest.raises(Empty): assert old_pool_queue is not None old_pool_queue.get(block=False) def test_url_from_pool(self) -> None: with connection_from_url("http://google.com:80") as pool: path = "path?query=foo" assert f"http://google.com:80/{path}" == _url_from_pool(pool, path) def test_ca_certs_default_cert_required(self) -> None: with connection_from_url("https://google.com:80", ca_certs=DEFAULT_CA) as pool: conn = pool._get_conn() assert conn.cert_reqs == ssl.CERT_REQUIRED # type: ignore[attr-defined] def test_cleanup_on_extreme_connection_error(self) -> None: """ This test validates that we clean up properly even on exceptions that we'd not otherwise catch, i.e. those that inherit from BaseException like KeyboardInterrupt or gevent.Timeout. See #805 for more details. """ class RealBad(BaseException): pass def kaboom(*args: typing.Any, **kwargs: typing.Any) -> None: raise RealBad() with connection_from_url("http://localhost:80") as c: with patch.object(c, "_make_request", kaboom): assert c.pool is not None initial_pool_size = c.pool.qsize() try: # We need to release_conn this way or we'd put it away # regardless. c.urlopen("GET", "/", release_conn=False) except RealBad: pass new_pool_size = c.pool.qsize() assert initial_pool_size == new_pool_size def test_release_conn_param_is_respected_after_http_error_retry(self) -> None: """For successful ```urlopen(release_conn=False)```, the connection isn't released, even after a retry. This is a regression test for issue #651 [1], where the connection would be released if the initial request failed, even if a retry succeeded. [1] """ class _raise_once_make_request_function: """Callable that can mimic `_make_request()`. Raises the given exception on its first call, but returns a successful response on subsequent calls. """ def __init__( self, ex: type[BaseException], pool: HTTPConnectionPool ) -> None: super().__init__() self._ex: type[BaseException] | None = ex self._pool = pool def __call__( self, conn: HTTPConnection, method: str, url: str, *args: typing.Any, retries: Retry, **kwargs: typing.Any, ) -> HTTPResponse: if self._ex: ex, self._ex = self._ex, None raise ex() httplib_response = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] httplib_response.fp = MockChunkedEncodingResponse([b"f", b"o", b"o"]) # type: ignore[assignment] httplib_response.headers = httplib_response.msg = httplib.HTTPMessage() response_conn: HTTPConnection | None = kwargs.get("response_conn") response = HTTPResponse( body=httplib_response, headers=httplib_response.headers, # type: ignore[arg-type] status=httplib_response.status, version=httplib_response.version, reason=httplib_response.reason, original_response=httplib_response, retries=retries, request_method=method, request_url=url, preload_content=False, connection=response_conn, pool=self._pool, ) return response def _test(exception: type[BaseException]) -> None: with HTTPConnectionPool(host="localhost", maxsize=1, block=True) as pool: # Verify that the request succeeds after two attempts, and that the # connection is left on the response object, instead of being # released back into the pool. with patch.object( pool, "_make_request", _raise_once_make_request_function(exception, pool), ): response = pool.urlopen( "GET", "/", retries=1, release_conn=False, preload_content=False, chunked=True, ) assert pool.pool is not None assert pool.pool.qsize() == 0 assert pool.num_connections == 2 assert response.connection is not None response.release_conn() assert pool.pool.qsize() == 1 assert response.connection is None # Run the test case for all the retriable exceptions. _test(TimeoutError) _test(HTTPException) _test(SocketError) _test(ProtocolError) def test_read_timeout_0_does_not_raise_bad_status_line_error(self) -> None: with HTTPConnectionPool(host="localhost", maxsize=1) as pool: conn = Mock(spec=HTTPConnection) # Needed to tell the pool that the connection is alive. conn.is_closed = False with patch.object(Timeout, "read_timeout", 0): timeout = Timeout(1, 1, 1) with pytest.raises(ReadTimeoutError): pool._make_request(conn, "", "", timeout=timeout) urllib3-2.0.7/test/test_exceptions.py0000644000000000000000000000416414513541732014605 0ustar00from __future__ import annotations import pickle from email.errors import MessageDefect from test import DUMMY_POOL import pytest from urllib3.connection import HTTPConnection from urllib3.connectionpool import HTTPConnectionPool from urllib3.exceptions import ( ClosedPoolError, ConnectTimeoutError, EmptyPoolError, HeaderParsingError, HostChangedError, HTTPError, LocationParseError, MaxRetryError, NewConnectionError, ReadTimeoutError, ) class TestPickle: @pytest.mark.parametrize( "exception", [ HTTPError(None), MaxRetryError(DUMMY_POOL, "", None), LocationParseError(""), ConnectTimeoutError(None), HTTPError("foo"), HTTPError("foo", IOError("foo")), MaxRetryError(HTTPConnectionPool("localhost"), "/", None), LocationParseError("fake location"), ClosedPoolError(HTTPConnectionPool("localhost"), ""), EmptyPoolError(HTTPConnectionPool("localhost"), ""), HostChangedError(HTTPConnectionPool("localhost"), "/", 0), ReadTimeoutError(HTTPConnectionPool("localhost"), "/", ""), ], ) def test_exceptions(self, exception: Exception) -> None: result = pickle.loads(pickle.dumps(exception)) assert isinstance(result, type(exception)) class TestFormat: def test_header_parsing_errors(self) -> None: hpe = HeaderParsingError([MessageDefect("defects")], "unparsed_data") assert "defects" in str(hpe) assert "unparsed_data" in str(hpe) class TestNewConnectionError: def test_pool_property_deprecation_warning(self) -> None: err = NewConnectionError(HTTPConnection("localhost"), "test") with pytest.warns(DeprecationWarning) as records: err_pool = err.pool assert err_pool is err.conn msg = ( "The 'pool' property is deprecated and will be removed " "in urllib3 v2.1.0. Use 'conn' instead." ) record = records[0] assert isinstance(record.message, Warning) assert record.message.args[0] == msg urllib3-2.0.7/test/test_fields.py0000644000000000000000000001052614513541732013671 0ustar00from __future__ import annotations import pytest from urllib3.fields import ( RequestField, format_header_param, format_header_param_html5, format_header_param_rfc2231, format_multipart_header_param, guess_content_type, ) class TestRequestField: @pytest.mark.parametrize( "filename, content_types", [ ("image.jpg", ["image/jpeg", "image/pjpeg"]), ("notsure", ["application/octet-stream"]), (None, ["application/octet-stream"]), ], ) def test_guess_content_type( self, filename: str | None, content_types: list[str] ) -> None: assert guess_content_type(filename) in content_types def test_create(self) -> None: simple_field = RequestField("somename", "data") assert simple_field.render_headers() == "\r\n" filename_field = RequestField("somename", "data", filename="somefile.txt") assert filename_field.render_headers() == "\r\n" headers_field = RequestField( "somename", "data", headers={"Content-Length": "4"} ) assert headers_field.render_headers() == "Content-Length: 4\r\n\r\n" def test_make_multipart(self) -> None: field = RequestField("somename", "data") field.make_multipart(content_type="image/jpg", content_location="/test") assert ( field.render_headers() == 'Content-Disposition: form-data; name="somename"\r\n' "Content-Type: image/jpg\r\n" "Content-Location: /test\r\n" "\r\n" ) def test_make_multipart_empty_filename(self) -> None: field = RequestField("somename", "data", "") field.make_multipart(content_type="application/octet-stream") assert ( field.render_headers() == 'Content-Disposition: form-data; name="somename"; filename=""\r\n' "Content-Type: application/octet-stream\r\n" "\r\n" ) def test_render_parts(self) -> None: field = RequestField("somename", "data") parts = field._render_parts({"name": "value", "filename": "value"}) assert 'name="value"' in parts assert 'filename="value"' in parts parts = field._render_parts([("name", "value"), ("filename", "value")]) assert parts == 'name="value"; filename="value"' @pytest.mark.parametrize( ("value", "expect"), [("näme", "filename*=utf-8''n%C3%A4me"), (b"name", 'filename="name"')], ) def test_format_header_param_rfc2231_deprecated( self, value: bytes | str, expect: str ) -> None: with pytest.deprecated_call(match=r"urllib3 v2\.1\.0"): param = format_header_param_rfc2231("filename", value) assert param == expect def test_format_header_param_html5_deprecated(self) -> None: with pytest.deprecated_call(match=r"urllib3 v2\.1\.0"): param2 = format_header_param_html5("filename", "name") with pytest.deprecated_call(match=r"urllib3 v2\.1\.0"): param1 = format_header_param("filename", "name") assert param1 == param2 @pytest.mark.parametrize( ("value", "expect"), [ ("name", "name"), ("näme", "näme"), (b"n\xc3\xa4me", "näme"), ("ski â›·.txt", "ski â›·.txt"), ("control \x1A\x1B\x1C", "control \x1A\x1B\x1C"), ("backslash \\", "backslash \\"), ("quotes '\"", "quotes '%22"), ("newline \n\r", "newline %0A%0D"), ], ) def test_format_multipart_header_param( self, value: bytes | str, expect: str ) -> None: param = format_multipart_header_param("filename", value) assert param == f'filename="{expect}"' def test_from_tuples(self) -> None: field = RequestField.from_tuples("file", ("スキー旅行.txt", "data")) cd = field.headers["Content-Disposition"] assert cd == 'form-data; name="file"; filename="スキー旅行.txt"' def test_from_tuples_rfc2231(self) -> None: with pytest.deprecated_call(match=r"urllib3 v2\.1\.0"): field = RequestField.from_tuples( "file", ("näme", "data"), header_formatter=format_header_param_rfc2231 ) cd = field.headers["Content-Disposition"] assert cd == "form-data; name=\"file\"; filename*=utf-8''n%C3%A4me" urllib3-2.0.7/test/test_filepost.py0000644000000000000000000000724714513541732014256 0ustar00from __future__ import annotations import pytest from urllib3.fields import RequestField from urllib3.filepost import _TYPE_FIELDS, encode_multipart_formdata BOUNDARY = "!! test boundary !!" BOUNDARY_BYTES = BOUNDARY.encode() class TestMultipartEncoding: @pytest.mark.parametrize( "fields", [dict(k="v", k2="v2"), [("k", "v"), ("k2", "v2")]] ) def test_input_datastructures(self, fields: _TYPE_FIELDS) -> None: encoded, _ = encode_multipart_formdata(fields, boundary=BOUNDARY) assert encoded.count(BOUNDARY_BYTES) == 3 @pytest.mark.parametrize( "fields", [ [("k", "v"), ("k2", "v2")], [("k", b"v"), ("k2", b"v2")], [("k", b"v"), ("k2", "v2")], ], ) def test_field_encoding(self, fields: _TYPE_FIELDS) -> None: encoded, content_type = encode_multipart_formdata(fields, boundary=BOUNDARY) expected = ( b"--" + BOUNDARY_BYTES + b"\r\n" b'Content-Disposition: form-data; name="k"\r\n' b"\r\n" b"v\r\n" b"--" + BOUNDARY_BYTES + b"\r\n" b'Content-Disposition: form-data; name="k2"\r\n' b"\r\n" b"v2\r\n" b"--" + BOUNDARY_BYTES + b"--\r\n" ) assert encoded == expected assert content_type == "multipart/form-data; boundary=" + str(BOUNDARY) def test_filename(self) -> None: fields = [("k", ("somename", b"v"))] encoded, content_type = encode_multipart_formdata(fields, boundary=BOUNDARY) expected = ( b"--" + BOUNDARY_BYTES + b"\r\n" b'Content-Disposition: form-data; name="k"; filename="somename"\r\n' b"Content-Type: application/octet-stream\r\n" b"\r\n" b"v\r\n" b"--" + BOUNDARY_BYTES + b"--\r\n" ) assert encoded == expected assert content_type == "multipart/form-data; boundary=" + str(BOUNDARY) def test_textplain(self) -> None: fields = [("k", ("somefile.txt", b"v"))] encoded, content_type = encode_multipart_formdata(fields, boundary=BOUNDARY) expected = ( b"--" + BOUNDARY_BYTES + b"\r\n" b'Content-Disposition: form-data; name="k"; filename="somefile.txt"\r\n' b"Content-Type: text/plain\r\n" b"\r\n" b"v\r\n" b"--" + BOUNDARY_BYTES + b"--\r\n" ) assert encoded == expected assert content_type == "multipart/form-data; boundary=" + str(BOUNDARY) def test_explicit(self) -> None: fields = [("k", ("somefile.txt", b"v", "image/jpeg"))] encoded, content_type = encode_multipart_formdata(fields, boundary=BOUNDARY) expected = ( b"--" + BOUNDARY_BYTES + b"\r\n" b'Content-Disposition: form-data; name="k"; filename="somefile.txt"\r\n' b"Content-Type: image/jpeg\r\n" b"\r\n" b"v\r\n" b"--" + BOUNDARY_BYTES + b"--\r\n" ) assert encoded == expected assert content_type == "multipart/form-data; boundary=" + str(BOUNDARY) def test_request_fields(self) -> None: fields = [ RequestField( "k", b"v", filename="somefile.txt", headers={"Content-Type": "image/jpeg"}, ) ] encoded, content_type = encode_multipart_formdata(fields, boundary=BOUNDARY) expected = ( b"--" + BOUNDARY_BYTES + b"\r\n" b"Content-Type: image/jpeg\r\n" b"\r\n" b"v\r\n" b"--" + BOUNDARY_BYTES + b"--\r\n" ) assert encoded == expected urllib3-2.0.7/test/test_no_ssl.py0000644000000000000000000000172214513541732013716 0ustar00""" Test what happens if Python was built without SSL * Everything that does not involve HTTPS should still work * HTTPS requests must fail with an error that points at the ssl module """ from __future__ import annotations import sys from test import ImportBlocker, ModuleStash import pytest ssl_blocker = ImportBlocker("ssl", "_ssl") module_stash = ModuleStash("urllib3") class TestWithoutSSL: @classmethod def setup_class(cls) -> None: sys.modules.pop("ssl", None) sys.modules.pop("_ssl", None) module_stash.stash() sys.meta_path.insert(0, ssl_blocker) @classmethod def teardown_class(cls) -> None: sys.meta_path.remove(ssl_blocker) module_stash.pop() class TestImportWithoutSSL(TestWithoutSSL): def test_cannot_import_ssl(self) -> None: with pytest.raises(ImportError): import ssl # noqa: F401 def test_import_urllib3(self) -> None: import urllib3 # noqa: F401 urllib3-2.0.7/test/test_poolmanager.py0000644000000000000000000004404014513541732014725 0ustar00from __future__ import annotations import gc import socket from test import resolvesLocalhostFQDN from unittest import mock from unittest.mock import MagicMock, patch import pytest from urllib3 import connection_from_url from urllib3.connectionpool import HTTPSConnectionPool from urllib3.exceptions import LocationValueError from urllib3.poolmanager import ( _DEFAULT_BLOCKSIZE, PoolKey, PoolManager, key_fn_by_scheme, ) from urllib3.util import retry, timeout from urllib3.util.url import Url class TestPoolManager: @resolvesLocalhostFQDN() def test_same_url(self) -> None: # Convince ourselves that normally we don't get the same object conn1 = connection_from_url("http://localhost:8081/foo") conn2 = connection_from_url("http://localhost:8081/bar") assert conn1 != conn2 # Now try again using the PoolManager p = PoolManager(1) conn1 = p.connection_from_url("http://localhost:8081/foo") conn2 = p.connection_from_url("http://localhost:8081/bar") assert conn1 == conn2 # Ensure that FQDNs are handled separately from relative domains p = PoolManager(2) conn1 = p.connection_from_url("http://localhost.:8081/foo") conn2 = p.connection_from_url("http://localhost:8081/bar") assert conn1 != conn2 def test_many_urls(self) -> None: urls = [ "http://localhost:8081/foo", "http://www.google.com/mail", "http://localhost:8081/bar", "https://www.google.com/", "https://www.google.com/mail", "http://yahoo.com", "http://bing.com", "http://yahoo.com/", ] connections = set() p = PoolManager(10) for url in urls: conn = p.connection_from_url(url) connections.add(conn) assert len(connections) == 5 def test_manager_clear(self) -> None: p = PoolManager(5) p.connection_from_url("http://google.com") assert len(p.pools) == 1 p.clear() assert len(p.pools) == 0 @pytest.mark.parametrize("url", ["http://@", None]) def test_nohost(self, url: str | None) -> None: p = PoolManager(5) with pytest.raises(LocationValueError): p.connection_from_url(url=url) # type: ignore[arg-type] def test_contextmanager(self) -> None: with PoolManager(1) as p: p.connection_from_url("http://google.com") assert len(p.pools) == 1 assert len(p.pools) == 0 def test_http_pool_key_fields(self) -> None: """Assert the HTTPPoolKey fields are honored when selecting a pool.""" connection_pool_kw = { "timeout": timeout.Timeout(3.14), "retries": retry.Retry(total=6, connect=2), "block": True, "source_address": "127.0.0.1", "blocksize": _DEFAULT_BLOCKSIZE + 1, } p = PoolManager() conn_pools = [ p.connection_from_url("http://example.com/"), p.connection_from_url("http://example.com:8000/"), p.connection_from_url("http://other.example.com/"), ] for key, value in connection_pool_kw.items(): p.connection_pool_kw[key] = value conn_pools.append(p.connection_from_url("http://example.com/")) assert all( x is not y for i, x in enumerate(conn_pools) for j, y in enumerate(conn_pools) if i != j ) assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_https_pool_key_fields(self) -> None: """Assert the HTTPSPoolKey fields are honored when selecting a pool.""" connection_pool_kw = { "timeout": timeout.Timeout(3.14), "retries": retry.Retry(total=6, connect=2), "block": True, "source_address": "127.0.0.1", "key_file": "/root/totally_legit.key", "cert_file": "/root/totally_legit.crt", "cert_reqs": "CERT_REQUIRED", "ca_certs": "/root/path_to_pem", "ssl_version": "SSLv23_METHOD", "blocksize": _DEFAULT_BLOCKSIZE + 1, } p = PoolManager() conn_pools = [ p.connection_from_url("https://example.com/"), p.connection_from_url("https://example.com:4333/"), p.connection_from_url("https://other.example.com/"), ] # Asking for a connection pool with the same key should give us an # existing pool. dup_pools = [] for key, value in connection_pool_kw.items(): p.connection_pool_kw[key] = value conn_pools.append(p.connection_from_url("https://example.com/")) dup_pools.append(p.connection_from_url("https://example.com/")) assert all( x is not y for i, x in enumerate(conn_pools) for j, y in enumerate(conn_pools) if i != j ) assert all(pool in conn_pools for pool in dup_pools) assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_default_pool_key_funcs_copy(self) -> None: """Assert each PoolManager gets a copy of ``pool_keys_by_scheme``.""" p = PoolManager() assert p.key_fn_by_scheme == p.key_fn_by_scheme assert p.key_fn_by_scheme is not key_fn_by_scheme def test_pools_keyed_with_from_host(self) -> None: """Assert pools are still keyed correctly with connection_from_host.""" ssl_kw = { "key_file": "/root/totally_legit.key", "cert_file": "/root/totally_legit.crt", "cert_reqs": "CERT_REQUIRED", "ca_certs": "/root/path_to_pem", "ssl_version": "SSLv23_METHOD", } p = PoolManager(5, **ssl_kw) # type: ignore[arg-type] conns = [p.connection_from_host("example.com", 443, scheme="https")] for k in ssl_kw: p.connection_pool_kw[k] = "newval" conns.append(p.connection_from_host("example.com", 443, scheme="https")) assert all( x is not y for i, x in enumerate(conns) for j, y in enumerate(conns) if i != j ) def test_https_connection_from_url_case_insensitive(self) -> None: """Assert scheme case is ignored when pooling HTTPS connections.""" p = PoolManager() pool = p.connection_from_url("https://example.com/") other_pool = p.connection_from_url("HTTPS://EXAMPLE.COM/") assert 1 == len(p.pools) assert pool is other_pool assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_https_connection_from_host_case_insensitive(self) -> None: """Assert scheme case is ignored when getting the https key class.""" p = PoolManager() pool = p.connection_from_host("example.com", scheme="https") other_pool = p.connection_from_host("EXAMPLE.COM", scheme="HTTPS") assert 1 == len(p.pools) assert pool is other_pool assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_https_connection_from_context_case_insensitive(self) -> None: """Assert scheme case is ignored when getting the https key class.""" p = PoolManager() context = {"scheme": "https", "host": "example.com", "port": "443"} other_context = {"scheme": "HTTPS", "host": "EXAMPLE.COM", "port": "443"} pool = p.connection_from_context(context) other_pool = p.connection_from_context(other_context) assert 1 == len(p.pools) assert pool is other_pool assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_http_connection_from_url_case_insensitive(self) -> None: """Assert scheme case is ignored when pooling HTTP connections.""" p = PoolManager() pool = p.connection_from_url("http://example.com/") other_pool = p.connection_from_url("HTTP://EXAMPLE.COM/") assert 1 == len(p.pools) assert pool is other_pool assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_http_connection_from_host_case_insensitive(self) -> None: """Assert scheme case is ignored when getting the https key class.""" p = PoolManager() pool = p.connection_from_host("example.com", scheme="http") other_pool = p.connection_from_host("EXAMPLE.COM", scheme="HTTP") assert 1 == len(p.pools) assert pool is other_pool assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_assert_hostname_and_fingerprint_flag(self) -> None: """Assert that pool manager can accept hostname and fingerprint flags.""" fingerprint = "92:81:FE:85:F7:0C:26:60:EC:D6:B3:BF:93:CF:F9:71:CC:07:7D:0A" p = PoolManager(assert_hostname=True, assert_fingerprint=fingerprint) pool = p.connection_from_url("https://example.com/") assert 1 == len(p.pools) assert isinstance(pool, HTTPSConnectionPool) assert pool.assert_hostname assert fingerprint == pool.assert_fingerprint def test_http_connection_from_context_case_insensitive(self) -> None: """Assert scheme case is ignored when getting the https key class.""" p = PoolManager() context = {"scheme": "http", "host": "example.com", "port": "8080"} other_context = {"scheme": "HTTP", "host": "EXAMPLE.COM", "port": "8080"} pool = p.connection_from_context(context) other_pool = p.connection_from_context(other_context) assert 1 == len(p.pools) assert pool is other_pool assert all(isinstance(key, PoolKey) for key in p.pools.keys()) def test_deprecated_no_scheme(self) -> None: p = PoolManager() with pytest.warns(DeprecationWarning) as records: p.request(method="GET", url="evil.com://good.com") msg = ( "URLs without a scheme (ie 'https://') are deprecated and will raise an error " "in a future version of urllib3. To avoid this DeprecationWarning ensure all URLs " "start with 'https://' or 'http://'. Read more in this issue: " "https://github.com/urllib3/urllib3/issues/2920" ) assert len(records) == 1 assert isinstance(records[0].message, DeprecationWarning) assert records[0].message.args[0] == msg @patch("urllib3.poolmanager.PoolManager.connection_from_pool_key") def test_connection_from_context_strict_param( self, connection_from_pool_key: mock.MagicMock ) -> None: p = PoolManager() context = { "scheme": "http", "host": "example.com", "port": 8080, "strict": True, } with pytest.warns(DeprecationWarning) as records: p.connection_from_context(context) msg = ( "The 'strict' parameter is no longer needed on Python 3+. " "This will raise an error in urllib3 v2.1.0." ) record = records[0] assert isinstance(record.message, Warning) assert record.message.args[0] == msg _, kwargs = connection_from_pool_key.call_args assert kwargs["request_context"] == { "scheme": "http", "host": "example.com", "port": 8080, } def test_custom_pool_key(self) -> None: """Assert it is possible to define a custom key function.""" p = PoolManager(10) p.key_fn_by_scheme["http"] = lambda x: tuple(x["key"]) # type: ignore[assignment] pool1 = p.connection_from_url( "http://example.com", pool_kwargs={"key": "value"} ) pool2 = p.connection_from_url( "http://example.com", pool_kwargs={"key": "other"} ) pool3 = p.connection_from_url( "http://example.com", pool_kwargs={"key": "value", "x": "y"} ) assert 2 == len(p.pools) assert pool1 is pool3 assert pool1 is not pool2 def test_override_pool_kwargs_url(self) -> None: """Assert overriding pool kwargs works with connection_from_url.""" p = PoolManager() pool_kwargs = {"retries": 100, "block": True} default_pool = p.connection_from_url("http://example.com/") override_pool = p.connection_from_url( "http://example.com/", pool_kwargs=pool_kwargs ) assert retry.Retry.DEFAULT == default_pool.retries assert not default_pool.block assert 100 == override_pool.retries assert override_pool.block def test_override_pool_kwargs_host(self) -> None: """Assert overriding pool kwargs works with connection_from_host""" p = PoolManager() pool_kwargs = {"retries": 100, "block": True} default_pool = p.connection_from_host("example.com", scheme="http") override_pool = p.connection_from_host( "example.com", scheme="http", pool_kwargs=pool_kwargs ) assert retry.Retry.DEFAULT == default_pool.retries assert not default_pool.block assert 100 == override_pool.retries assert override_pool.block def test_pool_kwargs_socket_options(self) -> None: """Assert passing socket options works with connection_from_host""" p = PoolManager(socket_options=[]) override_opts = [ (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1), (socket.IPPROTO_TCP, socket.TCP_NODELAY, 1), ] pool_kwargs = {"socket_options": override_opts} default_pool = p.connection_from_host("example.com", scheme="http") override_pool = p.connection_from_host( "example.com", scheme="http", pool_kwargs=pool_kwargs ) assert default_pool.conn_kw["socket_options"] == [] assert override_pool.conn_kw["socket_options"] == override_opts def test_merge_pool_kwargs(self) -> None: """Assert _merge_pool_kwargs works in the happy case""" p = PoolManager(retries=100) merged = p._merge_pool_kwargs({"new_key": "value"}) assert {"retries": 100, "new_key": "value"} == merged def test_merge_pool_kwargs_none(self) -> None: """Assert false-y values to _merge_pool_kwargs result in defaults""" p = PoolManager(retries=100) merged = p._merge_pool_kwargs({}) assert p.connection_pool_kw == merged merged = p._merge_pool_kwargs(None) assert p.connection_pool_kw == merged def test_merge_pool_kwargs_remove_key(self) -> None: """Assert keys can be removed with _merge_pool_kwargs""" p = PoolManager(retries=100) merged = p._merge_pool_kwargs({"retries": None}) assert "retries" not in merged def test_merge_pool_kwargs_invalid_key(self) -> None: """Assert removing invalid keys with _merge_pool_kwargs doesn't break""" p = PoolManager(retries=100) merged = p._merge_pool_kwargs({"invalid_key": None}) assert p.connection_pool_kw == merged def test_pool_manager_no_url_absolute_form(self) -> None: """Valides we won't send a request with absolute form without a proxy""" p = PoolManager() assert p._proxy_requires_url_absolute_form(Url("http://example.com")) is False assert p._proxy_requires_url_absolute_form(Url("https://example.com")) is False @pytest.mark.parametrize( "input_blocksize,expected_blocksize", [ (_DEFAULT_BLOCKSIZE, _DEFAULT_BLOCKSIZE), (None, _DEFAULT_BLOCKSIZE), (8192, 8192), ], ) def test_poolmanager_blocksize( self, input_blocksize: int, expected_blocksize: int ) -> None: """Assert PoolManager sets blocksize properly""" p = PoolManager() pool_blocksize = p.connection_from_url( "http://example.com", {"blocksize": input_blocksize} ) assert pool_blocksize.conn_kw["blocksize"] == expected_blocksize assert pool_blocksize._get_conn().blocksize == expected_blocksize @pytest.mark.parametrize( "url", [ "[a::b%zone]", "[a::b%25zone]", "http://[a::b%zone]", "http://[a::b%25zone]", ], ) @patch("urllib3.util.connection.create_connection") def test_e2e_connect_to_ipv6_scoped( self, create_connection: MagicMock, url: str ) -> None: """Checks that IPv6 scoped addresses are properly handled end-to-end. This is not strictly speaking a pool manager unit test - this test lives here in absence of a better code location for e2e/integration tests. """ p = PoolManager() conn_pool = p.connection_from_url(url) conn = conn_pool._get_conn() conn.connect() assert create_connection.call_args[0][0] == ("a::b%zone", 80) @patch("urllib3.connection.ssl_wrap_socket") @patch("urllib3.util.connection.create_connection") def test_e2e_connect_to_ipv6_scoped_tls( self, create_connection: MagicMock, ssl_wrap_socket: MagicMock ) -> None: p = PoolManager() conn_pool = p.connection_from_url( "https://[a::b%zone]", pool_kwargs={"assert_hostname": False} ) conn = conn_pool._get_conn() conn.connect() assert ssl_wrap_socket.call_args[1]["server_hostname"] == "a::b" def test_thread_safty(self) -> None: pool_manager = PoolManager(num_pools=2) # thread 1 gets a pool for host x pool_1 = pool_manager.connection_from_url("http://host_x:80/") # thread 2 gets a pool for host y pool_2 = pool_manager.connection_from_url("http://host_y:80/") # thread 3 gets a pool for host z pool_3 = pool_manager.connection_from_url("http://host_z:80") # None of the pools should be closed, since all of them are referenced. assert pool_1.pool is not None assert pool_2.pool is not None assert pool_3.pool is not None conn_queue = pool_1.pool assert conn_queue.qsize() > 0 # thread 1 stops. del pool_1 gc.collect() # Connection should be closed, because reference to pool_1 is gone. assert conn_queue.qsize() == 0 urllib3-2.0.7/test/test_proxymanager.py0000644000000000000000000000711114513541732015133 0ustar00from __future__ import annotations import pytest from urllib3.exceptions import MaxRetryError, NewConnectionError, ProxyError from urllib3.poolmanager import ProxyManager from urllib3.util.retry import Retry from urllib3.util.url import parse_url from .port_helpers import find_unused_port class TestProxyManager: @pytest.mark.parametrize("proxy_scheme", ["http", "https"]) def test_proxy_headers(self, proxy_scheme: str) -> None: url = "http://pypi.org/project/urllib3/" proxy_url = f"{proxy_scheme}://something:1234" with ProxyManager(proxy_url) as p: # Verify default headers default_headers = {"Accept": "*/*", "Host": "pypi.org"} headers = p._set_proxy_headers(url) assert headers == default_headers # Verify default headers don't overwrite provided headers provided_headers = { "Accept": "application/json", "custom": "header", "Host": "test.python.org", } headers = p._set_proxy_headers(url, provided_headers) assert headers == provided_headers # Verify proxy with nonstandard port provided_headers = {"Accept": "application/json"} expected_headers = provided_headers.copy() expected_headers.update({"Host": "pypi.org:8080"}) url_with_port = "http://pypi.org:8080/project/urllib3/" headers = p._set_proxy_headers(url_with_port, provided_headers) assert headers == expected_headers def test_default_port(self) -> None: with ProxyManager("http://something") as p: assert p.proxy is not None assert p.proxy.port == 80 with ProxyManager("https://something") as p: assert p.proxy is not None assert p.proxy.port == 443 def test_invalid_scheme(self) -> None: with pytest.raises(AssertionError): ProxyManager("invalid://host/p") with pytest.raises(ValueError): ProxyManager("invalid://host/p") def test_proxy_tunnel(self) -> None: http_url = parse_url("http://example.com") https_url = parse_url("https://example.com") with ProxyManager("http://proxy:8080") as p: assert p._proxy_requires_url_absolute_form(http_url) assert p._proxy_requires_url_absolute_form(https_url) is False with ProxyManager("https://proxy:8080") as p: assert p._proxy_requires_url_absolute_form(http_url) assert p._proxy_requires_url_absolute_form(https_url) is False with ProxyManager("https://proxy:8080", use_forwarding_for_https=True) as p: assert p._proxy_requires_url_absolute_form(http_url) assert p._proxy_requires_url_absolute_form(https_url) def test_proxy_connect_retry(self) -> None: retry = Retry(total=None, connect=False) port = find_unused_port() with ProxyManager(f"http://localhost:{port}") as p: with pytest.raises(ProxyError) as ei: p.urlopen("HEAD", url="http://localhost/", retries=retry) assert isinstance(ei.value.original_error, NewConnectionError) retry = Retry(total=None, connect=2) with ProxyManager(f"http://localhost:{port}") as p: with pytest.raises(MaxRetryError) as ei1: p.urlopen("HEAD", url="http://localhost/", retries=retry) assert ei1.value.reason is not None assert isinstance(ei1.value.reason, ProxyError) assert isinstance(ei1.value.reason.original_error, NewConnectionError) urllib3-2.0.7/test/test_queue_monkeypatch.py0000644000000000000000000000137114513541732016147 0ustar00from __future__ import annotations import queue from unittest import mock import pytest from urllib3 import HTTPConnectionPool from urllib3.exceptions import EmptyPoolError class BadError(Exception): """ This should not be raised. """ class TestMonkeypatchResistance: """ Test that connection pool works even with a monkey patched Queue module, see obspy/obspy#1599, psf/requests#3742, urllib3/urllib3#1061. """ def test_queue_monkeypatching(self) -> None: with mock.patch.object(queue, "Empty", BadError): with HTTPConnectionPool(host="localhost", block=True) as http: http._get_conn() with pytest.raises(EmptyPoolError): http._get_conn(timeout=0) urllib3-2.0.7/test/test_response.py0000644000000000000000000013057714513541732014272 0ustar00from __future__ import annotations import contextlib import http.client as httplib import socket import ssl import sys import typing import zlib from base64 import b64decode from http.client import IncompleteRead as httplib_IncompleteRead from io import BufferedReader, BytesIO, TextIOWrapper from test import onlyBrotli, onlyZstd from unittest import mock import pytest from urllib3 import HTTPHeaderDict from urllib3.exceptions import ( BodyNotHttplibCompatible, DecodeError, IncompleteRead, InvalidChunkLength, InvalidHeader, ProtocolError, ResponseNotChunked, SSLError, ) from urllib3.response import ( # type: ignore[attr-defined] BaseHTTPResponse, BytesQueueBuffer, HTTPResponse, brotli, zstd, ) from urllib3.util.response import is_fp_closed from urllib3.util.retry import RequestHistory, Retry class TestBytesQueueBuffer: def test_single_chunk(self) -> None: buffer = BytesQueueBuffer() assert len(buffer) == 0 with pytest.raises(RuntimeError, match="buffer is empty"): assert buffer.get(10) assert buffer.get(0) == b"" buffer.put(b"foo") with pytest.raises(ValueError, match="n should be > 0"): buffer.get(-1) assert buffer.get(1) == b"f" assert buffer.get(2) == b"oo" with pytest.raises(RuntimeError, match="buffer is empty"): assert buffer.get(10) def test_read_too_much(self) -> None: buffer = BytesQueueBuffer() buffer.put(b"foo") assert buffer.get(100) == b"foo" def test_multiple_chunks(self) -> None: buffer = BytesQueueBuffer() buffer.put(b"foo") buffer.put(b"bar") buffer.put(b"baz") assert len(buffer) == 9 assert buffer.get(1) == b"f" assert len(buffer) == 8 assert buffer.get(4) == b"ooba" assert len(buffer) == 4 assert buffer.get(4) == b"rbaz" assert len(buffer) == 0 @pytest.mark.skipif( sys.version_info < (3, 8), reason="pytest-memray requires Python 3.8+" ) @pytest.mark.limit_memory("12.5 MB") # assert that we're not doubling memory usage def test_memory_usage(self) -> None: # Allocate 10 1MiB chunks buffer = BytesQueueBuffer() for i in range(10): # This allocates 2MiB, putting the max at around 12MiB. Not sure why. buffer.put(bytes(2**20)) assert len(buffer.get(10 * 2**20)) == 10 * 2**20 # A known random (i.e, not-too-compressible) payload generated with: # "".join(random.choice(string.printable) for i in range(512)) # .encode("zlib").encode("base64") # Randomness in tests == bad, and fixing a seed may not be sufficient. ZLIB_PAYLOAD = b64decode( b"""\ eJwFweuaoQAAANDfineQhiKLUiaiCzvuTEmNNlJGiL5QhnGpZ99z8luQfe1AHoMioB+QSWHQu/L+ lzd7W5CipqYmeVTBjdgSATdg4l4Z2zhikbuF+EKn69Q0DTpdmNJz8S33odfJoVEexw/l2SS9nFdi pis7KOwXzfSqarSo9uJYgbDGrs1VNnQpT9f8zAorhYCEZronZQF9DuDFfNK3Hecc+WHLnZLQptwk nufw8S9I43sEwxsT71BiqedHo0QeIrFE01F/4atVFXuJs2yxIOak3bvtXjUKAA6OKnQJ/nNvDGKZ Khe5TF36JbnKVjdcL1EUNpwrWVfQpFYJ/WWm2b74qNeSZeQv5/xBhRdOmKTJFYgO96PwrHBlsnLn a3l0LwJsloWpMbzByU5WLbRE6X5INFqjQOtIwYz5BAlhkn+kVqJvWM5vBlfrwP42ifonM5yF4ciJ auHVks62997mNGOsM7WXNG3P98dBHPo2NhbTvHleL0BI5dus2JY81MUOnK3SGWLH8HeWPa1t5KcW S5moAj5HexY/g/F8TctpxwsvyZp38dXeLDjSQvEQIkF7XR3YXbeZgKk3V34KGCPOAeeuQDIgyVhV nP4HF2uWHA==""" ) @pytest.fixture def sock() -> typing.Generator[socket.socket, None, None]: s = socket.socket() yield s s.close() class TestLegacyResponse: def test_getheaders(self) -> None: headers = {"host": "example.com"} r = HTTPResponse(headers=headers) with pytest.warns( DeprecationWarning, match=r"HTTPResponse.getheaders\(\) is deprecated", ): assert r.getheaders() == HTTPHeaderDict(headers) def test_getheader(self) -> None: headers = {"host": "example.com"} r = HTTPResponse(headers=headers) with pytest.warns( DeprecationWarning, match=r"HTTPResponse.getheader\(\) is deprecated", ): assert r.getheader("host") == "example.com" class TestResponse: def test_cache_content(self) -> None: r = HTTPResponse(b"foo") assert r._body == b"foo" assert r.data == b"foo" assert r._body == b"foo" def test_cache_content_preload_false(self) -> None: fp = BytesIO(b"foo") r = HTTPResponse(fp, preload_content=False) assert not r._body assert r.data == b"foo" assert r._body == b"foo" assert r.data == b"foo" def test_default(self) -> None: r = HTTPResponse() assert r.data is None def test_none(self) -> None: r = HTTPResponse(None) # type: ignore[arg-type] assert r.data is None def test_preload(self) -> None: fp = BytesIO(b"foo") r = HTTPResponse(fp, preload_content=True) assert fp.tell() == len(b"foo") assert r.data == b"foo" def test_no_preload(self) -> None: fp = BytesIO(b"foo") r = HTTPResponse(fp, preload_content=False) assert fp.tell() == 0 assert r.data == b"foo" assert fp.tell() == len(b"foo") def test_decode_bad_data(self) -> None: fp = BytesIO(b"\x00" * 10) with pytest.raises(DecodeError): HTTPResponse(fp, headers={"content-encoding": "deflate"}) def test_reference_read(self) -> None: fp = BytesIO(b"foo") r = HTTPResponse(fp, preload_content=False) assert r.read(0) == b"" assert r.read(1) == b"f" assert r.read(2) == b"oo" assert r.read() == b"" assert r.read() == b"" def test_decode_deflate(self) -> None: data = zlib.compress(b"foo") fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "deflate"}) assert r.data == b"foo" def test_decode_deflate_case_insensitve(self) -> None: data = zlib.compress(b"foo") fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "DeFlAtE"}) assert r.data == b"foo" def test_chunked_decoding_deflate(self) -> None: data = zlib.compress(b"foo") fp = BytesIO(data) r = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) assert r.read(1) == b"f" assert r.read(2) == b"oo" assert r.read() == b"" assert r.read() == b"" def test_chunked_decoding_deflate2(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, -zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() fp = BytesIO(data) r = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) assert r.read(1) == b"f" assert r.read(2) == b"oo" assert r.read() == b"" assert r.read() == b"" def test_chunked_decoding_gzip(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() fp = BytesIO(data) r = HTTPResponse( fp, headers={"content-encoding": "gzip"}, preload_content=False ) assert r.read(1) == b"f" assert r.read(2) == b"oo" assert r.read() == b"" assert r.read() == b"" def test_decode_gzip_multi_member(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() data = data * 3 fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "gzip"}) assert r.data == b"foofoofoo" def test_decode_gzip_error(self) -> None: fp = BytesIO(b"foo") with pytest.raises(DecodeError): HTTPResponse(fp, headers={"content-encoding": "gzip"}) def test_decode_gzip_swallow_garbage(self) -> None: # When data comes from multiple calls to read(), data after # the first zlib error (here triggered by garbage) should be # ignored. compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() data = data * 3 + b"foo" fp = BytesIO(data) r = HTTPResponse( fp, headers={"content-encoding": "gzip"}, preload_content=False ) ret = b"" for _ in range(100): ret += r.read(1) if r.closed: break assert ret == b"foofoofoo" def test_chunked_decoding_gzip_swallow_garbage(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() data = data * 3 + b"foo" fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "gzip"}) assert r.data == b"foofoofoo" @onlyBrotli() def test_decode_brotli(self) -> None: data = brotli.compress(b"foo") fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "br"}) assert r.data == b"foo" @onlyBrotli() def test_chunked_decoding_brotli(self) -> None: data = brotli.compress(b"foobarbaz") fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "br"}, preload_content=False) ret = b"" for _ in range(100): ret += r.read(1) if r.closed: break assert ret == b"foobarbaz" @onlyBrotli() def test_decode_brotli_error(self) -> None: fp = BytesIO(b"foo") with pytest.raises(DecodeError): HTTPResponse(fp, headers={"content-encoding": "br"}) @onlyZstd() def test_decode_zstd(self) -> None: data = zstd.compress(b"foo") fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "zstd"}) assert r.data == b"foo" @onlyZstd() def test_decode_multiframe_zstd(self) -> None: data = ( # Zstandard frame zstd.compress(b"foo") # skippable frame (must be ignored) + bytes.fromhex( "50 2A 4D 18" # Magic_Number (little-endian) "07 00 00 00" # Frame_Size (little-endian) "00 00 00 00 00 00 00" # User_Data ) # Zstandard frame + zstd.compress(b"bar") ) fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "zstd"}) assert r.data == b"foobar" @onlyZstd() def test_chunked_decoding_zstd(self) -> None: data = zstd.compress(b"foobarbaz") fp = BytesIO(data) r = HTTPResponse( fp, headers={"content-encoding": "zstd"}, preload_content=False ) ret = b"" for _ in range(100): ret += r.read(1) if r.closed: break assert ret == b"foobarbaz" @onlyZstd() @pytest.mark.parametrize("data", [b"foo", b"x" * 100]) def test_decode_zstd_error(self, data: bytes) -> None: fp = BytesIO(data) with pytest.raises(DecodeError): HTTPResponse(fp, headers={"content-encoding": "zstd"}) @onlyZstd() @pytest.mark.parametrize("data", [b"foo", b"x" * 100]) def test_decode_zstd_incomplete(self, data: bytes) -> None: data = zstd.compress(data) fp = BytesIO(data[:-1]) with pytest.raises(DecodeError): HTTPResponse(fp, headers={"content-encoding": "zstd"}) def test_multi_decoding_deflate_deflate(self) -> None: data = zlib.compress(zlib.compress(b"foo")) fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "deflate, deflate"}) assert r.data == b"foo" def test_multi_decoding_deflate_gzip(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(zlib.compress(b"foo")) data += compress.flush() fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "deflate, gzip"}) assert r.data == b"foo" def test_multi_decoding_gzip_gzip(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(data) data += compress.flush() fp = BytesIO(data) r = HTTPResponse(fp, headers={"content-encoding": "gzip, gzip"}) assert r.data == b"foo" def test_read_multi_decoding_deflate_deflate(self) -> None: msg = b"foobarbaz" * 42 data = zlib.compress(zlib.compress(msg)) fp = BytesIO(data) r = HTTPResponse( fp, headers={"content-encoding": "deflate, deflate"}, preload_content=False ) assert r.read(3) == b"foo" assert r.read(3) == b"bar" assert r.read(3) == b"baz" assert r.read(9) == b"foobarbaz" assert r.read(9 * 3) == b"foobarbaz" * 3 assert r.read(9 * 37) == b"foobarbaz" * 37 assert r.read() == b"" def test_body_blob(self) -> None: resp = HTTPResponse(b"foo") assert resp.data == b"foo" assert resp.closed def test_base_io(self) -> None: resp = BaseHTTPResponse( status=200, version=11, reason=None, decode_content=False, request_url=None, ) assert not resp.closed assert not resp.readable() assert not resp.writable() with pytest.raises(NotImplementedError): resp.read() with pytest.raises(NotImplementedError): resp.close() def test_io(self, sock: socket.socket) -> None: fp = BytesIO(b"foo") resp = HTTPResponse(fp, preload_content=False) assert not resp.closed assert resp.readable() assert not resp.writable() with pytest.raises(IOError): resp.fileno() resp.close() assert resp.closed # Try closing with an `httplib.HTTPResponse`, because it has an # `isclosed` method. try: hlr = httplib.HTTPResponse(sock) resp2 = HTTPResponse(hlr, preload_content=False) assert not resp2.closed resp2.close() assert resp2.closed finally: hlr.close() # also try when only data is present. resp3 = HTTPResponse("foodata") with pytest.raises(IOError): resp3.fileno() resp3._fp = 2 # A corner case where _fp is present but doesn't have `closed`, # `isclosed`, or `fileno`. Unlikely, but possible. assert resp3.closed with pytest.raises(IOError): resp3.fileno() def test_io_closed_consistently(self, sock: socket.socket) -> None: try: hlr = httplib.HTTPResponse(sock) hlr.fp = BytesIO(b"foo") # type: ignore[assignment] hlr.chunked = 0 # type: ignore[assignment] hlr.length = 3 with HTTPResponse(hlr, preload_content=False) as resp: assert not resp.closed assert resp._fp is not None assert not resp._fp.isclosed() assert not is_fp_closed(resp._fp) assert not resp.isclosed() resp.read() assert resp.closed assert resp._fp.isclosed() assert is_fp_closed(resp._fp) assert resp.isclosed() finally: hlr.close() def test_io_bufferedreader(self) -> None: fp = BytesIO(b"foo") resp = HTTPResponse(fp, preload_content=False) br = BufferedReader(resp) # type: ignore[arg-type] assert br.read() == b"foo" br.close() assert resp.closed # HTTPResponse.read() by default closes the response # https://github.com/urllib3/urllib3/issues/1305 fp = BytesIO(b"hello\nworld") resp = HTTPResponse(fp, preload_content=False) with pytest.raises(ValueError, match="readline of closed file"): list(BufferedReader(resp)) # type: ignore[arg-type] b = b"fooandahalf" fp = BytesIO(b) resp = HTTPResponse(fp, preload_content=False) br = BufferedReader(resp, 5) # type: ignore[arg-type] br.read(1) # sets up the buffer, reading 5 assert len(fp.read()) == (len(b) - 5) # This is necessary to make sure the "no bytes left" part of `readinto` # gets tested. while not br.closed: br.read(5) def test_io_not_autoclose_bufferedreader(self) -> None: fp = BytesIO(b"hello\nworld") resp = HTTPResponse(fp, preload_content=False, auto_close=False) reader = BufferedReader(resp) # type: ignore[arg-type] assert list(reader) == [b"hello\n", b"world"] assert not reader.closed assert not resp.closed with pytest.raises(StopIteration): next(reader) reader.close() assert reader.closed assert resp.closed with pytest.raises(ValueError, match="readline of closed file"): next(reader) def test_io_textiowrapper(self) -> None: fp = BytesIO(b"\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f") resp = HTTPResponse(fp, preload_content=False) br = TextIOWrapper(resp, encoding="utf8") # type: ignore[arg-type] assert br.read() == "äöüß" br.close() assert resp.closed # HTTPResponse.read() by default closes the response # https://github.com/urllib3/urllib3/issues/1305 fp = BytesIO( b"\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f\n\xce\xb1\xce\xb2\xce\xb3\xce\xb4" ) resp = HTTPResponse(fp, preload_content=False) with pytest.raises(ValueError, match="I/O operation on closed file.?"): list(TextIOWrapper(resp)) # type: ignore[arg-type] def test_io_not_autoclose_textiowrapper(self) -> None: fp = BytesIO( b"\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f\n\xce\xb1\xce\xb2\xce\xb3\xce\xb4" ) resp = HTTPResponse(fp, preload_content=False, auto_close=False) reader = TextIOWrapper(resp, encoding="utf8") # type: ignore[arg-type] assert list(reader) == ["äöüß\n", "αβγδ"] assert not reader.closed assert not resp.closed with pytest.raises(StopIteration): next(reader) reader.close() assert reader.closed assert resp.closed with pytest.raises(ValueError, match="I/O operation on closed file.?"): next(reader) def test_read_with_illegal_mix_decode_toggle(self) -> None: data = zlib.compress(b"foo") fp = BytesIO(data) resp = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) assert resp.read(1) == b"f" with pytest.raises( RuntimeError, match=( r"Calling read\(decode_content=False\) is not supported after " r"read\(decode_content=True\) was called" ), ): resp.read(1, decode_content=False) with pytest.raises( RuntimeError, match=( r"Calling read\(decode_content=False\) is not supported after " r"read\(decode_content=True\) was called" ), ): resp.read(decode_content=False) def test_read_with_mix_decode_toggle(self) -> None: data = zlib.compress(b"foo") fp = BytesIO(data) resp = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) assert resp.read(2, decode_content=False) is not None assert resp.read(1, decode_content=True) == b"f" def test_streaming(self) -> None: fp = BytesIO(b"foo") resp = HTTPResponse(fp, preload_content=False) stream = resp.stream(2, decode_content=False) assert next(stream) == b"fo" assert next(stream) == b"o" with pytest.raises(StopIteration): next(stream) def test_streaming_tell(self) -> None: fp = BytesIO(b"foo") resp = HTTPResponse(fp, preload_content=False) stream = resp.stream(2, decode_content=False) position = 0 position += len(next(stream)) assert 2 == position assert position == resp.tell() position += len(next(stream)) assert 3 == position assert position == resp.tell() with pytest.raises(StopIteration): next(stream) def test_gzipped_streaming(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() fp = BytesIO(data) resp = HTTPResponse( fp, headers={"content-encoding": "gzip"}, preload_content=False ) stream = resp.stream(2) assert next(stream) == b"fo" assert next(stream) == b"o" with pytest.raises(StopIteration): next(stream) def test_gzipped_streaming_tell(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) uncompressed_data = b"foo" data = compress.compress(uncompressed_data) data += compress.flush() fp = BytesIO(data) resp = HTTPResponse( fp, headers={"content-encoding": "gzip"}, preload_content=False ) stream = resp.stream() # Read everything payload = next(stream) assert payload == uncompressed_data assert len(data) == resp.tell() with pytest.raises(StopIteration): next(stream) def test_deflate_streaming_tell_intermediate_point(self) -> None: # Ensure that ``tell()`` returns the correct number of bytes when # part-way through streaming compressed content. NUMBER_OF_READS = 10 PART_SIZE = 64 class MockCompressedDataReading(BytesIO): """ A BytesIO-like reader returning ``payload`` in ``NUMBER_OF_READS`` calls to ``read``. """ def __init__(self, payload: bytes, payload_part_size: int) -> None: self.payloads = [ payload[i * payload_part_size : (i + 1) * payload_part_size] for i in range(NUMBER_OF_READS + 1) ] assert b"".join(self.payloads) == payload def read(self, _: int) -> bytes: # type: ignore[override] # Amount is unused. if len(self.payloads) > 0: return self.payloads.pop(0) return b"" uncompressed_data = zlib.decompress(ZLIB_PAYLOAD) payload_part_size = len(ZLIB_PAYLOAD) // NUMBER_OF_READS fp = MockCompressedDataReading(ZLIB_PAYLOAD, payload_part_size) resp = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) stream = resp.stream(PART_SIZE) parts_positions = [(part, resp.tell()) for part in stream] end_of_stream = resp.tell() with pytest.raises(StopIteration): next(stream) parts, positions = zip(*parts_positions) # Check that the payload is equal to the uncompressed data payload = b"".join(parts) assert uncompressed_data == payload # Check that the positions in the stream are correct # It is difficult to determine programatically what the positions # returned by `tell` will be because the `HTTPResponse.read` method may # call socket `read` a couple of times if it doesn't have enough data # in the buffer or not call socket `read` at all if it has enough. All # this depends on the message, how it was compressed, what is # `PART_SIZE` and `payload_part_size`. # So for simplicity the expected values are hardcoded. expected = (92, 184, 230, 276, 322, 368, 414, 460) assert expected == positions # Check that the end of the stream is in the correct place assert len(ZLIB_PAYLOAD) == end_of_stream # Check that all parts have expected length expected_last_part_size = len(uncompressed_data) % PART_SIZE whole_parts = len(uncompressed_data) // PART_SIZE if expected_last_part_size == 0: expected_lengths = [PART_SIZE] * whole_parts else: expected_lengths = [PART_SIZE] * whole_parts + [expected_last_part_size] assert expected_lengths == [len(part) for part in parts] def test_deflate_streaming(self) -> None: data = zlib.compress(b"foo") fp = BytesIO(data) resp = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) stream = resp.stream(2) assert next(stream) == b"fo" assert next(stream) == b"o" with pytest.raises(StopIteration): next(stream) def test_deflate2_streaming(self) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, -zlib.MAX_WBITS) data = compress.compress(b"foo") data += compress.flush() fp = BytesIO(data) resp = HTTPResponse( fp, headers={"content-encoding": "deflate"}, preload_content=False ) stream = resp.stream(2) assert next(stream) == b"fo" assert next(stream) == b"o" with pytest.raises(StopIteration): next(stream) def test_empty_stream(self) -> None: fp = BytesIO(b"") resp = HTTPResponse(fp, preload_content=False) stream = resp.stream(2, decode_content=False) with pytest.raises(StopIteration): next(stream) @pytest.mark.parametrize( "preload_content, amt", [(True, None), (False, None), (False, 10 * 2**20)], ) @pytest.mark.limit_memory("25 MB") def test_buffer_memory_usage_decode_one_chunk( self, preload_content: bool, amt: int ) -> None: content_length = 10 * 2**20 # 10 MiB fp = BytesIO(zlib.compress(bytes(content_length))) resp = HTTPResponse( fp, preload_content=preload_content, headers={"content-encoding": "deflate"}, ) data = resp.data if preload_content else resp.read(amt) assert len(data) == content_length @pytest.mark.parametrize( "preload_content, amt", [(True, None), (False, None), (False, 10 * 2**20)], ) @pytest.mark.limit_memory("10.5 MB") def test_buffer_memory_usage_no_decoding( self, preload_content: bool, amt: int ) -> None: content_length = 10 * 2**20 # 10 MiB fp = BytesIO(bytes(content_length)) resp = HTTPResponse(fp, preload_content=preload_content, decode_content=False) data = resp.data if preload_content else resp.read(amt) assert len(data) == content_length def test_length_no_header(self) -> None: fp = BytesIO(b"12345") resp = HTTPResponse(fp, preload_content=False) assert resp.length_remaining is None def test_length_w_valid_header(self) -> None: headers = {"content-length": "5"} fp = BytesIO(b"12345") resp = HTTPResponse(fp, headers=headers, preload_content=False) assert resp.length_remaining == 5 def test_length_w_bad_header(self) -> None: garbage = {"content-length": "foo"} fp = BytesIO(b"12345") resp = HTTPResponse(fp, headers=garbage, preload_content=False) assert resp.length_remaining is None garbage["content-length"] = "-10" resp = HTTPResponse(fp, headers=garbage, preload_content=False) assert resp.length_remaining is None def test_length_when_chunked(self) -> None: # This is expressly forbidden in RFC 7230 sec 3.3.2 # We fall back to chunked in this case and try to # handle response ignoring content length. headers = {"content-length": "5", "transfer-encoding": "chunked"} fp = BytesIO(b"12345") resp = HTTPResponse(fp, headers=headers, preload_content=False) assert resp.length_remaining is None def test_length_with_multiple_content_lengths(self) -> None: headers = {"content-length": "5, 5, 5"} garbage = {"content-length": "5, 42"} fp = BytesIO(b"abcde") resp = HTTPResponse(fp, headers=headers, preload_content=False) assert resp.length_remaining == 5 with pytest.raises(InvalidHeader): HTTPResponse(fp, headers=garbage, preload_content=False) def test_length_after_read(self) -> None: headers = {"content-length": "5"} # Test no defined length fp = BytesIO(b"12345") resp = HTTPResponse(fp, preload_content=False) resp.read() assert resp.length_remaining is None # Test our update from content-length fp = BytesIO(b"12345") resp = HTTPResponse(fp, headers=headers, preload_content=False) resp.read() assert resp.length_remaining == 0 # Test partial read fp = BytesIO(b"12345") resp = HTTPResponse(fp, headers=headers, preload_content=False) data = resp.stream(2) next(data) assert resp.length_remaining == 3 def test_mock_httpresponse_stream(self) -> None: # Mock out a HTTP Request that does enough to make it through urllib3's # read() and close() calls, and also exhausts and underlying file # object. class MockHTTPRequest: def __init__(self) -> None: self.fp: BytesIO | None = None def read(self, amt: int) -> bytes: assert self.fp is not None data = self.fp.read(amt) if not data: self.fp = None return data def close(self) -> None: self.fp = None bio = BytesIO(b"foo") fp = MockHTTPRequest() fp.fp = bio resp = HTTPResponse(fp, preload_content=False) # type: ignore[arg-type] stream = resp.stream(2) assert next(stream) == b"fo" assert next(stream) == b"o" with pytest.raises(StopIteration): next(stream) def test_mock_transfer_encoding_chunked(self) -> None: stream = [b"fo", b"o", b"bar"] fp = MockChunkedEncodingResponse(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) for i, c in enumerate(resp.stream()): assert c == stream[i] def test_mock_gzipped_transfer_encoding_chunked_decoded(self) -> None: """Show that we can decode the gzipped and chunked body.""" def stream() -> typing.Generator[bytes, None, None]: # Set up a generator to chunk the gzipped body compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foobar") data += compress.flush() for i in range(0, len(data), 2): yield data[i : i + 2] fp = MockChunkedEncodingResponse(list(stream())) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] headers = {"transfer-encoding": "chunked", "content-encoding": "gzip"} resp = HTTPResponse(r, preload_content=False, headers=headers) data = b"" for c in resp.stream(decode_content=True): data += c assert b"foobar" == data def test_mock_transfer_encoding_chunked_custom_read(self) -> None: stream = [b"foooo", b"bbbbaaaaar"] fp = MockChunkedEncodingResponse(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] r.chunked = True r.chunk_left = None resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) expected_response = [b"fo", b"oo", b"o", b"bb", b"bb", b"aa", b"aa", b"ar"] response = list(resp.read_chunked(2)) assert expected_response == response def test_mock_transfer_encoding_chunked_unlmtd_read(self) -> None: stream = [b"foooo", b"bbbbaaaaar"] fp = MockChunkedEncodingResponse(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] r.chunked = True r.chunk_left = None resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) assert stream == list(resp.read_chunked()) def test_read_not_chunked_response_as_chunks(self) -> None: fp = BytesIO(b"foo") resp = HTTPResponse(fp, preload_content=False) r = resp.read_chunked() with pytest.raises(ResponseNotChunked): next(r) def test_read_chunked_not_supported(self) -> None: fp = BytesIO(b"foo") resp = HTTPResponse( fp, preload_content=False, headers={"transfer-encoding": "chunked"} ) r = resp.read_chunked() with pytest.raises(BodyNotHttplibCompatible): next(r) def test_buggy_incomplete_read(self) -> None: # Simulate buggy versions of Python (<2.7.4) # See http://bugs.python.org/issue16298 content_length = 1337 fp = BytesIO(b"") resp = HTTPResponse( fp, headers={"content-length": str(content_length)}, preload_content=False, enforce_content_length=True, ) with pytest.raises(ProtocolError) as ctx: resp.read(3) orig_ex = ctx.value.args[1] assert isinstance(orig_ex, IncompleteRead) assert orig_ex.partial == 0 # type: ignore[comparison-overlap] assert orig_ex.expected == content_length def test_incomplete_chunk(self) -> None: stream = [b"foooo", b"bbbbaaaaar"] fp = MockChunkedIncompleteRead(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] r.chunked = True r.chunk_left = None resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) with pytest.raises(ProtocolError) as ctx: next(resp.read_chunked()) orig_ex = ctx.value.args[1] assert isinstance(orig_ex, httplib_IncompleteRead) def test_invalid_chunk_length(self) -> None: stream = [b"foooo", b"bbbbaaaaar"] fp = MockChunkedInvalidChunkLength(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] r.chunked = True r.chunk_left = None resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) with pytest.raises(ProtocolError) as ctx: next(resp.read_chunked()) orig_ex = ctx.value.args[1] msg = ( "(\"Connection broken: InvalidChunkLength(got length b'ZZZ\\\\r\\\\n', 0 bytes read)\", " "InvalidChunkLength(got length b'ZZZ\\r\\n', 0 bytes read))" ) assert str(ctx.value) == msg assert isinstance(orig_ex, InvalidChunkLength) assert orig_ex.length == fp.BAD_LENGTH_LINE.encode() def test_chunked_response_without_crlf_on_end(self) -> None: stream = [b"foo", b"bar", b"baz"] fp = MockChunkedEncodingWithoutCRLFOnEnd(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] r.chunked = True r.chunk_left = None resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) assert stream == list(resp.stream()) def test_chunked_response_with_extensions(self) -> None: stream = [b"foo", b"bar"] fp = MockChunkedEncodingWithExtensions(stream) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] r.chunked = True r.chunk_left = None resp = HTTPResponse( r, preload_content=False, headers={"transfer-encoding": "chunked"} ) assert stream == list(resp.stream()) def test_chunked_head_response(self) -> None: r = httplib.HTTPResponse(MockSock, method="HEAD") # type: ignore[arg-type] r.chunked = True r.chunk_left = None resp = HTTPResponse( "", preload_content=False, headers={"transfer-encoding": "chunked"}, original_response=r, ) assert resp.chunked is True setattr(resp, "supports_chunked_reads", lambda: True) setattr(resp, "release_conn", mock.Mock()) for _ in resp.stream(): continue resp.release_conn.assert_called_once_with() # type: ignore[attr-defined] def test_get_case_insensitive_headers(self) -> None: headers = {"host": "example.com"} r = HTTPResponse(headers=headers) assert r.headers.get("host") == "example.com" assert r.headers.get("Host") == "example.com" def test_retries(self) -> None: fp = BytesIO(b"") resp = HTTPResponse(fp) assert resp.retries is None retry = Retry() resp = HTTPResponse(fp, retries=retry) assert resp.retries == retry def test_geturl(self) -> None: fp = BytesIO(b"") request_url = "https://example.com" resp = HTTPResponse(fp, request_url=request_url) assert resp.geturl() == request_url def test_url(self) -> None: fp = BytesIO(b"") request_url = "https://example.com" resp = HTTPResponse(fp, request_url=request_url) assert resp.url == request_url resp.url = "https://anotherurl.com" assert resp.url == "https://anotherurl.com" def test_geturl_retries(self) -> None: fp = BytesIO(b"") resp = HTTPResponse(fp, request_url="http://example.com") request_histories = ( RequestHistory( method="GET", url="http://example.com", error=None, status=301, redirect_location="https://example.com/", ), RequestHistory( method="GET", url="https://example.com/", error=None, status=301, redirect_location="https://www.example.com", ), ) retry = Retry(history=request_histories) resp = HTTPResponse(fp, retries=retry) assert resp.geturl() == "https://www.example.com" @pytest.mark.parametrize( ["payload", "expected_stream"], [ (b"", []), (b"\n", [b"\n"]), (b"\n\n\n", [b"\n", b"\n", b"\n"]), (b"abc\ndef", [b"abc\n", b"def"]), (b"Hello\nworld\n\n\n!", [b"Hello\n", b"world\n", b"\n", b"\n", b"!"]), ], ) def test__iter__(self, payload: bytes, expected_stream: list[bytes]) -> None: actual_stream = [] for chunk in HTTPResponse(BytesIO(payload), preload_content=False): actual_stream.append(chunk) assert actual_stream == expected_stream def test__iter__decode_content(self) -> None: def stream() -> typing.Generator[bytes, None, None]: # Set up a generator to chunk the gzipped body compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"foo\nbar") data += compress.flush() for i in range(0, len(data), 2): yield data[i : i + 2] fp = MockChunkedEncodingResponse(list(stream())) r = httplib.HTTPResponse(MockSock) # type: ignore[arg-type] r.fp = fp # type: ignore[assignment] headers = {"transfer-encoding": "chunked", "content-encoding": "gzip"} resp = HTTPResponse(r, preload_content=False, headers=headers) data = b"" for c in resp: data += c assert b"foo\nbar" == data def test_non_timeout_ssl_error_on_read(self) -> None: mac_error = ssl.SSLError( "SSL routines", "ssl3_get_record", "decryption failed or bad record mac" ) @contextlib.contextmanager def make_bad_mac_fp() -> typing.Generator[BytesIO, None, None]: fp = BytesIO(b"") with mock.patch.object(fp, "read") as fp_read: # mac/decryption error fp_read.side_effect = mac_error yield fp with make_bad_mac_fp() as fp: with pytest.raises(SSLError) as e: HTTPResponse(fp) assert e.value.args[0] == mac_error with make_bad_mac_fp() as fp: resp = HTTPResponse(fp, preload_content=False) with pytest.raises(SSLError) as e: resp.read() assert e.value.args[0] == mac_error class MockChunkedEncodingResponse: def __init__(self, content: list[bytes]) -> None: """ content: collection of str, each str is a chunk in response """ self.content = content self.index = 0 # This class iterates over self.content. self.closed = False self.cur_chunk = b"" self.chunks_exhausted = False def _encode_chunk(self, chunk: bytes) -> bytes: # In the general case, we can't decode the chunk to unicode length = f"{len(chunk):X}\r\n" return length.encode() + chunk + b"\r\n" def _pop_new_chunk(self) -> bytes: if self.chunks_exhausted: return b"" try: chunk = self.content[self.index] except IndexError: chunk = b"" self.chunks_exhausted = True else: self.index += 1 chunk = self._encode_chunk(chunk) if not isinstance(chunk, bytes): chunk = chunk.encode() assert isinstance(chunk, bytes) return chunk def pop_current_chunk(self, amt: int = -1, till_crlf: bool = False) -> bytes: if amt > 0 and till_crlf: raise ValueError("Can't specify amt and till_crlf.") if len(self.cur_chunk) <= 0: self.cur_chunk = self._pop_new_chunk() if till_crlf: try: i = self.cur_chunk.index(b"\r\n") except ValueError: # No CRLF in current chunk -- probably caused by encoder. self.cur_chunk = b"" return b"" else: chunk_part = self.cur_chunk[: i + 2] self.cur_chunk = self.cur_chunk[i + 2 :] return chunk_part elif amt <= -1: chunk_part = self.cur_chunk self.cur_chunk = b"" return chunk_part else: try: chunk_part = self.cur_chunk[:amt] except IndexError: chunk_part = self.cur_chunk self.cur_chunk = b"" else: self.cur_chunk = self.cur_chunk[amt:] return chunk_part def readline(self) -> bytes: return self.pop_current_chunk(till_crlf=True) def read(self, amt: int = -1) -> bytes: return self.pop_current_chunk(amt) def flush(self) -> None: # Python 3 wants this method. pass def close(self) -> None: self.closed = True class MockChunkedIncompleteRead(MockChunkedEncodingResponse): def _encode_chunk(self, chunk: bytes) -> bytes: return f"9999\r\n{chunk.decode()}\r\n".encode() class MockChunkedInvalidChunkLength(MockChunkedEncodingResponse): BAD_LENGTH_LINE = "ZZZ\r\n" def _encode_chunk(self, chunk: bytes) -> bytes: return f"{self.BAD_LENGTH_LINE}{chunk.decode()}\r\n".encode() class MockChunkedEncodingWithoutCRLFOnEnd(MockChunkedEncodingResponse): def _encode_chunk(self, chunk: bytes) -> bytes: return "{:X}\r\n{}{}".format( len(chunk), chunk.decode(), "\r\n" if len(chunk) > 0 else "", ).encode() class MockChunkedEncodingWithExtensions(MockChunkedEncodingResponse): def _encode_chunk(self, chunk: bytes) -> bytes: return f"{len(chunk):X};asd=qwe\r\n{chunk.decode()}\r\n".encode() class MockSock: @classmethod def makefile(cls, *args: typing.Any, **kwargs: typing.Any) -> None: return urllib3-2.0.7/test/test_retry.py0000644000000000000000000004040014513541732013562 0ustar00from __future__ import annotations import datetime from test import DUMMY_POOL from unittest import mock import pytest from urllib3.exceptions import ( ConnectTimeoutError, InvalidHeader, MaxRetryError, ReadTimeoutError, ResponseError, SSLError, ) from urllib3.response import HTTPResponse from urllib3.util.retry import RequestHistory, Retry class TestRetry: def test_string(self) -> None: """Retry string representation looks the way we expect""" retry = Retry() assert ( str(retry) == "Retry(total=10, connect=None, read=None, redirect=None, status=None)" ) for _ in range(3): retry = retry.increment(method="GET") assert ( str(retry) == "Retry(total=7, connect=None, read=None, redirect=None, status=None)" ) def test_retry_both_specified(self) -> None: """Total can win if it's lower than the connect value""" error = ConnectTimeoutError() retry = Retry(connect=3, total=2) retry = retry.increment(error=error) retry = retry.increment(error=error) with pytest.raises(MaxRetryError) as e: retry.increment(error=error) assert e.value.reason == error def test_retry_higher_total_loses(self) -> None: """A lower connect timeout than the total is honored""" error = ConnectTimeoutError() retry = Retry(connect=2, total=3) retry = retry.increment(error=error) retry = retry.increment(error=error) with pytest.raises(MaxRetryError): retry.increment(error=error) def test_retry_higher_total_loses_vs_read(self) -> None: """A lower read timeout than the total is honored""" error = ReadTimeoutError(DUMMY_POOL, "/", "read timed out") retry = Retry(read=2, total=3) retry = retry.increment(method="GET", error=error) retry = retry.increment(method="GET", error=error) with pytest.raises(MaxRetryError): retry.increment(method="GET", error=error) def test_retry_total_none(self) -> None: """if Total is none, connect error should take precedence""" error = ConnectTimeoutError() retry = Retry(connect=2, total=None) retry = retry.increment(error=error) retry = retry.increment(error=error) with pytest.raises(MaxRetryError) as e: retry.increment(error=error) assert e.value.reason == error timeout_error = ReadTimeoutError(DUMMY_POOL, "/", "read timed out") retry = Retry(connect=2, total=None) retry = retry.increment(method="GET", error=timeout_error) retry = retry.increment(method="GET", error=timeout_error) retry = retry.increment(method="GET", error=timeout_error) assert not retry.is_exhausted() def test_retry_default(self) -> None: """If no value is specified, should retry connects 3 times""" retry = Retry() assert retry.total == 10 assert retry.connect is None assert retry.read is None assert retry.redirect is None assert retry.other is None error = ConnectTimeoutError() retry = Retry(connect=1) retry = retry.increment(error=error) with pytest.raises(MaxRetryError): retry.increment(error=error) retry = Retry(connect=1) retry = retry.increment(error=error) assert not retry.is_exhausted() assert Retry(0).raise_on_redirect assert not Retry(False).raise_on_redirect def test_retry_other(self) -> None: """If an unexpected error is raised, should retry other times""" other_error = SSLError() retry = Retry(connect=1) retry = retry.increment(error=other_error) retry = retry.increment(error=other_error) assert not retry.is_exhausted() retry = Retry(other=1) retry = retry.increment(error=other_error) with pytest.raises(MaxRetryError) as e: retry.increment(error=other_error) assert e.value.reason == other_error def test_retry_read_zero(self) -> None: """No second chances on read timeouts, by default""" error = ReadTimeoutError(DUMMY_POOL, "/", "read timed out") retry = Retry(read=0) with pytest.raises(MaxRetryError) as e: retry.increment(method="GET", error=error) assert e.value.reason == error def test_status_counter(self) -> None: resp = HTTPResponse(status=400) retry = Retry(status=2) retry = retry.increment(response=resp) retry = retry.increment(response=resp) msg = ResponseError.SPECIFIC_ERROR.format(status_code=400) with pytest.raises(MaxRetryError, match=msg): retry.increment(response=resp) def test_backoff(self) -> None: """Backoff is computed correctly""" max_backoff = Retry.DEFAULT_BACKOFF_MAX retry = Retry(total=100, backoff_factor=0.2) assert retry.get_backoff_time() == 0 # First request retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0 # First retry retry = retry.increment(method="GET") assert retry.backoff_factor == 0.2 assert retry.total == 98 assert retry.get_backoff_time() == 0.4 # Start backoff retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0.8 retry = retry.increment(method="GET") assert retry.get_backoff_time() == 1.6 for _ in range(10): retry = retry.increment(method="GET") assert retry.get_backoff_time() == max_backoff def test_configurable_backoff_max(self) -> None: """Configurable backoff is computed correctly""" max_backoff = 1 retry = Retry(total=100, backoff_factor=0.2, backoff_max=max_backoff) assert retry.get_backoff_time() == 0 # First request retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0 # First retry retry = retry.increment(method="GET") assert retry.backoff_factor == 0.2 assert retry.total == 98 assert retry.get_backoff_time() == 0.4 # Start backoff retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0.8 retry = retry.increment(method="GET") assert retry.get_backoff_time() == max_backoff retry = retry.increment(method="GET") assert retry.get_backoff_time() == max_backoff def test_backoff_jitter(self) -> None: """Backoff with jitter is computed correctly""" max_backoff = 1 jitter = 0.4 retry = Retry( total=100, backoff_factor=0.2, backoff_max=max_backoff, backoff_jitter=jitter, ) assert retry.get_backoff_time() == 0 # First request retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0 # First retry retry = retry.increment(method="GET") assert retry.backoff_factor == 0.2 assert retry.total == 98 assert 0.4 <= retry.get_backoff_time() <= 0.8 # Start backoff retry = retry.increment(method="GET") assert 0.8 <= retry.get_backoff_time() <= max_backoff retry = retry.increment(method="GET") assert retry.get_backoff_time() == max_backoff retry = retry.increment(method="GET") assert retry.get_backoff_time() == max_backoff def test_zero_backoff(self) -> None: retry = Retry() assert retry.get_backoff_time() == 0 retry = retry.increment(method="GET") retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0 def test_backoff_reset_after_redirect(self) -> None: retry = Retry(total=100, redirect=5, backoff_factor=0.2) retry = retry.increment(method="GET") retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0.4 redirect_response = HTTPResponse(status=302, headers={"location": "test"}) retry = retry.increment(method="GET", response=redirect_response) assert retry.get_backoff_time() == 0 retry = retry.increment(method="GET") retry = retry.increment(method="GET") assert retry.get_backoff_time() == 0.4 def test_sleep(self) -> None: # sleep a very small amount of time so our code coverage is happy retry = Retry(backoff_factor=0.0001) retry = retry.increment(method="GET") retry = retry.increment(method="GET") retry.sleep() def test_status_forcelist(self) -> None: retry = Retry(status_forcelist=range(500, 600)) assert not retry.is_retry("GET", status_code=200) assert not retry.is_retry("GET", status_code=400) assert retry.is_retry("GET", status_code=500) retry = Retry(total=1, status_forcelist=[418]) assert not retry.is_retry("GET", status_code=400) assert retry.is_retry("GET", status_code=418) # String status codes are not matched. retry = Retry(total=1, status_forcelist=["418"]) # type: ignore[list-item] assert not retry.is_retry("GET", status_code=418) def test_allowed_methods_with_status_forcelist(self) -> None: # Falsey allowed_methods means to retry on any method. retry = Retry(status_forcelist=[500], allowed_methods=None) assert retry.is_retry("GET", status_code=500) assert retry.is_retry("POST", status_code=500) # Criteria of allowed_methods and status_forcelist are ANDed. retry = Retry(status_forcelist=[500], allowed_methods=["POST"]) assert not retry.is_retry("GET", status_code=500) assert retry.is_retry("POST", status_code=500) def test_exhausted(self) -> None: assert not Retry(0).is_exhausted() assert Retry(-1).is_exhausted() assert Retry(1).increment(method="GET").total == 0 @pytest.mark.parametrize("total", [-1, 0]) def test_disabled(self, total: int) -> None: with pytest.raises(MaxRetryError): Retry(total).increment(method="GET") def test_error_message(self) -> None: retry = Retry(total=0) with pytest.raises(MaxRetryError, match="read timed out") as e: retry = retry.increment( method="GET", error=ReadTimeoutError(DUMMY_POOL, "/", "read timed out") ) assert "Caused by redirect" not in str(e.value) retry = Retry(total=1) retry = retry.increment("POST", "/") with pytest.raises(MaxRetryError, match=ResponseError.GENERIC_ERROR) as e: retry = retry.increment("POST", "/") assert "Caused by redirect" not in str(e.value) assert isinstance(e.value.reason, ResponseError) retry = Retry(total=1) response = HTTPResponse(status=500) msg = ResponseError.SPECIFIC_ERROR.format(status_code=500) retry = retry.increment("POST", "/", response=response) with pytest.raises(MaxRetryError, match=msg) as e: retry = retry.increment("POST", "/", response=response) assert "Caused by redirect" not in str(e.value) retry = Retry(connect=1) retry = retry.increment(error=ConnectTimeoutError("conntimeout")) with pytest.raises(MaxRetryError, match="conntimeout") as e: retry = retry.increment(error=ConnectTimeoutError("conntimeout")) assert "Caused by redirect" not in str(e.value) def test_history(self) -> None: retry = Retry(total=10, allowed_methods=frozenset(["GET", "POST"])) assert retry.history == tuple() connection_error = ConnectTimeoutError("conntimeout") retry = retry.increment("GET", "/test1", None, connection_error) test_history1 = (RequestHistory("GET", "/test1", connection_error, None, None),) assert retry.history == test_history1 read_error = ReadTimeoutError(DUMMY_POOL, "/test2", "read timed out") retry = retry.increment("POST", "/test2", None, read_error) test_history2 = ( RequestHistory("GET", "/test1", connection_error, None, None), RequestHistory("POST", "/test2", read_error, None, None), ) assert retry.history == test_history2 response = HTTPResponse(status=500) retry = retry.increment("GET", "/test3", response, None) test_history3 = ( RequestHistory("GET", "/test1", connection_error, None, None), RequestHistory("POST", "/test2", read_error, None, None), RequestHistory("GET", "/test3", None, 500, None), ) assert retry.history == test_history3 def test_retry_method_not_allowed(self) -> None: error = ReadTimeoutError(DUMMY_POOL, "/", "read timed out") retry = Retry() with pytest.raises(ReadTimeoutError): retry.increment(method="POST", error=error) def test_retry_default_remove_headers_on_redirect(self) -> None: retry = Retry() assert retry.remove_headers_on_redirect == {"authorization", "cookie"} def test_retry_set_remove_headers_on_redirect(self) -> None: retry = Retry(remove_headers_on_redirect=["X-API-Secret"]) assert retry.remove_headers_on_redirect == {"x-api-secret"} @pytest.mark.parametrize("value", ["-1", "+1", "1.0", "\xb2"]) # \xb2 = ^2 def test_parse_retry_after_invalid(self, value: str) -> None: retry = Retry() with pytest.raises(InvalidHeader): retry.parse_retry_after(value) @pytest.mark.parametrize( "value, expected", [("0", 0), ("1000", 1000), ("\t42 ", 42)] ) def test_parse_retry_after(self, value: str, expected: int) -> None: retry = Retry() assert retry.parse_retry_after(value) == expected @pytest.mark.parametrize("respect_retry_after_header", [True, False]) def test_respect_retry_after_header_propagated( self, respect_retry_after_header: bool ) -> None: retry = Retry(respect_retry_after_header=respect_retry_after_header) new_retry = retry.new() assert new_retry.respect_retry_after_header == respect_retry_after_header @pytest.mark.parametrize( "retry_after_header,respect_retry_after_header,sleep_duration", [ ("3600", True, 3600), ("3600", False, None), # Will sleep due to header is 1 hour in future ("Mon, 3 Jun 2019 12:00:00 UTC", True, 3600), # Won't sleep due to not respecting header ("Mon, 3 Jun 2019 12:00:00 UTC", False, None), # Won't sleep due to current time reached ("Mon, 3 Jun 2019 11:00:00 UTC", True, None), # Won't sleep due to current time reached + not respecting header ("Mon, 3 Jun 2019 11:00:00 UTC", False, None), # Handle all the formats in RFC 7231 Section 7.1.1.1 ("Mon, 03 Jun 2019 11:30:12 GMT", True, 1812), ("Monday, 03-Jun-19 11:30:12 GMT", True, 1812), # Assume that datetimes without a timezone are in UTC per RFC 7231 ("Mon Jun 3 11:30:12 2019", True, 1812), ], ) @pytest.mark.parametrize( "stub_timezone", [ "UTC", "Asia/Jerusalem", None, ], indirect=True, ) @pytest.mark.usefixtures("stub_timezone") def test_respect_retry_after_header_sleep( self, retry_after_header: str, respect_retry_after_header: bool, sleep_duration: int | None, ) -> None: retry = Retry(respect_retry_after_header=respect_retry_after_header) with mock.patch( "time.time", return_value=datetime.datetime( 2019, 6, 3, 11, tzinfo=datetime.timezone.utc ).timestamp(), ), mock.patch("time.sleep") as sleep_mock: # for the default behavior, it must be in RETRY_AFTER_STATUS_CODES response = HTTPResponse( status=503, headers={"Retry-After": retry_after_header} ) retry.sleep(response) # The expected behavior is that we'll only sleep if respecting # this header (since we won't have any backoff sleep attempts) if respect_retry_after_header and sleep_duration is not None: sleep_mock.assert_called_with(sleep_duration) else: sleep_mock.assert_not_called() urllib3-2.0.7/test/test_ssl.py0000644000000000000000000001616114513541732013225 0ustar00from __future__ import annotations import ssl import typing from unittest import mock import pytest from urllib3.exceptions import ProxySchemeUnsupported, SSLError from urllib3.util import ssl_ class TestSSL: @pytest.mark.parametrize( "addr", [ # IPv6 "::1", "::", "FE80::8939:7684:D84b:a5A4%251", # IPv4 "127.0.0.1", "8.8.8.8", b"127.0.0.1", # IPv6 w/ Zone IDs "FE80::8939:7684:D84b:a5A4%251", b"FE80::8939:7684:D84b:a5A4%251", "FE80::8939:7684:D84b:a5A4%19", b"FE80::8939:7684:D84b:a5A4%19", ], ) def test_is_ipaddress_true(self, addr: bytes | str) -> None: assert ssl_.is_ipaddress(addr) @pytest.mark.parametrize( "addr", [ "www.python.org", b"www.python.org", "v2.sg.media-imdb.com", b"v2.sg.media-imdb.com", ], ) def test_is_ipaddress_false(self, addr: bytes | str) -> None: assert not ssl_.is_ipaddress(addr) def test_create_urllib3_context_set_ciphers( self, monkeypatch: pytest.MonkeyPatch ) -> None: ciphers = "ECDH+AESGCM:ECDH+CHACHA20" context = mock.create_autospec(ssl_.SSLContext) context.set_ciphers = mock.Mock() context.options = 0 monkeypatch.setattr(ssl_, "SSLContext", lambda *_, **__: context) assert ssl_.create_urllib3_context(ciphers=ciphers) is context assert context.set_ciphers.call_count == 1 assert context.set_ciphers.call_args == mock.call(ciphers) def test_create_urllib3_no_context(self) -> None: with mock.patch("urllib3.util.ssl_.SSLContext", None): with pytest.raises(TypeError): ssl_.create_urllib3_context() def test_wrap_socket_given_context_no_load_default_certs(self) -> None: context = mock.create_autospec(ssl_.SSLContext) context.load_default_certs = mock.Mock() sock = mock.Mock() ssl_.ssl_wrap_socket(sock, ssl_context=context) context.load_default_certs.assert_not_called() def test_wrap_socket_given_ca_certs_no_load_default_certs( self, monkeypatch: pytest.MonkeyPatch ) -> None: context = mock.create_autospec(ssl_.SSLContext) context.load_default_certs = mock.Mock() context.options = 0 monkeypatch.setattr(ssl_, "SSLContext", lambda *_, **__: context) sock = mock.Mock() ssl_.ssl_wrap_socket(sock, ca_certs="/tmp/fake-file") context.load_default_certs.assert_not_called() context.load_verify_locations.assert_called_with("/tmp/fake-file", None, None) def test_wrap_socket_default_loads_default_certs( self, monkeypatch: pytest.MonkeyPatch ) -> None: context = mock.create_autospec(ssl_.SSLContext) context.load_default_certs = mock.Mock() context.options = 0 monkeypatch.setattr(ssl_, "SSLContext", lambda *_, **__: context) sock = mock.Mock() ssl_.ssl_wrap_socket(sock) context.load_default_certs.assert_called_with() def test_wrap_socket_no_ssltransport(self) -> None: with mock.patch("urllib3.util.ssl_.SSLTransport", None): with pytest.raises(ProxySchemeUnsupported): sock = mock.Mock() ssl_.ssl_wrap_socket(sock, tls_in_tls=True) @pytest.mark.parametrize( ["pha", "expected_pha"], [(None, None), (False, True), (True, True)] ) def test_create_urllib3_context_pha( self, monkeypatch: pytest.MonkeyPatch, pha: bool | None, expected_pha: bool | None, ) -> None: context = mock.create_autospec(ssl_.SSLContext) context.set_ciphers = mock.Mock() context.options = 0 context.post_handshake_auth = pha monkeypatch.setattr(ssl_, "SSLContext", lambda *_, **__: context) assert ssl_.create_urllib3_context() is context assert context.post_handshake_auth == expected_pha def test_create_urllib3_context_default_ciphers( self, monkeypatch: pytest.MonkeyPatch ) -> None: context = mock.create_autospec(ssl_.SSLContext) context.set_ciphers = mock.Mock() context.options = 0 monkeypatch.setattr(ssl_, "SSLContext", lambda *_, **__: context) ssl_.create_urllib3_context() context.set_ciphers.assert_not_called() @pytest.mark.parametrize( "kwargs", [ { "ssl_version": ssl.PROTOCOL_TLSv1, "ssl_minimum_version": ssl.TLSVersion.MINIMUM_SUPPORTED, }, { "ssl_version": ssl.PROTOCOL_TLSv1, "ssl_maximum_version": ssl.TLSVersion.TLSv1, }, { "ssl_version": ssl.PROTOCOL_TLSv1, "ssl_minimum_version": ssl.TLSVersion.MINIMUM_SUPPORTED, "ssl_maximum_version": ssl.TLSVersion.MAXIMUM_SUPPORTED, }, ], ) def test_create_urllib3_context_ssl_version_and_ssl_min_max_version_errors( self, kwargs: dict[str, typing.Any] ) -> None: with pytest.raises(ValueError) as e: ssl_.create_urllib3_context(**kwargs) assert str(e.value) == ( "Can't specify both 'ssl_version' and either 'ssl_minimum_version' or 'ssl_maximum_version'" ) @pytest.mark.parametrize( "kwargs", [ { "ssl_version": ssl.PROTOCOL_TLS, "ssl_minimum_version": ssl.TLSVersion.MINIMUM_SUPPORTED, }, { "ssl_version": ssl.PROTOCOL_TLS_CLIENT, "ssl_minimum_version": ssl.TLSVersion.MINIMUM_SUPPORTED, }, { "ssl_version": None, "ssl_minimum_version": ssl.TLSVersion.MINIMUM_SUPPORTED, }, ], ) def test_create_urllib3_context_ssl_version_and_ssl_min_max_version_no_warning( self, kwargs: dict[str, typing.Any] ) -> None: ssl_.create_urllib3_context(**kwargs) @pytest.mark.parametrize( "kwargs", [ {"ssl_version": ssl.PROTOCOL_TLSv1, "ssl_minimum_version": None}, {"ssl_version": ssl.PROTOCOL_TLSv1, "ssl_maximum_version": None}, { "ssl_version": ssl.PROTOCOL_TLSv1, "ssl_minimum_version": None, "ssl_maximum_version": None, }, ], ) def test_create_urllib3_context_ssl_version_and_ssl_min_max_version_no_error( self, kwargs: dict[str, typing.Any] ) -> None: with pytest.warns( DeprecationWarning, match=r"'ssl_version' option is deprecated and will be removed in " r"urllib3 v2\.1\.0\. Instead use 'ssl_minimum_version'", ): ssl_.create_urllib3_context(**kwargs) def test_assert_fingerprint_raises_exception_on_none_cert(self) -> None: with pytest.raises(SSLError): ssl_.assert_fingerprint( cert=None, fingerprint="55:39:BF:70:05:12:43:FA:1F:D1:BF:4E:E8:1B:07:1D" ) urllib3-2.0.7/test/test_ssltransport.py0000644000000000000000000005041614513541732015203 0ustar00from __future__ import annotations import platform import select import socket import ssl import typing from unittest import mock import pytest from dummyserver.server import DEFAULT_CA, DEFAULT_CERTS from dummyserver.testcase import SocketDummyServerTestCase, consume_socket from urllib3.util import ssl_ from urllib3.util.ssltransport import SSLTransport if typing.TYPE_CHECKING: from typing_extensions import Literal # consume_socket can iterate forever, we add timeouts to prevent halting. PER_TEST_TIMEOUT = 60 def server_client_ssl_contexts() -> tuple[ssl.SSLContext, ssl.SSLContext]: if hasattr(ssl, "PROTOCOL_TLS_SERVER"): server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) server_context.load_cert_chain(DEFAULT_CERTS["certfile"], DEFAULT_CERTS["keyfile"]) if hasattr(ssl, "PROTOCOL_TLS_CLIENT"): client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) client_context.load_verify_locations(DEFAULT_CA) return server_context, client_context @typing.overload def sample_request(binary: Literal[True] = ...) -> bytes: ... @typing.overload def sample_request(binary: Literal[False]) -> str: ... def sample_request(binary: bool = True) -> bytes | str: request = ( b"GET http://www.testing.com/ HTTP/1.1\r\n" b"Host: www.testing.com\r\n" b"User-Agent: awesome-test\r\n" b"\r\n" ) return request if binary else request.decode("utf-8") def validate_request( provided_request: bytearray, binary: Literal[False, True] = True ) -> None: assert provided_request is not None expected_request = sample_request(binary) assert provided_request == expected_request @typing.overload def sample_response(binary: Literal[True] = ...) -> bytes: ... @typing.overload def sample_response(binary: Literal[False]) -> str: ... @typing.overload def sample_response(binary: bool = ...) -> bytes | str: ... def sample_response(binary: bool = True) -> bytes | str: response = b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n" return response if binary else response.decode("utf-8") def validate_response( provided_response: bytes | bytearray | str, binary: bool = True ) -> None: assert provided_response is not None expected_response = sample_response(binary) assert provided_response == expected_response def validate_peercert(ssl_socket: SSLTransport) -> None: binary_cert = ssl_socket.getpeercert(binary_form=True) assert type(binary_cert) == bytes assert len(binary_cert) > 0 cert = ssl_socket.getpeercert() assert type(cert) == dict assert "serialNumber" in cert assert cert["serialNumber"] != "" class SingleTLSLayerTestCase(SocketDummyServerTestCase): """ Uses the SocketDummyServer to validate a single TLS layer can be established through the SSLTransport. """ @classmethod def setup_class(cls) -> None: cls.server_context, cls.client_context = server_client_ssl_contexts() def start_dummy_server( self, handler: typing.Callable[[socket.socket], None] | None = None ) -> None: def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] try: with self.server_context.wrap_socket(sock, server_side=True) as ssock: request = consume_socket(ssock) validate_request(request) ssock.send(sample_response()) except (ConnectionAbortedError, ConnectionResetError): return chosen_handler = handler if handler else socket_handler self._start_server(chosen_handler) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_start_closed_socket(self) -> None: """Errors generated from an unconnected socket should bubble up.""" sock = socket.socket(socket.AF_INET) context = ssl.create_default_context() sock.close() with pytest.raises(OSError): SSLTransport(sock, context) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_close_after_handshake(self) -> None: """Socket errors should be bubbled up""" self.start_dummy_server() sock = socket.create_connection((self.host, self.port)) with SSLTransport( sock, self.client_context, server_hostname="localhost" ) as ssock: ssock.close() with pytest.raises(OSError): ssock.send(b"blaaargh") @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_wrap_existing_socket(self) -> None: """Validates a single TLS layer can be established.""" self.start_dummy_server() sock = socket.create_connection((self.host, self.port)) with SSLTransport( sock, self.client_context, server_hostname="localhost" ) as ssock: assert ssock.version() is not None ssock.send(sample_request()) response = consume_socket(ssock) validate_response(response) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_unbuffered_text_makefile(self) -> None: self.start_dummy_server() sock = socket.create_connection((self.host, self.port)) with SSLTransport( sock, self.client_context, server_hostname="localhost" ) as ssock: with pytest.raises(ValueError): ssock.makefile("r", buffering=0) ssock.send(sample_request()) response = consume_socket(ssock) validate_response(response) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_unwrap_existing_socket(self) -> None: """ Validates we can break up the TLS layer A full request/response is sent over TLS, and later over plain text. """ def shutdown_handler(listener: socket.socket) -> None: sock = listener.accept()[0] ssl_sock = self.server_context.wrap_socket(sock, server_side=True) request = consume_socket(ssl_sock) validate_request(request) ssl_sock.sendall(sample_response()) unwrapped_sock = ssl_sock.unwrap() request = consume_socket(unwrapped_sock) validate_request(request) unwrapped_sock.sendall(sample_response()) self.start_dummy_server(shutdown_handler) sock = socket.create_connection((self.host, self.port)) ssock = SSLTransport(sock, self.client_context, server_hostname="localhost") # request/response over TLS. ssock.sendall(sample_request()) response = consume_socket(ssock) validate_response(response) # request/response over plaintext after unwrap. ssock.unwrap() sock.sendall(sample_request()) response = consume_socket(sock) validate_response(response) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_ssl_object_attributes(self) -> None: """Ensures common ssl attributes are exposed""" self.start_dummy_server() sock = socket.create_connection((self.host, self.port)) with SSLTransport( sock, self.client_context, server_hostname="localhost" ) as ssock: cipher = ssock.cipher() assert type(cipher) == tuple # No chosen protocol through ALPN or NPN. assert ssock.selected_alpn_protocol() is None assert ssock.selected_npn_protocol() is None shared_ciphers = ssock.shared_ciphers() # SSLContext.shared_ciphers() changed behavior completely in a patch version. # See: https://github.com/python/cpython/issues/96931 assert shared_ciphers is None or ( type(shared_ciphers) is list and len(shared_ciphers) > 0 ) assert ssock.compression() is None validate_peercert(ssock) ssock.send(sample_request()) response = consume_socket(ssock) validate_response(response) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_socket_object_attributes(self) -> None: """Ensures common socket attributes are exposed""" self.start_dummy_server() sock = socket.create_connection((self.host, self.port)) with SSLTransport( sock, self.client_context, server_hostname="localhost" ) as ssock: assert ssock.fileno() is not None test_timeout = 10 ssock.settimeout(test_timeout) assert ssock.gettimeout() == test_timeout assert ssock.socket.gettimeout() == test_timeout ssock.send(sample_request()) response = consume_socket(ssock) validate_response(response) class SocketProxyDummyServer(SocketDummyServerTestCase): """ Simulates a proxy that performs a simple I/O loop on client/server socket. """ def __init__( self, destination_server_host: str, destination_server_port: int ) -> None: self.destination_server_host = destination_server_host self.destination_server_port = destination_server_port self.server_ctx, _ = server_client_ssl_contexts() def start_proxy_handler(self) -> None: """ Socket handler for the proxy. Terminates the first TLS layer and tunnels any bytes needed for client <-> server communicatin. """ def proxy_handler(listener: socket.socket) -> None: sock = listener.accept()[0] with self.server_ctx.wrap_socket(sock, server_side=True) as client_sock: upstream_sock = socket.create_connection( (self.destination_server_host, self.destination_server_port) ) self._read_write_loop(client_sock, upstream_sock) upstream_sock.close() client_sock.close() self._start_server(proxy_handler) def _read_write_loop( self, client_sock: socket.socket, server_sock: socket.socket, chunks: int = 65536, ) -> None: inputs = [client_sock, server_sock] output = [client_sock, server_sock] while inputs: readable, writable, exception = select.select(inputs, output, inputs) if exception: # Error occurred with either of the sockets, time to # wrap up, parent func will close sockets. break for s in readable: read_socket, write_socket = None, None if s == client_sock: read_socket = client_sock write_socket = server_sock else: read_socket = server_sock write_socket = client_sock # Ensure buffer is not full before writing if write_socket in writable: try: b = read_socket.recv(chunks) if len(b) == 0: # One of the sockets has EOFed, we return to close # both. return write_socket.send(b) except ssl.SSLEOFError: # It's possible, depending on shutdown order, that we'll # try to use a socket that was closed between select # calls. return class TlsInTlsTestCase(SocketDummyServerTestCase): """ Creates a TLS in TLS tunnel by chaining a 'SocketProxyDummyServer' and a `SocketDummyServerTestCase`. Client will first connect to the proxy, who will then proxy any bytes send to the destination server. First TLS layer terminates at the proxy, second TLS layer terminates at the destination server. """ @classmethod def setup_class(cls) -> None: cls.server_context, cls.client_context = server_client_ssl_contexts() @classmethod def start_proxy_server(cls) -> None: # Proxy server will handle the first TLS connection and create a # connection to the destination server. cls.proxy_server = SocketProxyDummyServer(cls.host, cls.port) cls.proxy_server.start_proxy_handler() @classmethod def teardown_class(cls) -> None: if hasattr(cls, "proxy_server"): cls.proxy_server.teardown_class() super().teardown_class() @classmethod def start_destination_server(cls) -> None: """ Socket handler for the destination_server. Terminates the second TLS layer and send a basic HTTP response. """ def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] try: with cls.server_context.wrap_socket(sock, server_side=True) as ssock: request = consume_socket(ssock) validate_request(request) ssock.send(sample_response()) except (ssl.SSLEOFError, ssl.SSLZeroReturnError, OSError): return sock.close() cls._start_server(socket_handler) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_tls_in_tls_tunnel(self) -> None: """ Basic communication over the TLS in TLS tunnel. """ self.start_destination_server() self.start_proxy_server() sock = socket.create_connection( (self.proxy_server.host, self.proxy_server.port) ) with self.client_context.wrap_socket( sock, server_hostname="localhost" ) as proxy_sock: with SSLTransport( proxy_sock, self.client_context, server_hostname="localhost" ) as destination_sock: assert destination_sock.version() is not None destination_sock.send(sample_request()) response = consume_socket(destination_sock) validate_response(response) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_wrong_sni_hint(self) -> None: """ Provides a wrong sni hint to validate an exception is thrown. """ self.start_destination_server() self.start_proxy_server() sock = socket.create_connection( (self.proxy_server.host, self.proxy_server.port) ) with self.client_context.wrap_socket( sock, server_hostname="localhost" ) as proxy_sock: with pytest.raises(ssl.SSLCertVerificationError): SSLTransport( proxy_sock, self.client_context, server_hostname="veryverywrong" ) @pytest.mark.timeout(PER_TEST_TIMEOUT) @pytest.mark.parametrize("buffering", [None, 0]) def test_tls_in_tls_makefile_raw_rw_binary(self, buffering: int | None) -> None: """ Uses makefile with read, write and binary modes without buffering. """ self.start_destination_server() self.start_proxy_server() sock = socket.create_connection( (self.proxy_server.host, self.proxy_server.port) ) with self.client_context.wrap_socket( sock, server_hostname="localhost" ) as proxy_sock: with SSLTransport( proxy_sock, self.client_context, server_hostname="localhost" ) as destination_sock: file = destination_sock.makefile("rwb", buffering) file.write(sample_request()) # type: ignore[call-overload] file.flush() response = bytearray(65536) wrote = file.readinto(response) # type: ignore[union-attr] assert wrote is not None # Allocated response is bigger than the actual response, we # rtrim remaining x00 bytes. str_response = response.decode("utf-8").rstrip("\x00") validate_response(str_response, binary=False) file.close() @pytest.mark.skipif( platform.system() == "Windows", reason="Skipping windows due to text makefile support", ) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_tls_in_tls_makefile_rw_text(self) -> None: """ Creates a separate buffer for reading and writing using text mode and utf-8 encoding. """ self.start_destination_server() self.start_proxy_server() sock = socket.create_connection( (self.proxy_server.host, self.proxy_server.port) ) with self.client_context.wrap_socket( sock, server_hostname="localhost" ) as proxy_sock: with SSLTransport( proxy_sock, self.client_context, server_hostname="localhost" ) as destination_sock: read = destination_sock.makefile("r", encoding="utf-8") write = destination_sock.makefile("w", encoding="utf-8") write.write(sample_request(binary=False)) # type: ignore[arg-type, call-overload] write.flush() response = read.read() assert isinstance(response, str) if "\r" not in response: # Carriage return will be removed when reading as a file on # some platforms. We add it before the comparison. assert isinstance(response, str) response = response.replace("\n", "\r\n") validate_response(response, binary=False) @pytest.mark.timeout(PER_TEST_TIMEOUT) def test_tls_in_tls_recv_into_sendall(self) -> None: """ Valides recv_into and sendall also work as expected. Other tests are using recv/send. """ self.start_destination_server() self.start_proxy_server() sock = socket.create_connection( (self.proxy_server.host, self.proxy_server.port) ) with self.client_context.wrap_socket( sock, server_hostname="localhost" ) as proxy_sock: with SSLTransport( proxy_sock, self.client_context, server_hostname="localhost" ) as destination_sock: destination_sock.sendall(sample_request()) response = bytearray(65536) destination_sock.recv_into(response) str_response = response.decode("utf-8").rstrip("\x00") validate_response(str_response, binary=False) class TestSSLTransportWithMock: def test_constructor_params(self) -> None: server_hostname = "example-domain.com" sock = mock.Mock() context = mock.create_autospec(ssl_.SSLContext) ssl_transport = SSLTransport( sock, context, server_hostname=server_hostname, suppress_ragged_eofs=False ) context.wrap_bio.assert_called_with( mock.ANY, mock.ANY, server_hostname=server_hostname ) assert not ssl_transport.suppress_ragged_eofs def test_various_flags_errors(self) -> None: server_hostname = "example-domain.com" sock = mock.Mock() context = mock.create_autospec(ssl_.SSLContext) ssl_transport = SSLTransport( sock, context, server_hostname=server_hostname, suppress_ragged_eofs=False ) with pytest.raises(ValueError): ssl_transport.recv(flags=1) with pytest.raises(ValueError): ssl_transport.recv_into(bytearray(), flags=1) with pytest.raises(ValueError): ssl_transport.sendall(bytearray(), flags=1) with pytest.raises(ValueError): ssl_transport.send(None, flags=1) # type: ignore[arg-type] def test_makefile_wrong_mode_error(self) -> None: server_hostname = "example-domain.com" sock = mock.Mock() context = mock.create_autospec(ssl_.SSLContext) ssl_transport = SSLTransport( sock, context, server_hostname=server_hostname, suppress_ragged_eofs=False ) with pytest.raises(ValueError): ssl_transport.makefile(mode="x") def test_wrap_ssl_read_error(self) -> None: server_hostname = "example-domain.com" sock = mock.Mock() context = mock.create_autospec(ssl_.SSLContext) ssl_transport = SSLTransport( sock, context, server_hostname=server_hostname, suppress_ragged_eofs=False ) with mock.patch.object(ssl_transport, "_ssl_io_loop") as _ssl_io_loop: _ssl_io_loop.side_effect = ssl.SSLError() with pytest.raises(ssl.SSLError): ssl_transport._wrap_ssl_read(1) urllib3-2.0.7/test/test_util.py0000644000000000000000000012556614513541732013413 0ustar00from __future__ import annotations import io import logging import socket import ssl import sys import typing import warnings from itertools import chain from test import ImportBlocker, ModuleStash, notBrotli, notZstd, onlyBrotli, onlyZstd from unittest import mock from unittest.mock import MagicMock, Mock, patch from urllib.parse import urlparse import pytest from urllib3 import add_stderr_logger, disable_warnings from urllib3.connection import ProxyConfig from urllib3.exceptions import ( InsecureRequestWarning, LocationParseError, TimeoutStateError, UnrewindableBodyError, ) from urllib3.util import is_fp_closed from urllib3.util.connection import _has_ipv6, allowed_gai_family, create_connection from urllib3.util.proxy import connection_requires_http_tunnel from urllib3.util.request import _FAILEDTELL, make_headers, rewind_body from urllib3.util.response import assert_header_parsing from urllib3.util.ssl_ import ( _TYPE_VERSION_INFO, _is_has_never_check_common_name_reliable, resolve_cert_reqs, resolve_ssl_version, ssl_wrap_socket, ) from urllib3.util.timeout import _DEFAULT_TIMEOUT, Timeout from urllib3.util.url import Url, _encode_invalid_chars, parse_url from urllib3.util.util import to_bytes, to_str from . import clear_warnings if typing.TYPE_CHECKING: from typing_extensions import Literal # This number represents a time in seconds, it doesn't mean anything in # isolation. Setting to a high-ish value to avoid conflicts with the smaller # numbers used for timeouts TIMEOUT_EPOCH = 1000 class TestUtil: url_host_map = [ # Hosts ("http://google.com/mail", ("http", "google.com", None)), ("http://google.com/mail/", ("http", "google.com", None)), ("google.com/mail", ("http", "google.com", None)), ("http://google.com/", ("http", "google.com", None)), ("http://google.com", ("http", "google.com", None)), ("http://www.google.com", ("http", "www.google.com", None)), ("http://mail.google.com", ("http", "mail.google.com", None)), ("http://google.com:8000/mail/", ("http", "google.com", 8000)), ("http://google.com:8000", ("http", "google.com", 8000)), ("https://google.com", ("https", "google.com", None)), ("https://google.com:8000", ("https", "google.com", 8000)), ("http://user:password@127.0.0.1:1234", ("http", "127.0.0.1", 1234)), ("http://google.com/foo=http://bar:42/baz", ("http", "google.com", None)), ("http://google.com?foo=http://bar:42/baz", ("http", "google.com", None)), ("http://google.com#foo=http://bar:42/baz", ("http", "google.com", None)), # IPv4 ("173.194.35.7", ("http", "173.194.35.7", None)), ("http://173.194.35.7", ("http", "173.194.35.7", None)), ("http://173.194.35.7/test", ("http", "173.194.35.7", None)), ("http://173.194.35.7:80", ("http", "173.194.35.7", 80)), ("http://173.194.35.7:80/test", ("http", "173.194.35.7", 80)), # IPv6 ("[2a00:1450:4001:c01::67]", ("http", "[2a00:1450:4001:c01::67]", None)), ("http://[2a00:1450:4001:c01::67]", ("http", "[2a00:1450:4001:c01::67]", None)), ( "http://[2a00:1450:4001:c01::67]/test", ("http", "[2a00:1450:4001:c01::67]", None), ), ( "http://[2a00:1450:4001:c01::67]:80", ("http", "[2a00:1450:4001:c01::67]", 80), ), ( "http://[2a00:1450:4001:c01::67]:80/test", ("http", "[2a00:1450:4001:c01::67]", 80), ), # More IPv6 from http://www.ietf.org/rfc/rfc2732.txt ( "http://[fedc:ba98:7654:3210:fedc:ba98:7654:3210]:8000/index.html", ("http", "[fedc:ba98:7654:3210:fedc:ba98:7654:3210]", 8000), ), ( "http://[1080:0:0:0:8:800:200c:417a]/index.html", ("http", "[1080:0:0:0:8:800:200c:417a]", None), ), ("http://[3ffe:2a00:100:7031::1]", ("http", "[3ffe:2a00:100:7031::1]", None)), ( "http://[1080::8:800:200c:417a]/foo", ("http", "[1080::8:800:200c:417a]", None), ), ("http://[::192.9.5.5]/ipng", ("http", "[::192.9.5.5]", None)), ( "http://[::ffff:129.144.52.38]:42/index.html", ("http", "[::ffff:129.144.52.38]", 42), ), ( "http://[2010:836b:4179::836b:4179]", ("http", "[2010:836b:4179::836b:4179]", None), ), # Scoped IPv6 (with ZoneID), both RFC 6874 compliant and not. ("http://[a::b%25zone]", ("http", "[a::b%zone]", None)), ("http://[a::b%zone]", ("http", "[a::b%zone]", None)), # Hosts ("HTTP://GOOGLE.COM/mail/", ("http", "google.com", None)), ("GOogle.COM/mail", ("http", "google.com", None)), ("HTTP://GoOgLe.CoM:8000/mail/", ("http", "google.com", 8000)), ("HTTP://user:password@EXAMPLE.COM:1234", ("http", "example.com", 1234)), ("173.194.35.7", ("http", "173.194.35.7", None)), ("HTTP://173.194.35.7", ("http", "173.194.35.7", None)), ( "HTTP://[2a00:1450:4001:c01::67]:80/test", ("http", "[2a00:1450:4001:c01::67]", 80), ), ( "HTTP://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:8000/index.html", ("http", "[fedc:ba98:7654:3210:fedc:ba98:7654:3210]", 8000), ), ( "HTTPS://[1080:0:0:0:8:800:200c:417A]/index.html", ("https", "[1080:0:0:0:8:800:200c:417a]", None), ), ("abOut://eXamPlE.com?info=1", ("about", "eXamPlE.com", None)), ( "http+UNIX://%2fvar%2frun%2fSOCKET/path", ("http+unix", "%2fvar%2frun%2fSOCKET", None), ), ] @pytest.mark.parametrize(["url", "scheme_host_port"], url_host_map) def test_scheme_host_port( self, url: str, scheme_host_port: tuple[str, str, int | None] ) -> None: parsed_url = parse_url(url) scheme, host, port = scheme_host_port assert (parsed_url.scheme or "http") == scheme assert parsed_url.hostname == parsed_url.host == host assert parsed_url.port == port def test_encode_invalid_chars_none(self) -> None: assert _encode_invalid_chars(None, set()) is None @pytest.mark.parametrize( "url", [ "http://google.com:foo", "http://::1/", "http://::1:80/", "http://google.com:-80", "http://google.com:65536", "http://google.com:\xb2\xb2", # \xb2 = ^2 # Invalid IDNA labels "http://\uD7FF.com", "http://â¤ï¸", # Unicode surrogates "http://\uD800.com", "http://\uDC00.com", ], ) def test_invalid_url(self, url: str) -> None: with pytest.raises(LocationParseError): parse_url(url) @pytest.mark.parametrize( "url, expected_normalized_url", [ ("HTTP://GOOGLE.COM/MAIL/", "http://google.com/MAIL/"), ( "http://user@domain.com:password@example.com/~tilde@?@", "http://user%40domain.com:password@example.com/~tilde@?@", ), ( "HTTP://JeremyCline:Hunter2@Example.com:8080/", "http://JeremyCline:Hunter2@example.com:8080/", ), ("HTTPS://Example.Com/?Key=Value", "https://example.com/?Key=Value"), ("Https://Example.Com/#Fragment", "https://example.com/#Fragment"), # IPv6 addresses with zone IDs. Both RFC 6874 (%25) as well as # non-standard (unquoted %) variants. ("[::1%zone]", "[::1%zone]"), ("[::1%25zone]", "[::1%zone]"), ("[::1%25]", "[::1%25]"), ("[::Ff%etH0%Ff]/%ab%Af", "[::ff%etH0%FF]/%AB%AF"), ( "http://user:pass@[AaAa::Ff%25etH0%Ff]/%ab%Af", "http://user:pass@[aaaa::ff%etH0%FF]/%AB%AF", ), # Invalid characters for the query/fragment getting encoded ( 'http://google.com/p[]?parameter[]="hello"#fragment#', "http://google.com/p%5B%5D?parameter%5B%5D=%22hello%22#fragment%23", ), # Percent encoding isn't applied twice despite '%' being invalid # but the percent encoding is still normalized. ( "http://google.com/p%5B%5d?parameter%5b%5D=%22hello%22#fragment%23", "http://google.com/p%5B%5D?parameter%5B%5D=%22hello%22#fragment%23", ), ], ) def test_parse_url_normalization( self, url: str, expected_normalized_url: str ) -> None: """Assert parse_url normalizes the scheme/host, and only the scheme/host""" actual_normalized_url = parse_url(url).url assert actual_normalized_url == expected_normalized_url @pytest.mark.parametrize("char", [chr(i) for i in range(0x00, 0x21)] + ["\x7F"]) def test_control_characters_are_percent_encoded(self, char: str) -> None: percent_char = "%" + (hex(ord(char))[2:].zfill(2).upper()) url = parse_url( f"http://user{char}@example.com/path{char}?query{char}#fragment{char}" ) assert url == Url( "http", auth="user" + percent_char, host="example.com", path="/path" + percent_char, query="query" + percent_char, fragment="fragment" + percent_char, ) parse_url_host_map = [ ("http://google.com/mail", Url("http", host="google.com", path="/mail")), ("http://google.com/mail/", Url("http", host="google.com", path="/mail/")), ("http://google.com/mail", Url("http", host="google.com", path="mail")), ("google.com/mail", Url(host="google.com", path="/mail")), ("http://google.com/", Url("http", host="google.com", path="/")), ("http://google.com", Url("http", host="google.com")), ("http://google.com?foo", Url("http", host="google.com", path="", query="foo")), # Path/query/fragment ("", Url()), ("/", Url(path="/")), ("#?/!google.com/?foo", Url(path="", fragment="?/!google.com/?foo")), ("/foo", Url(path="/foo")), ("/foo?bar=baz", Url(path="/foo", query="bar=baz")), ( "/foo?bar=baz#banana?apple/orange", Url(path="/foo", query="bar=baz", fragment="banana?apple/orange"), ), ( "/redirect?target=http://localhost:61020/", Url(path="redirect", query="target=http://localhost:61020/"), ), # Port ("http://google.com/", Url("http", host="google.com", path="/")), ("http://google.com:80/", Url("http", host="google.com", port=80, path="/")), ("http://google.com:80", Url("http", host="google.com", port=80)), # Auth ( "http://foo:bar@localhost/", Url("http", auth="foo:bar", host="localhost", path="/"), ), ("http://foo@localhost/", Url("http", auth="foo", host="localhost", path="/")), ( "http://foo:bar@localhost/", Url("http", auth="foo:bar", host="localhost", path="/"), ), ] non_round_tripping_parse_url_host_map = [ # Path/query/fragment ("?", Url(path="", query="")), ("#", Url(path="", fragment="")), # Path normalization ("/abc/../def", Url(path="/def")), # Empty Port ("http://google.com:", Url("http", host="google.com")), ("http://google.com:/", Url("http", host="google.com", path="/")), # Uppercase IRI ( "http://Königsgäßchen.de/straße", Url("http", host="xn--knigsgchen-b4a3dun.de", path="/stra%C3%9Fe"), ), # Percent-encode in userinfo ( "http://user@email.com:password@example.com/", Url("http", auth="user%40email.com:password", host="example.com", path="/"), ), ( 'http://user":quoted@example.com/', Url("http", auth="user%22:quoted", host="example.com", path="/"), ), # Unicode Surrogates ("http://google.com/\uD800", Url("http", host="google.com", path="%ED%A0%80")), ( "http://google.com?q=\uDC00", Url("http", host="google.com", path="", query="q=%ED%B0%80"), ), ( "http://google.com#\uDC00", Url("http", host="google.com", path="", fragment="%ED%B0%80"), ), ] @pytest.mark.parametrize( "url, expected_url", chain(parse_url_host_map, non_round_tripping_parse_url_host_map), ) def test_parse_url(self, url: str, expected_url: Url) -> None: returned_url = parse_url(url) assert returned_url == expected_url assert returned_url.hostname == returned_url.host == expected_url.host @pytest.mark.parametrize("url, expected_url", parse_url_host_map) def test_unparse_url(self, url: str, expected_url: Url) -> None: assert url == expected_url.url @pytest.mark.parametrize( ["url", "expected_url"], [ # RFC 3986 5.2.4 ("/abc/../def", Url(path="/def")), ("/..", Url(path="/")), ("/./abc/./def/", Url(path="/abc/def/")), ("/.", Url(path="/")), ("/./", Url(path="/")), ("/abc/./.././d/././e/.././f/./../../ghi", Url(path="/ghi")), ], ) def test_parse_and_normalize_url_paths(self, url: str, expected_url: Url) -> None: actual_url = parse_url(url) assert actual_url == expected_url assert actual_url.url == expected_url.url def test_parse_url_invalid_IPv6(self) -> None: with pytest.raises(LocationParseError): parse_url("[::1") def test_parse_url_negative_port(self) -> None: with pytest.raises(LocationParseError): parse_url("https://www.google.com:-80/") def test_parse_url_remove_leading_zeros(self) -> None: url = parse_url("https://example.com:0000000000080") assert url.port == 80 def test_parse_url_only_zeros(self) -> None: url = parse_url("https://example.com:0") assert url.port == 0 url = parse_url("https://example.com:000000000000") assert url.port == 0 def test_Url_str(self) -> None: U = Url("http", host="google.com") assert str(U) == U.url request_uri_map = [ ("http://google.com/mail", "/mail"), ("http://google.com/mail/", "/mail/"), ("http://google.com/", "/"), ("http://google.com", "/"), ("", "/"), ("/", "/"), ("?", "/?"), ("#", "/"), ("/foo?bar=baz", "/foo?bar=baz"), ] @pytest.mark.parametrize("url, expected_request_uri", request_uri_map) def test_request_uri(self, url: str, expected_request_uri: str) -> None: returned_url = parse_url(url) assert returned_url.request_uri == expected_request_uri url_authority_map: list[tuple[str, str | None]] = [ ("http://user:pass@google.com/mail", "user:pass@google.com"), ("http://user:pass@google.com:80/mail", "user:pass@google.com:80"), ("http://user@google.com:80/mail", "user@google.com:80"), ("http://user:pass@192.168.1.1/path", "user:pass@192.168.1.1"), ("http://user:pass@192.168.1.1:80/path", "user:pass@192.168.1.1:80"), ("http://user@192.168.1.1:80/path", "user@192.168.1.1:80"), ("http://user:pass@[::1]/path", "user:pass@[::1]"), ("http://user:pass@[::1]:80/path", "user:pass@[::1]:80"), ("http://user@[::1]:80/path", "user@[::1]:80"), ("http://user:pass@localhost/path", "user:pass@localhost"), ("http://user:pass@localhost:80/path", "user:pass@localhost:80"), ("http://user@localhost:80/path", "user@localhost:80"), ] url_netloc_map = [ ("http://google.com/mail", "google.com"), ("http://google.com:80/mail", "google.com:80"), ("http://192.168.0.1/path", "192.168.0.1"), ("http://192.168.0.1:80/path", "192.168.0.1:80"), ("http://[::1]/path", "[::1]"), ("http://[::1]:80/path", "[::1]:80"), ("http://localhost", "localhost"), ("http://localhost:80", "localhost:80"), ("google.com/foobar", "google.com"), ("google.com:12345", "google.com:12345"), ("/", None), ] combined_netloc_authority_map = url_authority_map + url_netloc_map # We compose this list due to variances between parse_url # and urlparse when URIs don't provide a scheme. url_authority_with_schemes_map = [ u for u in combined_netloc_authority_map if u[0].startswith("http") ] @pytest.mark.parametrize("url, expected_authority", combined_netloc_authority_map) def test_authority(self, url: str, expected_authority: str | None) -> None: assert parse_url(url).authority == expected_authority @pytest.mark.parametrize("url, expected_authority", url_authority_with_schemes_map) def test_authority_matches_urllib_netloc( self, url: str, expected_authority: str | None ) -> None: """Validate this matches the behavior of urlparse().netloc""" assert urlparse(url).netloc == expected_authority @pytest.mark.parametrize("url, expected_netloc", url_netloc_map) def test_netloc(self, url: str, expected_netloc: str | None) -> None: assert parse_url(url).netloc == expected_netloc url_vulnerabilities = [ # urlparse doesn't follow RFC 3986 Section 3.2 ( "http://google.com#@evil.com/", Url("http", host="google.com", path="", fragment="@evil.com/"), ), # CVE-2016-5699 ( "http://127.0.0.1%0d%0aConnection%3a%20keep-alive", Url("http", host="127.0.0.1%0d%0aconnection%3a%20keep-alive"), ), # NodeJS unicode -> double dot ( "http://google.com/\uff2e\uff2e/abc", Url("http", host="google.com", path="/%EF%BC%AE%EF%BC%AE/abc"), ), # Scheme without :// ( "javascript:a='@google.com:12345/';alert(0)", Url(scheme="javascript", path="a='@google.com:12345/';alert(0)"), ), ("//google.com/a/b/c", Url(host="google.com", path="/a/b/c")), # International URLs ( "http://ヒ:ã‚­@ヒ.abc.ニ/ヒ?ã‚­#ワ", Url( "http", host="xn--pdk.abc.xn--idk", auth="%E3%83%92:%E3%82%AD", path="/%E3%83%92", query="%E3%82%AD", fragment="%E3%83%AF", ), ), # Injected headers (CVE-2016-5699, CVE-2019-9740, CVE-2019-9947) ( "10.251.0.83:7777?a=1 HTTP/1.1\r\nX-injected: header", Url( host="10.251.0.83", port=7777, path="", query="a=1%20HTTP/1.1%0D%0AX-injected:%20header", ), ), ( "http://127.0.0.1:6379?\r\nSET test failure12\r\n:8080/test/?test=a", Url( scheme="http", host="127.0.0.1", port=6379, path="", query="%0D%0ASET%20test%20failure12%0D%0A:8080/test/?test=a", ), ), # See https://bugs.xdavidhu.me/google/2020/03/08/the-unexpected-google-wide-domain-check-bypass/ ( "https://user:pass@xdavidhu.me\\test.corp.google.com:8080/path/to/something?param=value#hash", Url( scheme="https", auth="user:pass", host="xdavidhu.me", path="/%5Ctest.corp.google.com:8080/path/to/something", query="param=value", fragment="hash", ), ), # Tons of '@' causing backtracking pytest.param( "https://" + ("@" * 10000) + "[", False, id="Tons of '@' causing backtracking 1", ), pytest.param( "https://user:" + ("@" * 10000) + "example.com", Url( scheme="https", auth="user:" + ("%40" * 9999), host="example.com", ), id="Tons of '@' causing backtracking 2", ), ] @pytest.mark.parametrize("url, expected_url", url_vulnerabilities) def test_url_vulnerabilities( self, url: str, expected_url: Literal[False] | Url ) -> None: if expected_url is False: with pytest.raises(LocationParseError): parse_url(url) else: assert parse_url(url) == expected_url def test_parse_url_bytes_type_error(self) -> None: with pytest.raises(TypeError): parse_url(b"https://www.google.com/") # type: ignore[arg-type] @pytest.mark.parametrize( "kwargs, expected", [ pytest.param( {"accept_encoding": True}, {"accept-encoding": "gzip,deflate,br,zstd"}, marks=[onlyBrotli(), onlyZstd()], # type: ignore[list-item] ), pytest.param( {"accept_encoding": True}, {"accept-encoding": "gzip,deflate,br"}, marks=[onlyBrotli(), notZstd()], # type: ignore[list-item] ), pytest.param( {"accept_encoding": True}, {"accept-encoding": "gzip,deflate,zstd"}, marks=[notBrotli(), onlyZstd()], # type: ignore[list-item] ), pytest.param( {"accept_encoding": True}, {"accept-encoding": "gzip,deflate"}, marks=[notBrotli(), notZstd()], # type: ignore[list-item] ), ({"accept_encoding": "foo,bar"}, {"accept-encoding": "foo,bar"}), ({"accept_encoding": ["foo", "bar"]}, {"accept-encoding": "foo,bar"}), pytest.param( {"accept_encoding": True, "user_agent": "banana"}, {"accept-encoding": "gzip,deflate,br,zstd", "user-agent": "banana"}, marks=[onlyBrotli(), onlyZstd()], # type: ignore[list-item] ), pytest.param( {"accept_encoding": True, "user_agent": "banana"}, {"accept-encoding": "gzip,deflate,br", "user-agent": "banana"}, marks=[onlyBrotli(), notZstd()], # type: ignore[list-item] ), pytest.param( {"accept_encoding": True, "user_agent": "banana"}, {"accept-encoding": "gzip,deflate,zstd", "user-agent": "banana"}, marks=[notBrotli(), onlyZstd()], # type: ignore[list-item] ), pytest.param( {"accept_encoding": True, "user_agent": "banana"}, {"accept-encoding": "gzip,deflate", "user-agent": "banana"}, marks=[notBrotli(), notZstd()], # type: ignore[list-item] ), ({"user_agent": "banana"}, {"user-agent": "banana"}), ({"keep_alive": True}, {"connection": "keep-alive"}), ({"basic_auth": "foo:bar"}, {"authorization": "Basic Zm9vOmJhcg=="}), ( {"proxy_basic_auth": "foo:bar"}, {"proxy-authorization": "Basic Zm9vOmJhcg=="}, ), ({"disable_cache": True}, {"cache-control": "no-cache"}), ], ) def test_make_headers( self, kwargs: dict[str, bool | str], expected: dict[str, str] ) -> None: assert make_headers(**kwargs) == expected # type: ignore[arg-type] def test_rewind_body(self) -> None: body = io.BytesIO(b"test data") assert body.read() == b"test data" # Assert the file object has been consumed assert body.read() == b"" # Rewind it back to just be b'data' rewind_body(body, 5) assert body.read() == b"data" def test_rewind_body_failed_tell(self) -> None: body = io.BytesIO(b"test data") body.read() # Consume body # Simulate failed tell() body_pos = _FAILEDTELL with pytest.raises(UnrewindableBodyError): rewind_body(body, body_pos) def test_rewind_body_bad_position(self) -> None: body = io.BytesIO(b"test data") body.read() # Consume body # Pass non-integer position with pytest.raises(ValueError): rewind_body(body, body_pos=None) # type: ignore[arg-type] with pytest.raises(ValueError): rewind_body(body, body_pos=object()) # type: ignore[arg-type] def test_rewind_body_failed_seek(self) -> None: class BadSeek(io.StringIO): def seek(self, offset: int, whence: int = 0) -> typing.NoReturn: raise OSError with pytest.raises(UnrewindableBodyError): rewind_body(BadSeek(), body_pos=2) def test_add_stderr_logger(self) -> None: handler = add_stderr_logger(level=logging.INFO) # Don't actually print debug logger = logging.getLogger("urllib3") assert handler in logger.handlers logger.debug("Testing add_stderr_logger") logger.removeHandler(handler) def test_disable_warnings(self) -> None: with warnings.catch_warnings(record=True) as w: clear_warnings() warnings.simplefilter("default", InsecureRequestWarning) warnings.warn("This is a test.", InsecureRequestWarning) assert len(w) == 1 disable_warnings() warnings.warn("This is a test.", InsecureRequestWarning) assert len(w) == 1 def _make_time_pass( self, seconds: int, timeout: Timeout, time_mock: Mock ) -> Timeout: """Make some time pass for the timeout object""" time_mock.return_value = TIMEOUT_EPOCH timeout.start_connect() time_mock.return_value = TIMEOUT_EPOCH + seconds return timeout @pytest.mark.parametrize( "kwargs, message", [ ({"total": -1}, "less than"), ({"connect": 2, "total": -1}, "less than"), ({"read": -1}, "less than"), ({"connect": False}, "cannot be a boolean"), ({"read": True}, "cannot be a boolean"), ({"connect": 0}, "less than or equal"), ({"read": "foo"}, "int, float or None"), ({"read": "1.0"}, "int, float or None"), ], ) def test_invalid_timeouts( self, kwargs: dict[str, int | bool], message: str ) -> None: with pytest.raises(ValueError, match=message): Timeout(**kwargs) @patch("time.monotonic") def test_timeout(self, time_monotonic: MagicMock) -> None: timeout = Timeout(total=3) # make 'no time' elapse timeout = self._make_time_pass( seconds=0, timeout=timeout, time_mock=time_monotonic ) assert timeout.read_timeout == 3 assert timeout.connect_timeout == 3 timeout = Timeout(total=3, connect=2) assert timeout.connect_timeout == 2 timeout = Timeout() assert timeout.connect_timeout == _DEFAULT_TIMEOUT # Connect takes 5 seconds, leaving 5 seconds for read timeout = Timeout(total=10, read=7) timeout = self._make_time_pass( seconds=5, timeout=timeout, time_mock=time_monotonic ) assert timeout.read_timeout == 5 # Connect takes 2 seconds, read timeout still 7 seconds timeout = Timeout(total=10, read=7) timeout = self._make_time_pass( seconds=2, timeout=timeout, time_mock=time_monotonic ) assert timeout.read_timeout == 7 timeout = Timeout(total=10, read=7) assert timeout.read_timeout == 7 timeout = Timeout(total=None, read=None, connect=None) assert timeout.connect_timeout is None assert timeout.read_timeout is None assert timeout.total is None timeout = Timeout(5) assert timeout.total == 5 def test_timeout_default_resolve(self) -> None: """The timeout default is resolved when read_timeout is accessed.""" timeout = Timeout() with patch("urllib3.util.timeout.getdefaulttimeout", return_value=2): assert timeout.read_timeout == 2 with patch("urllib3.util.timeout.getdefaulttimeout", return_value=3): assert timeout.read_timeout == 3 def test_timeout_str(self) -> None: timeout = Timeout(connect=1, read=2, total=3) assert str(timeout) == "Timeout(connect=1, read=2, total=3)" timeout = Timeout(connect=1, read=None, total=3) assert str(timeout) == "Timeout(connect=1, read=None, total=3)" @patch("time.monotonic") def test_timeout_elapsed(self, time_monotonic: MagicMock) -> None: time_monotonic.return_value = TIMEOUT_EPOCH timeout = Timeout(total=3) with pytest.raises(TimeoutStateError): timeout.get_connect_duration() timeout.start_connect() with pytest.raises(TimeoutStateError): timeout.start_connect() time_monotonic.return_value = TIMEOUT_EPOCH + 2 assert timeout.get_connect_duration() == 2 time_monotonic.return_value = TIMEOUT_EPOCH + 37 assert timeout.get_connect_duration() == 37 def test_is_fp_closed_object_supports_closed(self) -> None: class ClosedFile: @property def closed(self) -> Literal[True]: return True assert is_fp_closed(ClosedFile()) def test_is_fp_closed_object_has_none_fp(self) -> None: class NoneFpFile: @property def fp(self) -> None: return None assert is_fp_closed(NoneFpFile()) def test_is_fp_closed_object_has_fp(self) -> None: class FpFile: @property def fp(self) -> Literal[True]: return True assert not is_fp_closed(FpFile()) def test_is_fp_closed_object_has_neither_fp_nor_closed(self) -> None: class NotReallyAFile: pass with pytest.raises(ValueError): is_fp_closed(NotReallyAFile()) def test_has_ipv6_disabled_on_compile(self) -> None: with patch("socket.has_ipv6", False): assert not _has_ipv6("::1") def test_has_ipv6_enabled_but_fails(self) -> None: with patch("socket.has_ipv6", True): with patch("socket.socket") as mock: instance = mock.return_value instance.bind = Mock(side_effect=Exception("No IPv6 here!")) assert not _has_ipv6("::1") def test_has_ipv6_enabled_and_working(self) -> None: with patch("socket.has_ipv6", True): with patch("socket.socket") as mock: instance = mock.return_value instance.bind.return_value = True assert _has_ipv6("::1") def test_ip_family_ipv6_enabled(self) -> None: with patch("urllib3.util.connection.HAS_IPV6", True): assert allowed_gai_family() == socket.AF_UNSPEC def test_ip_family_ipv6_disabled(self) -> None: with patch("urllib3.util.connection.HAS_IPV6", False): assert allowed_gai_family() == socket.AF_INET @pytest.mark.parametrize("headers", [b"foo", None, object]) def test_assert_header_parsing_throws_typeerror_with_non_headers( self, headers: bytes | object | None ) -> None: with pytest.raises(TypeError): assert_header_parsing(headers) # type: ignore[arg-type] def test_connection_requires_http_tunnel_no_proxy(self) -> None: assert not connection_requires_http_tunnel( proxy_url=None, proxy_config=None, destination_scheme=None ) def test_connection_requires_http_tunnel_http_proxy(self) -> None: proxy = parse_url("http://proxy:8080") proxy_config = ProxyConfig( ssl_context=None, use_forwarding_for_https=False, assert_hostname=None, assert_fingerprint=None, ) destination_scheme = "http" assert not connection_requires_http_tunnel( proxy, proxy_config, destination_scheme ) destination_scheme = "https" assert connection_requires_http_tunnel(proxy, proxy_config, destination_scheme) def test_connection_requires_http_tunnel_https_proxy(self) -> None: proxy = parse_url("https://proxy:8443") proxy_config = ProxyConfig( ssl_context=None, use_forwarding_for_https=False, assert_hostname=None, assert_fingerprint=None, ) destination_scheme = "http" assert not connection_requires_http_tunnel( proxy, proxy_config, destination_scheme ) def test_assert_header_parsing_no_error_on_multipart(self) -> None: from http import client header_msg = io.BytesIO() header_msg.write( b'Content-Type: multipart/encrypted;protocol="application/' b'HTTP-SPNEGO-session-encrypted";boundary="Encrypted Boundary"' b"\nServer: Microsoft-HTTPAPI/2.0\nDate: Fri, 16 Aug 2019 19:28:01 GMT" b"\nContent-Length: 1895\n\n\n" ) header_msg.seek(0) assert_header_parsing(client.parse_headers(header_msg)) @pytest.mark.parametrize("host", [".localhost", "...", "t" * 64]) def test_create_connection_with_invalid_idna_labels(self, host: str) -> None: with pytest.raises( LocationParseError, match=f"Failed to parse: '{host}', label empty or too long", ): create_connection((host, 80)) @pytest.mark.parametrize( "host", [ "a.example.com", "localhost.", "[dead::beef]", "[dead::beef%en5]", "[dead::beef%en5.]", ], ) @patch("socket.getaddrinfo") @patch("socket.socket") def test_create_connection_with_valid_idna_labels( self, socket: MagicMock, getaddrinfo: MagicMock, host: str ) -> None: getaddrinfo.return_value = [(None, None, None, None, None)] socket.return_value = Mock() create_connection((host, 80)) @patch("socket.getaddrinfo") def test_create_connection_error(self, getaddrinfo: MagicMock) -> None: getaddrinfo.return_value = [] with pytest.raises(OSError, match="getaddrinfo returns an empty list"): create_connection(("example.com", 80)) @patch("socket.getaddrinfo") def test_dnsresolver_forced_error(self, getaddrinfo: MagicMock) -> None: getaddrinfo.side_effect = socket.gaierror() with pytest.raises(socket.gaierror): # dns is valid but we force the error just for the sake of the test create_connection(("example.com", 80)) def test_dnsresolver_expected_error(self) -> None: with pytest.raises(socket.gaierror): # windows: [Errno 11001] getaddrinfo failed in windows # linux: [Errno -2] Name or service not known # macos: [Errno 8] nodename nor servname provided, or not known create_connection(("badhost.invalid", 80)) @patch("socket.getaddrinfo") @patch("socket.socket") def test_create_connection_with_scoped_ipv6( self, socket: MagicMock, getaddrinfo: MagicMock ) -> None: # Check that providing create_connection with a scoped IPv6 address # properly propagates the scope to getaddrinfo, and that the returned # scoped ID makes it to the socket creation call. fake_scoped_sa6 = ("a::b", 80, 0, 42) getaddrinfo.return_value = [ ( socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", fake_scoped_sa6, ) ] socket.return_value = fake_sock = MagicMock() create_connection(("a::b%iface", 80)) assert getaddrinfo.call_args[0][0] == "a::b%iface" fake_sock.connect.assert_called_once_with(fake_scoped_sa6) @pytest.mark.parametrize( "input,params,expected", ( ("test", {}, "test"), # str input (b"test", {}, "test"), # bytes input (b"test", {"encoding": "utf-8"}, "test"), # bytes input with utf-8 (b"test", {"encoding": "ascii"}, "test"), # bytes input with ascii ), ) def test_to_str( self, input: bytes | str, params: dict[str, str], expected: str ) -> None: assert to_str(input, **params) == expected def test_to_str_error(self) -> None: with pytest.raises(TypeError, match="not expecting type int"): to_str(1) # type: ignore[arg-type] @pytest.mark.parametrize( "input,params,expected", ( (b"test", {}, b"test"), # str input ("test", {}, b"test"), # bytes input ("é", {}, b"\xc3\xa9"), # bytes input ("test", {"encoding": "utf-8"}, b"test"), # bytes input with utf-8 ("test", {"encoding": "ascii"}, b"test"), # bytes input with ascii ), ) def test_to_bytes( self, input: bytes | str, params: dict[str, str], expected: bytes ) -> None: assert to_bytes(input, **params) == expected def test_to_bytes_error(self) -> None: with pytest.raises(TypeError, match="not expecting type int"): to_bytes(1) # type: ignore[arg-type] class TestUtilSSL: """Test utils that use an SSL backend.""" @pytest.mark.parametrize( "candidate, requirements", [ (None, ssl.CERT_REQUIRED), (ssl.CERT_NONE, ssl.CERT_NONE), (ssl.CERT_REQUIRED, ssl.CERT_REQUIRED), ("REQUIRED", ssl.CERT_REQUIRED), ("CERT_REQUIRED", ssl.CERT_REQUIRED), ], ) def test_resolve_cert_reqs( self, candidate: int | str | None, requirements: int ) -> None: assert resolve_cert_reqs(candidate) == requirements @pytest.mark.parametrize( "candidate, version", [ (ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1), ("PROTOCOL_TLSv1", ssl.PROTOCOL_TLSv1), ("TLSv1", ssl.PROTOCOL_TLSv1), (ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23), ], ) def test_resolve_ssl_version(self, candidate: int | str, version: int) -> None: assert resolve_ssl_version(candidate) == version def test_ssl_wrap_socket_loads_the_cert_chain(self) -> None: socket = Mock() mock_context = Mock() ssl_wrap_socket( ssl_context=mock_context, sock=socket, certfile="/path/to/certfile" ) mock_context.load_cert_chain.assert_called_once_with("/path/to/certfile", None) @patch("urllib3.util.ssl_.create_urllib3_context") def test_ssl_wrap_socket_creates_new_context( self, create_urllib3_context: mock.MagicMock ) -> None: socket = Mock() ssl_wrap_socket(socket, cert_reqs=ssl.CERT_REQUIRED) create_urllib3_context.assert_called_once_with(None, 2, ciphers=None) def test_ssl_wrap_socket_loads_verify_locations(self) -> None: socket = Mock() mock_context = Mock() ssl_wrap_socket(ssl_context=mock_context, ca_certs="/path/to/pem", sock=socket) mock_context.load_verify_locations.assert_called_once_with( "/path/to/pem", None, None ) def test_ssl_wrap_socket_loads_certificate_directories(self) -> None: socket = Mock() mock_context = Mock() ssl_wrap_socket( ssl_context=mock_context, ca_cert_dir="/path/to/pems", sock=socket ) mock_context.load_verify_locations.assert_called_once_with( None, "/path/to/pems", None ) def test_ssl_wrap_socket_loads_certificate_data(self) -> None: socket = Mock() mock_context = Mock() ssl_wrap_socket( ssl_context=mock_context, ca_cert_data="TOTALLY PEM DATA", sock=socket ) mock_context.load_verify_locations.assert_called_once_with( None, None, "TOTALLY PEM DATA" ) def _wrap_socket_and_mock_warn( self, sock: socket.socket, server_hostname: str | None ) -> tuple[Mock, MagicMock]: mock_context = Mock() with patch("warnings.warn") as warn: ssl_wrap_socket( ssl_context=mock_context, sock=sock, server_hostname=server_hostname, ) return mock_context, warn def test_ssl_wrap_socket_sni_ip_address_no_warn(self) -> None: """Test that a warning is not made if server_hostname is an IP address.""" sock = Mock() context, warn = self._wrap_socket_and_mock_warn(sock, "8.8.8.8") context.wrap_socket.assert_called_once_with(sock, server_hostname="8.8.8.8") warn.assert_not_called() def test_ssl_wrap_socket_sni_none_no_warn(self) -> None: """Test that a warning is not made if server_hostname is not given.""" sock = Mock() context, warn = self._wrap_socket_and_mock_warn(sock, None) context.wrap_socket.assert_called_once_with(sock, server_hostname=None) warn.assert_not_called() @pytest.mark.parametrize( "openssl_version, openssl_version_number, implementation_name, version_info, pypy_version_info, reliable", [ # OpenSSL and Python OK -> reliable ("OpenSSL 1.1.1", 0x101010CF, "cpython", (3, 9, 3), None, True), # Python OK -> reliable ("OpenSSL 1.1.1", 0x10101000, "cpython", (3, 9, 3), None, True), # PyPy: depends on the version ("OpenSSL 1.1.1", 0x10101000, "pypy", (3, 6, 9), (7, 3, 7), False), ("OpenSSL 1.1.1", 0x10101000, "pypy", (3, 7, 13), (7, 3, 9), False), ("OpenSSL 1.1.1", 0x101010CF, "pypy", (3, 8, 12), (7, 3, 8), True), # OpenSSL OK -> reliable ("OpenSSL 1.1.1", 0x101010CF, "cpython", (3, 9, 2), None, True), # not OpenSSSL -> unreliable ("LibreSSL 2.8.3", 0x101010CF, "cpython", (3, 10, 0), None, False), # old OpenSSL and old Python, unreliable ("OpenSSL 1.1.0", 0x10101000, "cpython", (3, 9, 2), None, False), ], ) def test_is_has_never_check_common_name_reliable( self, openssl_version: str, openssl_version_number: int, implementation_name: str, version_info: _TYPE_VERSION_INFO, pypy_version_info: _TYPE_VERSION_INFO | None, reliable: bool, ) -> None: assert ( _is_has_never_check_common_name_reliable( openssl_version, openssl_version_number, implementation_name, version_info, pypy_version_info, ) == reliable ) idna_blocker = ImportBlocker("idna") module_stash = ModuleStash("urllib3") class TestUtilWithoutIdna: @classmethod def setup_class(cls) -> None: sys.modules.pop("idna", None) module_stash.stash() sys.meta_path.insert(0, idna_blocker) @classmethod def teardown_class(cls) -> None: sys.meta_path.remove(idna_blocker) module_stash.pop() def test_parse_url_without_idna(self) -> None: url = "http://\uD7FF.com" with pytest.raises(LocationParseError, match=f"Failed to parse: {url}"): parse_url(url) urllib3-2.0.7/test/test_wait.py0000644000000000000000000001355714513541732013376 0ustar00from __future__ import annotations import signal import threading import time import typing from socket import socket, socketpair from types import FrameType import pytest from urllib3.util.wait import ( _have_working_poll, poll_wait_for_socket, select_wait_for_socket, wait_for_read, wait_for_socket, wait_for_write, ) TYPE_SOCKET_PAIR = typing.Tuple[socket, socket] TYPE_WAIT_FOR = typing.Callable[..., bool] @pytest.fixture def spair() -> typing.Generator[TYPE_SOCKET_PAIR, None, None]: a, b = socketpair() yield a, b a.close() b.close() variants: list[TYPE_WAIT_FOR] = [wait_for_socket, select_wait_for_socket] if _have_working_poll(): variants.append(poll_wait_for_socket) @pytest.mark.parametrize("wfs", variants) def test_wait_for_socket(wfs: TYPE_WAIT_FOR, spair: TYPE_SOCKET_PAIR) -> None: a, b = spair with pytest.raises(RuntimeError): wfs(a, read=False, write=False) assert not wfs(a, read=True, timeout=0) assert wfs(a, write=True, timeout=0) b.send(b"x") assert wfs(a, read=True, timeout=0) assert wfs(a, read=True, timeout=10) assert wfs(a, read=True, timeout=None) # Fill up the socket with data a.setblocking(False) try: while True: a.send(b"x" * 999999) except OSError: pass # Now it's not writable anymore assert not wfs(a, write=True, timeout=0) # But if we ask for read-or-write, that succeeds assert wfs(a, read=True, write=True, timeout=0) # Unless we read from it assert a.recv(1) == b"x" assert not wfs(a, read=True, write=True, timeout=0) # But if the remote peer closes the socket, then it becomes readable b.close() assert wfs(a, read=True, timeout=0) # Waiting for a socket that's actually been closed is just a bug, and # raises some kind of helpful exception (exact details depend on the # platform). with pytest.raises(Exception): wfs(b, read=True) def test_wait_for_read_write(spair: TYPE_SOCKET_PAIR) -> None: a, b = spair assert not wait_for_read(a, 0) assert wait_for_write(a, 0) b.send(b"x") assert wait_for_read(a, 0) assert wait_for_write(a, 0) # Fill up the socket with data a.setblocking(False) try: while True: a.send(b"x" * 999999) except OSError: pass # Now it's not writable anymore assert not wait_for_write(a, 0) @pytest.mark.skipif(not hasattr(signal, "setitimer"), reason="need setitimer() support") @pytest.mark.parametrize("wfs", variants) def test_eintr(wfs: TYPE_WAIT_FOR, spair: TYPE_SOCKET_PAIR) -> None: a, b = spair interrupt_count = [0] def handler(sig: int, frame: FrameType | None) -> typing.Any: assert sig == signal.SIGALRM interrupt_count[0] += 1 old_handler = signal.signal(signal.SIGALRM, handler) try: assert not wfs(a, read=True, timeout=0) start = time.monotonic() try: # Start delivering SIGALRM 10 times per second signal.setitimer(signal.ITIMER_REAL, 0.1, 0.1) # Sleep for 1 second (we hope!) wfs(a, read=True, timeout=1) finally: # Stop delivering SIGALRM signal.setitimer(signal.ITIMER_REAL, 0) end = time.monotonic() dur = end - start assert 0.9 < dur < 3 finally: signal.signal(signal.SIGALRM, old_handler) assert interrupt_count[0] > 0 @pytest.mark.skipif(not hasattr(signal, "setitimer"), reason="need setitimer() support") @pytest.mark.parametrize("wfs", variants) def test_eintr_zero_timeout(wfs: TYPE_WAIT_FOR, spair: TYPE_SOCKET_PAIR) -> None: a, b = spair interrupt_count = [0] def handler(sig: int, frame: FrameType | None) -> typing.Any: assert sig == signal.SIGALRM interrupt_count[0] += 1 old_handler = signal.signal(signal.SIGALRM, handler) try: assert not wfs(a, read=True, timeout=0) try: # Start delivering SIGALRM 1000 times per second, # to trigger race conditions such as # https://github.com/urllib3/urllib3/issues/1396. signal.setitimer(signal.ITIMER_REAL, 0.001, 0.001) # Hammer the system call for a while to trigger the # race. end = time.monotonic() + 5 for i in range(100000): wfs(a, read=True, timeout=0) if time.monotonic() >= end: break finally: # Stop delivering SIGALRM signal.setitimer(signal.ITIMER_REAL, 0) finally: signal.signal(signal.SIGALRM, old_handler) assert interrupt_count[0] > 0 @pytest.mark.skipif(not hasattr(signal, "setitimer"), reason="need setitimer() support") @pytest.mark.parametrize("wfs", variants) def test_eintr_infinite_timeout(wfs: TYPE_WAIT_FOR, spair: TYPE_SOCKET_PAIR) -> None: a, b = spair interrupt_count = [0] def handler(sig: int, frame: FrameType | None) -> typing.Any: assert sig == signal.SIGALRM interrupt_count[0] += 1 def make_a_readable_after_one_second() -> None: time.sleep(1) b.send(b"x") old_handler = signal.signal(signal.SIGALRM, handler) try: assert not wfs(a, read=True, timeout=0) start = time.monotonic() try: # Start delivering SIGALRM 10 times per second signal.setitimer(signal.ITIMER_REAL, 0.1, 0.1) # Sleep for 1 second (we hope!) thread = threading.Thread(target=make_a_readable_after_one_second) thread.start() wfs(a, read=True) finally: # Stop delivering SIGALRM signal.setitimer(signal.ITIMER_REAL, 0) thread.join() end = time.monotonic() dur = end - start assert 0.9 < dur < 3 finally: signal.signal(signal.SIGALRM, old_handler) assert interrupt_count[0] > 0 urllib3-2.0.7/test/tz_stub.py0000644000000000000000000000227314513541732013056 0ustar00from __future__ import annotations import datetime import os import time import typing from contextlib import contextmanager import pytest try: import zoneinfo # type: ignore[import] except ImportError: # Python < 3.9 from backports import zoneinfo # type: ignore[no-redef] @contextmanager def stub_timezone_ctx(tzname: str | None) -> typing.Generator[None, None, None]: """ Switch to a locally-known timezone specified by `tzname`. On exit, restore the previous timezone. If `tzname` is `None`, do nothing. """ if tzname is None: yield return # Only supported on Unix if not hasattr(time, "tzset"): pytest.skip("Timezone patching is not supported") # Make sure the new timezone exists try: zoneinfo.ZoneInfo(tzname) except zoneinfo.ZoneInfoNotFoundError: raise ValueError(f"Invalid timezone specified: {tzname!r}") # Get the current timezone old_tzname = datetime.datetime.now().astimezone().tzname() if old_tzname is None: raise OSError("Cannot determine current timezone") os.environ["TZ"] = tzname time.tzset() yield os.environ["TZ"] = old_tzname time.tzset() urllib3-2.0.7/test/contrib/__init__.py0000644000000000000000000000000014513541732014545 0ustar00urllib3-2.0.7/test/contrib/duplicate_san.pem0000644000000000000000000000235114513541732015765 0ustar00-----BEGIN CERTIFICATE----- MIIDcjCCAlqgAwIBAgICAwkwDQYJKoZIhvcNAQEFBQAwVzELMAkGA1UEBhMCVVMx DjAMBgNVBAgMBVRleGFzMQ8wDQYDVQQHDAZBdXN0aW4xDTALBgNVBAoMBFB5Q0Ex GDAWBgNVBAMMD2NyeXB0b2dyYXBoeS5pbzAeFw0wMjAxMDExMjAxMDBaFw0zMDEy MzEwODMwMDBaMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEPMA0GA1UE BwwGQXVzdGluMQ0wCwYDVQQKDARQeUNBMRgwFgYDVQQDDA9jcnlwdG9ncmFwaHku aW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBevx+d0dMqlqoMDYV ij/797UhaFG6IjDl1qv8wcbP71npI+oTMLxZO3OAKrYIpuSjMGUjoxFrpao5ZhRR dOE7bEnpt4Bi5EnXLvsQ/UnpH6CLltBR54Lp9avFtab3mEgnrbjnPaAPIrLv3Nt2 6rRu2tmO1lZidD/cbA4zal0M26p9wp5TY14kyHpbLEIVloBjzetoqXK6u8Hjz/AP uagONypNDCySDR6M7jM85HDcLoFFrbBb8pruHSTxQejMeEmJxYf8b7rNl58/IWPB 1ymbNlvHL/4oSOlnrtHkjcxRWzpQ7U3gT9BThGyhCiI7EMyEHMgP3r7kTzEUwT6I avWDAgMBAAGjSDBGMAwGA1UdEwEB/wQCMAAwGgYDVR0RBBMwEYIPY3J5cHRvZ3Jh cGh5LmlvMBoGA1UdEQQTMBGCD2NyeXB0b2dyYXBoeS5pbzANBgkqhkiG9w0BAQUF AAOCAQEAAzAU5U814RrWYHiRQKDBu0710/ch5Q7z9fuJx7JWcX9pr152vCiPas4D s2YWZS6sNFh6g8h4AaDXjScNar15s2WWmWV3tS5InI9eFxn4TBHSjH4a/Lxa1W0V RA1XWNZZjm7Y9psRKjwiJmnK4Lm0Xk3KWMa03SlOpXNMTCQ/7vQX54zvlnynpo2E +pN6M28rbuA5SXkMRVv77W9kYZITAPqVLI99fSA8xaYW6NpyPtWPFSGbTwmt+U9w h/Acs8RYAR5NaS/aCELTqjsKjQmZAHlmxzbMo0ueqBUyQlOFX+G7TzBW1NFrSJNj 1TbuzMOELwTO7/l+m112lNpSWiP+UQ== -----END CERTIFICATE----- urllib3-2.0.7/test/contrib/test_pyopenssl.py0000644000000000000000000000567714513541732016132 0ustar00from __future__ import annotations import os from unittest import mock import pytest try: from cryptography import x509 from OpenSSL.crypto import FILETYPE_PEM, load_certificate # type: ignore[import] from urllib3.contrib.pyopenssl import _dnsname_to_stdlib, get_subj_alt_name except ImportError: pass def setup_module() -> None: try: from urllib3.contrib.pyopenssl import inject_into_urllib3 inject_into_urllib3() except ImportError as e: pytest.skip(f"Could not import PyOpenSSL: {e!r}") def teardown_module() -> None: try: from urllib3.contrib.pyopenssl import extract_from_urllib3 extract_from_urllib3() except ImportError: pass from ..test_ssl import TestSSL # noqa: E402, F401 from ..test_util import TestUtilSSL # noqa: E402, F401 from ..with_dummyserver.test_https import ( # noqa: E402, F401 TestHTTPS, TestHTTPS_IPV4SAN, TestHTTPS_IPV6SAN, TestHTTPS_TLSv1, TestHTTPS_TLSv1_1, TestHTTPS_TLSv1_2, TestHTTPS_TLSv1_3, ) from ..with_dummyserver.test_socketlevel import ( # noqa: E402, F401 TestClientCerts, TestSNI, TestSocketClosing, ) from ..with_dummyserver.test_socketlevel import ( # noqa: E402, F401 TestSSL as TestSocketSSL, ) class TestPyOpenSSLHelpers: """ Tests for PyOpenSSL helper functions. """ def test_dnsname_to_stdlib_simple(self) -> None: """ We can convert a dnsname to a native string when the domain is simple. """ name = "उदाहरण.परीक" expected_result = "xn--p1b6ci4b4b3a.xn--11b5bs8d" assert _dnsname_to_stdlib(name) == expected_result def test_dnsname_to_stdlib_leading_period(self) -> None: """ If there is a . in front of the domain name we correctly encode it. """ name = ".उदाहरण.परीक" expected_result = ".xn--p1b6ci4b4b3a.xn--11b5bs8d" assert _dnsname_to_stdlib(name) == expected_result def test_dnsname_to_stdlib_leading_splat(self) -> None: """ If there's a wildcard character in the front of the string we handle it appropriately. """ name = "*.उदाहरण.परीक" expected_result = "*.xn--p1b6ci4b4b3a.xn--11b5bs8d" assert _dnsname_to_stdlib(name) == expected_result @mock.patch("urllib3.contrib.pyopenssl.log.warning") def test_get_subj_alt_name(self, mock_warning: mock.MagicMock) -> None: """ If a certificate has two subject alternative names, cryptography raises an x509.DuplicateExtension exception. """ path = os.path.join(os.path.dirname(__file__), "duplicate_san.pem") with open(path) as fp: cert = load_certificate(FILETYPE_PEM, fp.read()) assert get_subj_alt_name(cert) == [] assert mock_warning.call_count == 1 assert isinstance(mock_warning.call_args[0][1], x509.DuplicateExtension) urllib3-2.0.7/test/contrib/test_pyopenssl_dependencies.py0000644000000000000000000000370414513541732020625 0ustar00from __future__ import annotations from unittest.mock import Mock, patch import pytest try: from urllib3.contrib.pyopenssl import extract_from_urllib3, inject_into_urllib3 except ImportError: pass def setup_module() -> None: try: from urllib3.contrib.pyopenssl import inject_into_urllib3 inject_into_urllib3() except ImportError as e: pytest.skip(f"Could not import PyOpenSSL: {e!r}") def teardown_module() -> None: try: from urllib3.contrib.pyopenssl import extract_from_urllib3 extract_from_urllib3() except ImportError: pass class TestPyOpenSSLInjection: """ Tests for error handling in pyopenssl's 'inject_into urllib3' """ def test_inject_validate_fail_cryptography(self) -> None: """ Injection should not be supported if cryptography is too old. """ try: with patch("cryptography.x509.extensions.Extensions") as mock: del mock.get_extension_for_class with pytest.raises(ImportError): inject_into_urllib3() finally: # `inject_into_urllib3` is not supposed to succeed. # If it does, this test should fail, but we need to # clean up so that subsequent tests are unaffected. extract_from_urllib3() def test_inject_validate_fail_pyopenssl(self) -> None: """ Injection should not be supported if pyOpenSSL is too old. """ try: return_val = Mock() del return_val._x509 with patch("OpenSSL.crypto.X509", return_value=return_val): with pytest.raises(ImportError): inject_into_urllib3() finally: # `inject_into_urllib3` is not supposed to succeed. # If it does, this test should fail, but we need to # clean up so that subsequent tests are unaffected. extract_from_urllib3() urllib3-2.0.7/test/contrib/test_securetransport.py0000644000000000000000000000323214513541732017322 0ustar00from __future__ import annotations import base64 import contextlib import socket import ssl import pytest try: from urllib3.contrib.securetransport import WrappedSocket except ImportError: pass def setup_module() -> None: try: from urllib3.contrib.securetransport import inject_into_urllib3 inject_into_urllib3() except ImportError as e: pytest.skip(f"Could not import SecureTransport: {repr(e)}") def teardown_module() -> None: try: from urllib3.contrib.securetransport import extract_from_urllib3 extract_from_urllib3() except ImportError: pass from ..test_util import TestUtilSSL # noqa: E402, F401 # SecureTransport does not support TLSv1.3 # https://github.com/urllib3/urllib3/issues/1674 from ..with_dummyserver.test_https import ( # noqa: E402, F401 TestHTTPS, TestHTTPS_TLSv1, TestHTTPS_TLSv1_1, TestHTTPS_TLSv1_2, ) from ..with_dummyserver.test_socketlevel import ( # noqa: E402, F401 TestClientCerts, TestSNI, TestSocketClosing, TestSSL, ) def test_no_crash_with_empty_trust_bundle() -> None: with contextlib.closing(socket.socket()) as s: ws = WrappedSocket(s) with pytest.raises(ssl.SSLError): ws._custom_validate(True, b"") def test_no_crash_with_invalid_trust_bundle() -> None: invalid_cert = base64.b64encode(b"invalid-cert") cert_bundle = ( b"-----BEGIN CERTIFICATE-----\n" + invalid_cert + b"\n-----END CERTIFICATE-----" ) with contextlib.closing(socket.socket()) as s: ws = WrappedSocket(s) with pytest.raises(ssl.SSLError): ws._custom_validate(True, cert_bundle) urllib3-2.0.7/test/contrib/test_socks.py0000644000000000000000000006246414513541732015215 0ustar00from __future__ import annotations import socket import threading import typing from socket import getaddrinfo as real_getaddrinfo from socket import timeout as SocketTimeout from test import SHORT_TIMEOUT from unittest.mock import Mock, patch import pytest import socks as py_socks # type: ignore[import] from dummyserver.server import DEFAULT_CA, DEFAULT_CERTS from dummyserver.testcase import IPV4SocketDummyServerTestCase from urllib3.contrib import socks from urllib3.exceptions import ConnectTimeoutError, NewConnectionError try: import ssl from urllib3.util import ssl_ as better_ssl HAS_SSL = True except ImportError: ssl = None # type: ignore[assignment] better_ssl = None # type: ignore[assignment] HAS_SSL = False SOCKS_NEGOTIATION_NONE = b"\x00" SOCKS_NEGOTIATION_PASSWORD = b"\x02" SOCKS_VERSION_SOCKS4 = b"\x04" SOCKS_VERSION_SOCKS5 = b"\x05" def _get_free_port(host: str) -> int: """ Gets a free port by opening a socket, binding it, checking the assigned port, and then closing it. """ s = socket.socket() s.bind((host, 0)) port = s.getsockname()[1] s.close() return port # type: ignore[no-any-return] def _read_exactly(sock: socket.socket, amt: int) -> bytes: """ Read *exactly* ``amt`` bytes from the socket ``sock``. """ data = b"" while amt > 0: chunk = sock.recv(amt) data += chunk amt -= len(chunk) return data def _read_until(sock: socket.socket, char: bytes) -> bytes: """ Read from the socket until the character is received. """ chunks = [] while True: chunk = sock.recv(1) chunks.append(chunk) if chunk == char: break return b"".join(chunks) def _address_from_socket(sock: socket.socket) -> bytes | str: """ Returns the address from the SOCKS socket """ addr_type = sock.recv(1) if addr_type == b"\x01": ipv4_addr = _read_exactly(sock, 4) return socket.inet_ntoa(ipv4_addr) elif addr_type == b"\x04": ipv6_addr = _read_exactly(sock, 16) return socket.inet_ntop(socket.AF_INET6, ipv6_addr) elif addr_type == b"\x03": addr_len = ord(sock.recv(1)) return _read_exactly(sock, addr_len) else: raise RuntimeError(f"Unexpected addr type: {addr_type!r}") def _set_up_fake_getaddrinfo(monkeypatch: pytest.MonkeyPatch) -> None: # Work around https://github.com/urllib3/urllib3/pull/2034 # Nothing prevents localhost to point to two different IPs. For example, in the # Ubuntu set up by GitHub Actions, localhost points both to 127.0.0.1 and ::1. # # In case of failure, PySocks will try the same request on both IPs, but our # handle_socks[45]_negotiation functions don't handle retries, which leads either to # a deadlock or a timeout in case of a failure on the first address. # # However, some tests need to exercise failure. We don't want retries there, but # can't affect PySocks retries via its API. Instead, we monkeypatch PySocks so that # it only sees a single address, which effectively disables retries. def fake_getaddrinfo( addr: str, port: int, family: int, socket_type: int ) -> list[ tuple[ socket.AddressFamily, socket.SocketKind, int, str, tuple[str, int] | tuple[str, int, int, int], ] ]: gai_list = real_getaddrinfo(addr, port, family, socket_type) gai_list = [gai for gai in gai_list if gai[0] == socket.AF_INET] return gai_list[:1] monkeypatch.setattr(py_socks.socket, "getaddrinfo", fake_getaddrinfo) def handle_socks5_negotiation( sock: socket.socket, negotiate: bool, username: bytes | None = None, password: bytes | None = None, ) -> typing.Generator[tuple[bytes | str, int], bool, None]: """ Handle the SOCKS5 handshake. Returns a generator object that allows us to break the handshake into steps so that the test code can intervene at certain useful points. """ received_version = sock.recv(1) assert received_version == SOCKS_VERSION_SOCKS5 nmethods = ord(sock.recv(1)) methods = _read_exactly(sock, nmethods) if negotiate: assert SOCKS_NEGOTIATION_PASSWORD in methods send_data = SOCKS_VERSION_SOCKS5 + SOCKS_NEGOTIATION_PASSWORD sock.sendall(send_data) # This is the password negotiation. negotiation_version = sock.recv(1) assert negotiation_version == b"\x01" ulen = ord(sock.recv(1)) provided_username = _read_exactly(sock, ulen) plen = ord(sock.recv(1)) provided_password = _read_exactly(sock, plen) if username == provided_username and password == provided_password: sock.sendall(b"\x01\x00") else: sock.sendall(b"\x01\x01") sock.close() return else: assert SOCKS_NEGOTIATION_NONE in methods send_data = SOCKS_VERSION_SOCKS5 + SOCKS_NEGOTIATION_NONE sock.sendall(send_data) # Client sends where they want to go. received_version = sock.recv(1) command = sock.recv(1) reserved = sock.recv(1) addr = _address_from_socket(sock) port_raw = _read_exactly(sock, 2) port = (ord(port_raw[0:1]) << 8) + (ord(port_raw[1:2])) # Check some basic stuff. assert received_version == SOCKS_VERSION_SOCKS5 assert command == b"\x01" # Only support connect, not bind. assert reserved == b"\x00" # Yield the address port tuple. succeed = yield addr, port if succeed: # Hard-coded response for now. response = SOCKS_VERSION_SOCKS5 + b"\x00\x00\x01\x7f\x00\x00\x01\xea\x60" else: # Hard-coded response for now. response = SOCKS_VERSION_SOCKS5 + b"\x01\00" sock.sendall(response) def handle_socks4_negotiation( sock: socket.socket, username: bytes | None = None ) -> typing.Generator[tuple[bytes | str, int], bool, None]: """ Handle the SOCKS4 handshake. Returns a generator object that allows us to break the handshake into steps so that the test code can intervene at certain useful points. """ received_version = sock.recv(1) command = sock.recv(1) port_raw = _read_exactly(sock, 2) port = (ord(port_raw[0:1]) << 8) + (ord(port_raw[1:2])) addr_raw = _read_exactly(sock, 4) provided_username = _read_until(sock, b"\x00")[:-1] # Strip trailing null. addr: bytes | str if addr_raw == b"\x00\x00\x00\x01": # Magic string: means DNS name. addr = _read_until(sock, b"\x00")[:-1] # Strip trailing null. else: addr = socket.inet_ntoa(addr_raw) # Check some basic stuff. assert received_version == SOCKS_VERSION_SOCKS4 assert command == b"\x01" # Only support connect, not bind. if username is not None and username != provided_username: sock.sendall(b"\x00\x5d\x00\x00\x00\x00\x00\x00") sock.close() return # Yield the address port tuple. succeed = yield addr, port if succeed: response = b"\x00\x5a\xea\x60\x7f\x00\x00\x01" else: response = b"\x00\x5b\x00\x00\x00\x00\x00\x00" sock.sendall(response) class TestSOCKSProxyManager: def test_invalid_socks_version_is_valueerror(self) -> None: with pytest.raises(ValueError, match="Unable to determine SOCKS version"): socks.SOCKSProxyManager(proxy_url="http://example.org") class TestSocks5Proxy(IPV4SocketDummyServerTestCase): """ Test the SOCKS proxy in SOCKS5 mode. """ def test_basic_request(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation(sock, negotiate=False) addr, port = next(handler) assert addr == "16.17.18.19" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks5://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://16.17.18.19") assert response.status == 200 assert response.data == b"" assert response.headers["Server"] == "SocksTestServer" def test_local_dns(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation(sock, negotiate=False) addr, port = next(handler) assert addr in ["127.0.0.1", "::1"] assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks5://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://localhost") assert response.status == 200 assert response.data == b"" assert response.headers["Server"] == "SocksTestServer" def test_correct_header_line(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation(sock, negotiate=False) addr, port = next(handler) assert addr == b"example.com" assert port == 80 with pytest.raises(StopIteration): handler.send(True) buf = b"" while True: buf += sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break assert buf.startswith(b"GET / HTTP/1.1") assert b"Host: example.com" in buf sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://example.com") assert response.status == 200 def test_connection_timeouts(self) -> None: event = threading.Event() def request_handler(listener: socket.socket) -> None: event.wait() self._start_server(request_handler) proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: with pytest.raises(ConnectTimeoutError): pm.request( "GET", "http://example.com", timeout=SHORT_TIMEOUT, retries=False ) event.set() @patch("socks.create_connection") def test_socket_timeout(self, create_connection: Mock) -> None: create_connection.side_effect = SocketTimeout() proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: with pytest.raises(ConnectTimeoutError, match="timed out"): pm.request("GET", "http://example.com", retries=False) def test_connection_failure(self) -> None: event = threading.Event() def request_handler(listener: socket.socket) -> None: listener.close() event.set() self._start_server(request_handler) proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: event.wait() with pytest.raises(NewConnectionError): pm.request("GET", "http://example.com", retries=False) def test_proxy_rejection(self, monkeypatch: pytest.MonkeyPatch) -> None: _set_up_fake_getaddrinfo(monkeypatch) evt = threading.Event() def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation(sock, negotiate=False) addr, port = next(handler) with pytest.raises(StopIteration): handler.send(False) evt.wait() sock.close() self._start_server(request_handler) proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: with pytest.raises(NewConnectionError): pm.request("GET", "http://example.com", retries=False) evt.set() def test_socks_with_password(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation( sock, negotiate=True, username=b"user", password=b"pass" ) addr, port = next(handler) assert addr == "16.17.18.19" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks5://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url, username="user", password="pass") as pm: response = pm.request("GET", "http://16.17.18.19") assert response.status == 200 assert response.data == b"" assert response.headers["Server"] == "SocksTestServer" def test_socks_with_auth_in_url(self) -> None: """ Test when we have auth info in url, i.e. socks5://user:pass@host:port and no username/password as params """ def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation( sock, negotiate=True, username=b"user", password=b"pass" ) addr, port = next(handler) assert addr == "16.17.18.19" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks5://user:pass@{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://16.17.18.19") assert response.status == 200 assert response.data == b"" assert response.headers["Server"] == "SocksTestServer" def test_socks_with_invalid_password(self, monkeypatch: pytest.MonkeyPatch) -> None: _set_up_fake_getaddrinfo(monkeypatch) def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation( sock, negotiate=True, username=b"user", password=b"pass" ) with pytest.raises(StopIteration): next(handler) self._start_server(request_handler) proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager( proxy_url, username="user", password="badpass" ) as pm: with pytest.raises( NewConnectionError, match="SOCKS5 authentication failed" ): pm.request("GET", "http://example.com", retries=False) def test_source_address_works(self) -> None: expected_port = _get_free_port(self.host) def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] assert sock.getpeername()[0] == "127.0.0.1" assert sock.getpeername()[1] == expected_port handler = handle_socks5_negotiation(sock, negotiate=False) addr, port = next(handler) assert addr == "16.17.18.19" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks5://{self.host}:{self.port}" with socks.SOCKSProxyManager( proxy_url, source_address=("127.0.0.1", expected_port) ) as pm: response = pm.request("GET", "http://16.17.18.19") assert response.status == 200 class TestSOCKS4Proxy(IPV4SocketDummyServerTestCase): """ Test the SOCKS proxy in SOCKS4 mode. Has relatively fewer tests than the SOCKS5 case, mostly because once the negotiation is done the two cases behave identically. """ def test_basic_request(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks4_negotiation(sock) addr, port = next(handler) assert addr == "16.17.18.19" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks4://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://16.17.18.19") assert response.status == 200 assert response.headers["Server"] == "SocksTestServer" assert response.data == b"" def test_local_dns(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks4_negotiation(sock) addr, port = next(handler) assert addr == "127.0.0.1" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks4://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://localhost") assert response.status == 200 assert response.headers["Server"] == "SocksTestServer" assert response.data == b"" def test_correct_header_line(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks4_negotiation(sock) addr, port = next(handler) assert addr == b"example.com" assert port == 80 with pytest.raises(StopIteration): handler.send(True) buf = b"" while True: buf += sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break assert buf.startswith(b"GET / HTTP/1.1") assert b"Host: example.com" in buf sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks4a://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: response = pm.request("GET", "http://example.com") assert response.status == 200 def test_proxy_rejection(self, monkeypatch: pytest.MonkeyPatch) -> None: _set_up_fake_getaddrinfo(monkeypatch) evt = threading.Event() def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks4_negotiation(sock) addr, port = next(handler) with pytest.raises(StopIteration): handler.send(False) evt.wait() sock.close() self._start_server(request_handler) proxy_url = f"socks4a://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url) as pm: with pytest.raises(NewConnectionError): pm.request("GET", "http://example.com", retries=False) evt.set() def test_socks4_with_username(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks4_negotiation(sock, username=b"user") addr, port = next(handler) assert addr == "16.17.18.19" assert port == 80 with pytest.raises(StopIteration): handler.send(True) while True: buf = sock.recv(65535) if buf.endswith(b"\r\n\r\n"): break sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(request_handler) proxy_url = f"socks4://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url, username="user") as pm: response = pm.request("GET", "http://16.17.18.19") assert response.status == 200 assert response.data == b"" assert response.headers["Server"] == "SocksTestServer" def test_socks_with_invalid_username(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks4_negotiation(sock, username=b"user") next(handler, None) self._start_server(request_handler) proxy_url = f"socks4a://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url, username="baduser") as pm: with pytest.raises(NewConnectionError, match="different user-ids"): pm.request("GET", "http://example.com", retries=False) class TestSOCKSWithTLS(IPV4SocketDummyServerTestCase): """ Test that TLS behaves properly for SOCKS proxies. """ @pytest.mark.skipif(not HAS_SSL, reason="No TLS available") def test_basic_request(self) -> None: def request_handler(listener: socket.socket) -> None: sock = listener.accept()[0] handler = handle_socks5_negotiation(sock, negotiate=False) addr, port = next(handler) assert addr == b"localhost" assert port == 443 with pytest.raises(StopIteration): handler.send(True) # Wrap in TLS context = better_ssl.SSLContext(ssl.PROTOCOL_SSLv23) # type: ignore[misc] context.load_cert_chain(DEFAULT_CERTS["certfile"], DEFAULT_CERTS["keyfile"]) tls = context.wrap_socket(sock, server_side=True) buf = b"" while True: buf += tls.recv(65535) if buf.endswith(b"\r\n\r\n"): break assert buf.startswith(b"GET / HTTP/1.1\r\n") tls.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: SocksTestServer\r\n" b"Content-Length: 0\r\n" b"\r\n" ) tls.close() sock.close() self._start_server(request_handler) proxy_url = f"socks5h://{self.host}:{self.port}" with socks.SOCKSProxyManager(proxy_url, ca_certs=DEFAULT_CA) as pm: response = pm.request("GET", "https://localhost") assert response.status == 200 assert response.data == b"" assert response.headers["Server"] == "SocksTestServer" urllib3-2.0.7/test/with_dummyserver/__init__.py0000644000000000000000000000000014513541732016522 0ustar00urllib3-2.0.7/test/with_dummyserver/test_chunked_transfer.py0000644000000000000000000002466614513541732021377 0ustar00from __future__ import annotations import socket import pytest from dummyserver.testcase import ( ConnectionMarker, SocketDummyServerTestCase, consume_socket, ) from urllib3 import HTTPConnectionPool from urllib3.util import SKIP_HEADER from urllib3.util.retry import Retry class TestChunkedTransfer(SocketDummyServerTestCase): def start_chunked_handler(self) -> None: self.buffer = b"" def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] while not self.buffer.endswith(b"\r\n0\r\n\r\n"): self.buffer += sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-type: text/plain\r\n" b"Content-Length: 0\r\n" b"\r\n" ) sock.close() self._start_server(socket_handler) @pytest.mark.parametrize( "chunks", [ ["foo", "bar", "", "bazzzzzzzzzzzzzzzzzzzzzz"], [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"], ], ) def test_chunks(self, chunks: list[bytes | str]) -> None: self.start_chunked_handler() with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen("GET", "/", body=chunks, headers=dict(DNT="1"), chunked=True) # type: ignore[arg-type] assert b"Transfer-Encoding" in self.buffer body = self.buffer.split(b"\r\n\r\n", 1)[1] lines = body.split(b"\r\n") # Empty chunks should have been skipped, as this could not be distinguished # from terminating the transmission for i, chunk in enumerate( [c.decode() if isinstance(c, bytes) else c for c in chunks if c] ): assert lines[i * 2] == hex(len(chunk))[2:].encode("utf-8") assert lines[i * 2 + 1] == chunk.encode("utf-8") def _test_body(self, data: bytes | str | None) -> None: self.start_chunked_handler() with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen("GET", "/", data, chunked=True) header, body = self.buffer.split(b"\r\n\r\n", 1) assert b"Transfer-Encoding: chunked" in header.split(b"\r\n") if data: bdata = data if isinstance(data, bytes) else data.encode("utf-8") assert b"\r\n" + bdata + b"\r\n" in body assert body.endswith(b"\r\n0\r\n\r\n") len_str = body.split(b"\r\n", 1)[0] stated_len = int(len_str, 16) assert stated_len == len(bdata) else: assert body == b"0\r\n\r\n" def test_bytestring_body(self) -> None: self._test_body(b"thisshouldbeonechunk\r\nasdf") def test_unicode_body(self) -> None: self._test_body("thisshouldbeonechunk\r\näöüß") def test_empty_body(self) -> None: self._test_body(None) def test_empty_string_body(self) -> None: self._test_body("") def test_empty_iterable_body(self) -> None: self._test_body(None) def _get_header_lines(self, prefix: bytes) -> list[bytes]: header_block = self.buffer.split(b"\r\n\r\n", 1)[0].lower() header_lines = header_block.split(b"\r\n")[1:] return [x for x in header_lines if x.startswith(prefix)] def test_removes_duplicate_host_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen( "GET", "/", body=chunks, headers={"Host": "test.org"}, chunked=True ) host_headers = self._get_header_lines(b"host") assert len(host_headers) == 1 def test_provides_default_host_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen("GET", "/", body=chunks, chunked=True) host_headers = self._get_header_lines(b"host") assert len(host_headers) == 1 def test_provides_default_user_agent_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen("GET", "/", body=chunks, chunked=True) ua_headers = self._get_header_lines(b"user-agent") assert len(ua_headers) == 1 def test_preserve_user_agent_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen( "GET", "/", body=chunks, headers={"user-Agent": "test-agent"}, chunked=True, ) ua_headers = self._get_header_lines(b"user-agent") # Validate that there is only one User-Agent header. assert len(ua_headers) == 1 # Validate that the existing User-Agent header is the one that was # provided. assert ua_headers[0] == b"user-agent: test-agent" def test_remove_user_agent_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen( "GET", "/", body=chunks, headers={"User-Agent": SKIP_HEADER}, chunked=True, ) ua_headers = self._get_header_lines(b"user-agent") assert len(ua_headers) == 0 def test_provides_default_transfer_encoding_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen("GET", "/", body=chunks, chunked=True) te_headers = self._get_header_lines(b"transfer-encoding") assert len(te_headers) == 1 def test_preserve_transfer_encoding_header(self) -> None: self.start_chunked_handler() chunks = [b"foo", b"bar", b"", b"bazzzzzzzzzzzzzzzzzzzzzz"] with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.urlopen( "GET", "/", body=chunks, headers={"transfer-Encoding": "test-transfer-encoding"}, chunked=True, ) te_headers = self._get_header_lines(b"transfer-encoding") # Validate that there is only one Transfer-Encoding header. assert len(te_headers) == 1 # Validate that the existing Transfer-Encoding header is the one that # was provided. assert te_headers[0] == b"transfer-encoding: test-transfer-encoding" def test_preserve_chunked_on_retry_after(self) -> None: self.chunked_requests = 0 self.socks: list[socket.socket] = [] def socket_handler(listener: socket.socket) -> None: for _ in range(2): sock = listener.accept()[0] self.socks.append(sock) request = consume_socket(sock) if b"Transfer-Encoding: chunked" in request.split(b"\r\n"): self.chunked_requests += 1 sock.send( b"HTTP/1.1 429 Too Many Requests\r\n" b"Content-Type: text/plain\r\n" b"Retry-After: 1\r\n" b"Content-Length: 0\r\n" b"Connection: close\r\n" b"\r\n" ) self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: retries = Retry(total=1) pool.urlopen("GET", "/", chunked=True, retries=retries) for sock in self.socks: sock.close() assert self.chunked_requests == 2 def test_preserve_chunked_on_redirect( self, monkeypatch: pytest.MonkeyPatch ) -> None: self.chunked_requests = 0 def socket_handler(listener: socket.socket) -> None: for i in range(2): sock = listener.accept()[0] request = ConnectionMarker.consume_request(sock) if b"Transfer-Encoding: chunked" in request.split(b"\r\n"): self.chunked_requests += 1 if i == 0: sock.sendall( b"HTTP/1.1 301 Moved Permanently\r\n" b"Location: /redirect\r\n\r\n" ) else: sock.sendall(b"HTTP/1.1 200 OK\r\n\r\n") sock.close() self._start_server(socket_handler) with ConnectionMarker.mark(monkeypatch): with HTTPConnectionPool(self.host, self.port) as pool: retries = Retry(redirect=1) pool.urlopen( "GET", "/", chunked=True, preload_content=False, retries=retries ) assert self.chunked_requests == 2 def test_preserve_chunked_on_broken_connection( self, monkeypatch: pytest.MonkeyPatch ) -> None: self.chunked_requests = 0 def socket_handler(listener: socket.socket) -> None: for i in range(2): sock = listener.accept()[0] request = ConnectionMarker.consume_request(sock) if b"Transfer-Encoding: chunked" in request.split(b"\r\n"): self.chunked_requests += 1 if i == 0: # Bad HTTP version will trigger a connection close sock.sendall(b"HTTP/0.5 200 OK\r\n\r\n") else: sock.sendall(b"HTTP/1.1 200 OK\r\n\r\n") sock.close() self._start_server(socket_handler) with ConnectionMarker.mark(monkeypatch): with HTTPConnectionPool(self.host, self.port) as pool: retries = Retry(read=1) pool.urlopen( "GET", "/", chunked=True, preload_content=False, retries=retries ) assert self.chunked_requests == 2 urllib3-2.0.7/test/with_dummyserver/test_connection.py0000644000000000000000000001046314513541732020177 0ustar00from __future__ import annotations import sys import typing from http.client import ResponseNotReady from unittest import mock import pytest from dummyserver.testcase import HTTPDummyServerTestCase as server from urllib3 import HTTPConnectionPool from urllib3.response import HTTPResponse @pytest.fixture() def pool() -> typing.Generator[HTTPConnectionPool, None, None]: server.setup_class() with HTTPConnectionPool(server.host, server.port) as pool: yield pool server.teardown_class() def test_returns_urllib3_HTTPResponse(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() method = "GET" path = "/" conn.request(method, path) response = conn.getresponse() assert isinstance(response, HTTPResponse) @pytest.mark.skipif(not hasattr(sys, "audit"), reason="requires python 3.8+") @mock.patch("urllib3.connection.sys.audit") def test_audit_event(audit_mock: mock.Mock, pool: HTTPConnectionPool) -> None: conn = pool._get_conn() conn.request("GET", "/") audit_mock.assert_any_call("http.client.connect", conn, conn.host, conn.port) # Ensure the event is raised only once. connect_events = [ call for call in audit_mock.mock_calls if call.args[0] == "http.client.connect" ] assert len(connect_events) == 1 def test_does_not_release_conn(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() method = "GET" path = "/" conn.request(method, path) response = conn.getresponse() response.release_conn() assert pool.pool.qsize() == 0 # type: ignore[union-attr] def test_releases_conn(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() assert conn is not None method = "GET" path = "/" conn.request(method, path) response = conn.getresponse() # If these variables are set by the pool # then the response can release the connection # back into the pool. response._pool = pool # type: ignore[attr-defined] response._connection = conn # type: ignore[attr-defined] response.release_conn() assert pool.pool.qsize() == 1 # type: ignore[union-attr] def test_double_getresponse(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() method = "GET" path = "/" conn.request(method, path) _ = conn.getresponse() # Calling getrepsonse() twice should cause an error with pytest.raises(ResponseNotReady): conn.getresponse() def test_connection_state_properties(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() assert conn.is_closed is True assert conn.is_connected is False assert conn.has_connected_to_proxy is False assert conn.is_verified is False assert conn.proxy_is_verified is None conn.connect() assert conn.is_closed is False assert conn.is_connected is True assert conn.has_connected_to_proxy is False assert conn.is_verified is False assert conn.proxy_is_verified is None conn.request("GET", "/") resp = conn.getresponse() assert resp.status == 200 conn.close() assert conn.is_closed is True assert conn.is_connected is False assert conn.has_connected_to_proxy is False assert conn.is_verified is False assert conn.proxy_is_verified is None def test_set_tunnel_is_reset(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() assert conn.is_closed is True assert conn.is_connected is False assert conn.has_connected_to_proxy is False assert conn.is_verified is False assert conn.proxy_is_verified is None conn.set_tunnel(host="host", port=8080, scheme="http") assert conn._tunnel_host == "host" # type: ignore[attr-defined] assert conn._tunnel_port == 8080 # type: ignore[attr-defined] assert conn._tunnel_scheme == "http" # type: ignore[attr-defined] conn.close() assert conn._tunnel_host is None # type: ignore[attr-defined] assert conn._tunnel_port is None # type: ignore[attr-defined] assert conn._tunnel_scheme is None # type: ignore[attr-defined] def test_invalid_tunnel_scheme(pool: HTTPConnectionPool) -> None: conn = pool._get_conn() with pytest.raises(ValueError) as e: conn.set_tunnel(host="host", port=8080, scheme="socks") assert ( str(e.value) == "Invalid proxy scheme for tunneling: 'socks', must be either 'http' or 'https'" ) urllib3-2.0.7/test/with_dummyserver/test_connectionpool.py0000644000000000000000000016043314513541732021074 0ustar00from __future__ import annotations import io import socket import time import typing import warnings from test import LONG_TIMEOUT, SHORT_TIMEOUT from threading import Event from unittest import mock from urllib.parse import urlencode import pytest from dummyserver.server import HAS_IPV6_AND_DNS, NoIPv6Warning from dummyserver.testcase import HTTPDummyServerTestCase, SocketDummyServerTestCase from urllib3 import HTTPConnectionPool, encode_multipart_formdata from urllib3._collections import HTTPHeaderDict from urllib3.connection import _get_default_user_agent from urllib3.exceptions import ( ConnectTimeoutError, DecodeError, EmptyPoolError, MaxRetryError, NameResolutionError, NewConnectionError, ReadTimeoutError, UnrewindableBodyError, ) from urllib3.fields import _TYPE_FIELD_VALUE_TUPLE from urllib3.util import SKIP_HEADER, SKIPPABLE_HEADERS from urllib3.util.retry import RequestHistory, Retry from urllib3.util.timeout import _TYPE_TIMEOUT, Timeout from .. import INVALID_SOURCE_ADDRESSES, TARPIT_HOST, VALID_SOURCE_ADDRESSES from ..port_helpers import find_unused_port def wait_for_socket(ready_event: Event) -> None: ready_event.wait() ready_event.clear() class TestConnectionPoolTimeouts(SocketDummyServerTestCase): def test_timeout_float(self) -> None: block_event = Event() ready_event = self.start_basic_handler(block_send=block_event, num=2) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: wait_for_socket(ready_event) with pytest.raises(ReadTimeoutError): pool.request("GET", "/", timeout=SHORT_TIMEOUT) block_event.set() # Release block # Shouldn't raise this time wait_for_socket(ready_event) block_event.set() # Pre-release block pool.request("GET", "/", timeout=LONG_TIMEOUT) def test_conn_closed(self) -> None: block_event = Event() self.start_basic_handler(block_send=block_event, num=1) with HTTPConnectionPool( self.host, self.port, timeout=SHORT_TIMEOUT, retries=False ) as pool: conn = pool._get_conn() pool._put_conn(conn) try: with pytest.raises(ReadTimeoutError): pool.urlopen("GET", "/") if not conn.is_closed: with pytest.raises(socket.error): conn.sock.recv(1024) # type: ignore[attr-defined] finally: pool._put_conn(conn) block_event.set() def test_timeout(self) -> None: # Requests should time out when expected block_event = Event() ready_event = self.start_basic_handler(block_send=block_event, num=3) # Pool-global timeout short_timeout = Timeout(read=SHORT_TIMEOUT) with HTTPConnectionPool( self.host, self.port, timeout=short_timeout, retries=False ) as pool: wait_for_socket(ready_event) block_event.clear() with pytest.raises(ReadTimeoutError): pool.request("GET", "/") block_event.set() # Release request # Request-specific timeouts should raise errors with HTTPConnectionPool( self.host, self.port, timeout=short_timeout, retries=False ) as pool: wait_for_socket(ready_event) now = time.time() with pytest.raises(ReadTimeoutError): pool.request("GET", "/", timeout=LONG_TIMEOUT) delta = time.time() - now message = "timeout was pool-level SHORT_TIMEOUT rather than request-level LONG_TIMEOUT" assert delta >= LONG_TIMEOUT, message block_event.set() # Release request # Timeout passed directly to request should raise a request timeout wait_for_socket(ready_event) with pytest.raises(ReadTimeoutError): pool.request("GET", "/", timeout=SHORT_TIMEOUT) block_event.set() # Release request def test_connect_timeout(self) -> None: url = "/" host, port = TARPIT_HOST, 80 timeout = Timeout(connect=SHORT_TIMEOUT) # Pool-global timeout with HTTPConnectionPool(host, port, timeout=timeout) as pool: conn = pool._get_conn() with pytest.raises(ConnectTimeoutError): pool._make_request(conn, "GET", url) # Retries retries = Retry(connect=0) with pytest.raises(MaxRetryError): pool.request("GET", url, retries=retries) # Request-specific connection timeouts big_timeout = Timeout(read=LONG_TIMEOUT, connect=LONG_TIMEOUT) with HTTPConnectionPool(host, port, timeout=big_timeout, retries=False) as pool: conn = pool._get_conn() with pytest.raises(ConnectTimeoutError): pool._make_request(conn, "GET", url, timeout=timeout) pool._put_conn(conn) with pytest.raises(ConnectTimeoutError): pool.request("GET", url, timeout=timeout) def test_total_applies_connect(self) -> None: host, port = TARPIT_HOST, 80 timeout = Timeout(total=None, connect=SHORT_TIMEOUT) with HTTPConnectionPool(host, port, timeout=timeout) as pool: conn = pool._get_conn() try: with pytest.raises(ConnectTimeoutError): pool._make_request(conn, "GET", "/") finally: conn.close() timeout = Timeout(connect=3, read=5, total=SHORT_TIMEOUT) with HTTPConnectionPool(host, port, timeout=timeout) as pool: conn = pool._get_conn() try: with pytest.raises(ConnectTimeoutError): pool._make_request(conn, "GET", "/") finally: conn.close() def test_total_timeout(self) -> None: block_event = Event() ready_event = self.start_basic_handler(block_send=block_event, num=2) wait_for_socket(ready_event) # This will get the socket to raise an EAGAIN on the read timeout = Timeout(connect=3, read=SHORT_TIMEOUT) with HTTPConnectionPool( self.host, self.port, timeout=timeout, retries=False ) as pool: with pytest.raises(ReadTimeoutError): pool.request("GET", "/") block_event.set() wait_for_socket(ready_event) block_event.clear() # The connect should succeed and this should hit the read timeout timeout = Timeout(connect=3, read=5, total=SHORT_TIMEOUT) with HTTPConnectionPool( self.host, self.port, timeout=timeout, retries=False ) as pool: with pytest.raises(ReadTimeoutError): pool.request("GET", "/") def test_create_connection_timeout(self) -> None: self.start_basic_handler(block_send=Event(), num=0) # needed for self.port timeout = Timeout(connect=SHORT_TIMEOUT, total=LONG_TIMEOUT) with HTTPConnectionPool( TARPIT_HOST, self.port, timeout=timeout, retries=False ) as pool: conn = pool._new_conn() with pytest.raises(ConnectTimeoutError): conn.connect() class TestConnectionPool(HTTPDummyServerTestCase): def test_get(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/specific_method", fields={"method": "GET"}) assert r.status == 200, r.data def test_post_url(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("POST", "/specific_method", fields={"method": "POST"}) assert r.status == 200, r.data def test_urlopen_put(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.urlopen("PUT", "/specific_method?method=PUT") assert r.status == 200, r.data def test_wrong_specific_method(self) -> None: # To make sure the dummy server is actually returning failed responses with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/specific_method", fields={"method": "POST"}) assert r.status == 400, r.data with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("POST", "/specific_method", fields={"method": "GET"}) assert r.status == 400, r.data def test_upload(self) -> None: data = "I'm in ur multipart form-data, hazing a cheezburgr" fields: dict[str, _TYPE_FIELD_VALUE_TUPLE] = { "upload_param": "filefield", "upload_filename": "lolcat.txt", "filefield": ("lolcat.txt", data), } fields["upload_size"] = len(data) # type: ignore with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("POST", "/upload", fields=fields) assert r.status == 200, r.data def test_one_name_multiple_values(self) -> None: fields = [("foo", "a"), ("foo", "b")] with HTTPConnectionPool(self.host, self.port) as pool: # urlencode r = pool.request("GET", "/echo", fields=fields) assert r.data == b"foo=a&foo=b" # multipart r = pool.request("POST", "/echo", fields=fields) assert r.data.count(b'name="foo"') == 2 def test_request_method_body(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: body = b"hi" r = pool.request("POST", "/echo", body=body) assert r.data == body fields = [("hi", "hello")] with pytest.raises(TypeError): pool.request("POST", "/echo", body=body, fields=fields) def test_unicode_upload(self) -> None: fieldname = "myfile" filename = "\xe2\x99\xa5.txt" data = "\xe2\x99\xa5".encode() size = len(data) fields: dict[str, _TYPE_FIELD_VALUE_TUPLE] = { "upload_param": fieldname, "upload_filename": filename, fieldname: (filename, data), } fields["upload_size"] = size # type: ignore with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("POST", "/upload", fields=fields) assert r.status == 200, r.data def test_nagle(self) -> None: """Test that connections have TCP_NODELAY turned on""" # This test needs to be here in order to be run. socket.create_connection actually tries # to connect to the host provided so we need a dummyserver to be running. with HTTPConnectionPool(self.host, self.port) as pool: conn = pool._get_conn() try: pool._make_request(conn, "GET", "/") tcp_nodelay_setting = conn.sock.getsockopt( # type: ignore[attr-defined] socket.IPPROTO_TCP, socket.TCP_NODELAY ) assert tcp_nodelay_setting finally: conn.close() @pytest.mark.parametrize( "socket_options", [ [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)], ((socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),), ], ) def test_socket_options(self, socket_options: tuple[int, int, int]) -> None: """Test that connections accept socket options.""" # This test needs to be here in order to be run. socket.create_connection actually tries to # connect to the host provided so we need a dummyserver to be running. with HTTPConnectionPool( self.host, self.port, socket_options=socket_options, ) as pool: # Get the socket of a new connection. s = pool._new_conn()._new_conn() # type: ignore[attr-defined] try: using_keepalive = ( s.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE) > 0 ) assert using_keepalive finally: s.close() @pytest.mark.parametrize("socket_options", [None, []]) def test_disable_default_socket_options( self, socket_options: list[int] | None ) -> None: """Test that passing None or empty list disables all socket options.""" # This test needs to be here in order to be run. socket.create_connection actually tries # to connect to the host provided so we need a dummyserver to be running. with HTTPConnectionPool( self.host, self.port, socket_options=socket_options ) as pool: s = pool._new_conn()._new_conn() # type: ignore[attr-defined] try: using_nagle = s.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY) == 0 assert using_nagle finally: s.close() def test_defaults_are_applied(self) -> None: """Test that modifying the default socket options works.""" # This test needs to be here in order to be run. socket.create_connection actually tries # to connect to the host provided so we need a dummyserver to be running. with HTTPConnectionPool(self.host, self.port) as pool: # Get the HTTPConnection instance conn = pool._new_conn() try: # Update the default socket options assert conn.socket_options is not None conn.socket_options += [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)] # type: ignore[operator] s = conn._new_conn() # type: ignore[attr-defined] nagle_disabled = ( s.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY) > 0 ) using_keepalive = ( s.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE) > 0 ) assert nagle_disabled assert using_keepalive finally: conn.close() s.close() def test_connection_error_retries(self) -> None: """ECONNREFUSED error should raise a connection error, with retries""" port = find_unused_port() with HTTPConnectionPool(self.host, port) as pool: with pytest.raises(MaxRetryError) as e: pool.request("GET", "/", retries=Retry(connect=3)) assert type(e.value.reason) == NewConnectionError def test_timeout_success(self) -> None: timeout = Timeout(connect=3, read=5, total=None) with HTTPConnectionPool(self.host, self.port, timeout=timeout) as pool: pool.request("GET", "/") # This should not raise a "Timeout already started" error pool.request("GET", "/") with HTTPConnectionPool(self.host, self.port, timeout=timeout) as pool: # This should also not raise a "Timeout already started" error pool.request("GET", "/") timeout = Timeout(total=None) with HTTPConnectionPool(self.host, self.port, timeout=timeout) as pool: pool.request("GET", "/") socket_timeout_reuse_testdata = pytest.mark.parametrize( ["timeout", "expect_settimeout_calls"], [ (1, (1, 1)), (None, (None, None)), (Timeout(read=4), (None, 4)), (Timeout(read=4, connect=5), (5, 4)), (Timeout(connect=6), (6, None)), ], ) @socket_timeout_reuse_testdata def test_socket_timeout_updated_on_reuse_constructor( self, timeout: _TYPE_TIMEOUT, expect_settimeout_calls: typing.Sequence[float | None], ) -> None: with HTTPConnectionPool(self.host, self.port, timeout=timeout) as pool: # Make a request to create a new connection. pool.urlopen("GET", "/") # Grab the connection and mock the inner socket. assert pool.pool is not None conn = pool.pool.get_nowait() conn_sock = mock.Mock(wraps=conn.sock) conn.sock = conn_sock pool._put_conn(conn) # Assert that sock.settimeout() is called with the new connect timeout, then the read timeout. pool.urlopen("GET", "/", timeout=timeout) conn_sock.settimeout.assert_has_calls( [mock.call(x) for x in expect_settimeout_calls] ) @socket_timeout_reuse_testdata def test_socket_timeout_updated_on_reuse_parameter( self, timeout: _TYPE_TIMEOUT, expect_settimeout_calls: typing.Sequence[float | None], ) -> None: with HTTPConnectionPool(self.host, self.port) as pool: # Make a request to create a new connection. pool.urlopen("GET", "/", timeout=LONG_TIMEOUT) # Grab the connection and mock the inner socket. assert pool.pool is not None conn = pool.pool.get_nowait() conn_sock = mock.Mock(wraps=conn.sock) conn.sock = conn_sock pool._put_conn(conn) # Assert that sock.settimeout() is called with the new connect timeout, then the read timeout. pool.urlopen("GET", "/", timeout=timeout) conn_sock.settimeout.assert_has_calls( [mock.call(x) for x in expect_settimeout_calls] ) def test_tunnel(self) -> None: # note the actual httplib.py has no tests for this functionality timeout = Timeout(total=None) with HTTPConnectionPool(self.host, self.port, timeout=timeout) as pool: conn = pool._get_conn() try: conn.set_tunnel(self.host, self.port) with mock.patch.object( conn, "_tunnel", create=True, return_value=None ) as conn_tunnel: pool._make_request(conn, "GET", "/") conn_tunnel.assert_called_once_with() finally: conn.close() # test that it's not called when tunnel is not set timeout = Timeout(total=None) with HTTPConnectionPool(self.host, self.port, timeout=timeout) as pool: conn = pool._get_conn() try: with mock.patch.object( conn, "_tunnel", create=True, return_value=None ) as conn_tunnel: pool._make_request(conn, "GET", "/") assert not conn_tunnel.called finally: conn.close() def test_redirect_relative_url_no_deprecation(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: with warnings.catch_warnings(): warnings.simplefilter("error", DeprecationWarning) pool.request("GET", "/redirect", fields={"target": "/"}) def test_redirect(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/redirect", fields={"target": "/"}, redirect=False) assert r.status == 303 r = pool.request("GET", "/redirect", fields={"target": "/"}) assert r.status == 200 assert r.data == b"Dummy server!" def test_303_redirect_makes_request_lose_body(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request( "POST", "/redirect", fields={"target": "/headers_and_params", "status": "303 See Other"}, ) data = response.json() assert data["params"] == {} assert "Content-Type" not in HTTPHeaderDict(data["headers"]) def test_bad_connect(self) -> None: with HTTPConnectionPool("badhost.invalid", self.port) as pool: with pytest.raises(MaxRetryError) as e: pool.request("GET", "/", retries=5) assert type(e.value.reason) == NameResolutionError def test_keepalive(self) -> None: with HTTPConnectionPool(self.host, self.port, block=True, maxsize=1) as pool: r = pool.request("GET", "/keepalive?close=0") r = pool.request("GET", "/keepalive?close=0") assert r.status == 200 assert pool.num_connections == 1 assert pool.num_requests == 2 def test_keepalive_close(self) -> None: with HTTPConnectionPool( self.host, self.port, block=True, maxsize=1, timeout=2 ) as pool: r = pool.request( "GET", "/keepalive?close=1", retries=0, headers={"Connection": "close"} ) assert pool.num_connections == 1 # The dummyserver will have responded with Connection:close, # and httplib will properly cleanup the socket. # We grab the HTTPConnection object straight from the Queue, # because _get_conn() is where the check & reset occurs assert pool.pool is not None conn = pool.pool.get() assert conn.sock is None pool._put_conn(conn) # Now with keep-alive r = pool.request( "GET", "/keepalive?close=0", retries=0, headers={"Connection": "keep-alive"}, ) # The dummyserver responded with Connection:keep-alive, the connection # persists. conn = pool.pool.get() assert conn.sock is not None pool._put_conn(conn) # Another request asking the server to close the connection. This one # should get cleaned up for the next request. r = pool.request( "GET", "/keepalive?close=1", retries=0, headers={"Connection": "close"} ) assert r.status == 200 conn = pool.pool.get() assert conn.sock is None pool._put_conn(conn) # Next request r = pool.request("GET", "/keepalive?close=0") def test_post_with_urlencode(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: data = {"banana": "hammock", "lol": "cat"} r = pool.request("POST", "/echo", fields=data, encode_multipart=False) assert r.data.decode("utf-8") == urlencode(data) def test_post_with_multipart(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: data = {"banana": "hammock", "lol": "cat"} r = pool.request("POST", "/echo", fields=data, encode_multipart=True) body = r.data.split(b"\r\n") encoded_data = encode_multipart_formdata(data)[0] expected_body = encoded_data.split(b"\r\n") # TODO: Get rid of extra parsing stuff when you can specify # a custom boundary to encode_multipart_formdata """ We need to loop the return lines because a timestamp is attached from within encode_multipart_formdata. When the server echos back the data, it has the timestamp from when the data was encoded, which is not equivalent to when we run encode_multipart_formdata on the data again. """ for i, line in enumerate(body): if line.startswith(b"--"): continue assert body[i] == expected_body[i] def test_post_with_multipart__iter__(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: data = {"hello": "world"} r = pool.request( "POST", "/echo", fields=data, preload_content=False, multipart_boundary="boundary", encode_multipart=True, ) chunks = [chunk for chunk in r] assert chunks == [ b"--boundary\r\n", b'Content-Disposition: form-data; name="hello"\r\n', b"\r\n", b"world\r\n", b"--boundary--\r\n", ] def test_check_gzip(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request( "GET", "/encodingrequest", headers={"accept-encoding": "gzip"} ) assert r.headers.get("content-encoding") == "gzip" assert r.data == b"hello, world!" def test_check_deflate(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request( "GET", "/encodingrequest", headers={"accept-encoding": "deflate"} ) assert r.headers.get("content-encoding") == "deflate" assert r.data == b"hello, world!" def test_bad_decode(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: with pytest.raises(DecodeError): pool.request( "GET", "/encodingrequest", headers={"accept-encoding": "garbage-deflate"}, ) with pytest.raises(DecodeError): pool.request( "GET", "/encodingrequest", headers={"accept-encoding": "garbage-gzip"}, ) def test_connection_count(self) -> None: with HTTPConnectionPool(self.host, self.port, maxsize=1) as pool: pool.request("GET", "/") pool.request("GET", "/") pool.request("GET", "/") assert pool.num_connections == 1 assert pool.num_requests == 3 def test_connection_count_bigpool(self) -> None: with HTTPConnectionPool(self.host, self.port, maxsize=16) as http_pool: http_pool.request("GET", "/") http_pool.request("GET", "/") http_pool.request("GET", "/") assert http_pool.num_connections == 1 assert http_pool.num_requests == 3 def test_partial_response(self) -> None: with HTTPConnectionPool(self.host, self.port, maxsize=1) as pool: req_data = {"lol": "cat"} resp_data = urlencode(req_data).encode("utf-8") r = pool.request("GET", "/echo", fields=req_data, preload_content=False) assert r.read(5) == resp_data[:5] assert r.read() == resp_data[5:] def test_lazy_load_twice(self) -> None: # This test is sad and confusing. Need to figure out what's # going on with partial reads and socket reuse. with HTTPConnectionPool( self.host, self.port, block=True, maxsize=1, timeout=2 ) as pool: payload_size = 1024 * 2 first_chunk = 512 boundary = "foo" req_data = {"count": "a" * payload_size} resp_data = encode_multipart_formdata(req_data, boundary=boundary)[0] req2_data = {"count": "b" * payload_size} resp2_data = encode_multipart_formdata(req2_data, boundary=boundary)[0] r1 = pool.request( "POST", "/echo", fields=req_data, multipart_boundary=boundary, preload_content=False, ) assert r1.read(first_chunk) == resp_data[:first_chunk] try: r2 = pool.request( "POST", "/echo", fields=req2_data, multipart_boundary=boundary, preload_content=False, pool_timeout=0.001, ) # This branch should generally bail here, but maybe someday it will # work? Perhaps by some sort of magic. Consider it a TODO. assert r2.read(first_chunk) == resp2_data[:first_chunk] assert r1.read() == resp_data[first_chunk:] assert r2.read() == resp2_data[first_chunk:] assert pool.num_requests == 2 except EmptyPoolError: assert r1.read() == resp_data[first_chunk:] assert pool.num_requests == 1 assert pool.num_connections == 1 def test_for_double_release(self) -> None: MAXSIZE = 5 # Check default state with HTTPConnectionPool(self.host, self.port, maxsize=MAXSIZE) as pool: assert pool.num_connections == 0 assert pool.pool is not None assert pool.pool.qsize() == MAXSIZE # Make an empty slot for testing pool.pool.get() assert pool.pool.qsize() == MAXSIZE - 1 # Check state after simple request pool.urlopen("GET", "/") assert pool.pool.qsize() == MAXSIZE - 1 # Check state without release pool.urlopen("GET", "/", preload_content=False) assert pool.pool.qsize() == MAXSIZE - 2 pool.urlopen("GET", "/") assert pool.pool.qsize() == MAXSIZE - 2 # Check state after read pool.urlopen("GET", "/").data assert pool.pool.qsize() == MAXSIZE - 2 pool.urlopen("GET", "/") assert pool.pool.qsize() == MAXSIZE - 2 def test_release_conn_parameter(self) -> None: MAXSIZE = 5 with HTTPConnectionPool(self.host, self.port, maxsize=MAXSIZE) as pool: assert pool.pool is not None assert pool.pool.qsize() == MAXSIZE # Make request without releasing connection pool.request("GET", "/", release_conn=False, preload_content=False) assert pool.pool.qsize() == MAXSIZE - 1 def test_dns_error(self) -> None: with HTTPConnectionPool( "thishostdoesnotexist.invalid", self.port, timeout=0.001 ) as pool: with pytest.raises(MaxRetryError): pool.request("GET", "/test", retries=2) @pytest.mark.parametrize("char", [" ", "\r", "\n", "\x00"]) def test_invalid_method_not_allowed(self, char: str) -> None: with pytest.raises(ValueError): with HTTPConnectionPool(self.host, self.port) as pool: pool.request("GET" + char, "/") def test_percent_encode_invalid_target_chars(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/echo_params?q=\r&k=\n \n") assert r.data == b"[('k', '\\n \\n'), ('q', '\\r')]" def test_source_address(self) -> None: for addr, is_ipv6 in VALID_SOURCE_ADDRESSES: if is_ipv6 and not HAS_IPV6_AND_DNS: warnings.warn("No IPv6 support: skipping.", NoIPv6Warning) continue with HTTPConnectionPool( self.host, self.port, source_address=addr, retries=False ) as pool: r = pool.request("GET", "/source_address") assert r.data == addr[0].encode() @pytest.mark.parametrize( "invalid_source_address, is_ipv6", INVALID_SOURCE_ADDRESSES ) def test_source_address_error( self, invalid_source_address: tuple[str, int], is_ipv6: bool ) -> None: with HTTPConnectionPool( self.host, self.port, source_address=invalid_source_address, retries=False ) as pool: if is_ipv6: with pytest.raises(NameResolutionError): pool.request("GET", f"/source_address?{invalid_source_address}") else: with pytest.raises(NewConnectionError): pool.request("GET", f"/source_address?{invalid_source_address}") def test_stream_keepalive(self) -> None: x = 2 with HTTPConnectionPool(self.host, self.port) as pool: for _ in range(x): response = pool.request( "GET", "/chunked", headers={"Connection": "keep-alive"}, preload_content=False, retries=False, ) for chunk in response.stream(): assert chunk == b"123" assert pool.num_connections == 1 assert pool.num_requests == x def test_read_chunked_short_circuit(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request("GET", "/chunked", preload_content=False) response.read() with pytest.raises(StopIteration): next(response.read_chunked()) def test_read_chunked_on_closed_response(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request("GET", "/chunked", preload_content=False) response.close() with pytest.raises(StopIteration): next(response.read_chunked()) def test_chunked_gzip(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request( "GET", "/chunked_gzip", preload_content=False, decode_content=True ) assert b"123" * 4 == response.read() def test_cleanup_on_connection_error(self) -> None: """ Test that connections are recycled to the pool on connection errors where no http response is received. """ poolsize = 3 with HTTPConnectionPool( self.host, self.port, maxsize=poolsize, block=True ) as http: assert http.pool is not None assert http.pool.qsize() == poolsize # force a connection error by supplying a non-existent # url. We won't get a response for this and so the # conn won't be implicitly returned to the pool. with pytest.raises(MaxRetryError): http.request( "GET", "/redirect", fields={"target": "/"}, release_conn=False, retries=0, ) r = http.request( "GET", "/redirect", fields={"target": "/"}, release_conn=False, retries=1, ) r.release_conn() # the pool should still contain poolsize elements assert http.pool.qsize() == http.pool.maxsize def test_mixed_case_hostname(self) -> None: with HTTPConnectionPool("LoCaLhOsT", self.port) as pool: response = pool.request("GET", f"http://LoCaLhOsT:{self.port}/") assert response.status == 200 def test_preserves_path_dot_segments(self) -> None: """ConnectionPool preserves dot segments in the URI""" with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request("GET", "/echo_uri/seg0/../seg2") assert response.data == b"/echo_uri/seg0/../seg2" def test_default_user_agent_header(self) -> None: """ConnectionPool has a default user agent""" default_ua = _get_default_user_agent() custom_ua = "I'm not a web scraper, what are you talking about?" custom_ua2 = "Yet Another User Agent" with HTTPConnectionPool(self.host, self.port) as pool: # Use default user agent if no user agent was specified. r = pool.request("GET", "/headers") request_headers = r.json() assert request_headers.get("User-Agent") == _get_default_user_agent() # Prefer the request user agent over the default. headers = {"UsEr-AGENt": custom_ua} r = pool.request("GET", "/headers", headers=headers) request_headers = r.json() assert request_headers.get("User-Agent") == custom_ua # Do not modify pool headers when using the default user agent. pool_headers = {"foo": "bar"} pool.headers = pool_headers r = pool.request("GET", "/headers") request_headers = r.json() assert request_headers.get("User-Agent") == default_ua assert "User-Agent" not in pool_headers pool.headers.update({"User-Agent": custom_ua2}) r = pool.request("GET", "/headers") request_headers = r.json() assert request_headers.get("User-Agent") == custom_ua2 @pytest.mark.parametrize( "headers", [ None, {}, {"User-Agent": "key"}, {"user-agent": "key"}, {b"uSeR-AgEnT": b"key"}, {b"user-agent": "key"}, ], ) @pytest.mark.parametrize("chunked", [True, False]) def test_user_agent_header_not_sent_twice( self, headers: dict[str, str] | None, chunked: bool ) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/headers", headers=headers, chunked=chunked) request_headers = r.json() if not headers: assert request_headers["User-Agent"].startswith("python-urllib3/") assert "key" not in request_headers["User-Agent"] else: assert request_headers["User-Agent"] == "key" def test_no_user_agent_header(self) -> None: """ConnectionPool can suppress sending a user agent header""" custom_ua = "I'm not a web scraper, what are you talking about?" with HTTPConnectionPool(self.host, self.port) as pool: # Suppress user agent in the request headers. no_ua_headers = {"User-Agent": SKIP_HEADER} r = pool.request("GET", "/headers", headers=no_ua_headers) request_headers = r.json() assert "User-Agent" not in request_headers assert no_ua_headers["User-Agent"] == SKIP_HEADER # Suppress user agent in the pool headers. pool.headers = no_ua_headers r = pool.request("GET", "/headers") request_headers = r.json() assert "User-Agent" not in request_headers assert no_ua_headers["User-Agent"] == SKIP_HEADER # Request headers override pool headers. pool_headers = {"User-Agent": custom_ua} pool.headers = pool_headers r = pool.request("GET", "/headers", headers=no_ua_headers) request_headers = r.json() assert "User-Agent" not in request_headers assert no_ua_headers["User-Agent"] == SKIP_HEADER assert pool_headers.get("User-Agent") == custom_ua @pytest.mark.parametrize( "accept_encoding", [ "Accept-Encoding", "accept-encoding", b"Accept-Encoding", b"accept-encoding", None, ], ) @pytest.mark.parametrize("host", ["Host", "host", b"Host", b"host", None]) @pytest.mark.parametrize( "user_agent", ["User-Agent", "user-agent", b"User-Agent", b"user-agent", None] ) @pytest.mark.parametrize("chunked", [True, False]) def test_skip_header( self, accept_encoding: str | None, host: str | None, user_agent: str | None, chunked: bool, ) -> None: headers = {} if accept_encoding is not None: headers[accept_encoding] = SKIP_HEADER if host is not None: headers[host] = SKIP_HEADER if user_agent is not None: headers[user_agent] = SKIP_HEADER with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/headers", headers=headers, chunked=chunked) request_headers = r.json() if accept_encoding is None: assert "Accept-Encoding" in request_headers else: assert accept_encoding not in request_headers if host is None: assert "Host" in request_headers else: assert host not in request_headers if user_agent is None: assert "User-Agent" in request_headers else: assert user_agent not in request_headers @pytest.mark.parametrize("header", ["Content-Length", "content-length"]) @pytest.mark.parametrize("chunked", [True, False]) def test_skip_header_non_supported(self, header: str, chunked: bool) -> None: with HTTPConnectionPool(self.host, self.port) as pool: with pytest.raises( ValueError, match="urllib3.util.SKIP_HEADER only supports 'Accept-Encoding', 'Host', 'User-Agent'", ) as e: pool.request( "GET", "/headers", headers={header: SKIP_HEADER}, chunked=chunked ) # Ensure that the error message stays up to date with 'SKIP_HEADER_SUPPORTED_HEADERS' assert all( ("'" + header.title() + "'") in str(e.value) for header in SKIPPABLE_HEADERS ) @pytest.mark.parametrize("chunked", [True, False]) @pytest.mark.parametrize("pool_request", [True, False]) @pytest.mark.parametrize("header_type", [dict, HTTPHeaderDict]) def test_headers_not_modified_by_request( self, chunked: bool, pool_request: bool, header_type: type[dict[str, str] | HTTPHeaderDict], ) -> None: # Test that the .request*() methods of ConnectionPool and HTTPConnection # don't modify the given 'headers' structure, instead they should # make their own internal copies at request time. headers = header_type() headers["key"] = "val" with HTTPConnectionPool(self.host, self.port) as pool: pool.headers = headers if pool_request: pool.request("GET", "/headers", chunked=chunked) else: conn = pool._get_conn() conn.request("GET", "/headers", chunked=chunked) assert pool.headers == {"key": "val"} assert isinstance(pool.headers, header_type) with HTTPConnectionPool(self.host, self.port) as pool: if pool_request: pool.request("GET", "/headers", headers=headers, chunked=chunked) else: conn = pool._get_conn() conn.request("GET", "/headers", headers=headers, chunked=chunked) assert headers == {"key": "val"} def test_request_chunked_is_deprecated( self, ) -> None: with HTTPConnectionPool(self.host, self.port) as pool: conn = pool._get_conn() with pytest.warns(DeprecationWarning) as w: conn.request_chunked("GET", "/headers") # type: ignore[attr-defined] assert len(w) == 1 and str(w[0].message) == ( "HTTPConnection.request_chunked() is deprecated and will be removed in urllib3 v2.1.0. " "Instead use HTTPConnection.request(..., chunked=True)." ) resp = conn.getresponse() assert resp.status == 200 assert resp.json()["Transfer-Encoding"] == "chunked" def test_bytes_header(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: headers = {"User-Agent": "test header"} r = pool.request("GET", "/headers", headers=headers) request_headers = r.json() assert "User-Agent" in request_headers assert request_headers["User-Agent"] == "test header" @pytest.mark.parametrize( "user_agent", ["Schönefeld/1.18.0", "Schönefeld/1.18.0".encode("iso-8859-1")] ) def test_user_agent_non_ascii_user_agent(self, user_agent: str) -> None: with HTTPConnectionPool(self.host, self.port, retries=False) as pool: r = pool.urlopen( "GET", "/headers", headers={"User-Agent": user_agent}, ) request_headers = r.json() assert "User-Agent" in request_headers assert request_headers["User-Agent"] == "Schönefeld/1.18.0" class TestRetry(HTTPDummyServerTestCase): def test_max_retry(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: with pytest.raises(MaxRetryError): pool.request("GET", "/redirect", fields={"target": "/"}, retries=0) def test_disabled_retry(self) -> None: """Disabled retries should disable redirect handling.""" with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/redirect", fields={"target": "/"}, retries=False) assert r.status == 303 r = pool.request( "GET", "/redirect", fields={"target": "/"}, retries=Retry(redirect=False), ) assert r.status == 303 with HTTPConnectionPool( "thishostdoesnotexist.invalid", self.port, timeout=0.001 ) as pool: with pytest.raises(NameResolutionError): pool.request("GET", "/test", retries=False) def test_read_retries(self) -> None: """Should retry for status codes in the forcelist""" with HTTPConnectionPool(self.host, self.port) as pool: retry = Retry(read=1, status_forcelist=[418]) resp = pool.request( "GET", "/successful_retry", headers={"test-name": "test_read_retries"}, retries=retry, ) assert resp.status == 200 def test_read_total_retries(self) -> None: """HTTP response w/ status code in the forcelist should be retried""" with HTTPConnectionPool(self.host, self.port) as pool: headers = {"test-name": "test_read_total_retries"} retry = Retry(total=1, status_forcelist=[418]) resp = pool.request( "GET", "/successful_retry", headers=headers, retries=retry ) assert resp.status == 200 def test_retries_wrong_forcelist(self) -> None: """HTTP response w/ status code not in forcelist shouldn't be retried""" with HTTPConnectionPool(self.host, self.port) as pool: retry = Retry(total=1, status_forcelist=[202]) resp = pool.request( "GET", "/successful_retry", headers={"test-name": "test_wrong_forcelist"}, retries=retry, ) assert resp.status == 418 def test_default_method_forcelist_retried(self) -> None: """urllib3 should retry methods in the default method forcelist""" with HTTPConnectionPool(self.host, self.port) as pool: retry = Retry(total=1, status_forcelist=[418]) resp = pool.request( "OPTIONS", "/successful_retry", headers={"test-name": "test_default_forcelist"}, retries=retry, ) assert resp.status == 200 def test_retries_wrong_method_list(self) -> None: """Method not in our allowed list should not be retried, even if code matches""" with HTTPConnectionPool(self.host, self.port) as pool: headers = {"test-name": "test_wrong_allowed_method"} retry = Retry(total=1, status_forcelist=[418], allowed_methods=["POST"]) resp = pool.request( "GET", "/successful_retry", headers=headers, retries=retry ) assert resp.status == 418 def test_read_retries_unsuccessful(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: headers = {"test-name": "test_read_retries_unsuccessful"} resp = pool.request("GET", "/successful_retry", headers=headers, retries=1) assert resp.status == 418 def test_retry_reuse_safe(self) -> None: """It should be possible to reuse a Retry object across requests""" with HTTPConnectionPool(self.host, self.port) as pool: headers = {"test-name": "test_retry_safe"} retry = Retry(total=1, status_forcelist=[418]) resp = pool.request( "GET", "/successful_retry", headers=headers, retries=retry ) assert resp.status == 200 with HTTPConnectionPool(self.host, self.port) as pool: resp = pool.request( "GET", "/successful_retry", headers=headers, retries=retry ) assert resp.status == 200 def test_retry_return_in_response(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: headers = {"test-name": "test_retry_return_in_response"} retry = Retry(total=2, status_forcelist=[418]) resp = pool.request( "GET", "/successful_retry", headers=headers, retries=retry ) assert resp.status == 200 assert resp.retries is not None assert resp.retries.total == 1 assert resp.retries.history == ( RequestHistory("GET", "/successful_retry", None, 418, None), ) def test_retry_redirect_history(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: resp = pool.request("GET", "/redirect", fields={"target": "/"}) assert resp.status == 200 assert resp.retries is not None assert resp.retries.history == ( RequestHistory("GET", "/redirect?target=%2F", None, 303, "/"), ) def test_multi_redirect_history(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request( "GET", "/multi_redirect", fields={"redirect_codes": "303,302,200"}, redirect=False, ) assert r.status == 303 assert r.retries is not None assert r.retries.history == tuple() with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request( "GET", "/multi_redirect", retries=10, fields={"redirect_codes": "303,302,301,307,302,200"}, ) assert r.status == 200 assert r.data == b"Done redirecting" expected = [ (303, "/multi_redirect?redirect_codes=302,301,307,302,200"), (302, "/multi_redirect?redirect_codes=301,307,302,200"), (301, "/multi_redirect?redirect_codes=307,302,200"), (307, "/multi_redirect?redirect_codes=302,200"), (302, "/multi_redirect?redirect_codes=200"), ] assert r.retries is not None actual = [ (history.status, history.redirect_location) for history in r.retries.history ] assert actual == expected class TestRetryAfter(HTTPDummyServerTestCase): def test_retry_after(self) -> None: # Request twice in a second to get a 429 response. with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request( "GET", "/retry_after", fields={"status": "429 Too Many Requests"}, retries=False, ) r = pool.request( "GET", "/retry_after", fields={"status": "429 Too Many Requests"}, retries=False, ) assert r.status == 429 r = pool.request( "GET", "/retry_after", fields={"status": "429 Too Many Requests"}, retries=True, ) assert r.status == 200 # Request twice in a second to get a 503 response. r = pool.request( "GET", "/retry_after", fields={"status": "503 Service Unavailable"}, retries=False, ) r = pool.request( "GET", "/retry_after", fields={"status": "503 Service Unavailable"}, retries=False, ) assert r.status == 503 r = pool.request( "GET", "/retry_after", fields={"status": "503 Service Unavailable"}, retries=True, ) assert r.status == 200 # Ignore Retry-After header on status which is not defined in # Retry.RETRY_AFTER_STATUS_CODES. r = pool.request( "GET", "/retry_after", fields={"status": "418 I'm a teapot"}, retries=True, ) assert r.status == 418 def test_redirect_after(self) -> None: with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/redirect_after", retries=False) assert r.status == 303 t = time.time() r = pool.request("GET", "/redirect_after") assert r.status == 200 delta = time.time() - t assert delta >= 1 t = time.time() timestamp = t + 2 r = pool.request("GET", "/redirect_after?date=" + str(timestamp)) assert r.status == 200 delta = time.time() - t assert delta >= 1 # Retry-After is past t = time.time() timestamp = t - 1 r = pool.request("GET", "/redirect_after?date=" + str(timestamp)) delta = time.time() - t assert r.status == 200 assert delta < 1 class TestFileBodiesOnRetryOrRedirect(HTTPDummyServerTestCase): def test_retries_put_filehandle(self) -> None: """HTTP PUT retry with a file-like object should not timeout""" with HTTPConnectionPool(self.host, self.port, timeout=0.1) as pool: retry = Retry(total=3, status_forcelist=[418]) # httplib reads in 8k chunks; use a larger content length content_length = 65535 data = b"A" * content_length uploaded_file = io.BytesIO(data) headers = { "test-name": "test_retries_put_filehandle", "Content-Length": str(content_length), } resp = pool.urlopen( "PUT", "/successful_retry", headers=headers, retries=retry, body=uploaded_file, assert_same_host=False, redirect=False, ) assert resp.status == 200 def test_redirect_put_file(self) -> None: """PUT with file object should work with a redirection response""" with HTTPConnectionPool(self.host, self.port, timeout=0.1) as pool: retry = Retry(total=3, status_forcelist=[418]) # httplib reads in 8k chunks; use a larger content length content_length = 65535 data = b"A" * content_length uploaded_file = io.BytesIO(data) headers = { "test-name": "test_redirect_put_file", "Content-Length": str(content_length), } url = "/redirect?target=/echo&status=307" resp = pool.urlopen( "PUT", url, headers=headers, retries=retry, body=uploaded_file, assert_same_host=False, redirect=True, ) assert resp.status == 200 assert resp.data == data def test_redirect_with_failed_tell(self) -> None: """Abort request if failed to get a position from tell()""" class BadTellObject(io.BytesIO): def tell(self) -> typing.NoReturn: raise OSError body = BadTellObject(b"the data") url = "/redirect?target=/successful_retry" # httplib uses fileno if Content-Length isn't supplied, # which is unsupported by BytesIO. headers = {"Content-Length": "8"} with HTTPConnectionPool(self.host, self.port, timeout=0.1) as pool: with pytest.raises( UnrewindableBodyError, match="Unable to record file position for" ): pool.urlopen("PUT", url, headers=headers, body=body) class TestRetryPoolSize(HTTPDummyServerTestCase): def test_pool_size_retry(self) -> None: retries = Retry(total=1, raise_on_status=False, status_forcelist=[404]) with HTTPConnectionPool( self.host, self.port, maxsize=10, retries=retries, block=True ) as pool: pool.urlopen("GET", "/not_found", preload_content=False) assert pool.num_connections == 1 class TestRedirectPoolSize(HTTPDummyServerTestCase): def test_pool_size_redirect(self) -> None: retries = Retry( total=1, raise_on_status=False, status_forcelist=[404], redirect=True ) with HTTPConnectionPool( self.host, self.port, maxsize=10, retries=retries, block=True ) as pool: pool.urlopen("GET", "/redirect", preload_content=False) assert pool.num_connections == 1 urllib3-2.0.7/test/with_dummyserver/test_https.py0000644000000000000000000012721514513541732017206 0ustar00from __future__ import annotations import contextlib import datetime import os.path import shutil import ssl import sys import tempfile import warnings from pathlib import Path from test import ( LONG_TIMEOUT, SHORT_TIMEOUT, TARPIT_HOST, notSecureTransport, requires_network, requires_ssl_context_keyfile_password, resolvesLocalhostFQDN, ) from test.conftest import ServerConfig from unittest import mock import pytest import trustme import urllib3.util as util import urllib3.util.ssl_ from dummyserver.server import ( DEFAULT_CA, DEFAULT_CA_KEY, DEFAULT_CERTS, encrypt_key_pem, ) from dummyserver.testcase import HTTPSDummyServerTestCase from urllib3 import HTTPSConnectionPool from urllib3.connection import RECENT_DATE, HTTPSConnection, VerifiedHTTPSConnection from urllib3.exceptions import ( ConnectTimeoutError, InsecureRequestWarning, MaxRetryError, ProtocolError, SSLError, SystemTimeWarning, ) from urllib3.util.ssl_match_hostname import CertificateError from urllib3.util.timeout import Timeout from .. import has_alpn TLSv1_CERTS = DEFAULT_CERTS.copy() TLSv1_CERTS["ssl_version"] = getattr(ssl, "PROTOCOL_TLSv1", None) TLSv1_1_CERTS = DEFAULT_CERTS.copy() TLSv1_1_CERTS["ssl_version"] = getattr(ssl, "PROTOCOL_TLSv1_1", None) TLSv1_2_CERTS = DEFAULT_CERTS.copy() TLSv1_2_CERTS["ssl_version"] = getattr(ssl, "PROTOCOL_TLSv1_2", None) TLSv1_3_CERTS = DEFAULT_CERTS.copy() TLSv1_3_CERTS["ssl_version"] = getattr(ssl, "PROTOCOL_TLS", None) CLIENT_INTERMEDIATE_PEM = "client_intermediate.pem" CLIENT_NO_INTERMEDIATE_PEM = "client_no_intermediate.pem" CLIENT_INTERMEDIATE_KEY = "client_intermediate.key" PASSWORD_CLIENT_KEYFILE = "client_password.key" CLIENT_CERT = CLIENT_INTERMEDIATE_PEM class TestHTTPS(HTTPSDummyServerTestCase): tls_protocol_name: str | None = None def tls_protocol_not_default(self) -> bool: return self.tls_protocol_name in {"TLSv1", "TLSv1.1"} def tls_version(self) -> ssl.TLSVersion: if self.tls_protocol_name is None: return pytest.skip("Skipping base test class") try: from ssl import TLSVersion except ImportError: return pytest.skip("ssl.TLSVersion isn't available") return TLSVersion[self.tls_protocol_name.replace(".", "_")] def ssl_version(self) -> int: if self.tls_protocol_name is None: return pytest.skip("Skipping base test class") attribute = f"PROTOCOL_{self.tls_protocol_name.replace('.', '_')}" ssl_version = getattr(ssl, attribute, None) if ssl_version is None: return pytest.skip(f"ssl.{attribute} isn't available") return ssl_version # type: ignore[no-any-return] @classmethod def setup_class(cls) -> None: super().setup_class() cls.certs_dir = tempfile.mkdtemp() # Start from existing root CA as we don't want to change the server certificate yet with open(DEFAULT_CA, "rb") as crt, open(DEFAULT_CA_KEY, "rb") as key: root_ca = trustme.CA.from_pem(crt.read(), key.read()) # Generate another CA to test verification failure bad_ca = trustme.CA() cls.bad_ca_path = os.path.join(cls.certs_dir, "ca_bad.pem") bad_ca.cert_pem.write_to_path(cls.bad_ca_path) # client cert chain intermediate_ca = root_ca.create_child_ca() cert = intermediate_ca.issue_cert("example.com") encrypted_key = encrypt_key_pem(cert.private_key_pem, b"letmein") cert.private_key_pem.write_to_path( os.path.join(cls.certs_dir, CLIENT_INTERMEDIATE_KEY) ) encrypted_key.write_to_path( os.path.join(cls.certs_dir, PASSWORD_CLIENT_KEYFILE) ) # Write the client cert and the intermediate CA client_cert = os.path.join(cls.certs_dir, CLIENT_INTERMEDIATE_PEM) cert.cert_chain_pems[0].write_to_path(client_cert) cert.cert_chain_pems[1].write_to_path(client_cert, append=True) # Write only the client cert cert.cert_chain_pems[0].write_to_path( os.path.join(cls.certs_dir, CLIENT_NO_INTERMEDIATE_PEM) ) @classmethod def teardown_class(cls) -> None: super().teardown_class() shutil.rmtree(cls.certs_dir) def test_simple(self) -> None: with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: r = https_pool.request("GET", "/") assert r.status == 200, r.data @resolvesLocalhostFQDN() def test_dotted_fqdn(self) -> None: with HTTPSConnectionPool( self.host + ".", self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as pool: r = pool.request("GET", "/") assert r.status == 200, r.data def test_client_intermediate(self) -> None: """Check that certificate chains work well with client certs We generate an intermediate CA from the root CA, and issue a client certificate from that intermediate CA. Since the server only knows about the root CA, we need to send it the certificate *and* the intermediate CA, so that it can check the whole chain. """ with HTTPSConnectionPool( self.host, self.port, key_file=os.path.join(self.certs_dir, CLIENT_INTERMEDIATE_KEY), cert_file=os.path.join(self.certs_dir, CLIENT_INTERMEDIATE_PEM), ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: r = https_pool.request("GET", "/certificate") subject = r.json() assert subject["organizationalUnitName"].startswith("Testing cert") def test_client_no_intermediate(self) -> None: """Check that missing links in certificate chains indeed break The only difference with test_client_intermediate is that we don't send the intermediate CA to the server, only the client cert. """ with HTTPSConnectionPool( self.host, self.port, cert_file=os.path.join(self.certs_dir, CLIENT_NO_INTERMEDIATE_PEM), key_file=os.path.join(self.certs_dir, CLIENT_INTERMEDIATE_KEY), ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.raises((SSLError, ProtocolError)): https_pool.request("GET", "/certificate", retries=False) @requires_ssl_context_keyfile_password() def test_client_key_password(self) -> None: with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, key_file=os.path.join(self.certs_dir, PASSWORD_CLIENT_KEYFILE), cert_file=os.path.join(self.certs_dir, CLIENT_CERT), key_password="letmein", ssl_minimum_version=self.tls_version(), ) as https_pool: r = https_pool.request("GET", "/certificate") subject = r.json() assert subject["organizationalUnitName"].startswith("Testing cert") @requires_ssl_context_keyfile_password() def test_client_encrypted_key_requires_password(self) -> None: with HTTPSConnectionPool( self.host, self.port, key_file=os.path.join(self.certs_dir, PASSWORD_CLIENT_KEYFILE), cert_file=os.path.join(self.certs_dir, CLIENT_CERT), key_password=None, ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.raises(MaxRetryError, match="password is required") as e: https_pool.request("GET", "/certificate") assert isinstance(e.value.reason, SSLError) def test_verified(self) -> None: with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: conn = https_pool._new_conn() assert conn.__class__ == VerifiedHTTPSConnection with warnings.catch_warnings(record=True) as w: r = https_pool.request("GET", "/") assert r.status == 200 assert [str(wm) for wm in w] == [] def test_verified_with_context(self) -> None: ctx = util.ssl_.create_urllib3_context( cert_reqs=ssl.CERT_REQUIRED, ssl_minimum_version=self.tls_version() ) ctx.load_verify_locations(cafile=DEFAULT_CA) with HTTPSConnectionPool(self.host, self.port, ssl_context=ctx) as https_pool: conn = https_pool._new_conn() assert conn.__class__ == VerifiedHTTPSConnection with mock.patch("warnings.warn") as warn: r = https_pool.request("GET", "/") assert r.status == 200 assert not warn.called, warn.call_args_list def test_context_combines_with_ca_certs(self) -> None: ctx = util.ssl_.create_urllib3_context( cert_reqs=ssl.CERT_REQUIRED, ssl_minimum_version=self.tls_version() ) with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_context=ctx ) as https_pool: conn = https_pool._new_conn() assert conn.__class__ == VerifiedHTTPSConnection with mock.patch("warnings.warn") as warn: r = https_pool.request("GET", "/") assert r.status == 200 assert not warn.called, warn.call_args_list @notSecureTransport() # SecureTransport does not support cert directories def test_ca_dir_verified(self, tmp_path: Path) -> None: # OpenSSL looks up certificates by the hash for their name, see c_rehash # TODO infer the bytes using `cryptography.x509.Name.public_bytes`. # https://github.com/pyca/cryptography/pull/3236 shutil.copyfile(DEFAULT_CA, str(tmp_path / "81deb5f7.0")) with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ca_cert_dir=str(tmp_path), ssl_minimum_version=self.tls_version(), ) as https_pool: conn = https_pool._new_conn() assert conn.__class__ == VerifiedHTTPSConnection with warnings.catch_warnings(record=True) as w: r = https_pool.request("GET", "/") assert r.status == 200 assert [str(wm) for wm in w] == [] def test_invalid_common_name(self) -> None: with HTTPSConnectionPool( "127.0.0.1", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.raises(MaxRetryError) as e: https_pool.request("GET", "/", retries=0) assert isinstance(e.value.reason, SSLError) assert "doesn't match" in str( e.value.reason ) or "certificate verify failed" in str(e.value.reason) def test_verified_with_bad_ca_certs(self) -> None: with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ca_certs=self.bad_ca_path, ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.raises(MaxRetryError) as e: https_pool.request("GET", "/") assert isinstance(e.value.reason, SSLError) assert ( "certificate verify failed" in str(e.value.reason) # PyPy is more specific or "self signed certificate in certificate chain" in str(e.value.reason) ), f"Expected 'certificate verify failed', instead got: {e.value.reason!r}" def test_wrap_socket_failure_resource_leak(self) -> None: with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ca_certs=self.bad_ca_path, ssl_minimum_version=self.tls_version(), ) as https_pool: conn = https_pool._get_conn() try: with pytest.raises(ssl.SSLError): conn.connect() assert conn.sock is not None # type: ignore[attr-defined] finally: conn.close() def test_verified_without_ca_certs(self) -> None: # default is cert_reqs=None which is ssl.CERT_NONE with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.raises(MaxRetryError) as e: https_pool.request("GET", "/") assert isinstance(e.value.reason, SSLError) # there is a different error message depending on whether or # not pyopenssl is injected assert ( "No root certificates specified" in str(e.value.reason) # PyPy is more specific or "self signed certificate in certificate chain" in str(e.value.reason) # PyPy sometimes uses all-caps here or "certificate verify failed" in str(e.value.reason).lower() or "invalid certificate chain" in str(e.value.reason) ), ( "Expected 'No root certificates specified', " "'certificate verify failed', or " "'invalid certificate chain', " "instead got: %r" % e.value.reason ) def test_no_ssl(self) -> None: with HTTPSConnectionPool(self.host, self.port) as pool: pool.ConnectionCls = None # type: ignore[assignment] with pytest.raises(ImportError): pool._new_conn() with pytest.raises(ImportError): pool.request("GET", "/", retries=0) def test_unverified_ssl(self) -> None: """Test that bare HTTPSConnection can connect, make requests""" with HTTPSConnectionPool( self.host, self.port, cert_reqs=ssl.CERT_NONE, ssl_minimum_version=self.tls_version(), ) as pool: with mock.patch("warnings.warn") as warn: r = pool.request("GET", "/") assert r.status == 200 assert warn.called # Modern versions of Python, or systems using PyOpenSSL, only emit # the unverified warning. Older systems may also emit other # warnings, which we want to ignore here. calls = warn.call_args_list assert InsecureRequestWarning in [x[0][1] for x in calls] def test_ssl_unverified_with_ca_certs(self) -> None: with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_NONE", ca_certs=self.bad_ca_path, ssl_minimum_version=self.tls_version(), ) as pool: with mock.patch("warnings.warn") as warn: r = pool.request("GET", "/") assert r.status == 200 assert warn.called # Modern versions of Python, or systems using PyOpenSSL, only emit # the unverified warning. Older systems may also emit other # warnings, which we want to ignore here. calls = warn.call_args_list category = calls[0][0][1] assert category == InsecureRequestWarning def test_assert_hostname_false(self) -> None: with HTTPSConnectionPool( "localhost", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.assert_hostname = False https_pool.request("GET", "/") def test_assert_specific_hostname(self) -> None: with HTTPSConnectionPool( "localhost", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.assert_hostname = "localhost" https_pool.request("GET", "/") def test_server_hostname(self) -> None: with HTTPSConnectionPool( "127.0.0.1", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, server_hostname="localhost", ssl_minimum_version=self.tls_version(), ) as https_pool: conn = https_pool._new_conn() conn.request("GET", "/") # Assert the wrapping socket is using the passed-through SNI name. # pyopenssl doesn't let you pull the server_hostname back off the # socket, so only add this assertion if the attribute is there (i.e. # the python ssl module). if hasattr(conn.sock, "server_hostname"): # type: ignore[attr-defined] assert conn.sock.server_hostname == "localhost" # type: ignore[attr-defined] def test_assert_fingerprint_md5(self) -> None: with HTTPSConnectionPool( "localhost", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, assert_fingerprint=("55:39:BF:70:05:12:43:FA:1F:D1:BF:4E:E8:1B:07:1D"), ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.request("GET", "/") def test_assert_fingerprint_sha1(self) -> None: with HTTPSConnectionPool( "localhost", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, assert_fingerprint=( "72:8B:55:4C:9A:FC:1E:88:A1:1C:AD:1B:B2:E7:CC:3E:DB:C8:F9:8A" ), ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.request("GET", "/") def test_assert_fingerprint_sha256(self) -> None: with HTTPSConnectionPool( "localhost", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, assert_fingerprint=( "E3:59:8E:69:FF:C5:9F:C7:88:87:44:58:22:7F:90:8D:D9:BC:12:C4:90:79:D5:" "DC:A8:5D:4F:60:40:1E:A6:D2" ), ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.request("GET", "/") def test_assert_invalid_fingerprint(self) -> None: def _test_request(pool: HTTPSConnectionPool) -> SSLError: with pytest.raises(MaxRetryError) as cm: pool.request("GET", "/", retries=0) assert isinstance(cm.value.reason, SSLError) return cm.value.reason with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.assert_fingerprint = ( "AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA" ) e = _test_request(https_pool) expected = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" got = "728b554c9afc1e88a11cad1bb2e7cc3edbc8f98a" assert ( str(e) == f'Fingerprints did not match. Expected "{expected}", got "{got}"' ) # Uneven length https_pool.assert_fingerprint = "AA:A" e = _test_request(https_pool) assert "Fingerprint of invalid length:" in str(e) # Invalid length https_pool.assert_fingerprint = "AA" e = _test_request(https_pool) assert "Fingerprint of invalid length:" in str(e) def test_verify_none_and_bad_fingerprint(self) -> None: with HTTPSConnectionPool( "127.0.0.1", self.port, cert_reqs="CERT_NONE", assert_hostname=False, assert_fingerprint=( "AA:8B:55:4C:9A:FC:1E:88:A1:1C:AD:1B:B2:E7:CC:3E:DB:C8:F9:8A" ), ) as https_pool: with pytest.raises(MaxRetryError) as cm: https_pool.request("GET", "/", retries=0) assert isinstance(cm.value.reason, SSLError) def test_verify_none_and_good_fingerprint(self) -> None: with HTTPSConnectionPool( "127.0.0.1", self.port, cert_reqs="CERT_NONE", assert_hostname=False, assert_fingerprint=( "72:8B:55:4C:9A:FC:1E:88:A1:1C:AD:1B:B2:E7:CC:3E:DB:C8:F9:8A" ), ) as https_pool: https_pool.request("GET", "/") @notSecureTransport() def test_good_fingerprint_and_hostname_mismatch(self) -> None: # This test doesn't run with SecureTransport because we don't turn off # hostname validation without turning off all validation, which this # test doesn't do (deliberately). We should revisit this if we make # new decisions. with HTTPSConnectionPool( "127.0.0.1", self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, assert_fingerprint=( "72:8B:55:4C:9A:FC:1E:88:A1:1C:AD:1B:B2:E7:CC:3E:DB:C8:F9:8A" ), ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.request("GET", "/") @requires_network() def test_https_timeout(self) -> None: timeout = Timeout(total=None, connect=SHORT_TIMEOUT) with HTTPSConnectionPool( TARPIT_HOST, self.port, timeout=timeout, retries=False, cert_reqs="CERT_REQUIRED", ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.raises(ConnectTimeoutError): https_pool.request("GET", "/") timeout = Timeout(read=0.01) with HTTPSConnectionPool( self.host, self.port, timeout=timeout, retries=False, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, assert_fingerprint=( "72:8B:55:4C:9A:FC:1E:88:A1:1C:AD:1B:B2:E7:CC:3E:DB:C8:F9:8A" ), ssl_minimum_version=self.tls_version(), ) as https_pool: # TODO This was removed in https://github.com/urllib3/urllib3/pull/703/files # We need to put something back or remove this block. pass timeout = Timeout(total=None) with HTTPSConnectionPool( self.host, self.port, timeout=timeout, cert_reqs="CERT_NONE", ssl_minimum_version=self.tls_version(), ) as https_pool: with pytest.warns(InsecureRequestWarning): https_pool.request("GET", "/") def test_tunnel(self) -> None: """test the _tunnel behavior""" timeout = Timeout(total=None) with HTTPSConnectionPool( self.host, self.port, timeout=timeout, cert_reqs="CERT_NONE", ssl_minimum_version=self.tls_version(), ) as https_pool: conn = https_pool._new_conn() try: conn.set_tunnel(self.host, self.port) with mock.patch.object( conn, "_tunnel", create=True, return_value=None ) as conn_tunnel: with pytest.warns(InsecureRequestWarning): https_pool._make_request(conn, "GET", "/") conn_tunnel.assert_called_once_with() finally: conn.close() @requires_network() def test_enhanced_timeout(self) -> None: with HTTPSConnectionPool( TARPIT_HOST, self.port, timeout=Timeout(connect=SHORT_TIMEOUT), retries=False, cert_reqs="CERT_REQUIRED", ) as https_pool: conn = https_pool._new_conn() try: with pytest.raises(ConnectTimeoutError): https_pool.request("GET", "/") with pytest.raises(ConnectTimeoutError): https_pool._make_request(conn, "GET", "/") finally: conn.close() with HTTPSConnectionPool( TARPIT_HOST, self.port, timeout=Timeout(connect=LONG_TIMEOUT), retries=False, cert_reqs="CERT_REQUIRED", ) as https_pool: with pytest.raises(ConnectTimeoutError): https_pool.request("GET", "/", timeout=Timeout(connect=SHORT_TIMEOUT)) with HTTPSConnectionPool( TARPIT_HOST, self.port, timeout=Timeout(total=None), retries=False, cert_reqs="CERT_REQUIRED", ) as https_pool: conn = https_pool._new_conn() try: with pytest.raises(ConnectTimeoutError): https_pool.request( "GET", "/", timeout=Timeout(total=None, connect=SHORT_TIMEOUT) ) finally: conn.close() def test_enhanced_ssl_connection(self) -> None: fingerprint = "72:8B:55:4C:9A:FC:1E:88:A1:1C:AD:1B:B2:E7:CC:3E:DB:C8:F9:8A" with HTTPSConnectionPool( self.host, self.port, cert_reqs="CERT_REQUIRED", ca_certs=DEFAULT_CA, assert_fingerprint=fingerprint, ssl_minimum_version=self.tls_version(), ) as https_pool: r = https_pool.request("GET", "/") assert r.status == 200 def test_ssl_correct_system_time(self) -> None: with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.cert_reqs = "CERT_REQUIRED" https_pool.ca_certs = DEFAULT_CA w = self._request_without_resource_warnings("GET", "/") assert [] == w def test_ssl_wrong_system_time(self) -> None: with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.cert_reqs = "CERT_REQUIRED" https_pool.ca_certs = DEFAULT_CA with mock.patch("urllib3.connection.datetime") as mock_date: mock_date.date.today.return_value = datetime.date(1970, 1, 1) w = self._request_without_resource_warnings("GET", "/") assert len(w) == 1 warning = w[0] assert SystemTimeWarning == warning.category assert isinstance(warning.message, Warning) assert str(RECENT_DATE) in warning.message.args[0] def _request_without_resource_warnings( self, method: str, url: str ) -> list[warnings.WarningMessage]: with warnings.catch_warnings(record=True) as w: warnings.simplefilter("always") with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: https_pool.request(method, url) w = [x for x in w if not isinstance(x.message, ResourceWarning)] return w def test_set_ssl_version_to_tls_version(self) -> None: if self.tls_protocol_name is None: pytest.skip("Skipping base test class") with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA ) as https_pool: https_pool.ssl_version = ssl_version = self.certs["ssl_version"] if ssl_version is getattr(ssl, "PROTOCOL_TLS", object()): cmgr: contextlib.AbstractContextManager[ object ] = contextlib.nullcontext() else: cmgr = pytest.warns( DeprecationWarning, match=r"'ssl_version' option is deprecated and will be removed " r"in urllib3 v2\.1\.0\. Instead use 'ssl_minimum_version'", ) with cmgr: r = https_pool.request("GET", "/") assert r.status == 200, r.data def test_set_cert_default_cert_required(self) -> None: conn = VerifiedHTTPSConnection(self.host, self.port) with pytest.warns(DeprecationWarning) as w: conn.set_cert() assert conn.cert_reqs == ssl.CERT_REQUIRED assert len(w) == 1 and str(w[0].message) == ( "HTTPSConnection.set_cert() is deprecated and will be removed in urllib3 v2.1.0. " "Instead provide the parameters to the HTTPSConnection constructor." ) @pytest.mark.parametrize("verify_mode", [ssl.CERT_NONE, ssl.CERT_REQUIRED]) def test_set_cert_inherits_cert_reqs_from_ssl_context( self, verify_mode: int ) -> None: ssl_context = urllib3.util.ssl_.create_urllib3_context(cert_reqs=verify_mode) assert ssl_context.verify_mode == verify_mode conn = HTTPSConnection(self.host, self.port, ssl_context=ssl_context) with pytest.warns(DeprecationWarning) as w: conn.set_cert() assert conn.cert_reqs == verify_mode assert ( conn.ssl_context is not None and conn.ssl_context.verify_mode == verify_mode ) assert len(w) == 1 and str(w[0].message) == ( "HTTPSConnection.set_cert() is deprecated and will be removed in urllib3 v2.1.0. " "Instead provide the parameters to the HTTPSConnection constructor." ) def test_tls_protocol_name_of_socket(self) -> None: if self.tls_protocol_name is None: pytest.skip("Skipping base test class") with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: conn = https_pool._get_conn() try: conn.connect() if not hasattr(conn.sock, "version"): # type: ignore[attr-defined] pytest.skip("SSLSocket.version() not available") assert conn.sock.version() == self.tls_protocol_name # type: ignore[attr-defined] finally: conn.close() def test_ssl_version_is_deprecated(self) -> None: if self.tls_protocol_name is None: pytest.skip("Skipping base test class") with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_version=self.ssl_version() ) as https_pool: conn = https_pool._get_conn() try: with pytest.warns(DeprecationWarning) as w: conn.connect() finally: conn.close() assert len(w) >= 1 assert any(x.category == DeprecationWarning for x in w) assert any( str(x.message) == ( "'ssl_version' option is deprecated and will be removed in " "urllib3 v2.1.0. Instead use 'ssl_minimum_version'" ) for x in w ) @pytest.mark.parametrize( "ssl_version", [None, ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS_CLIENT] ) def test_ssl_version_with_protocol_tls_or_client_not_deprecated( self, ssl_version: int | None ) -> None: if self.tls_protocol_name is None: pytest.skip("Skipping base test class") if self.tls_protocol_not_default(): pytest.skip( f"Skipping because '{self.tls_protocol_name}' isn't set by default" ) with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_version=ssl_version ) as https_pool: conn = https_pool._get_conn() try: with warnings.catch_warnings(record=True) as w: conn.connect() finally: conn.close() assert [str(wm) for wm in w if wm.category != ResourceWarning] == [] def test_no_tls_version_deprecation_with_ssl_context(self) -> None: if self.tls_protocol_name is None: pytest.skip("Skipping base test class") ctx = util.ssl_.create_urllib3_context(ssl_minimum_version=self.tls_version()) with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_context=ctx, ) as https_pool: conn = https_pool._get_conn() try: with warnings.catch_warnings(record=True) as w: conn.connect() finally: conn.close() assert [str(wm) for wm in w if wm.category != ResourceWarning] == [] def test_tls_version_maximum_and_minimum(self) -> None: if self.tls_protocol_name is None: pytest.skip("Skipping base test class") from ssl import TLSVersion min_max_versions = [ (self.tls_version(), self.tls_version()), (TLSVersion.MINIMUM_SUPPORTED, self.tls_version()), (TLSVersion.MINIMUM_SUPPORTED, TLSVersion.MAXIMUM_SUPPORTED), ] for minimum_version, maximum_version in min_max_versions: with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=minimum_version, ssl_maximum_version=maximum_version, ) as https_pool: conn = https_pool._get_conn() try: conn.connect() assert conn.sock.version() == self.tls_protocol_name # type: ignore[attr-defined] finally: conn.close() @pytest.mark.skipif(sys.version_info < (3, 8), reason="requires python 3.8+") def test_sslkeylogfile( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: if not hasattr(util.SSLContext, "keylog_filename"): pytest.skip("requires OpenSSL 1.1.1+") keylog_file = tmp_path / "keylogfile.txt" monkeypatch.setenv("SSLKEYLOGFILE", str(keylog_file)) with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as https_pool: r = https_pool.request("GET", "/") assert r.status == 200, r.data assert keylog_file.is_file(), "keylogfile '%s' should exist" % str( keylog_file ) assert keylog_file.read_text().startswith( "# TLS secrets log file" ), "keylogfile '%s' should start with '# TLS secrets log file'" % str( keylog_file ) @pytest.mark.parametrize("sslkeylogfile", [None, ""]) def test_sslkeylogfile_empty( self, monkeypatch: pytest.MonkeyPatch, sslkeylogfile: str | None ) -> None: # Assert that an HTTPS connection doesn't error out when given # no SSLKEYLOGFILE or an empty value (ie 'SSLKEYLOGFILE=') if sslkeylogfile is not None: monkeypatch.setenv("SSLKEYLOGFILE", sslkeylogfile) else: monkeypatch.delenv("SSLKEYLOGFILE", raising=False) with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as pool: r = pool.request("GET", "/") assert r.status == 200, r.data def test_alpn_default(self) -> None: """Default ALPN protocols are sent by default.""" if not has_alpn() or not has_alpn(ssl.SSLContext): pytest.skip("ALPN-support not available") with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, ssl_minimum_version=self.tls_version(), ) as pool: r = pool.request("GET", "/alpn_protocol", retries=0) assert r.status == 200 assert r.data.decode("utf-8") == util.ALPN_PROTOCOLS[0] def test_default_ssl_context_ssl_min_max_versions(self) -> None: ctx = urllib3.util.ssl_.create_urllib3_context() assert ctx.minimum_version == ssl.TLSVersion.TLSv1_2 # urllib3 sets a default maximum version only when it is # injected with PyOpenSSL- or SecureTransport-backed # SSL-support. # Otherwise, the default maximum version is set by Python's # `ssl.SSLContext`. The value respects OpenSSL configuration and # can be different from `ssl.TLSVersion.MAXIMUM_SUPPORTED`. # https://github.com/urllib3/urllib3/issues/2477#issuecomment-1151452150 if util.IS_PYOPENSSL or util.IS_SECURETRANSPORT: expected_maximum_version = ssl.TLSVersion.MAXIMUM_SUPPORTED else: expected_maximum_version = ssl.SSLContext( ssl.PROTOCOL_TLS_CLIENT ).maximum_version assert ctx.maximum_version == expected_maximum_version def test_ssl_context_ssl_version_uses_ssl_min_max_versions(self) -> None: with pytest.warns( DeprecationWarning, match=r"'ssl_version' option is deprecated and will be removed in " r"urllib3 v2\.1\.0\. Instead use 'ssl_minimum_version'", ): ctx = urllib3.util.ssl_.create_urllib3_context( ssl_version=self.ssl_version() ) assert ctx.minimum_version == self.tls_version() assert ctx.maximum_version == self.tls_version() @pytest.mark.usefixtures("requires_tlsv1") class TestHTTPS_TLSv1(TestHTTPS): tls_protocol_name = "TLSv1" certs = TLSv1_CERTS @pytest.mark.usefixtures("requires_tlsv1_1") class TestHTTPS_TLSv1_1(TestHTTPS): tls_protocol_name = "TLSv1.1" certs = TLSv1_1_CERTS @pytest.mark.usefixtures("requires_tlsv1_2") class TestHTTPS_TLSv1_2(TestHTTPS): tls_protocol_name = "TLSv1.2" certs = TLSv1_2_CERTS @pytest.mark.usefixtures("requires_tlsv1_3") class TestHTTPS_TLSv1_3(TestHTTPS): tls_protocol_name = "TLSv1.3" certs = TLSv1_3_CERTS class TestHTTPS_Hostname: def test_can_validate_san(self, san_server: ServerConfig) -> None: """Ensure that urllib3 can validate SANs with IP addresses in them.""" with HTTPSConnectionPool( san_server.host, san_server.port, cert_reqs="CERT_REQUIRED", ca_certs=san_server.ca_certs, ) as https_pool: r = https_pool.request("GET", "/") assert r.status == 200 def test_common_name_without_san_fails(self, no_san_server: ServerConfig) -> None: with HTTPSConnectionPool( no_san_server.host, no_san_server.port, cert_reqs="CERT_REQUIRED", ca_certs=no_san_server.ca_certs, ) as https_pool: with pytest.raises( MaxRetryError, ) as e: https_pool.request("GET", "/") assert "mismatch, certificate is not valid" in str( e.value ) or "no appropriate subjectAltName" in str(e.value) def test_common_name_without_san_with_different_common_name( self, no_san_server_with_different_commmon_name: ServerConfig ) -> None: ctx = urllib3.util.ssl_.create_urllib3_context() try: ctx.hostname_checks_common_name = True except AttributeError: pytest.skip("Couldn't set 'SSLContext.hostname_checks_common_name'") with HTTPSConnectionPool( no_san_server_with_different_commmon_name.host, no_san_server_with_different_commmon_name.port, cert_reqs="CERT_REQUIRED", ca_certs=no_san_server_with_different_commmon_name.ca_certs, ssl_context=ctx, ) as https_pool: with pytest.raises(MaxRetryError) as e: https_pool.request("GET", "/") assert "mismatch, certificate is not valid for 'localhost'" in str( e.value ) or "hostname 'localhost' doesn't match 'example.com'" in str(e.value) @pytest.mark.parametrize("use_assert_hostname", [True, False]) def test_hostname_checks_common_name_respected( self, no_san_server: ServerConfig, use_assert_hostname: bool ) -> None: ctx = urllib3.util.ssl_.create_urllib3_context() if not hasattr(ctx, "hostname_checks_common_name"): pytest.skip("Test requires 'SSLContext.hostname_checks_common_name'") ctx.load_verify_locations(no_san_server.ca_certs) try: ctx.hostname_checks_common_name = True except AttributeError: pytest.skip("Couldn't set 'SSLContext.hostname_checks_common_name'") err: MaxRetryError | None try: with HTTPSConnectionPool( no_san_server.host, no_san_server.port, cert_reqs="CERT_REQUIRED", ssl_context=ctx, assert_hostname=no_san_server.host if use_assert_hostname else None, ) as https_pool: https_pool.request("GET", "/") except MaxRetryError as e: err = e else: err = None # commonName is only valid for DNS names, not IP addresses. if no_san_server.host == "localhost": assert err is None # IP addresses should fail for commonName. else: assert err is not None assert type(err.reason) == SSLError assert isinstance( err.reason.args[0], (ssl.SSLCertVerificationError, CertificateError) ) def test_assert_hostname_invalid_san( self, no_localhost_san_server: ServerConfig ) -> None: """Ensure SAN errors are not raised while assert_hostname is false""" with HTTPSConnectionPool( no_localhost_san_server.host, no_localhost_san_server.port, cert_reqs="CERT_REQUIRED", ca_certs=no_localhost_san_server.ca_certs, assert_hostname=False, ) as https_pool: https_pool.request("GET", "/") def test_assert_hostname_invalid_cn( self, no_san_server_with_different_commmon_name: ServerConfig ) -> None: """Ensure CN errors are not raised while assert_hostname is false""" with HTTPSConnectionPool( no_san_server_with_different_commmon_name.host, no_san_server_with_different_commmon_name.port, cert_reqs="CERT_REQUIRED", ca_certs=no_san_server_with_different_commmon_name.ca_certs, assert_hostname=False, ) as https_pool: https_pool.request("GET", "/") class TestHTTPS_IPV4SAN: def test_can_validate_ip_san(self, ipv4_san_server: ServerConfig) -> None: """Ensure that urllib3 can validate SANs with IP addresses in them.""" with HTTPSConnectionPool( ipv4_san_server.host, ipv4_san_server.port, cert_reqs="CERT_REQUIRED", ca_certs=ipv4_san_server.ca_certs, ) as https_pool: r = https_pool.request("GET", "/") assert r.status == 200 class TestHTTPS_IPV6SAN: @pytest.mark.parametrize("host", ["::1", "[::1]"]) def test_can_validate_ipv6_san( self, ipv6_san_server: ServerConfig, host: str ) -> None: """Ensure that urllib3 can validate SANs with IPv6 addresses in them.""" with HTTPSConnectionPool( host, ipv6_san_server.port, cert_reqs="CERT_REQUIRED", ca_certs=ipv6_san_server.ca_certs, ) as https_pool: r = https_pool.request("GET", "/") assert r.status == 200 urllib3-2.0.7/test/with_dummyserver/test_no_ssl.py0000644000000000000000000000212014513541732017324 0ustar00""" Test connections without the builtin ssl module Note: Import urllib3 inside the test functions to get the importblocker to work """ from __future__ import annotations import pytest import urllib3 from dummyserver.testcase import HTTPDummyServerTestCase, HTTPSDummyServerTestCase from urllib3.exceptions import InsecureRequestWarning from ..test_no_ssl import TestWithoutSSL class TestHTTPWithoutSSL(HTTPDummyServerTestCase, TestWithoutSSL): def test_simple(self) -> None: with urllib3.HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/") assert r.status == 200, r.data class TestHTTPSWithoutSSL(HTTPSDummyServerTestCase, TestWithoutSSL): def test_simple(self) -> None: with urllib3.HTTPSConnectionPool( self.host, self.port, cert_reqs="NONE" ) as pool: with pytest.warns(InsecureRequestWarning): try: pool.request("GET", "/") except urllib3.exceptions.SSLError as e: assert "SSL module is not available" in str(e) urllib3-2.0.7/test/with_dummyserver/test_poolmanager.py0000644000000000000000000005700014513541732020342 0ustar00from __future__ import annotations import gzip import typing from test import LONG_TIMEOUT from unittest import mock import pytest from dummyserver.server import HAS_IPV6 from dummyserver.testcase import HTTPDummyServerTestCase, IPv6HTTPDummyServerTestCase from urllib3 import HTTPHeaderDict, HTTPResponse, request from urllib3.connectionpool import port_by_scheme from urllib3.exceptions import MaxRetryError, URLSchemeUnknown from urllib3.poolmanager import PoolManager from urllib3.util.retry import Retry class TestPoolManager(HTTPDummyServerTestCase): @classmethod def setup_class(cls) -> None: super().setup_class() cls.base_url = f"http://{cls.host}:{cls.port}" cls.base_url_alt = f"http://{cls.host_alt}:{cls.port}" def test_redirect(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/"}, redirect=False, ) assert r.status == 303 r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/"}, ) assert r.status == 200 assert r.data == b"Dummy server!" def test_redirect_twice(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/redirect"}, redirect=False, ) assert r.status == 303 r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/redirect?target={self.base_url}/"}, ) assert r.status == 200 assert r.data == b"Dummy server!" def test_redirect_to_relative_url(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": "/redirect"}, redirect=False, ) assert r.status == 303 r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": "/redirect"} ) assert r.status == 200 assert r.data == b"Dummy server!" def test_cross_host_redirect(self) -> None: with PoolManager() as http: cross_host_location = f"{self.base_url_alt}/echo?a=b" with pytest.raises(MaxRetryError): http.request( "GET", f"{self.base_url}/redirect", fields={"target": cross_host_location}, timeout=LONG_TIMEOUT, retries=0, ) r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url_alt}/echo?a=b"}, timeout=LONG_TIMEOUT, retries=1, ) assert isinstance(r, HTTPResponse) assert r._pool is not None assert r._pool.host == self.host_alt def test_too_many_redirects(self) -> None: with PoolManager() as http: with pytest.raises(MaxRetryError): http.request( "GET", f"{self.base_url}/redirect", fields={ "target": f"{self.base_url}/redirect?target={self.base_url}/" }, retries=1, preload_content=False, ) with pytest.raises(MaxRetryError): http.request( "GET", f"{self.base_url}/redirect", fields={ "target": f"{self.base_url}/redirect?target={self.base_url}/" }, retries=Retry(total=None, redirect=1), preload_content=False, ) # Even with preload_content=False and raise on redirects, we reused the same # connection assert len(http.pools) == 1 pool = http.connection_from_host(self.host, self.port) assert pool.num_connections == 1 def test_redirect_cross_host_remove_headers(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url_alt}/headers"}, headers={"Authorization": "foo", "Cookie": "foo=bar"}, ) assert r.status == 200 data = r.json() assert "Authorization" not in data assert "Cookie" not in data r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url_alt}/headers"}, headers={"authorization": "foo", "cookie": "foo=bar"}, ) assert r.status == 200 data = r.json() assert "authorization" not in data assert "Authorization" not in data assert "cookie" not in data assert "Cookie" not in data def test_redirect_cross_host_no_remove_headers(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url_alt}/headers"}, headers={"Authorization": "foo", "Cookie": "foo=bar"}, retries=Retry(remove_headers_on_redirect=[]), ) assert r.status == 200 data = r.json() assert data["Authorization"] == "foo" assert data["Cookie"] == "foo=bar" def test_redirect_cross_host_set_removed_headers(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url_alt}/headers"}, headers={ "X-API-Secret": "foo", "Authorization": "bar", "Cookie": "foo=bar", }, retries=Retry(remove_headers_on_redirect=["X-API-Secret"]), ) assert r.status == 200 data = r.json() assert "X-API-Secret" not in data assert data["Authorization"] == "bar" assert data["Cookie"] == "foo=bar" headers = { "x-api-secret": "foo", "authorization": "bar", "cookie": "foo=bar", } r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url_alt}/headers"}, headers=headers, retries=Retry(remove_headers_on_redirect=["X-API-Secret"]), ) assert r.status == 200 data = r.json() assert "x-api-secret" not in data assert "X-API-Secret" not in data assert data["Authorization"] == "bar" assert data["Cookie"] == "foo=bar" # Ensure the header argument itself is not modified in-place. assert headers == { "x-api-secret": "foo", "authorization": "bar", "cookie": "foo=bar", } def test_redirect_without_preload_releases_connection(self) -> None: with PoolManager(block=True, maxsize=2) as http: r = http.request("GET", f"{self.base_url}/redirect", preload_content=False) assert isinstance(r, HTTPResponse) assert r._pool is not None assert r._pool.num_requests == 2 assert r._pool.num_connections == 1 assert len(http.pools) == 1 def test_303_redirect_makes_request_lose_body(self) -> None: with PoolManager() as http: response = http.request( "POST", f"{self.base_url}/redirect", fields={ "target": f"{self.base_url}/headers_and_params", "status": "303 See Other", }, ) data = response.json() assert data["params"] == {} assert "Content-Type" not in HTTPHeaderDict(data["headers"]) def test_unknown_scheme(self) -> None: with PoolManager() as http: unknown_scheme = "unknown" unknown_scheme_url = f"{unknown_scheme}://host" with pytest.raises(URLSchemeUnknown) as e: r = http.request("GET", unknown_scheme_url) assert e.value.scheme == unknown_scheme r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": unknown_scheme_url}, redirect=False, ) assert r.status == 303 assert r.headers.get("Location") == unknown_scheme_url with pytest.raises(URLSchemeUnknown) as e: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": unknown_scheme_url}, ) assert e.value.scheme == unknown_scheme def test_raise_on_redirect(self) -> None: with PoolManager() as http: r = http.request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/redirect?target={self.base_url}/"}, retries=Retry(total=None, redirect=1, raise_on_redirect=False), ) assert r.status == 303 def test_raise_on_status(self) -> None: with PoolManager() as http: with pytest.raises(MaxRetryError): # the default is to raise r = http.request( "GET", f"{self.base_url}/status", fields={"status": "500 Internal Server Error"}, retries=Retry(total=1, status_forcelist=range(500, 600)), ) with pytest.raises(MaxRetryError): # raise explicitly r = http.request( "GET", f"{self.base_url}/status", fields={"status": "500 Internal Server Error"}, retries=Retry( total=1, status_forcelist=range(500, 600), raise_on_status=True ), ) # don't raise r = http.request( "GET", f"{self.base_url}/status", fields={"status": "500 Internal Server Error"}, retries=Retry( total=1, status_forcelist=range(500, 600), raise_on_status=False ), ) assert r.status == 500 def test_missing_port(self) -> None: # Can a URL that lacks an explicit port like ':80' succeed, or # will all such URLs fail with an error? with PoolManager() as http: # By globally adjusting `port_by_scheme` we pretend for a moment # that HTTP's default port is not 80, but is the port at which # our test server happens to be listening. port_by_scheme["http"] = self.port try: r = http.request("GET", f"http://{self.host}/", retries=0) finally: port_by_scheme["http"] = 80 assert r.status == 200 assert r.data == b"Dummy server!" def test_headers(self) -> None: with PoolManager(headers={"Foo": "bar"}) as http: r = http.request("GET", f"{self.base_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" r = http.request("POST", f"{self.base_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" r = http.request_encode_url("GET", f"{self.base_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" r = http.request_encode_body("POST", f"{self.base_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" r = http.request_encode_url( "GET", f"{self.base_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" r = http.request_encode_body( "GET", f"{self.base_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" def test_headers_http_header_dict(self) -> None: # Test uses a list of headers to assert the order # that headers are sent in the request too. headers = HTTPHeaderDict() headers.add("Foo", "bar") headers.add("Multi", "1") headers.add("Baz", "quux") headers.add("Multi", "2") with PoolManager(headers=headers) as http: r = http.request("GET", f"{self.base_url}/multi_headers") returned_headers = r.json()["headers"] assert returned_headers[-4:] == [ ["Foo", "bar"], ["Multi", "1"], ["Multi", "2"], ["Baz", "quux"], ] r = http.request( "GET", f"{self.base_url}/multi_headers", headers={ **headers, "Extra": "extra", "Foo": "new", }, ) returned_headers = r.json()["headers"] assert returned_headers[-4:] == [ ["Foo", "new"], ["Multi", "1, 2"], ["Baz", "quux"], ["Extra", "extra"], ] def test_merge_headers_with_pool_manager_headers(self) -> None: headers = HTTPHeaderDict() headers.add("Cookie", "choc-chip") headers.add("Cookie", "oatmeal-raisin") orig = headers.copy() added_headers = {"Cookie": "tim-tam"} with PoolManager(headers=headers) as http: r = http.request( "GET", f"{self.base_url}/multi_headers", headers=typing.cast(HTTPHeaderDict, http.headers) | added_headers, ) returned_headers = r.json()["headers"] assert returned_headers[-3:] == [ ["Cookie", "choc-chip"], ["Cookie", "oatmeal-raisin"], ["Cookie", "tim-tam"], ] # make sure the pool headers weren't modified assert http.headers == orig def test_headers_http_multi_header_multipart(self) -> None: headers = HTTPHeaderDict() headers.add("Multi", "1") headers.add("Multi", "2") old_headers = headers.copy() with PoolManager(headers=headers) as http: r = http.request( "POST", f"{self.base_url}/multi_headers", fields={"k": "v"}, multipart_boundary="b", encode_multipart=True, ) returned_headers = r.json()["headers"] assert returned_headers[4:] == [ ["Multi", "1"], ["Multi", "2"], ["Content-Type", "multipart/form-data; boundary=b"], ] # Assert that the previous headers weren't modified. assert headers == old_headers # Set a default value for the Content-Type headers["Content-Type"] = "multipart/form-data; boundary=b; field=value" r = http.request( "POST", f"{self.base_url}/multi_headers", fields={"k": "v"}, multipart_boundary="b", encode_multipart=True, ) returned_headers = r.json()["headers"] assert returned_headers[4:] == [ ["Multi", "1"], ["Multi", "2"], # Uses the set value, not the one that would be generated. ["Content-Type", "multipart/form-data; boundary=b; field=value"], ] def test_body(self) -> None: with PoolManager() as http: r = http.request("POST", f"{self.base_url}/echo", body=b"test") assert r.data == b"test" def test_http_with_ssl_keywords(self) -> None: with PoolManager(ca_certs="REQUIRED") as http: r = http.request("GET", f"http://{self.host}:{self.port}/") assert r.status == 200 def test_http_with_server_hostname(self) -> None: with PoolManager(server_hostname="example.com") as http: r = http.request("GET", f"http://{self.host}:{self.port}/") assert r.status == 200 def test_http_with_ca_cert_dir(self) -> None: with PoolManager(ca_certs="REQUIRED", ca_cert_dir="/nosuchdir") as http: r = http.request("GET", f"http://{self.host}:{self.port}/") assert r.status == 200 @pytest.mark.parametrize( ["target", "expected_target"], [ ("/echo_uri?q=1#fragment", b"/echo_uri?q=1"), ("/echo_uri?#", b"/echo_uri?"), ("/echo_uri#?", b"/echo_uri"), ("/echo_uri#?#", b"/echo_uri"), ("/echo_uri??#", b"/echo_uri??"), ("/echo_uri?%3f#", b"/echo_uri?%3F"), ("/echo_uri?%3F#", b"/echo_uri?%3F"), ("/echo_uri?[]", b"/echo_uri?%5B%5D"), ], ) def test_encode_http_target(self, target: str, expected_target: bytes) -> None: with PoolManager() as http: url = f"http://{self.host}:{self.port}{target}" r = http.request("GET", url) assert r.data == expected_target def test_top_level_request(self) -> None: r = request("GET", f"{self.base_url}/") assert r.status == 200 assert r.data == b"Dummy server!" def test_top_level_request_without_keyword_args(self) -> None: body = "" with pytest.raises(TypeError): request("GET", f"{self.base_url}/", body) # type: ignore[misc] def test_top_level_request_with_body(self) -> None: r = request("POST", f"{self.base_url}/echo", body=b"test") assert r.status == 200 assert r.data == b"test" def test_top_level_request_with_preload_content(self) -> None: r = request("GET", f"{self.base_url}/echo", preload_content=False) assert r.status == 200 assert r.connection is not None r.data assert r.connection is None def test_top_level_request_with_decode_content(self) -> None: r = request( "GET", f"{self.base_url}/encodingrequest", headers={"accept-encoding": "gzip"}, decode_content=False, ) assert r.status == 200 assert gzip.decompress(r.data) == b"hello, world!" r = request( "GET", f"{self.base_url}/encodingrequest", headers={"accept-encoding": "gzip"}, decode_content=True, ) assert r.status == 200 assert r.data == b"hello, world!" def test_top_level_request_with_redirect(self) -> None: r = request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/"}, redirect=False, ) assert r.status == 303 r = request( "GET", f"{self.base_url}/redirect", fields={"target": f"{self.base_url}/"}, redirect=True, ) assert r.status == 200 assert r.data == b"Dummy server!" def test_top_level_request_with_retries(self) -> None: r = request("GET", f"{self.base_url}/redirect", retries=False) assert r.status == 303 r = request("GET", f"{self.base_url}/redirect", retries=3) assert r.status == 200 def test_top_level_request_with_timeout(self) -> None: with mock.patch("urllib3.poolmanager.RequestMethods.request") as mockRequest: mockRequest.return_value = HTTPResponse(status=200) r = request("GET", f"{self.base_url}/redirect", timeout=2.5) assert r.status == 200 mockRequest.assert_called_with( "GET", f"{self.base_url}/redirect", body=None, fields=None, headers=None, preload_content=True, decode_content=True, redirect=True, retries=None, timeout=2.5, json=None, ) @pytest.mark.parametrize( "headers", [ None, {"content-Type": "application/json"}, {"content-Type": "text/plain"}, {"attribute": "value", "CONTENT-TYPE": "application/json"}, HTTPHeaderDict(cookie="foo, bar"), ], ) def test_request_with_json(self, headers: HTTPHeaderDict) -> None: body = {"attribute": "value"} r = request( method="POST", url=f"{self.base_url}/echo_json", headers=headers, json=body ) assert r.status == 200 assert r.json() == body if headers is not None and "application/json" not in headers.values(): assert "text/plain" in r.headers["Content-Type"].replace(" ", "").split(",") else: assert "application/json" in r.headers["Content-Type"].replace( " ", "" ).split(",") def test_top_level_request_with_json_with_httpheaderdict(self) -> None: body = {"attribute": "value"} header = HTTPHeaderDict(cookie="foo, bar") with PoolManager(headers=header) as http: r = http.request(method="POST", url=f"{self.base_url}/echo_json", json=body) assert r.status == 200 assert r.json() == body assert "application/json" in r.headers["Content-Type"].replace( " ", "" ).split(",") def test_top_level_request_with_body_and_json(self) -> None: match = "request got values for both 'body' and 'json' parameters which are mutually exclusive" with pytest.raises(TypeError, match=match): body = {"attribute": "value"} request(method="POST", url=f"{self.base_url}/echo", body="", json=body) def test_top_level_request_with_invalid_body(self) -> None: class BadBody: def __repr__(self) -> str: return "" with pytest.raises(TypeError) as e: request( method="POST", url=f"{self.base_url}/echo", body=BadBody(), # type: ignore[arg-type] ) assert str(e.value) == ( "'body' must be a bytes-like object, file-like " "object, or iterable. Instead was " ) @pytest.mark.skipif(not HAS_IPV6, reason="IPv6 is not supported on this system") class TestIPv6PoolManager(IPv6HTTPDummyServerTestCase): @classmethod def setup_class(cls) -> None: super().setup_class() cls.base_url = f"http://[{cls.host}]:{cls.port}" def test_ipv6(self) -> None: with PoolManager() as http: http.request("GET", self.base_url) urllib3-2.0.7/test/with_dummyserver/test_proxy_poolmanager.py0000644000000000000000000010132114513541732021577 0ustar00from __future__ import annotations import binascii import hashlib import ipaddress import os.path import pathlib import shutil import socket import ssl import tempfile from test import LONG_TIMEOUT, SHORT_TIMEOUT, onlySecureTransport, withPyOpenSSL from test.conftest import ServerConfig import pytest import trustme import urllib3.exceptions from dummyserver.server import DEFAULT_CA, HAS_IPV6, get_unreachable_address from dummyserver.testcase import HTTPDummyProxyTestCase, IPv6HTTPDummyProxyTestCase from urllib3 import HTTPResponse from urllib3._collections import HTTPHeaderDict from urllib3.connection import VerifiedHTTPSConnection from urllib3.connectionpool import connection_from_url from urllib3.exceptions import ( ConnectTimeoutError, InsecureRequestWarning, MaxRetryError, ProxyError, ProxySchemeUnknown, ProxySchemeUnsupported, ReadTimeoutError, SSLError, ) from urllib3.poolmanager import ProxyManager, proxy_from_url from urllib3.util.ssl_ import create_urllib3_context from urllib3.util.timeout import Timeout from .. import TARPIT_HOST, requires_network class TestHTTPProxyManager(HTTPDummyProxyTestCase): @classmethod def setup_class(cls) -> None: super().setup_class() cls.http_url = f"http://{cls.http_host}:{int(cls.http_port)}" cls.http_url_alt = f"http://{cls.http_host_alt}:{int(cls.http_port)}" cls.https_url = f"https://{cls.https_host}:{int(cls.https_port)}" cls.https_url_alt = f"https://{cls.https_host_alt}:{int(cls.https_port)}" cls.proxy_url = f"http://{cls.proxy_host}:{int(cls.proxy_port)}" cls.https_proxy_url = f"https://{cls.proxy_host}:{int(cls.https_proxy_port)}" # Generate another CA to test verification failure cls.certs_dir = tempfile.mkdtemp() bad_ca = trustme.CA() cls.bad_ca_path = os.path.join(cls.certs_dir, "ca_bad.pem") bad_ca.cert_pem.write_to_path(cls.bad_ca_path) @classmethod def teardown_class(cls) -> None: super().teardown_class() shutil.rmtree(cls.certs_dir) def test_basic_proxy(self) -> None: with proxy_from_url(self.proxy_url, ca_certs=DEFAULT_CA) as http: r = http.request("GET", f"{self.http_url}/") assert r.status == 200 r = http.request("GET", f"{self.https_url}/") assert r.status == 200 def test_https_proxy(self) -> None: with proxy_from_url(self.https_proxy_url, ca_certs=DEFAULT_CA) as https: r = https.request("GET", f"{self.https_url}/") assert r.status == 200 r = https.request("GET", f"{self.http_url}/") assert r.status == 200 def test_https_proxy_with_proxy_ssl_context(self) -> None: proxy_ssl_context = create_urllib3_context() proxy_ssl_context.load_verify_locations(DEFAULT_CA) with proxy_from_url( self.https_proxy_url, proxy_ssl_context=proxy_ssl_context, ca_certs=DEFAULT_CA, ) as https: r = https.request("GET", f"{self.https_url}/") assert r.status == 200 r = https.request("GET", f"{self.http_url}/") assert r.status == 200 @withPyOpenSSL def test_https_proxy_pyopenssl_not_supported(self) -> None: with proxy_from_url(self.https_proxy_url, ca_certs=DEFAULT_CA) as https: r = https.request("GET", f"{self.http_url}/") assert r.status == 200 with pytest.raises( ProxySchemeUnsupported, match="isn't available on non-native SSLContext" ): https.request("GET", f"{self.https_url}/") @onlySecureTransport() def test_https_proxy_securetransport_not_supported(self) -> None: with proxy_from_url(self.https_proxy_url, ca_certs=DEFAULT_CA) as https: r = https.request("GET", f"{self.http_url}/") assert r.status == 200 with pytest.raises( ProxySchemeUnsupported, match="isn't available on non-native SSLContext" ): https.request("GET", f"{self.https_url}/") def test_https_proxy_forwarding_for_https(self) -> None: with proxy_from_url( self.https_proxy_url, ca_certs=DEFAULT_CA, use_forwarding_for_https=True, ) as https: r = https.request("GET", f"{self.http_url}/") assert r.status == 200 r = https.request("GET", f"{self.https_url}/") assert r.status == 200 def test_nagle_proxy(self) -> None: """Test that proxy connections do not have TCP_NODELAY turned on""" with ProxyManager(self.proxy_url) as http: hc2 = http.connection_from_host(self.http_host, self.http_port) conn = hc2._get_conn() try: hc2._make_request(conn, "GET", "/") tcp_nodelay_setting = conn.sock.getsockopt( # type: ignore[attr-defined] socket.IPPROTO_TCP, socket.TCP_NODELAY ) assert tcp_nodelay_setting == 0, ( "Expected TCP_NODELAY for proxies to be set " "to zero, instead was %s" % tcp_nodelay_setting ) finally: conn.close() @pytest.mark.parametrize("proxy_scheme", ["http", "https"]) @pytest.mark.parametrize("target_scheme", ["http", "https"]) def test_proxy_conn_fail_from_dns( self, proxy_scheme: str, target_scheme: str ) -> None: host, port = get_unreachable_address() with proxy_from_url( f"{proxy_scheme}://{host}:{port}/", retries=1, timeout=LONG_TIMEOUT ) as http: if target_scheme == "https": target_url = self.https_url else: target_url = self.http_url with pytest.raises(MaxRetryError) as e: http.request("GET", f"{target_url}/") assert type(e.value.reason) == ProxyError assert ( type(e.value.reason.original_error) == urllib3.exceptions.NameResolutionError ) def test_oldapi(self) -> None: with ProxyManager( connection_from_url(self.proxy_url), ca_certs=DEFAULT_CA # type: ignore[arg-type] ) as http: r = http.request("GET", f"{self.http_url}/") assert r.status == 200 r = http.request("GET", f"{self.https_url}/") assert r.status == 200 def test_proxy_verified(self) -> None: with proxy_from_url( self.proxy_url, cert_reqs="REQUIRED", ca_certs=self.bad_ca_path ) as http: https_pool = http._new_pool("https", self.https_host, self.https_port) with pytest.raises(MaxRetryError) as e: https_pool.request("GET", "/", retries=0) assert isinstance(e.value.reason, SSLError) assert ( "certificate verify failed" in str(e.value.reason) # PyPy is more specific or "self signed certificate in certificate chain" in str(e.value.reason) ), f"Expected 'certificate verify failed', instead got: {e.value.reason!r}" http = proxy_from_url( self.proxy_url, cert_reqs="REQUIRED", ca_certs=DEFAULT_CA ) https_pool = http._new_pool("https", self.https_host, self.https_port) conn = https_pool._new_conn() assert conn.__class__ == VerifiedHTTPSConnection https_pool.request("GET", "/") # Should succeed without exceptions. http = proxy_from_url( self.proxy_url, cert_reqs="REQUIRED", ca_certs=DEFAULT_CA ) https_fail_pool = http._new_pool("https", "127.0.0.1", self.https_port) with pytest.raises( MaxRetryError, match="doesn't match|IP address mismatch" ) as e: https_fail_pool.request("GET", "/", retries=0) assert isinstance(e.value.reason, SSLError) def test_redirect(self) -> None: with proxy_from_url(self.proxy_url) as http: r = http.request( "GET", f"{self.http_url}/redirect", fields={"target": f"{self.http_url}/"}, redirect=False, ) assert r.status == 303 r = http.request( "GET", f"{self.http_url}/redirect", fields={"target": f"{self.http_url}/"}, ) assert r.status == 200 assert r.data == b"Dummy server!" def test_cross_host_redirect(self) -> None: with proxy_from_url(self.proxy_url) as http: cross_host_location = f"{self.http_url_alt}/echo?a=b" with pytest.raises(MaxRetryError): http.request( "GET", f"{self.http_url}/redirect", fields={"target": cross_host_location}, retries=0, ) r = http.request( "GET", f"{self.http_url}/redirect", fields={"target": f"{self.http_url_alt}/echo?a=b"}, retries=1, ) assert isinstance(r, HTTPResponse) assert r._pool is not None assert r._pool.host != self.http_host_alt def test_cross_protocol_redirect(self) -> None: with proxy_from_url(self.proxy_url, ca_certs=DEFAULT_CA) as http: cross_protocol_location = f"{self.https_url}/echo?a=b" with pytest.raises(MaxRetryError): http.request( "GET", f"{self.http_url}/redirect", fields={"target": cross_protocol_location}, retries=0, ) r = http.request( "GET", f"{self.http_url}/redirect", fields={"target": f"{self.https_url}/echo?a=b"}, retries=1, ) assert isinstance(r, HTTPResponse) assert r._pool is not None assert r._pool.host == self.https_host def test_headers(self) -> None: with proxy_from_url( self.proxy_url, headers={"Foo": "bar"}, proxy_headers={"Hickory": "dickory"}, ca_certs=DEFAULT_CA, ) as http: r = http.request_encode_url("GET", f"{self.http_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") == "dickory" assert returned_headers.get("Host") == f"{self.http_host}:{self.http_port}" r = http.request_encode_url("GET", f"{self.http_url_alt}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") == "dickory" assert ( returned_headers.get("Host") == f"{self.http_host_alt}:{self.http_port}" ) r = http.request_encode_url("GET", f"{self.https_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") is None assert ( returned_headers.get("Host") == f"{self.https_host}:{self.https_port}" ) r = http.request_encode_body("POST", f"{self.http_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") == "dickory" assert returned_headers.get("Host") == f"{self.http_host}:{self.http_port}" r = http.request_encode_url( "GET", f"{self.http_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" assert returned_headers.get("Hickory") == "dickory" assert returned_headers.get("Host") == f"{self.http_host}:{self.http_port}" r = http.request_encode_url( "GET", f"{self.https_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" assert returned_headers.get("Hickory") is None assert ( returned_headers.get("Host") == f"{self.https_host}:{self.https_port}" ) r = http.request_encode_body( "GET", f"{self.http_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" assert returned_headers.get("Hickory") == "dickory" assert returned_headers.get("Host") == f"{self.http_host}:{self.http_port}" r = http.request_encode_body( "GET", f"{self.https_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" assert returned_headers.get("Hickory") is None assert ( returned_headers.get("Host") == f"{self.https_host}:{self.https_port}" ) def test_https_headers(self) -> None: with proxy_from_url( self.https_proxy_url, headers={"Foo": "bar"}, proxy_headers={"Hickory": "dickory"}, ca_certs=DEFAULT_CA, ) as http: r = http.request_encode_url("GET", f"{self.http_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") == "dickory" assert returned_headers.get("Host") == f"{self.http_host}:{self.http_port}" r = http.request_encode_url("GET", f"{self.http_url_alt}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") == "dickory" assert ( returned_headers.get("Host") == f"{self.http_host_alt}:{self.http_port}" ) r = http.request_encode_body( "GET", f"{self.https_url}/headers", headers={"Baz": "quux"} ) returned_headers = r.json() assert returned_headers.get("Foo") is None assert returned_headers.get("Baz") == "quux" assert returned_headers.get("Hickory") is None assert ( returned_headers.get("Host") == f"{self.https_host}:{self.https_port}" ) def test_https_headers_forwarding_for_https(self) -> None: with proxy_from_url( self.https_proxy_url, headers={"Foo": "bar"}, proxy_headers={"Hickory": "dickory"}, ca_certs=DEFAULT_CA, use_forwarding_for_https=True, ) as http: r = http.request_encode_url("GET", f"{self.https_url}/headers") returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Hickory") == "dickory" assert ( returned_headers.get("Host") == f"{self.https_host}:{self.https_port}" ) def test_headerdict(self) -> None: default_headers = HTTPHeaderDict(a="b") proxy_headers = HTTPHeaderDict() proxy_headers.add("foo", "bar") with proxy_from_url( self.proxy_url, headers=default_headers, proxy_headers=proxy_headers ) as http: request_headers = HTTPHeaderDict(baz="quux") r = http.request("GET", f"{self.http_url}/headers", headers=request_headers) returned_headers = r.json() assert returned_headers.get("Foo") == "bar" assert returned_headers.get("Baz") == "quux" def test_proxy_pooling(self) -> None: with proxy_from_url(self.proxy_url, cert_reqs="NONE") as http: for x in range(2): http.urlopen("GET", self.http_url) assert len(http.pools) == 1 for x in range(2): http.urlopen("GET", self.http_url_alt) assert len(http.pools) == 1 for x in range(2): with pytest.warns(InsecureRequestWarning): http.urlopen("GET", self.https_url) assert len(http.pools) == 2 for x in range(2): with pytest.warns(InsecureRequestWarning): http.urlopen("GET", self.https_url_alt) assert len(http.pools) == 3 def test_proxy_pooling_ext(self) -> None: with proxy_from_url(self.proxy_url) as http: hc1 = http.connection_from_url(self.http_url) hc2 = http.connection_from_host(self.http_host, self.http_port) hc3 = http.connection_from_url(self.http_url_alt) hc4 = http.connection_from_host(self.http_host_alt, self.http_port) assert hc1 == hc2 assert hc2 == hc3 assert hc3 == hc4 sc1 = http.connection_from_url(self.https_url) sc2 = http.connection_from_host( self.https_host, self.https_port, scheme="https" ) sc3 = http.connection_from_url(self.https_url_alt) sc4 = http.connection_from_host( self.https_host_alt, self.https_port, scheme="https" ) assert sc1 == sc2 assert sc2 != sc3 assert sc3 == sc4 @requires_network() @pytest.mark.parametrize( ["proxy_scheme", "target_scheme", "use_forwarding_for_https"], [ ("http", "http", False), ("https", "http", False), # 'use_forwarding_for_https' is only valid for HTTPS+HTTPS. ("https", "https", True), ], ) def test_forwarding_proxy_request_timeout( self, proxy_scheme: str, target_scheme: str, use_forwarding_for_https: bool ) -> None: proxy_url = self.https_proxy_url if proxy_scheme == "https" else self.proxy_url target_url = f"{target_scheme}://{TARPIT_HOST}" with proxy_from_url( proxy_url, ca_certs=DEFAULT_CA, use_forwarding_for_https=use_forwarding_for_https, ) as proxy: with pytest.raises(MaxRetryError) as e: timeout = Timeout(connect=LONG_TIMEOUT, read=SHORT_TIMEOUT) proxy.request("GET", target_url, timeout=timeout) # We sent the request to the proxy but didn't get any response # so we're not sure if that's being caused by the proxy or the # target so we put the blame on the target. assert type(e.value.reason) == ReadTimeoutError @requires_network() @pytest.mark.parametrize( ["proxy_scheme", "target_scheme"], [("http", "https"), ("https", "https")] ) def test_tunneling_proxy_request_timeout( self, proxy_scheme: str, target_scheme: str ) -> None: proxy_url = self.https_proxy_url if proxy_scheme == "https" else self.proxy_url target_url = f"{target_scheme}://{TARPIT_HOST}" with proxy_from_url( proxy_url, ca_certs=DEFAULT_CA, ) as proxy: with pytest.raises(MaxRetryError) as e: timeout = Timeout(connect=LONG_TIMEOUT, read=SHORT_TIMEOUT) proxy.request("GET", target_url, timeout=timeout) assert type(e.value.reason) == ReadTimeoutError @requires_network() @pytest.mark.parametrize( ["proxy_scheme", "target_scheme", "use_forwarding_for_https"], [ ("http", "http", False), ("https", "http", False), # 'use_forwarding_for_https' is only valid for HTTPS+HTTPS. ("https", "https", True), ], ) def test_forwarding_proxy_connect_timeout( self, proxy_scheme: str, target_scheme: str, use_forwarding_for_https: bool ) -> None: proxy_url = f"{proxy_scheme}://{TARPIT_HOST}" target_url = self.https_url if target_scheme == "https" else self.http_url with proxy_from_url( proxy_url, ca_certs=DEFAULT_CA, timeout=SHORT_TIMEOUT, use_forwarding_for_https=use_forwarding_for_https, ) as proxy: with pytest.raises(MaxRetryError) as e: proxy.request("GET", target_url) assert type(e.value.reason) == ProxyError assert type(e.value.reason.original_error) == ConnectTimeoutError @requires_network() @pytest.mark.parametrize( ["proxy_scheme", "target_scheme"], [("http", "https"), ("https", "https")] ) def test_tunneling_proxy_connect_timeout( self, proxy_scheme: str, target_scheme: str ) -> None: proxy_url = f"{proxy_scheme}://{TARPIT_HOST}" target_url = self.https_url if target_scheme == "https" else self.http_url with proxy_from_url( proxy_url, ca_certs=DEFAULT_CA, timeout=SHORT_TIMEOUT ) as proxy: with pytest.raises(MaxRetryError) as e: proxy.request("GET", target_url) assert type(e.value.reason) == ProxyError assert type(e.value.reason.original_error) == ConnectTimeoutError @requires_network() @pytest.mark.parametrize( ["target_scheme", "use_forwarding_for_https"], [ ("http", False), ("https", False), ("https", True), ], ) def test_https_proxy_tls_error( self, target_scheme: str, use_forwarding_for_https: str ) -> None: target_url = self.https_url if target_scheme == "https" else self.http_url proxy_ctx = ssl.create_default_context() with proxy_from_url( self.https_proxy_url, proxy_ssl_context=proxy_ctx, use_forwarding_for_https=use_forwarding_for_https, ) as proxy: with pytest.raises(MaxRetryError) as e: proxy.request("GET", target_url) assert type(e.value.reason) == ProxyError assert type(e.value.reason.original_error) == SSLError @requires_network() @pytest.mark.parametrize( ["proxy_scheme", "use_forwarding_for_https"], [ ("http", False), ("https", False), ("https", True), ], ) def test_proxy_https_target_tls_error( self, proxy_scheme: str, use_forwarding_for_https: str ) -> None: if proxy_scheme == "https" and use_forwarding_for_https: pytest.skip("Test is expected to fail due to urllib3/urllib3#2577") proxy_url = self.https_proxy_url if proxy_scheme == "https" else self.proxy_url proxy_ctx = ssl.create_default_context() proxy_ctx.load_verify_locations(DEFAULT_CA) ctx = ssl.create_default_context() with proxy_from_url( proxy_url, proxy_ssl_context=proxy_ctx, ssl_context=ctx, use_forwarding_for_https=use_forwarding_for_https, ) as proxy: with pytest.raises(MaxRetryError) as e: proxy.request("GET", self.https_url) assert type(e.value.reason) == SSLError def test_scheme_host_case_insensitive(self) -> None: """Assert that upper-case schemes and hosts are normalized.""" with proxy_from_url(self.proxy_url.upper(), ca_certs=DEFAULT_CA) as http: r = http.request("GET", f"{self.http_url.upper()}/") assert r.status == 200 r = http.request("GET", f"{self.https_url.upper()}/") assert r.status == 200 @pytest.mark.parametrize( "url, error_msg", [ ( "127.0.0.1", "Proxy URL had no scheme, should start with http:// or https://", ), ( "localhost:8080", "Proxy URL had no scheme, should start with http:// or https://", ), ( "ftp://google.com", "Proxy URL had unsupported scheme ftp, should use http:// or https://", ), ], ) def test_invalid_schema(self, url: str, error_msg: str) -> None: with pytest.raises(ProxySchemeUnknown, match=error_msg): proxy_from_url(url) @pytest.mark.skipif(not HAS_IPV6, reason="Only runs on IPv6 systems") class TestIPv6HTTPProxyManager(IPv6HTTPDummyProxyTestCase): @classmethod def setup_class(cls) -> None: HTTPDummyProxyTestCase.setup_class() cls.http_url = f"http://{cls.http_host}:{int(cls.http_port)}" cls.http_url_alt = f"http://{cls.http_host_alt}:{int(cls.http_port)}" cls.https_url = f"https://{cls.https_host}:{int(cls.https_port)}" cls.https_url_alt = f"https://{cls.https_host_alt}:{int(cls.https_port)}" cls.proxy_url = f"http://[{cls.proxy_host}]:{int(cls.proxy_port)}" def test_basic_ipv6_proxy(self) -> None: with proxy_from_url(self.proxy_url, ca_certs=DEFAULT_CA) as http: r = http.request("GET", f"{self.http_url}/") assert r.status == 200 r = http.request("GET", f"{self.https_url}/") assert r.status == 200 class TestHTTPSProxyVerification: @staticmethod def _get_proxy_fingerprint_md5(ca_path: str) -> str: proxy_pem_path = pathlib.Path(ca_path).parent / "proxy.pem" proxy_der = ssl.PEM_cert_to_DER_cert(proxy_pem_path.read_text()) proxy_hashed = hashlib.md5(proxy_der).digest() fingerprint = binascii.hexlify(proxy_hashed).decode("ascii") return fingerprint @staticmethod def _get_certificate_formatted_proxy_host(host: str) -> str: try: addr = ipaddress.ip_address(host) except ValueError: return host if addr.version != 6: return host # Transform ipv6 like '::1' to 0:0:0:0:0:0:0:1 via '0000:0000:0000:0000:0000:0000:0000:0001' return addr.exploded.replace("0000", "0").replace("000", "") def test_https_proxy_assert_fingerprint_md5( self, no_san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = no_san_proxy_with_server proxy_url = f"https://{proxy.host}:{proxy.port}" destination_url = f"https://{server.host}:{server.port}" proxy_fingerprint = self._get_proxy_fingerprint_md5(proxy.ca_certs) with proxy_from_url( proxy_url, ca_certs=proxy.ca_certs, proxy_assert_fingerprint=proxy_fingerprint, ) as https: https.request("GET", destination_url) def test_https_proxy_assert_fingerprint_md5_non_matching( self, no_san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = no_san_proxy_with_server proxy_url = f"https://{proxy.host}:{proxy.port}" destination_url = f"https://{server.host}:{server.port}" proxy_fingerprint = self._get_proxy_fingerprint_md5(proxy.ca_certs) new_char = "b" if proxy_fingerprint[5] == "a" else "a" proxy_fingerprint = proxy_fingerprint[:5] + new_char + proxy_fingerprint[6:] with proxy_from_url( proxy_url, ca_certs=proxy.ca_certs, proxy_assert_fingerprint=proxy_fingerprint, ) as https: with pytest.raises(MaxRetryError) as e: https.request("GET", destination_url) assert "Fingerprints did not match" in str(e) def test_https_proxy_assert_hostname( self, san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = san_proxy_with_server destination_url = f"https://{server.host}:{server.port}" with proxy_from_url( proxy.base_url, ca_certs=proxy.ca_certs, proxy_assert_hostname=proxy.host ) as https: https.request("GET", destination_url) def test_https_proxy_assert_hostname_non_matching( self, san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = san_proxy_with_server destination_url = f"https://{server.host}:{server.port}" proxy_hostname = "example.com" with proxy_from_url( proxy.base_url, ca_certs=proxy.ca_certs, proxy_assert_hostname=proxy_hostname, ) as https: with pytest.raises(MaxRetryError) as e: https.request("GET", destination_url) proxy_host = self._get_certificate_formatted_proxy_host(proxy.host) msg = f"hostname \\'{proxy_hostname}\\' doesn\\'t match \\'{proxy_host}\\'" assert msg in str(e) def test_https_proxy_hostname_verification( self, no_localhost_san_server: ServerConfig ) -> None: bad_server = no_localhost_san_server bad_proxy_url = f"https://{bad_server.host}:{bad_server.port}" # An exception will be raised before we contact the destination domain. test_url = "testing.com" with proxy_from_url(bad_proxy_url, ca_certs=bad_server.ca_certs) as https: with pytest.raises(MaxRetryError) as e: https.request("GET", "http://%s/" % test_url) assert isinstance(e.value.reason, ProxyError) ssl_error = e.value.reason.original_error assert isinstance(ssl_error, SSLError) assert "hostname 'localhost' doesn't match" in str( ssl_error ) or "Hostname mismatch" in str(ssl_error) with pytest.raises(MaxRetryError) as e: https.request("GET", "https://%s/" % test_url) assert isinstance(e.value.reason, ProxyError) ssl_error = e.value.reason.original_error assert isinstance(ssl_error, SSLError) assert "hostname 'localhost' doesn't match" in str( ssl_error ) or "Hostname mismatch" in str(ssl_error) def test_https_proxy_ipv4_san( self, ipv4_san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = ipv4_san_proxy_with_server proxy_url = f"https://{proxy.host}:{proxy.port}" destination_url = f"https://{server.host}:{server.port}" with proxy_from_url(proxy_url, ca_certs=proxy.ca_certs) as https: r = https.request("GET", destination_url) assert r.status == 200 def test_https_proxy_ipv6_san( self, ipv6_san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = ipv6_san_proxy_with_server proxy_url = f"https://[{proxy.host}]:{proxy.port}" destination_url = f"https://{server.host}:{server.port}" with proxy_from_url(proxy_url, ca_certs=proxy.ca_certs) as https: r = https.request("GET", destination_url) assert r.status == 200 @pytest.mark.parametrize("target_scheme", ["http", "https"]) def test_https_proxy_no_san( self, no_san_proxy_with_server: tuple[ServerConfig, ServerConfig], target_scheme: str, ) -> None: proxy, server = no_san_proxy_with_server proxy_url = f"https://{proxy.host}:{proxy.port}" destination_url = f"{target_scheme}://{server.host}:{server.port}" with proxy_from_url(proxy_url, ca_certs=proxy.ca_certs) as https: with pytest.raises(MaxRetryError) as e: https.request("GET", destination_url) assert isinstance(e.value.reason, ProxyError) ssl_error = e.value.reason.original_error assert isinstance(ssl_error, SSLError) assert "no appropriate subjectAltName fields were found" in str( ssl_error ) or "Hostname mismatch, certificate is not valid for 'localhost'" in str( ssl_error ) def test_https_proxy_no_san_hostname_checks_common_name( self, no_san_proxy_with_server: tuple[ServerConfig, ServerConfig] ) -> None: proxy, server = no_san_proxy_with_server proxy_url = f"https://{proxy.host}:{proxy.port}" destination_url = f"https://{server.host}:{server.port}" proxy_ctx = urllib3.util.ssl_.create_urllib3_context() try: proxy_ctx.hostname_checks_common_name = True # PyPy doesn't like us setting 'hostname_checks_common_name' # but also has it enabled by default so we need to handle that. except AttributeError: pass if getattr(proxy_ctx, "hostname_checks_common_name", False) is not True: pytest.skip("Test requires 'SSLContext.hostname_checks_common_name=True'") with proxy_from_url( proxy_url, ca_certs=proxy.ca_certs, proxy_ssl_context=proxy_ctx ) as https: https.request("GET", destination_url) urllib3-2.0.7/test/with_dummyserver/test_socketlevel.py0000644000000000000000000024770114513541732020367 0ustar00# TODO: Break this module up into pieces. Maybe group by functionality tested # rather than the socket level-ness of it. from __future__ import annotations import contextlib import errno import io import os import os.path import select import shutil import socket import ssl import sys import tempfile import time import typing import zlib from collections import OrderedDict from pathlib import Path from test import ( LONG_TIMEOUT, SHORT_TIMEOUT, notSecureTransport, notWindows, requires_ssl_context_keyfile_password, resolvesLocalhostFQDN, ) from threading import Event from unittest import mock import pytest import trustme from dummyserver.server import ( DEFAULT_CA, DEFAULT_CERTS, encrypt_key_pem, get_unreachable_address, ) from dummyserver.testcase import SocketDummyServerTestCase, consume_socket from urllib3 import HTTPConnectionPool, HTTPSConnectionPool, ProxyManager, util from urllib3._collections import HTTPHeaderDict from urllib3.connection import HTTPConnection, _get_default_user_agent from urllib3.connectionpool import _url_from_pool from urllib3.exceptions import ( InsecureRequestWarning, MaxRetryError, ProtocolError, ProxyError, ReadTimeoutError, SSLError, ) from urllib3.poolmanager import proxy_from_url from urllib3.util import ssl_, ssl_wrap_socket from urllib3.util.retry import Retry from urllib3.util.timeout import Timeout from .. import LogRecorder, has_alpn if typing.TYPE_CHECKING: from _typeshed import StrOrBytesPath else: StrOrBytesPath = object class TestCookies(SocketDummyServerTestCase): def test_multi_setcookie(self) -> None: def multicookie_response_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" b"Set-Cookie: foo=1\r\n" b"Set-Cookie: bar=1\r\n" b"\r\n" ) sock.close() self._start_server(multicookie_response_handler) with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/", retries=0) assert r.headers == {"set-cookie": "foo=1, bar=1"} assert r.headers.getlist("set-cookie") == ["foo=1", "bar=1"] class TestSNI(SocketDummyServerTestCase): def test_hostname_in_first_request_packet(self) -> None: done_receiving = Event() self.buf = b"" def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] self.buf = sock.recv(65536) # We only accept one packet done_receiving.set() # let the test know it can proceed sock.close() self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port) as pool: try: pool.request("GET", "/", retries=0) except MaxRetryError: # We are violating the protocol pass successful = done_receiving.wait(LONG_TIMEOUT) assert successful, "Timed out waiting for connection accept" assert ( self.host.encode("ascii") in self.buf ), "missing hostname in SSL handshake" class TestALPN(SocketDummyServerTestCase): def test_alpn_protocol_in_first_request_packet(self) -> None: if not has_alpn(): pytest.skip("ALPN-support not available") done_receiving = Event() self.buf = b"" def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] self.buf = sock.recv(65536) # We only accept one packet done_receiving.set() # let the test know it can proceed sock.close() self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port) as pool: try: pool.request("GET", "/", retries=0) except MaxRetryError: # We are violating the protocol pass successful = done_receiving.wait(LONG_TIMEOUT) assert successful, "Timed out waiting for connection accept" for protocol in util.ALPN_PROTOCOLS: assert ( protocol.encode("ascii") in self.buf ), "missing ALPN protocol in SSL handshake" def original_ssl_wrap_socket( sock: socket.socket, keyfile: StrOrBytesPath | None = None, certfile: StrOrBytesPath | None = None, server_side: bool = False, cert_reqs: ssl.VerifyMode = ssl.CERT_NONE, ssl_version: int = ssl.PROTOCOL_TLS, ca_certs: str | None = None, do_handshake_on_connect: bool = True, suppress_ragged_eofs: bool = True, ciphers: str | None = None, ) -> ssl.SSLSocket: if server_side and not certfile: raise ValueError("certfile must be specified for server-side operations") if keyfile and not certfile: raise ValueError("certfile must be specified") context = ssl.SSLContext(ssl_version) context.verify_mode = cert_reqs if ca_certs: context.load_verify_locations(ca_certs) if certfile: context.load_cert_chain(certfile, keyfile) if ciphers: context.set_ciphers(ciphers) return context.wrap_socket( sock=sock, server_side=server_side, do_handshake_on_connect=do_handshake_on_connect, suppress_ragged_eofs=suppress_ragged_eofs, ) class TestClientCerts(SocketDummyServerTestCase): """ Tests for client certificate support. """ @classmethod def setup_class(cls) -> None: cls.tmpdir = tempfile.mkdtemp() ca = trustme.CA() cert = ca.issue_cert("localhost") encrypted_key = encrypt_key_pem(cert.private_key_pem, b"letmein") cls.ca_path = os.path.join(cls.tmpdir, "ca.pem") cls.cert_combined_path = os.path.join(cls.tmpdir, "server.combined.pem") cls.cert_path = os.path.join(cls.tmpdir, "server.pem") cls.key_path = os.path.join(cls.tmpdir, "key.pem") cls.password_key_path = os.path.join(cls.tmpdir, "password_key.pem") ca.cert_pem.write_to_path(cls.ca_path) cert.private_key_and_cert_chain_pem.write_to_path(cls.cert_combined_path) cert.cert_chain_pems[0].write_to_path(cls.cert_path) cert.private_key_pem.write_to_path(cls.key_path) encrypted_key.write_to_path(cls.password_key_path) @classmethod def teardown_class(cls) -> None: shutil.rmtree(cls.tmpdir) def _wrap_in_ssl(self, sock: socket.socket) -> ssl.SSLSocket: """ Given a single socket, wraps it in TLS. """ return original_ssl_wrap_socket( sock, ssl_version=ssl.PROTOCOL_SSLv23, cert_reqs=ssl.CERT_REQUIRED, ca_certs=self.ca_path, certfile=self.cert_path, keyfile=self.key_path, server_side=True, ) def test_client_certs_two_files(self) -> None: """ Having a client cert in a separate file to its associated key works properly. """ done_receiving = Event() client_certs = [] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] sock = self._wrap_in_ssl(sock) client_certs.append(sock.getpeercert()) data = b"" while not data.endswith(b"\r\n\r\n"): data += sock.recv(8192) sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: testsocket\r\n" b"Connection: close\r\n" b"Content-Length: 6\r\n" b"\r\n" b"Valid!" ) done_receiving.wait(5) sock.close() self._start_server(socket_handler) with HTTPSConnectionPool( self.host, self.port, cert_file=self.cert_path, key_file=self.key_path, cert_reqs="REQUIRED", ca_certs=self.ca_path, ) as pool: pool.request("GET", "/", retries=0) done_receiving.set() assert len(client_certs) == 1 def test_client_certs_one_file(self) -> None: """ Having a client cert and its associated private key in just one file works properly. """ done_receiving = Event() client_certs = [] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] sock = self._wrap_in_ssl(sock) client_certs.append(sock.getpeercert()) data = b"" while not data.endswith(b"\r\n\r\n"): data += sock.recv(8192) sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: testsocket\r\n" b"Connection: close\r\n" b"Content-Length: 6\r\n" b"\r\n" b"Valid!" ) done_receiving.wait(5) sock.close() self._start_server(socket_handler) with HTTPSConnectionPool( self.host, self.port, cert_file=self.cert_combined_path, cert_reqs="REQUIRED", ca_certs=self.ca_path, ) as pool: pool.request("GET", "/", retries=0) done_receiving.set() assert len(client_certs) == 1 def test_missing_client_certs_raises_error(self) -> None: """ Having client certs not be present causes an error. """ done_receiving = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] try: self._wrap_in_ssl(sock) except ssl.SSLError: pass done_receiving.wait(5) sock.close() self._start_server(socket_handler) with HTTPSConnectionPool( self.host, self.port, cert_reqs="REQUIRED", ca_certs=self.ca_path ) as pool: with pytest.raises(MaxRetryError): pool.request("GET", "/", retries=0) done_receiving.set() done_receiving.set() @requires_ssl_context_keyfile_password() def test_client_cert_with_string_password(self) -> None: self.run_client_cert_with_password_test("letmein") @requires_ssl_context_keyfile_password() def test_client_cert_with_bytes_password(self) -> None: self.run_client_cert_with_password_test(b"letmein") def run_client_cert_with_password_test(self, password: bytes | str) -> None: """ Tests client certificate password functionality """ done_receiving = Event() client_certs = [] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] sock = self._wrap_in_ssl(sock) client_certs.append(sock.getpeercert()) data = b"" while not data.endswith(b"\r\n\r\n"): data += sock.recv(8192) sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: testsocket\r\n" b"Connection: close\r\n" b"Content-Length: 6\r\n" b"\r\n" b"Valid!" ) done_receiving.wait(5) sock.close() self._start_server(socket_handler) assert ssl_.SSLContext is not None ssl_context = ssl_.SSLContext(ssl_.PROTOCOL_SSLv23) ssl_context.load_cert_chain( certfile=self.cert_path, keyfile=self.password_key_path, password=password ) with HTTPSConnectionPool( self.host, self.port, ssl_context=ssl_context, cert_reqs="REQUIRED", ca_certs=self.ca_path, ) as pool: pool.request("GET", "/", retries=0) done_receiving.set() assert len(client_certs) == 1 @requires_ssl_context_keyfile_password() def test_load_keyfile_with_invalid_password(self) -> None: assert ssl_.SSLContext is not None context = ssl_.SSLContext(ssl_.PROTOCOL_SSLv23) with pytest.raises(ssl.SSLError): context.load_cert_chain( certfile=self.cert_path, keyfile=self.password_key_path, password=b"letmei", ) # For SecureTransport, the validation that would raise an error in # this case is deferred. @notSecureTransport() def test_load_invalid_cert_file(self) -> None: assert ssl_.SSLContext is not None context = ssl_.SSLContext(ssl_.PROTOCOL_SSLv23) with pytest.raises(ssl.SSLError): context.load_cert_chain(certfile=self.password_key_path) class TestSocketClosing(SocketDummyServerTestCase): def test_recovery_when_server_closes_connection(self) -> None: # Does the pool work seamlessly if an open connection in the # connection pool gets hung up on by the server, then reaches # the front of the queue again? done_closing = Event() def socket_handler(listener: socket.socket) -> None: for i in 0, 1: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65536) body = f"Response {int(i)}" sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(body), body) ).encode("utf-8") ) sock.close() # simulate a server timing out, closing socket done_closing.set() # let the test know it can proceed self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request("GET", "/", retries=0) assert response.status == 200 assert response.data == b"Response 0" done_closing.wait() # wait until the socket in our pool gets closed response = pool.request("GET", "/", retries=0) assert response.status == 200 assert response.data == b"Response 1" def test_connection_refused(self) -> None: # Does the pool retry if there is no listener on the port? host, port = get_unreachable_address() with HTTPConnectionPool(host, port, maxsize=3, block=True) as http: with pytest.raises(MaxRetryError): http.request("GET", "/", retries=0, release_conn=False) assert http.pool is not None assert http.pool.qsize() == http.pool.maxsize def test_connection_read_timeout(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] while not sock.recv(65536).endswith(b"\r\n\r\n"): pass timed_out.wait() sock.close() self._start_server(socket_handler) with HTTPConnectionPool( self.host, self.port, timeout=SHORT_TIMEOUT, retries=False, maxsize=3, block=True, ) as http: try: with pytest.raises(ReadTimeoutError): http.request("GET", "/", release_conn=False) finally: timed_out.set() assert http.pool is not None assert http.pool.qsize() == http.pool.maxsize def test_read_timeout_dont_retry_method_not_in_allowlist(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] sock.recv(65536) timed_out.wait() sock.close() self._start_server(socket_handler) with HTTPConnectionPool( self.host, self.port, timeout=LONG_TIMEOUT, retries=True ) as pool: try: with pytest.raises(ReadTimeoutError): pool.request("POST", "/") finally: timed_out.set() def test_https_connection_read_timeout(self) -> None: """Handshake timeouts should fail with a Timeout""" timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] while not sock.recv(65536): pass timed_out.wait() sock.close() # first ReadTimeoutError due to SocketTimeout self._start_server(socket_handler) with HTTPSConnectionPool( self.host, self.port, timeout=LONG_TIMEOUT, retries=False ) as pool: try: with pytest.raises(ReadTimeoutError): pool.request("GET", "/") finally: timed_out.set() # second ReadTimeoutError due to errno with HTTPSConnectionPool(host=self.host): err = OSError() err.errno = errno.EAGAIN with pytest.raises(ReadTimeoutError): pool._raise_timeout(err, "", 0) def test_timeout_errors_cause_retries(self) -> None: def socket_handler(listener: socket.socket) -> None: sock_timeout = listener.accept()[0] # Wait for a second request before closing the first socket. sock = listener.accept()[0] sock_timeout.close() # Second request. buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) # Now respond immediately. body = "Response 2" sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(body), body) ).encode("utf-8") ) sock.close() # In situations where the main thread throws an exception, the server # thread can hang on an accept() call. This ensures everything times # out within 1 second. This should be long enough for any socket # operations in the test suite to complete default_timeout = socket.getdefaulttimeout() socket.setdefaulttimeout(1) try: self._start_server(socket_handler) t = Timeout(connect=LONG_TIMEOUT, read=LONG_TIMEOUT) with HTTPConnectionPool(self.host, self.port, timeout=t) as pool: response = pool.request("GET", "/", retries=1) assert response.status == 200 assert response.data == b"Response 2" finally: socket.setdefaulttimeout(default_timeout) def test_delayed_body_read_timeout(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" body = "Hi" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65536) sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" % len(body) ).encode("utf-8") ) timed_out.wait() sock.send(body.encode("utf-8")) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: response = pool.urlopen( "GET", "/", retries=0, preload_content=False, timeout=Timeout(connect=1, read=LONG_TIMEOUT), ) try: with pytest.raises(ReadTimeoutError): response.read() finally: timed_out.set() def test_delayed_body_read_timeout_with_preload(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" body = "Hi" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" % len(body) ).encode("utf-8") ) timed_out.wait(5) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: try: with pytest.raises(ReadTimeoutError): timeout = Timeout(connect=LONG_TIMEOUT, read=SHORT_TIMEOUT) pool.urlopen("GET", "/", retries=False, timeout=timeout) finally: timed_out.set() def test_incomplete_response(self) -> None: body = "Response" partial_body = body[:2] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] # Consume request buf = b"" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65536) # Send partial response and close socket. sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(body), partial_body) ).encode("utf-8") ) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request("GET", "/", retries=0, preload_content=False) with pytest.raises(ProtocolError): response.read() def test_retry_weird_http_version(self) -> None: """Retry class should handle httplib.BadStatusLine errors properly""" def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] # First request. # Pause before responding so the first request times out. buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) # send unknown http protocol body = "bad http 0.5 response" sock.send( ( "HTTP/0.5 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(body), body) ).encode("utf-8") ) sock.close() # Second request. sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) # Now respond immediately. sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "foo" % (len("foo")) ).encode("utf-8") ) sock.close() # Close the socket. self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: retry = Retry(read=1) response = pool.request("GET", "/", retries=retry) assert response.status == 200 assert response.data == b"foo" def test_connection_cleanup_on_read_timeout(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" body = "Hi" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65536) sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" % len(body) ).encode("utf-8") ) timed_out.wait() sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: assert pool.pool is not None poolsize = pool.pool.qsize() response = pool.urlopen( "GET", "/", retries=0, preload_content=False, timeout=LONG_TIMEOUT ) try: with pytest.raises(ReadTimeoutError): response.read() assert poolsize == pool.pool.qsize() finally: timed_out.set() def test_connection_cleanup_on_protocol_error_during_read(self) -> None: body = "Response" partial_body = body[:2] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] # Consume request buf = b"" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65536) # Send partial response and close socket. sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(body), partial_body) ).encode("utf-8") ) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: assert pool.pool is not None poolsize = pool.pool.qsize() response = pool.request("GET", "/", retries=0, preload_content=False) with pytest.raises(ProtocolError): response.read() assert poolsize == pool.pool.qsize() def test_connection_closed_on_read_timeout_preload_false(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] # Consume request buf = b"" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65535) # Send partial chunked response and then hang. sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Transfer-Encoding: chunked\r\n" b"\r\n" b"8\r\n" b"12345678\r\n" ) timed_out.wait(5) # Expect a new request, but keep hold of the old socket to avoid # leaking it. Because we don't want to hang this thread, we # actually use select.select to confirm that a new request is # coming in: this lets us time the thread out. rlist, _, _ = select.select([listener], [], [], 1) assert rlist new_sock = listener.accept()[0] # Consume request buf = b"" while not buf.endswith(b"\r\n\r\n"): buf = new_sock.recv(65535) # Send complete chunked response. new_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Transfer-Encoding: chunked\r\n" b"\r\n" b"8\r\n" b"12345678\r\n" b"0\r\n\r\n" ) new_sock.close() sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: # First request should fail. response = pool.urlopen( "GET", "/", retries=0, preload_content=False, timeout=LONG_TIMEOUT ) try: with pytest.raises(ReadTimeoutError): response.read() finally: timed_out.set() # Second should succeed. response = pool.urlopen( "GET", "/", retries=0, preload_content=False, timeout=LONG_TIMEOUT ) assert len(response.read()) == 8 def test_closing_response_actually_closes_connection(self) -> None: done_closing = Event() complete = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf = sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 0\r\n" b"\r\n" ) # Wait for the socket to close. done_closing.wait(timeout=LONG_TIMEOUT) # Look for the empty string to show that the connection got closed. # Don't get stuck in a timeout. sock.settimeout(LONG_TIMEOUT) new_data = sock.recv(65536) assert not new_data sock.close() complete.set() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: response = pool.request("GET", "/", retries=0, preload_content=False) assert response.status == 200 response.close() done_closing.set() # wait until the socket in our pool gets closed successful = complete.wait(timeout=LONG_TIMEOUT) assert successful, "Timed out waiting for connection close" def test_release_conn_param_is_respected_after_timeout_retry(self) -> None: """For successful ```urlopen(release_conn=False)```, the connection isn't released, even after a retry. This test allows a retry: one request fails, the next request succeeds. This is a regression test for issue #651 [1], where the connection would be released if the initial request failed, even if a retry succeeded. [1] """ def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] consume_socket(sock) # Close the connection, without sending any response (not even the # HTTP status line). This will trigger a `Timeout` on the client, # inside `urlopen()`. sock.close() # Expect a new request. Because we don't want to hang this thread, # we actually use select.select to confirm that a new request is # coming in: this lets us time the thread out. rlist, _, _ = select.select([listener], [], [], 5) assert rlist sock = listener.accept()[0] consume_socket(sock) # Send complete chunked response. sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Transfer-Encoding: chunked\r\n" b"\r\n" b"8\r\n" b"12345678\r\n" b"0\r\n\r\n" ) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port, maxsize=1) as pool: # First request should fail, but the timeout and `retries=1` should # save it. response = pool.urlopen( "GET", "/", retries=1, release_conn=False, preload_content=False, timeout=LONG_TIMEOUT, ) # The connection should still be on the response object, and none # should be in the pool. We opened two though. assert pool.num_connections == 2 assert pool.pool is not None assert pool.pool.qsize() == 0 assert response.connection is not None # Consume the data. This should put the connection back. response.read() assert pool.pool.qsize() == 1 assert response.connection is None def test_socket_close_socket_then_file(self) -> None: def consume_ssl_socket(listener: socket.socket) -> None: try: with listener.accept()[0] as sock, original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) as ssl_sock: consume_socket(ssl_sock) except (ConnectionResetError, ConnectionAbortedError, OSError): pass self._start_server(consume_ssl_socket) with socket.create_connection( (self.host, self.port) ) as sock, contextlib.closing( ssl_wrap_socket(sock, server_hostname=self.host, ca_certs=DEFAULT_CA) ) as ssl_sock, ssl_sock.makefile( "rb" ) as f: ssl_sock.close() f.close() # SecureTransport is supposed to raise OSError but raises # ssl.SSLError when closed because ssl_sock.context is None with pytest.raises((OSError, ssl.SSLError)): ssl_sock.sendall(b"hello") assert ssl_sock.fileno() == -1 def test_socket_close_stays_open_with_makefile_open(self) -> None: def consume_ssl_socket(listener: socket.socket) -> None: try: with listener.accept()[0] as sock, original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) as ssl_sock: consume_socket(ssl_sock) except (ConnectionResetError, ConnectionAbortedError, OSError): pass self._start_server(consume_ssl_socket) with socket.create_connection( (self.host, self.port) ) as sock, contextlib.closing( ssl_wrap_socket(sock, server_hostname=self.host, ca_certs=DEFAULT_CA) ) as ssl_sock, ssl_sock.makefile( "rb" ): ssl_sock.close() ssl_sock.close() ssl_sock.sendall(b"hello") assert ssl_sock.fileno() > 0 class TestProxyManager(SocketDummyServerTestCase): def test_simple(self) -> None: def echo_socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(buf), buf.decode("utf-8")) ).encode("utf-8") ) sock.close() self._start_server(echo_socket_handler) base_url = f"http://{self.host}:{self.port}" with proxy_from_url(base_url) as proxy: r = proxy.request("GET", "http://google.com/") assert r.status == 200 # FIXME: The order of the headers is not predictable right now. We # should fix that someday (maybe when we migrate to # OrderedDict/MultiDict). assert sorted(r.data.split(b"\r\n")) == sorted( [ b"GET http://google.com/ HTTP/1.1", b"Host: google.com", b"Accept-Encoding: identity", b"Accept: */*", b"User-Agent: " + _get_default_user_agent().encode("utf-8"), b"", b"", ] ) def test_headers(self) -> None: def echo_socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(buf), buf.decode("utf-8")) ).encode("utf-8") ) sock.close() self._start_server(echo_socket_handler) base_url = f"http://{self.host}:{self.port}" # Define some proxy headers. proxy_headers = HTTPHeaderDict({"For The Proxy": "YEAH!"}) with proxy_from_url(base_url, proxy_headers=proxy_headers) as proxy: conn = proxy.connection_from_url("http://www.google.com/") r = conn.urlopen("GET", "http://www.google.com/", assert_same_host=False) assert r.status == 200 # FIXME: The order of the headers is not predictable right now. We # should fix that someday (maybe when we migrate to # OrderedDict/MultiDict). assert b"For The Proxy: YEAH!\r\n" in r.data def test_retries(self) -> None: close_event = Event() def echo_socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] # First request, which should fail sock.close() # Second request sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( ( "HTTP/1.1 200 OK\r\n" "Content-Type: text/plain\r\n" "Content-Length: %d\r\n" "\r\n" "%s" % (len(buf), buf.decode("utf-8")) ).encode("utf-8") ) sock.close() close_event.set() self._start_server(echo_socket_handler) base_url = f"http://{self.host}:{self.port}" with proxy_from_url(base_url) as proxy: conn = proxy.connection_from_url("http://www.google.com") r = conn.urlopen( "GET", "http://www.google.com", assert_same_host=False, retries=1 ) assert r.status == 200 close_event.wait(timeout=LONG_TIMEOUT) with pytest.raises(ProxyError): conn.urlopen( "GET", "http://www.google.com", assert_same_host=False, retries=False, ) def test_connect_reconn(self) -> None: def proxy_ssl_one(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) s = buf.decode("utf-8") if not s.startswith("CONNECT "): sock.send(b"HTTP/1.1 405 Method not allowed\r\nAllow: CONNECT\r\n\r\n") sock.close() return if not s.startswith(f"CONNECT {self.host}:443"): sock.send(b"HTTP/1.1 403 Forbidden\r\n\r\n") sock.close() return sock.send(b"HTTP/1.1 200 Connection Established\r\n\r\n") ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 2\r\n" b"Connection: close\r\n" b"\r\n" b"Hi" ) ssl_sock.close() def echo_socket_handler(listener: socket.socket) -> None: proxy_ssl_one(listener) proxy_ssl_one(listener) self._start_server(echo_socket_handler) base_url = f"http://{self.host}:{self.port}" with proxy_from_url(base_url, ca_certs=DEFAULT_CA) as proxy: url = f"https://{self.host}" conn = proxy.connection_from_url(url) r = conn.urlopen("GET", url, retries=0) assert r.status == 200 r = conn.urlopen("GET", url, retries=0) assert r.status == 200 def test_connect_ipv6_addr(self) -> None: ipv6_addr = "2001:4998:c:a06::2:4008" def echo_socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) s = buf.decode("utf-8") if s.startswith(f"CONNECT [{ipv6_addr}]:443"): sock.send(b"HTTP/1.1 200 Connection Established\r\n\r\n") ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ) buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 2\r\n" b"Connection: close\r\n" b"\r\n" b"Hi" ) ssl_sock.close() else: sock.close() self._start_server(echo_socket_handler) base_url = f"http://{self.host}:{self.port}" with proxy_from_url(base_url, cert_reqs="NONE") as proxy: url = f"https://[{ipv6_addr}]" conn = proxy.connection_from_url(url) try: with pytest.warns(InsecureRequestWarning): r = conn.urlopen("GET", url, retries=0) assert r.status == 200 except MaxRetryError: pytest.fail("Invalid IPv6 format in HTTP CONNECT request") @pytest.mark.parametrize("target_scheme", ["http", "https"]) def test_https_proxymanager_connected_to_http_proxy( self, target_scheme: str ) -> None: errored = Event() def http_socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] sock.send(b"HTTP/1.0 501 Not Implemented\r\nConnection: close\r\n\r\n") errored.wait() sock.close() self._start_server(http_socket_handler) base_url = f"https://{self.host}:{self.port}" with ProxyManager(base_url, cert_reqs="NONE") as proxy: with pytest.raises(MaxRetryError) as e: proxy.request("GET", f"{target_scheme}://example.com", retries=0) errored.set() # Avoid a ConnectionAbortedError on Windows. assert type(e.value.reason) == ProxyError assert "Your proxy appears to only use HTTP and not HTTPS" in str( e.value.reason ) class TestSSL(SocketDummyServerTestCase): def test_ssl_failure_midway_through_conn(self) -> None: def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] sock2 = sock.dup() ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) # Deliberately send from the non-SSL socket. sock2.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 2\r\n" b"\r\n" b"Hi" ) sock2.close() ssl_sock.close() self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port, ca_certs=DEFAULT_CA) as pool: with pytest.raises( SSLError, match=r"(wrong version number|record overflow)" ): pool.request("GET", "/", retries=False) @notSecureTransport() def test_ssl_read_timeout(self) -> None: timed_out = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] # disable Nagle's algorithm so there's no delay in sending a partial body sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, True) ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ) buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) # Send incomplete message (note Content-Length) ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 10\r\n" b"\r\n" b"Hi-" ) timed_out.wait() sock.close() ssl_sock.close() self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port, ca_certs=DEFAULT_CA) as pool: response = pool.urlopen( "GET", "/", retries=0, preload_content=False, timeout=LONG_TIMEOUT ) try: with pytest.raises(ReadTimeoutError): response.read() finally: timed_out.set() def test_ssl_failed_fingerprint_verification(self) -> None: def socket_handler(listener: socket.socket) -> None: for i in range(2): sock = listener.accept()[0] try: ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) except (ssl.SSLError, ConnectionResetError): if i == 1: raise return ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 5\r\n\r\n" b"Hello" ) ssl_sock.close() sock.close() self._start_server(socket_handler) # GitHub's fingerprint. Valid, but not matching. fingerprint = "A0:C4:A7:46:00:ED:A7:2D:C0:BE:CB:9A:8C:B6:07:CA:58:EE:74:5E" def request() -> None: pool = HTTPSConnectionPool( self.host, self.port, assert_fingerprint=fingerprint ) try: timeout = Timeout(connect=LONG_TIMEOUT, read=SHORT_TIMEOUT) response = pool.urlopen( "GET", "/", preload_content=False, retries=0, timeout=timeout ) response.read() finally: pool.close() with pytest.raises(MaxRetryError) as cm: request() assert isinstance(cm.value.reason, SSLError) # Should not hang, see https://github.com/urllib3/urllib3/issues/529 with pytest.raises(MaxRetryError): request() def test_retry_ssl_error(self) -> None: def socket_handler(listener: socket.socket) -> None: # first request, trigger an SSLError sock = listener.accept()[0] sock2 = sock.dup() ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ) buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) # Deliberately send from the non-SSL socket to trigger an SSLError sock2.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 4\r\n" b"\r\n" b"Fail" ) sock2.close() ssl_sock.close() # retried request sock = listener.accept()[0] ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ) buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 7\r\n\r\n" b"Success" ) ssl_sock.close() self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port, ca_certs=DEFAULT_CA) as pool: response = pool.urlopen("GET", "/", retries=1) assert response.data == b"Success" def test_ssl_load_default_certs_when_empty(self) -> None: def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] try: ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) except (ssl.SSLError, OSError): return buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 5\r\n\r\n" b"Hello" ) ssl_sock.close() sock.close() context = mock.create_autospec(ssl_.SSLContext) context.load_default_certs = mock.Mock() context.options = 0 with mock.patch("urllib3.util.ssl_.SSLContext", lambda *_, **__: context): self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port) as pool: # Without a proper `SSLContext`, this request will fail in some # arbitrary way, but we only want to know if load_default_certs() was # called, which is why we accept any `Exception` here. with pytest.raises(Exception): pool.request("GET", "/", timeout=SHORT_TIMEOUT) context.load_default_certs.assert_called_with() def test_ssl_dont_load_default_certs_when_given(self) -> None: def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] try: ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) except (ssl.SSLError, OSError): return buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += ssl_sock.recv(65536) ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: 5\r\n\r\n" b"Hello" ) ssl_sock.close() sock.close() context = mock.create_autospec(ssl_.SSLContext) context.load_default_certs = mock.Mock() context.options = 0 with mock.patch("urllib3.util.ssl_.SSLContext", lambda *_, **__: context): for kwargs in [ {"ca_certs": "/a"}, {"ca_cert_dir": "/a"}, {"ca_certs": "a", "ca_cert_dir": "a"}, {"ssl_context": context}, ]: self._start_server(socket_handler) with HTTPSConnectionPool(self.host, self.port, **kwargs) as pool: with pytest.raises(Exception): pool.request("GET", "/", timeout=SHORT_TIMEOUT) context.load_default_certs.assert_not_called() def test_load_verify_locations_exception(self) -> None: """ Ensure that load_verify_locations raises SSLError for all backends """ with pytest.raises(SSLError): ssl_wrap_socket(None, ca_certs="/tmp/fake-file") # type: ignore[call-overload] def test_ssl_custom_validation_failure_terminates(self, tmpdir: Path) -> None: """ Ensure that the underlying socket is terminated if custom validation fails. """ server_closed = Event() def is_closed_socket(sock: socket.socket) -> bool: try: sock.settimeout(SHORT_TIMEOUT) except OSError: return True return False def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] try: _ = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) except ConnectionResetError: return except ssl.SSLError as e: assert "alert unknown ca" in str(e) if is_closed_socket(sock): server_closed.set() self._start_server(socket_handler) # client uses a different ca other_ca = trustme.CA() other_ca_path = str(tmpdir / "ca.pem") other_ca.cert_pem.write_to_path(other_ca_path) with HTTPSConnectionPool( self.host, self.port, cert_reqs="REQUIRED", ca_certs=other_ca_path ) as pool: with pytest.raises(SSLError): pool.request("GET", "/", retries=False, timeout=LONG_TIMEOUT) assert server_closed.wait(LONG_TIMEOUT), "The socket was not terminated" # SecureTransport can read only small pieces of data at the moment. # https://github.com/urllib3/urllib3/pull/2674 @notSecureTransport() @pytest.mark.skipif( os.environ.get("CI") == "true" and sys.implementation.name == "pypy", reason="too slow to run in CI", ) @pytest.mark.parametrize( "preload_content,read_amt", [(True, None), (False, None), (False, 2**31)] ) def test_requesting_large_resources_via_ssl( self, preload_content: bool, read_amt: int | None ) -> None: """ Ensure that it is possible to read 2 GiB or more via an SSL socket. https://github.com/urllib3/urllib3/issues/2513 """ content_length = 2**31 # (`int` max value in C) + 1. ssl_ready = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] ssl_sock = original_ssl_wrap_socket( sock, server_side=True, keyfile=DEFAULT_CERTS["keyfile"], certfile=DEFAULT_CERTS["certfile"], ca_certs=DEFAULT_CA, ) ssl_ready.set() while not ssl_sock.recv(65536).endswith(b"\r\n\r\n"): continue ssl_sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/plain\r\n" b"Content-Length: %d\r\n\r\n" % content_length ) chunks = 2 for i in range(chunks): ssl_sock.sendall(bytes(content_length // chunks)) ssl_sock.close() sock.close() self._start_server(socket_handler) ssl_ready.wait(5) with HTTPSConnectionPool( self.host, self.port, ca_certs=DEFAULT_CA, retries=False ) as pool: response = pool.request("GET", "/", preload_content=preload_content) data = response.data if preload_content else response.read(read_amt) assert len(data) == content_length class TestErrorWrapping(SocketDummyServerTestCase): def test_bad_statusline(self) -> None: self.start_response_handler( b"HTTP/1.1 Omg What Is This?\r\n" b"Content-Length: 0\r\n" b"\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: with pytest.raises(ProtocolError): pool.request("GET", "/") def test_unknown_protocol(self) -> None: self.start_response_handler( b"HTTP/1000 200 OK\r\n" b"Content-Length: 0\r\n" b"\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: with pytest.raises(ProtocolError): pool.request("GET", "/") class TestHeaders(SocketDummyServerTestCase): def test_httplib_headers_case_insensitive(self) -> None: self.start_response_handler( b"HTTP/1.1 200 OK\r\n" b"Content-Length: 0\r\n" b"Content-type: text/plain\r\n" b"\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: HEADERS = {"Content-Length": "0", "Content-type": "text/plain"} r = pool.request("GET", "/") assert HEADERS == dict(r.headers.items()) # to preserve case sensitivity def start_parsing_handler(self) -> None: self.parsed_headers: typing.OrderedDict[str, str] = OrderedDict() self.received_headers: list[bytes] = [] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) self.received_headers = [ header for header in buf.split(b"\r\n")[1:] if header ] for header in self.received_headers: (key, value) = header.split(b": ") self.parsed_headers[key.decode("ascii")] = value.decode("ascii") sock.send(b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n") sock.close() self._start_server(socket_handler) def test_headers_are_sent_with_the_original_case(self) -> None: headers = {"foo": "bar", "bAz": "quux"} self.start_parsing_handler() expected_headers = { "Accept-Encoding": "identity", "Host": f"{self.host}:{self.port}", "User-Agent": _get_default_user_agent(), } expected_headers.update(headers) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.request("GET", "/", headers=HTTPHeaderDict(headers)) assert expected_headers == self.parsed_headers def test_ua_header_can_be_overridden(self) -> None: headers = {"uSeR-AgENt": "Definitely not urllib3!"} self.start_parsing_handler() expected_headers = { "Accept-Encoding": "identity", "Host": f"{self.host}:{self.port}", } expected_headers.update(headers) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.request("GET", "/", headers=HTTPHeaderDict(headers)) assert expected_headers == self.parsed_headers def test_request_headers_are_sent_in_the_original_order(self) -> None: # NOTE: Probability this test gives a false negative is 1/(K!) K = 16 # NOTE: Provide headers in non-sorted order (i.e. reversed) # so that if the internal implementation tries to sort them, # a change will be detected. expected_request_headers = [ (f"X-Header-{int(i)}", str(i)) for i in reversed(range(K)) ] def filter_non_x_headers( d: typing.OrderedDict[str, str] ) -> list[tuple[str, str]]: return [(k, v) for (k, v) in d.items() if k.startswith("X-Header-")] self.start_parsing_handler() with HTTPConnectionPool(self.host, self.port, retries=False) as pool: pool.request("GET", "/", headers=OrderedDict(expected_request_headers)) assert expected_request_headers == filter_non_x_headers(self.parsed_headers) @resolvesLocalhostFQDN() def test_request_host_header_ignores_fqdn_dot(self) -> None: self.start_parsing_handler() with HTTPConnectionPool(self.host + ".", self.port, retries=False) as pool: pool.request("GET", "/") self.assert_header_received( self.received_headers, "Host", f"{self.host}:{self.port}" ) def test_response_headers_are_returned_in_the_original_order(self) -> None: # NOTE: Probability this test gives a false negative is 1/(K!) K = 16 # NOTE: Provide headers in non-sorted order (i.e. reversed) # so that if the internal implementation tries to sort them, # a change will be detected. expected_response_headers = [ (f"X-Header-{int(i)}", str(i)) for i in reversed(range(K)) ] def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" + b"\r\n".join( [ (k.encode("utf8") + b": " + v.encode("utf8")) for (k, v) in expected_response_headers ] ) + b"\r\n" ) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("GET", "/", retries=0) actual_response_headers = [ (k, v) for (k, v) in r.headers.items() if k.startswith("X-Header-") ] assert expected_response_headers == actual_response_headers @pytest.mark.parametrize( "method_type, body_type", [ ("GET", None), ("POST", None), ("POST", "bytes"), ("POST", "bytes-io"), ], ) def test_headers_sent_with_add( self, method_type: str, body_type: str | None ) -> None: """ Confirm that when adding headers with combine=True that we simply append to the most recent value, rather than create a new header line. """ body: None | bytes | io.BytesIO if body_type is None: body = None elif body_type == "bytes": body = b"my-body" elif body_type == "bytes-io": body = io.BytesIO(b"bytes-io-body") body.seek(0, 0) else: raise ValueError("Unknonw body type") buffer: bytes = b"" def socket_handler(listener: socket.socket) -> None: nonlocal buffer sock = listener.accept()[0] sock.settimeout(0) start = time.time() while time.time() - start < (LONG_TIMEOUT / 2): try: buffer += sock.recv(65536) except OSError: continue sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: example.com\r\n" b"Content-Length: 0\r\n\r\n" ) sock.close() self._start_server(socket_handler) headers = HTTPHeaderDict() headers.add("A", "1") headers.add("C", "3") headers.add("B", "2") headers.add("B", "3") headers.add("A", "4", combine=False) headers.add("C", "5", combine=True) headers.add("C", "6") with HTTPConnectionPool(self.host, self.port, retries=False) as pool: r = pool.request( method_type, "/", body=body, headers=headers, ) assert r.status == 200 assert b"A: 1\r\nA: 4\r\nC: 3, 5\r\nC: 6\r\nB: 2\r\nB: 3" in buffer class TestBrokenHeaders(SocketDummyServerTestCase): def _test_broken_header_parsing( self, headers: list[bytes], unparsed_data_check: str | None = None ) -> None: self.start_response_handler( ( b"HTTP/1.1 200 OK\r\n" b"Content-Length: 0\r\n" b"Content-type: text/plain\r\n" ) + b"\r\n".join(headers) + b"\r\n\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: with LogRecorder() as logs: pool.request("GET", "/") for record in logs: if ( "Failed to parse headers" in record.msg and isinstance(record.args, tuple) and _url_from_pool(pool, "/") == record.args[0] ): if ( unparsed_data_check is None or unparsed_data_check in record.getMessage() ): return pytest.fail("Missing log about unparsed headers") def test_header_without_name(self) -> None: self._test_broken_header_parsing([b": Value", b"Another: Header"]) def test_header_without_name_or_value(self) -> None: self._test_broken_header_parsing([b":", b"Another: Header"]) def test_header_without_colon_or_value(self) -> None: self._test_broken_header_parsing( [b"Broken Header", b"Another: Header"], "Broken Header" ) class TestHeaderParsingContentType(SocketDummyServerTestCase): def _test_okay_header_parsing(self, header: bytes) -> None: self.start_response_handler( (b"HTTP/1.1 200 OK\r\n" b"Content-Length: 0\r\n") + header + b"\r\n\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: with LogRecorder() as logs: pool.request("GET", "/") for record in logs: assert "Failed to parse headers" not in record.msg def test_header_text_plain(self) -> None: self._test_okay_header_parsing(b"Content-type: text/plain") def test_header_message_rfc822(self) -> None: self._test_okay_header_parsing(b"Content-type: message/rfc822") class TestHEAD(SocketDummyServerTestCase): def test_chunked_head_response_does_not_hang(self) -> None: self.start_response_handler( b"HTTP/1.1 200 OK\r\n" b"Transfer-Encoding: chunked\r\n" b"Content-type: text/plain\r\n" b"\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: r = pool.request("HEAD", "/", timeout=LONG_TIMEOUT, preload_content=False) # stream will use the read_chunked method here. assert [] == list(r.stream()) def test_empty_head_response_does_not_hang(self) -> None: self.start_response_handler( b"HTTP/1.1 200 OK\r\n" b"Content-Length: 256\r\n" b"Content-type: text/plain\r\n" b"\r\n" ) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: r = pool.request("HEAD", "/", timeout=LONG_TIMEOUT, preload_content=False) # stream will use the read method here. assert [] == list(r.stream()) class TestStream(SocketDummyServerTestCase): def test_stream_none_unchunked_response_does_not_hang(self) -> None: done_event = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Length: 12\r\n" b"Content-type: text/plain\r\n" b"\r\n" b"hello, world" ) done_event.wait(5) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: r = pool.request("GET", "/", timeout=LONG_TIMEOUT, preload_content=False) # Stream should read to the end. assert [b"hello, world"] == list(r.stream(None)) done_event.set() def test_large_compressed_stream(self) -> None: done_event = Event() expected_total_length = 296085 def socket_handler(listener: socket.socket) -> None: compress = zlib.compressobj(6, zlib.DEFLATED, 16 + zlib.MAX_WBITS) data = compress.compress(b"x" * expected_total_length) data += compress.flush() sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Content-Length: %d\r\n" b"Content-Encoding: gzip\r\n" b"\r\n" % (len(data),) + data ) done_event.wait(5) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port, retries=False) as pool: r = pool.request("GET", "/", timeout=LONG_TIMEOUT, preload_content=False) # Chunks must all be equal or less than 10240 # and only the last chunk is allowed to be smaller # than 10240. total_length = 0 chunks_smaller_than_10240 = 0 for chunk in r.stream(10240, decode_content=True): assert 0 < len(chunk) <= 10240 if len(chunk) < 10240: chunks_smaller_than_10240 += 1 else: assert chunks_smaller_than_10240 == 0 total_length += len(chunk) assert chunks_smaller_than_10240 == 1 assert expected_total_length == total_length done_event.set() class TestBadContentLength(SocketDummyServerTestCase): def test_enforce_content_length_get(self) -> None: done_event = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Length: 22\r\n" b"Content-type: text/plain\r\n" b"\r\n" b"hello, world" ) done_event.wait(LONG_TIMEOUT) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port, maxsize=1) as conn: # Test stream read when content length less than headers claim get_response = conn.request( "GET", url="/", preload_content=False, enforce_content_length=True ) data = get_response.stream(100) with pytest.raises(ProtocolError, match="12 bytes read, 10 more expected"): next(data) done_event.set() def test_enforce_content_length_no_body(self) -> None: done_event = Event() def socket_handler(listener: socket.socket) -> None: sock = listener.accept()[0] buf = b"" while not buf.endswith(b"\r\n\r\n"): buf += sock.recv(65536) sock.send( b"HTTP/1.1 200 OK\r\n" b"Content-Length: 22\r\n" b"Content-type: text/plain\r\n" b"\r\n" ) done_event.wait(1) sock.close() self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port, maxsize=1) as conn: # Test stream on 0 length body head_response = conn.request( "HEAD", url="/", preload_content=False, enforce_content_length=True ) data = [chunk for chunk in head_response.stream(1)] assert len(data) == 0 done_event.set() class TestRetryPoolSizeDrainFail(SocketDummyServerTestCase): def test_pool_size_retry_drain_fail(self) -> None: def socket_handler(listener: socket.socket) -> None: for _ in range(2): sock = listener.accept()[0] while not sock.recv(65536).endswith(b"\r\n\r\n"): pass # send a response with an invalid content length -- this causes # a ProtocolError to raise when trying to drain the connection sock.send( b"HTTP/1.1 404 NOT FOUND\r\n" b"Content-Length: 1000\r\n" b"Content-Type: text/plain\r\n" b"\r\n" ) sock.close() self._start_server(socket_handler) retries = Retry(total=1, raise_on_status=False, status_forcelist=[404]) with HTTPConnectionPool( self.host, self.port, maxsize=10, retries=retries, block=True ) as pool: pool.urlopen("GET", "/not_found", preload_content=False) assert pool.num_connections == 1 class TestBrokenPipe(SocketDummyServerTestCase): @notWindows() def test_ignore_broken_pipe_errors(self, monkeypatch: pytest.MonkeyPatch) -> None: # On Windows an aborted connection raises an error on # attempts to read data out of a socket that's been closed. sock_shut = Event() orig_connect = HTTPConnection.connect # a buffer that will cause two sendall calls buf = "a" * 1024 * 1024 * 4 def connect_and_wait(*args: typing.Any, **kw: typing.Any) -> None: ret = orig_connect(*args, **kw) assert sock_shut.wait(5) return ret def socket_handler(listener: socket.socket) -> None: for i in range(2): sock = listener.accept()[0] sock.send( b"HTTP/1.1 404 Not Found\r\n" b"Connection: close\r\n" b"Content-Length: 10\r\n" b"\r\n" b"xxxxxxxxxx" ) sock.shutdown(socket.SHUT_RDWR) sock_shut.set() sock.close() monkeypatch.setattr(HTTPConnection, "connect", connect_and_wait) self._start_server(socket_handler) with HTTPConnectionPool(self.host, self.port) as pool: r = pool.request("POST", "/", body=buf) assert r.status == 404 assert r.headers["content-length"] == "10" assert r.data == b"xxxxxxxxxx" r = pool.request("POST", "/admin", chunked=True, body=buf) assert r.status == 404 assert r.headers["content-length"] == "10" assert r.data == b"xxxxxxxxxx" class TestMultipartResponse(SocketDummyServerTestCase): def test_multipart_assert_header_parsing_no_defects(self) -> None: def socket_handler(listener: socket.socket) -> None: for _ in range(2): sock = listener.accept()[0] while not sock.recv(65536).endswith(b"\r\n\r\n"): pass sock.sendall( b"HTTP/1.1 404 Not Found\r\n" b"Server: example.com\r\n" b"Content-Type: multipart/mixed; boundary=36eeb8c4e26d842a\r\n" b"Content-Length: 73\r\n" b"\r\n" b"--36eeb8c4e26d842a\r\n" b"Content-Type: text/plain\r\n" b"\r\n" b"1\r\n" b"--36eeb8c4e26d842a--\r\n", ) sock.close() self._start_server(socket_handler) from urllib3.connectionpool import log with mock.patch.object(log, "warning") as log_warning: with HTTPConnectionPool(self.host, self.port, timeout=LONG_TIMEOUT) as pool: resp = pool.urlopen("GET", "/") assert resp.status == 404 assert ( resp.headers["content-type"] == "multipart/mixed; boundary=36eeb8c4e26d842a" ) assert len(resp.data) == 73 log_warning.assert_not_called() class TestContentFraming(SocketDummyServerTestCase): @pytest.mark.parametrize("content_length", [None, 0]) @pytest.mark.parametrize("method", ["POST", "PUT", "PATCH"]) def test_content_length_0_by_default( self, method: str, content_length: int | None ) -> None: buffer = bytearray() def socket_handler(listener: socket.socket) -> None: nonlocal buffer sock = listener.accept()[0] while not buffer.endswith(b"\r\n\r\n"): buffer += sock.recv(65536) sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: example.com\r\n" b"Content-Length: 0\r\n\r\n" ) sock.close() self._start_server(socket_handler) headers = {} if content_length is not None: headers["Content-Length"] = str(content_length) with HTTPConnectionPool(self.host, self.port, timeout=3) as pool: resp = pool.request(method, "/") assert resp.status == 200 sent_bytes = bytes(buffer) assert b"Accept-Encoding: identity\r\n" in sent_bytes assert b"Content-Length: 0\r\n" in sent_bytes assert b"transfer-encoding" not in sent_bytes.lower() @pytest.mark.parametrize("chunked", [True, False]) @pytest.mark.parametrize("method", ["POST", "PUT", "PATCH"]) @pytest.mark.parametrize("body_type", ["file", "generator", "bytes"]) def test_chunked_specified( self, method: str, chunked: bool, body_type: str ) -> None: buffer = bytearray() def socket_handler(listener: socket.socket) -> None: nonlocal buffer sock = listener.accept()[0] sock.settimeout(0) start = time.time() while time.time() - start < (LONG_TIMEOUT / 2): try: buffer += sock.recv(65536) except OSError: continue sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: example.com\r\n" b"Content-Length: 0\r\n\r\n" ) sock.close() self._start_server(socket_handler) body: typing.Any if body_type == "generator": def body_generator() -> typing.Generator[bytes, None, None]: yield b"x" * 10 body = body_generator() elif body_type == "file": body = io.BytesIO(b"x" * 10) body.seek(0, 0) else: if chunked is False: pytest.skip("urllib3 uses Content-Length in this case") body = b"x" * 10 with HTTPConnectionPool( self.host, self.port, timeout=LONG_TIMEOUT, retries=False ) as pool: resp = pool.request(method, "/", chunked=chunked, body=body) assert resp.status == 200 sent_bytes = bytes(buffer) assert sent_bytes.count(b":") == 5 assert b"Host: localhost:" in sent_bytes assert b"Accept-Encoding: identity\r\n" in sent_bytes assert b"Transfer-Encoding: chunked\r\n" in sent_bytes assert b"User-Agent: python-urllib3/" in sent_bytes assert b"content-length" not in sent_bytes.lower() assert b"\r\n\r\na\r\nxxxxxxxxxx\r\n0\r\n\r\n" in sent_bytes @pytest.mark.parametrize("method", ["POST", "PUT", "PATCH"]) @pytest.mark.parametrize( "body_type", ["file", "generator", "bytes", "bytearray", "file_text"] ) def test_chunked_not_specified(self, method: str, body_type: str) -> None: buffer = bytearray() def socket_handler(listener: socket.socket) -> None: nonlocal buffer sock = listener.accept()[0] sock.settimeout(0) start = time.time() while time.time() - start < (LONG_TIMEOUT / 2): try: buffer += sock.recv(65536) except OSError: continue sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: example.com\r\n" b"Content-Length: 0\r\n\r\n" ) sock.close() self._start_server(socket_handler) body: typing.Any if body_type == "generator": def body_generator() -> typing.Generator[bytes, None, None]: yield b"x" * 10 body = body_generator() should_be_chunked = True elif body_type == "file": body = io.BytesIO(b"x" * 10) body.seek(0, 0) should_be_chunked = True elif body_type == "file_text": body = io.StringIO("x" * 10) body.seek(0, 0) should_be_chunked = True elif body_type == "bytearray": body = bytearray(b"x" * 10) should_be_chunked = False else: body = b"x" * 10 should_be_chunked = False with HTTPConnectionPool( self.host, self.port, timeout=LONG_TIMEOUT, retries=False ) as pool: resp = pool.request(method, "/", body=body) assert resp.status == 200 sent_bytes = bytes(buffer) assert sent_bytes.count(b":") == 5 assert b"Host: localhost:" in sent_bytes assert b"Accept-Encoding: identity\r\n" in sent_bytes assert b"User-Agent: python-urllib3/" in sent_bytes if should_be_chunked: assert b"content-length" not in sent_bytes.lower() assert b"Transfer-Encoding: chunked\r\n" in sent_bytes assert b"\r\n\r\na\r\nxxxxxxxxxx\r\n0\r\n\r\n" in sent_bytes else: assert b"Content-Length: 10\r\n" in sent_bytes assert b"transfer-encoding" not in sent_bytes.lower() assert sent_bytes.endswith(b"\r\n\r\nxxxxxxxxxx") @pytest.mark.parametrize( "header_transform", [str.lower, str.title, str.upper], ) @pytest.mark.parametrize( ["header", "header_value", "expected"], [ ("content-length", "10", b": 10\r\n\r\nxxxxxxxx"), ( "transfer-encoding", "chunked", b": chunked\r\n\r\n8\r\nxxxxxxxx\r\n0\r\n\r\n", ), ], ) def test_framing_set_via_headers( self, header_transform: typing.Callable[[str], str], header: str, header_value: str, expected: bytes, ) -> None: buffer = bytearray() def socket_handler(listener: socket.socket) -> None: nonlocal buffer sock = listener.accept()[0] sock.settimeout(0) start = time.time() while time.time() - start < (LONG_TIMEOUT / 2): try: buffer += sock.recv(65536) except OSError: continue sock.sendall( b"HTTP/1.1 200 OK\r\n" b"Server: example.com\r\n" b"Content-Length: 0\r\n\r\n" ) sock.close() self._start_server(socket_handler) with HTTPConnectionPool( self.host, self.port, timeout=LONG_TIMEOUT, retries=False ) as pool: resp = pool.request( "POST", "/", body=b"xxxxxxxx", headers={header_transform(header): header_value}, ) assert resp.status == 200 sent_bytes = bytes(buffer) assert sent_bytes.endswith(expected) urllib3-2.0.7/.gitignore0000644000000000000000000000012514513541732012015 0ustar00.* !/.github/ *.py[co] *.egg *.egg-info *.log /dist /build /docs/_build coverage.xml urllib3-2.0.7/LICENSE.txt0000644000000000000000000000210514513541732011650 0ustar00MIT License Copyright (c) 2008-2020 Andrey Petrov and contributors. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. urllib3-2.0.7/README.md0000644000000000000000000001041314513541732011305 0ustar00

![urllib3](https://github.com/urllib3/urllib3/raw/main/docs/_static/banner_github.svg)

PyPI Version Python Versions Join our Discord Coverage Status Build Status on GitHub Documentation Status
OpenSSF Scorecard SLSA 3 CII Best Practices

urllib3 is a powerful, *user-friendly* HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many critical features that are missing from the Python standard libraries: - Thread safety. - Connection pooling. - Client-side SSL/TLS verification. - File uploads with multipart encoding. - Helpers for retrying requests and dealing with HTTP redirects. - Support for gzip, deflate, brotli, and zstd encoding. - Proxy support for HTTP and SOCKS. - 100% test coverage. urllib3 is powerful and easy to use: ```python3 >>> import urllib3 >>> resp = urllib3.request("GET", "http://httpbin.org/robots.txt") >>> resp.status 200 >>> resp.data b"User-agent: *\nDisallow: /deny\n" ``` ## Installing urllib3 can be installed with [pip](https://pip.pypa.io): ```bash $ python -m pip install urllib3 ``` Alternatively, you can grab the latest source code from [GitHub](https://github.com/urllib3/urllib3): ```bash $ git clone https://github.com/urllib3/urllib3.git $ cd urllib3 $ pip install . ``` ## Documentation urllib3 has usage and reference documentation at [urllib3.readthedocs.io](https://urllib3.readthedocs.io). ## Community urllib3 has a [community Discord channel](https://discord.gg/urllib3) for asking questions and collaborating with other contributors. Drop by and say hello 👋 ## Contributing urllib3 happily accepts contributions. Please see our [contributing documentation](https://urllib3.readthedocs.io/en/latest/contributing.html) for some tips on getting started. ## Security Disclosures To report a security vulnerability, please use the [Tidelift security contact](https://tidelift.com/security). Tidelift will coordinate the fix and disclosure with maintainers. ## Maintainers - [@sethmlarson](https://github.com/sethmlarson) (Seth M. Larson) - [@pquentin](https://github.com/pquentin) (Quentin Pradet) - [@theacodes](https://github.com/theacodes) (Thea Flowers) - [@haikuginger](https://github.com/haikuginger) (Jess Shapiro) - [@lukasa](https://github.com/lukasa) (Cory Benfield) - [@sigmavirus24](https://github.com/sigmavirus24) (Ian Stapleton Cordasco) - [@shazow](https://github.com/shazow) (Andrey Petrov) 👋 ## Sponsorship If your company benefits from this library, please consider [sponsoring its development](https://urllib3.readthedocs.io/en/latest/sponsors.html). ## For Enterprise Professional support for urllib3 is available as part of the [Tidelift Subscription][1]. Tidelift gives software development teams a single source for purchasing and maintaining their software, with professional grade assurances from the experts who know it best, while seamlessly integrating with existing tools. [1]: https://tidelift.com/subscription/pkg/pypi-urllib3?utm_source=pypi-urllib3&utm_medium=referral&utm_campaign=readme urllib3-2.0.7/pyproject.toml0000644000000000000000000000764114513541732012753 0ustar00# This file is protected via CODEOWNERS [build-system] requires = ["hatchling>=1.6.0,<2"] build-backend = "hatchling.build" [project] name = "urllib3" description = "HTTP library with thread-safe connection pooling, file post, and more." readme = "README.md" keywords = ["urllib", "httplib", "threadsafe", "filepost", "http", "https", "ssl", "pooling"] authors = [ {name = "Andrey Petrov", email = "andrey.petrov@shazow.net"} ] maintainers = [ {name = "Seth Michael Larson", email="sethmichaellarson@gmail.com"}, {name = "Quentin Pradet", email="quentin@pradet.me"}, ] classifiers = [ "Environment :: Web Environment", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Internet :: WWW/HTTP", "Topic :: Software Development :: Libraries", ] requires-python = ">=3.7" dynamic = ["version"] [project.optional-dependencies] brotli = [ "brotli>=1.0.9; platform_python_implementation == 'CPython'", "brotlicffi>=0.8.0; platform_python_implementation != 'CPython'" ] zstd = [ "zstandard>=0.18.0", ] secure = [ "pyOpenSSL>=17.1.0", "cryptography>=1.9", "idna>=2.0.0", "certifi", "urllib3-secure-extra", ] socks = [ "PySocks>=1.5.6,<2.0,!=1.5.7", ] [project.urls] "Changelog" = "https://github.com/urllib3/urllib3/blob/main/CHANGES.rst" "Documentation" = "https://urllib3.readthedocs.io" "Code" = "https://github.com/urllib3/urllib3" "Issue tracker" = "https://github.com/urllib3/urllib3/issues" [tool.hatch.version] path = "src/urllib3/_version.py" [tool.hatch.build.targets.sdist] include = [ "/docs", "/dummyserver", "/src", "/test", "/dev-requirements.txt", "/CHANGES.rst", "/README.md", "/LICENSE.txt", ] [tool.pytest.ini_options] xfail_strict = true python_classes = ["Test", "*TestCase"] markers = ["limit_memory"] log_level = "DEBUG" filterwarnings = [ "error", '''default:urllib3 v2.0 only supports OpenSSL 1.1.1+.*''', '''default:'urllib3\[secure\]' extra is deprecated and will be removed in urllib3 v2\.1\.0.*:DeprecationWarning''', '''default:'urllib3\.contrib\.securetransport' module is deprecated and will be removed in urllib3 v2\.1\.0.*:DeprecationWarning''', '''default:No IPv6 support. Falling back to IPv4:urllib3.exceptions.HTTPWarning''', '''default:No IPv6 support. skipping:urllib3.exceptions.HTTPWarning''', '''default:ssl\.TLSVersion\.TLSv1 is deprecated:DeprecationWarning''', '''default:ssl\.PROTOCOL_TLS is deprecated:DeprecationWarning''', '''default:ssl\.PROTOCOL_TLSv1 is deprecated:DeprecationWarning''', '''default:ssl\.TLSVersion\.TLSv1_1 is deprecated:DeprecationWarning''', '''default:ssl\.PROTOCOL_TLSv1_1 is deprecated:DeprecationWarning''', '''default:ssl\.PROTOCOL_TLSv1_2 is deprecated:DeprecationWarning''', '''default:unclosed .*:ResourceWarning''', '''default:ssl NPN is deprecated, use ALPN instead:DeprecationWarning''', ] [tool.isort] profile = "black" add_imports = "from __future__ import annotations" [tool.mypy] mypy_path = "src" check_untyped_defs = true disallow_any_generics = true disallow_incomplete_defs = true disallow_subclassing_any = true disallow_untyped_calls = true disallow_untyped_decorators = true disallow_untyped_defs = true no_implicit_optional = true no_implicit_reexport = true show_error_codes = true strict_equality = true warn_redundant_casts = true warn_return_any = true warn_unused_configs = true warn_unused_ignores = true urllib3-2.0.7/PKG-INFO0000644000000000000000000001467714513541732011143 0ustar00Metadata-Version: 2.1 Name: urllib3 Version: 2.0.7 Summary: HTTP library with thread-safe connection pooling, file post, and more. Project-URL: Changelog, https://github.com/urllib3/urllib3/blob/main/CHANGES.rst Project-URL: Documentation, https://urllib3.readthedocs.io Project-URL: Code, https://github.com/urllib3/urllib3 Project-URL: Issue tracker, https://github.com/urllib3/urllib3/issues Author-email: Andrey Petrov Maintainer-email: Seth Michael Larson , Quentin Pradet License-File: LICENSE.txt Keywords: filepost,http,httplib,https,pooling,ssl,threadsafe,urllib Classifier: Environment :: Web Environment Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Internet :: WWW/HTTP Classifier: Topic :: Software Development :: Libraries Requires-Python: >=3.7 Provides-Extra: brotli Requires-Dist: brotli>=1.0.9; platform_python_implementation == 'CPython' and extra == 'brotli' Requires-Dist: brotlicffi>=0.8.0; platform_python_implementation != 'CPython' and extra == 'brotli' Provides-Extra: secure Requires-Dist: certifi; extra == 'secure' Requires-Dist: cryptography>=1.9; extra == 'secure' Requires-Dist: idna>=2.0.0; extra == 'secure' Requires-Dist: pyopenssl>=17.1.0; extra == 'secure' Requires-Dist: urllib3-secure-extra; extra == 'secure' Provides-Extra: socks Requires-Dist: pysocks!=1.5.7,<2.0,>=1.5.6; extra == 'socks' Provides-Extra: zstd Requires-Dist: zstandard>=0.18.0; extra == 'zstd' Description-Content-Type: text/markdown

![urllib3](https://github.com/urllib3/urllib3/raw/main/docs/_static/banner_github.svg)

PyPI Version Python Versions Join our Discord Coverage Status Build Status on GitHub Documentation Status
OpenSSF Scorecard SLSA 3 CII Best Practices

urllib3 is a powerful, *user-friendly* HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many critical features that are missing from the Python standard libraries: - Thread safety. - Connection pooling. - Client-side SSL/TLS verification. - File uploads with multipart encoding. - Helpers for retrying requests and dealing with HTTP redirects. - Support for gzip, deflate, brotli, and zstd encoding. - Proxy support for HTTP and SOCKS. - 100% test coverage. urllib3 is powerful and easy to use: ```python3 >>> import urllib3 >>> resp = urllib3.request("GET", "http://httpbin.org/robots.txt") >>> resp.status 200 >>> resp.data b"User-agent: *\nDisallow: /deny\n" ``` ## Installing urllib3 can be installed with [pip](https://pip.pypa.io): ```bash $ python -m pip install urllib3 ``` Alternatively, you can grab the latest source code from [GitHub](https://github.com/urllib3/urllib3): ```bash $ git clone https://github.com/urllib3/urllib3.git $ cd urllib3 $ pip install . ``` ## Documentation urllib3 has usage and reference documentation at [urllib3.readthedocs.io](https://urllib3.readthedocs.io). ## Community urllib3 has a [community Discord channel](https://discord.gg/urllib3) for asking questions and collaborating with other contributors. Drop by and say hello 👋 ## Contributing urllib3 happily accepts contributions. Please see our [contributing documentation](https://urllib3.readthedocs.io/en/latest/contributing.html) for some tips on getting started. ## Security Disclosures To report a security vulnerability, please use the [Tidelift security contact](https://tidelift.com/security). Tidelift will coordinate the fix and disclosure with maintainers. ## Maintainers - [@sethmlarson](https://github.com/sethmlarson) (Seth M. Larson) - [@pquentin](https://github.com/pquentin) (Quentin Pradet) - [@theacodes](https://github.com/theacodes) (Thea Flowers) - [@haikuginger](https://github.com/haikuginger) (Jess Shapiro) - [@lukasa](https://github.com/lukasa) (Cory Benfield) - [@sigmavirus24](https://github.com/sigmavirus24) (Ian Stapleton Cordasco) - [@shazow](https://github.com/shazow) (Andrey Petrov) 👋 ## Sponsorship If your company benefits from this library, please consider [sponsoring its development](https://urllib3.readthedocs.io/en/latest/sponsors.html). ## For Enterprise Professional support for urllib3 is available as part of the [Tidelift Subscription][1]. Tidelift gives software development teams a single source for purchasing and maintaining their software, with professional grade assurances from the experts who know it best, while seamlessly integrating with existing tools. [1]: https://tidelift.com/subscription/pkg/pypi-urllib3?utm_source=pypi-urllib3&utm_medium=referral&utm_campaign=readme